fithia2/dev/phase7_deliverables/production_hardening_and_governance.md

Production Hardening and Governance

Objective

This document defines operational governance for live trading. It assumes trading can lose money quickly when operational discipline is weak. Governance must therefore be explicit, logged, and testable.

Governance Domains

1. Runtime safety
2. Access control
3. Config integrity
4. Release approval
5. Incident response
6. Auditability
7. Reporting and review

Runtime Safety Controls

Mandatory controls:

• kill switch with immediate new-order disable
• no-trade mode
• degraded mode classification by source and subsystem
• hard daily loss limit
• hard gross exposure limit
• hard net exposure limit
• per-symbol and per-sector caps
• stale-data rejection for mandatory inputs
• duplicate order prevention
• restart-safe state recovery

Access Control

Minimum roles:

• Operator: may start/stop jobs and review health, but may not change strategy logic.
• Approver: may approve live rollout stage changes and exceptional trades.
• Engineer: may deploy code through approved process, but not bypass approvals.
• Auditor: read-only access to logs, configs, incidents, and trade history.

Configuration Governance

• All live configs must be versioned and immutable after activation.
• Any manual override must create an audit log.
• Runtime should record exact config hash used for each trade.
• Config changes may not be applied mid-session unless emergency procedure is used.

Release Gates

A release may move to live only if:

• code passed all required tests
• migration steps were rehearsed
• rollback steps are documented
• a release note exists with risk classification
• dry-run or replay evidence is attached
• approval is recorded

Emergency Procedures

Emergency actions include:

• disable new orders
• cancel open non-protective orders
• force no-trade mode
• disable specific adapters or overlays
• revert to previous known-good config
• pause stage promotion

Each emergency action must be logged with who, when, why, and expected blast radius.

Incident Governance

Incidents are classified by severity:

• Sev-1: material financial or safety risk; immediate intervention required.
• Sev-2: significant malfunction but contained.
• Sev-3: degraded behavior with workaround.
• Sev-4: cosmetic or low-risk issue.

For every incident, capture:

• detection time
• impacted systems
• first symptom
• trades affected
• capital at risk
• mitigation taken
• root cause
• corrective action
• prevention action

Audit Requirements

The system must retain for each trade:

• strategy id
• config version/hash
• code version/build id
• parser version
• feature version
• approval state
• order lifecycle events
• reconciliation outcome
• post-trade review id

Governance Meetings

• Daily ops review: incidents, fills, reconciliation, drift alerts.
• Weekly strategy ops review: attribution, execution drift, parser quality, config exceptions.
• Monthly governance review: stage eligibility, risk policy adherence, release quality, recurring issues.