On 2026-05-08 19:45 UTC a transient Oracle FRED-proxy 5xx storm caused
paper_engine_vix_fred_unavailable to fire (3 sequential 500s; the 4th
attempt returned 200 OK with VIX=17.08). The pre-fix engine just left
VIXCLS missing from the macro dict, which the selector at
libs/backtest/selector.py:1171-1173 already treats strict-conservatively
(None → veto). So the 5/8 incident vetoed v7.356 PEAD candidates for
~1 minute with no money-loss exposure. But:
- Log severity was thin (info-level "unavailable", no escalation).
- No tolerance for short outages — every 500 cost the gate's signal.
- EventDetector PostgreSQL rows do not pre-populate macro_vix per
engine.py:2810-2812 comment, so live trading depends entirely on
the FRED fetch path.
Fix: in-memory session-scoped cache + 3-tier fallback in
PaperTradingEngine._fetch_macro:
Tier 1 fetch ok → cache (value, now_utc), log ..._ok (info)
Tier 2 fail, cache <24h → return cached value, log ..._stale_fallback
(warning) with staleness_sec
Tier 3 fail, cache stale → None, log ..._unavailable_blocking (error)
with reason={no_cache,cache_too_stale}
The None-veto path through the selector is preserved exactly, so no
silent-pass on unknown VIX. Empty/0 observations now treated as outage
to defend against an upstream regression flipping "missing→veto" into
"0→always-pass".
The thin libs/oracle_client/fred.py is intentionally untouched — fallback
policy belongs in the engine, not the generic client.
6 new tests in tests/unit/paper_trader/test_vix_fred_fallback.py
(success/cache-write, 500+stale-<24h, 500+stale->24h-blocks,
no-cache+500-blocks, empty-observations-blocks, recovery-refresh).
All 18 tests in -k "vix or fred" pass; full paper_trader suite 45/45.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- v7.356 config: swap dataset_snapshot_id from manual_only ftb_fix_v2 to
auto_full_rebuild base canonical so paper trader can refresh snapshot
(root cause of processed_events=0 for 30 days)
- Multi-session order isolation (1.A.2/1.A.3): tag client_order_id with
pt-{session_id[:8]}-{uuid} prefix on all entry orders; _cancel_stale_orders
filters by own session prefix so one session no longer ghost-cancels another's
orders on shared Alpaca account
- Pipeline halt on failure (1.B.1): _run_pipeline returns bool and stops on
first subprocess failure instead of silently progressing with stale data
- Daemon restart window skip (2.2): run_open/run_close only marked completed
if processed_phases DB confirms prior execution — no more trading-less days
after mid-day restart
- event_parser: periodic batch commits every 500 docs (hypothesis fix for
3h hangs; unverified — may just be slow serial Oracle calls)
- Tests updated for _verify_order_fill tuple return + new cross-session
isolation test; all 23 paper_trader unit tests green
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- ReconciliationReport dataclass tracking orphaned/ghost positions and stale orders
- _cancel_stale_orders(): cancel leftover open orders at daily run start
- _reconcile_positions(): detect Alpaca vs local state mismatches; auto-close ghost positions with RECONCILED exit reason
- _verify_order_fill(): poll broker up to 2s to confirm market order fill before saving state
- _check_kill_switch(): activate and persist kill switch at 25% drawdown; blocks new entries
- run_daily() and _process_entries() wired with all safety checks
- 18 unit tests covering all reconciliation scenarios
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
I Luk Kim