Pre-market label_generator runs request future-dated price windows from
Stock Oracle, which correctly returns 404 because the data does not yet
exist. The labeler was swallowing this as label_status='unavailable' with
entry_date=None. Snapshot export then filtered these rows out, so live
PEAD trading silently lost candidates whose entry_dates fell on
later trading days (e.g., post-market 8-K filings late Friday → Monday
open entry). This explains today's missed RKLB/SNDK/AKAM/MNST/AMD/MRNA
even though their 8-Ks parsed correctly.
Changes:
- libs/labeler/label_generator.py: in 404/empty-bars path, when
entry_date >= today, preserve entry_date and mark label_status='pending'.
New log event label_price_pending_future_window distinguishes from real
data-unavailable failures (past dates still log label_price_unavailable).
- libs/export/snapshot_export.py: include 'pending' in the
label_status filter so today's not-yet-labeled events flow into the
live snapshot.
- apps/pipeline/label_generator/main.py: regeneration logic also
retries existing 'unavailable' rows whose entry_date is null or future
to recover events already mis-labeled in the DB.
- tests/unit/test_labeler.py: regression test reproducing the
RKLB/SNDK/AKAM failure mode and asserting label_status='pending' with
entry_date preserved.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
event_volume (and potentially other columns) can arrive as int64 from the
pipeline while the stored snapshot uses double, causing pa.concat_tables to
fail with "incompatible types" every run and silently fall back to a full
rebuild. _coerce_schema() casts new rows to the existing snapshot's types
before concatenation so incremental works without a full rebuild.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- enrich_tier2: prefetch price bars (parallel ThreadPool) and short ratio
(single batch DB query) instead of per-row HTTP/DB calls (~20min → ~2min)
- canonical_snapshots: add PYTHONUNBUFFERED=1 to enrichment subprocesses
so progress output is visible in real time
- backtest_sim: use incremental_update_canonical_snapshot when existing
snapshot is present, falling back to full rebuild only when needed
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds earnings surprise extraction to parser/features/labeler pipeline,
improves filing fetcher robustness, and extends snapshot export with
new field support.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Previously only used rows[0].keys() — columns present in later rows
(like earnings_surprise_pct from sparse features) were silently dropped.
Now collects all unique keys across all rows.
YoY earnings surprise tested: WR spread only 2.5pp (55.2% vs 52.7%).
Not actionable — YoY growth != analyst consensus surprise.
v6new.30 remains the framework optimum.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
New features:
- _enrich_macro_features() in snapshot_export: adds macro_vix, macro_hy_spread from FRED
- compute_return_max_long_score_v10: +12% bonus in favorable regime (VIX>18+HY>3.25)
- _macro_regime_score(): regime-aware scoring component
Findings:
- VIX signal is statistically strong: 62.3% WR (VIX>18+HY>3.25) vs 50.8% (other)
- But scoring bonus promotes marginal trades, diluting OOS quality
- Same pattern as eps_growth, drift bonus: raw signal ≠ scoring improvement
- v6new.17 SQS 59.1 < v6new.9 SQS 63.3
v6new.9 remains best at SQS 63.3 after 17 experiments.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
I Luk Kim