Apply libs.parser.event_type_normalizer (added in commit 722e5cf) to
existing events whose parser_version LIKE 'oracle-fallback%' so historical
rows match the forward-going normalization wired into the parser.
Implementation:
- New renormalize_oracle_fallback_events() in apps/pipeline/event_parser/main.py
- SELECT filter Event.parser_version.like("oracle-fallback%") — broader
than a hardcoded IN list, so already-aligned values skip naturally and
future additions to _ORACLE_TO_STRATEGY get picked up automatically
- New --renormalize-oracle-fallback CLI flag, chainable with --reparse
- JobRun row written (job_name=event_parser_renormalize_oracle)
- Per-row renormalize_event_updated INFO log + final renormalize_done
summary with transition counters
Live DB run: seen=2937 / updated=1069 / skipped=1868 / errors=0.
Wall ~2 sec (pure DB UPDATEs, no Oracle calls).
Transitions:
earnings_result -> earnings_release : 412
shareholder_vote -> other_material_event : 409
regulation_fd -> guidance_update : 237
acquisition_disposition -> other_material_event : 7
other -> other_material_event : 4
Unmapped Oracle values (financial_obligation 225, articles_amendment 81,
contract_termination 48, etc.) preserved verbatim — honest filter-drop.
Stale-by-design (mirrors existing reparse_events convention):
- Event.event_id PK still embeds old raw event_type substring
- EventParse.output_json["event_type"] still carries raw Oracle value
Strategies read Event.event_type, not those fields. Avoids cascading
PK rewrites across event_parses/feature_snapshots/event_labels tables.
Integration test: tests/integration/test_renormalize_oracle_fallback.py
inserts 4 fixtures, drives _apply_oracle_renormalization() against the
rolled-back db_session, asserts updated/skipped/error counts and final
row state.
Snapshot rebuild not run — nightly auto-rebuild picks up normalized
values incrementally.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
When the rule parser can't classify an 8-K and falls back to Stock Oracle's
filing-events API, Oracle's vocabulary (e.g. earnings_result, shareholder_vote,
regulation_fd) was being written verbatim into events.event_type. The DB has
no CHECK constraint (libs/db/models.py:189), so 22 distinct Oracle values
silently leaked into a column the strategy's engine filters expect to be in
its 4-event vocabulary. Result: ~1,069 live rows silently dropped from
strategy candidate pool.
Files:
- NEW libs/parser/event_type_normalizer.py: normalize_oracle_event_type()
with conservative synonym map; normalize_oracle_event() additionally
uses _classify_event_type from rule_parser when an item_number is
present (item-code path is more reliable than Oracle's event taxonomy)
- MOD apps/pipeline/event_parser/main.py: oracle-fallback branch (~line
140) now calls normalize_oracle_event before writing to DB; emits
oracle_event_type_normalized log event when value changes
- NEW tests/unit/test_event_type_normalizer.py: 60 tests covering
identity, synonyms, case/separator insensitivity, None/empty,
non-string, item_number-precedence
Mapping highlights (justifications in test docstrings):
earnings_result/earnings_announcement/earnings -> earnings_release
guidance_revision/guidance_change/regulation_fd -> guidance_update
material_definitive_agreement/definitive_agreement -> material_contract
shareholder_vote/acquisition_disposition/bankruptcy/other -> other_material_event
Reg FD -> guidance_update mirrors rule_parser's Item 7.01 mapping for
internal consistency. Debatable but auditable.
Conservative pass-through for ambiguous values (financial_obligation,
articles_amendment, contract_termination, etc., 14 distinct values).
Visible filter-drop > silent re-tag.
Live DB counts that would reclassify on a future --reparse pass:
412 earnings_result -> earnings_release
409 shareholder_vote -> other_material_event
237 regulation_fd -> guidance_update
7 acquisition_disposition -> other_material_event
4 other -> other_material_event
TOTAL 1,069 rows currently in oracle-fallback dead-zone.
60/60 normalizer tests pass; combined parser+schema validator suite 86/86.
Follow-up flagged: run --reparse on historical oracle-fallback rows after
extending reparse_events() to also re-normalize known oracle-fallback
values (currently only re-parses event_type='unknown').
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Pre-market label_generator runs request future-dated price windows from
Stock Oracle, which correctly returns 404 because the data does not yet
exist. The labeler was swallowing this as label_status='unavailable' with
entry_date=None. Snapshot export then filtered these rows out, so live
PEAD trading silently lost candidates whose entry_dates fell on
later trading days (e.g., post-market 8-K filings late Friday → Monday
open entry). This explains today's missed RKLB/SNDK/AKAM/MNST/AMD/MRNA
even though their 8-Ks parsed correctly.
Changes:
- libs/labeler/label_generator.py: in 404/empty-bars path, when
entry_date >= today, preserve entry_date and mark label_status='pending'.
New log event label_price_pending_future_window distinguishes from real
data-unavailable failures (past dates still log label_price_unavailable).
- libs/export/snapshot_export.py: include 'pending' in the
label_status filter so today's not-yet-labeled events flow into the
live snapshot.
- apps/pipeline/label_generator/main.py: regeneration logic also
retries existing 'unavailable' rows whose entry_date is null or future
to recover events already mis-labeled in the DB.
- tests/unit/test_labeler.py: regression test reproducing the
RKLB/SNDK/AKAM failure mode and asserting label_status='pending' with
entry_date preserved.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- v7.356 config: swap dataset_snapshot_id from manual_only ftb_fix_v2 to
auto_full_rebuild base canonical so paper trader can refresh snapshot
(root cause of processed_events=0 for 30 days)
- Multi-session order isolation (1.A.2/1.A.3): tag client_order_id with
pt-{session_id[:8]}-{uuid} prefix on all entry orders; _cancel_stale_orders
filters by own session prefix so one session no longer ghost-cancels another's
orders on shared Alpaca account
- Pipeline halt on failure (1.B.1): _run_pipeline returns bool and stops on
first subprocess failure instead of silently progressing with stale data
- Daemon restart window skip (2.2): run_open/run_close only marked completed
if processed_phases DB confirms prior execution — no more trading-less days
after mid-day restart
- event_parser: periodic batch commits every 500 docs (hypothesis fix for
3h hangs; unverified — may just be slow serial Oracle calls)
- Tests updated for _verify_order_fill tuple return + new cross-session
isolation test; all 23 paper_trader unit tests green
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- AutoScheduler._run_catchup: if server starts after 9:35 AM ET but before
market close (16:00 ET), and run_open hasn't already run today
(checked via processed_phases), run it immediately instead of silently
skipping it — prevents AVGO/event entries being missed on late starts
- filing_poller: log exc_type alongside error so empty-string exceptions
(e.g. HTTPError()) are still identifiable by their type
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds earnings surprise extraction to parser/features/labeler pipeline,
improves filing fetcher robustness, and extends snapshot export with
new field support.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add `think: False` and `num_ctx: 8192` to Ollama payload:
Qwen3.5 extended thinking mode generated 1300+ internal reasoning
tokens before each response, adding 30-60s latency per LLM call.
Disabling it reduces parse time from 600s timeout to ~13s.
- Rewrite OllamaClient to use sync httpx.Client inside asyncio.to_thread():
Async httpx inside an active asyncpg SQLAlchemy session context on
Python 3.13 hung indefinitely. Synchronous httpx in a thread pool
completely isolates Ollama I/O from the asyncio event loop.
- Fix filing_poller to set issuer_id/symbol_id on Document records:
Missing FK caused feature_builder to reject all events with
event_no_symbol warning. Now looks up IssuerMaster/SymbolMaster
by ticker before creating Document rows.
- Update test_llm_client to mock _sync_call instead of _client attr.
- Raise ollama_timeout default to 600s for large document processing.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- filing_poller: add --start-date/--end-date CLI args for historical backfill
(defaults to 7 days ago when omitted)
- OracleClient.get/post: apply with_retry(max_attempts=3) so transient
connection errors, timeouts, and 5xx responses are automatically retried
with exponential backoff (0.1s→0.2s→fail)
- financial_features: new compute_financial_features() extracting latest_eps,
latest_gross_margin, latest_operating_margin, eps_growth_qoq,
revenue_growth_qoq from FinancialDataResponse
- feature_builder: wire FinancialService into build_features_for_event(),
persisting financial_v1 FeatureSnapshot (non-fatal if unavailable)
- tests: 94 pass (81→89 unit + 5 replay); +8 new tests covering financial
features and retry success path
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
I Luk Kim