stock-oracle/CHANGELOG.md

# Changelog

All notable changes to Stock Oracle API will be documented in this file.

## [3.1.1] - 2026-04-26

### Changed (breaking — News v2 only, pre-GA)
- **`SessionAggregateItem` social fields nested**: `social_message_count` / `social_bull_count` / `social_bear_count` removed; replaced by nested `social: {message_count, bull_count, bear_count, bull_bear_ratio}`. Adds derived `bull_bear_ratio = bull/(bull+bear)` (null when no directional messages). Restores spec compliance — fithia2 integration test caught the deviation pre-GA, no client traffic yet.

## [3.1.0] - 2026-04-26

### Added
- **News v2 — multi-source structured ingest** (`/api/v1/news/v2/*`): premium news/social signal designed for backtest/forward-test consumers (fithia2 V49 ORB).
  - Sources: Alpaca News (Benzinga backend, P0), StockTwits public API (P1), Finnhub free tier (P2). Existing `/news/{ticker}` aggregator unchanged for UI use.
  - `GET /news/v2/headlines` — raw rows with symbols/start/end/sources/limit/cursor
  - `GET /news/v2/session_aggregate` — single (ticker, session_date, window) Redis-cached aggregate
  - `POST /news/v2/session_aggregate/batch` — many tickers in one call (no server cache; client disk-cache assumed)
  - `GET /news/v2/coverage` — per-source ingest depth probe
- **Unified 22-term category taxonomy** with regex headline overrides for FDA approval/rejection split, analyst rating direction, etc.
- **Session windows** via `pandas_market_calendars` (XNYS) — premarket/intraday/post/full_session with NYSE holiday + early-close handling. `post` ends at next trading day's premarket start (04:00 ET) to remain disjoint from next session's premarket.
- **PIT safety**: aggregates filter `ingested_at <= window_end_utc` so backtests don't see lookahead headlines.
- **APScheduler jobs** (opt-in via `NEWS_INGEST_ENABLED=true`): Alpaca 5-min poll + 04:30 ET daily backfill, StockTwits 09:00 ET universe refresh + 5-min poll, Finnhub 05:00 ET daily backfill.
- **Fail-fast**: `NEWS_INGEST_ENABLED=true` with neither `ALPACA_API_KEY/SECRET` nor `FINNHUB_API_KEY` → scheduler refuses to start (StockTwits-only is too low-signal).
- **StockTwits dynamic universe**: `(last 14 days V49 union) ∪ (today's premarket gap movers > 2%)`, capped at 300 tickers.
- **Manual backfill script**: `scripts/news_backfill.py --source finnhub --tickers ... --start --end --chunk monthly`.
- **Stock Oracle Python client**: `get_news_headlines`, `get_news_session_aggregate`, `get_news_session_aggregate_batch`, `get_news_coverage`.
- **New table**: `news_headline` (UUID PK, ARRAY columns, `(source, source_id, ticker)` unique constraint for idempotent multi-source ingest).
- **New env vars**: `NEWS_INGEST_ENABLED`, `NEWS_INGEST_TIMEZONE`, `ALPACA_NEWS_BASE_URL`, `FINNHUB_API_KEY`, `FINNHUB_BASE_URL`, `STOCKTWITS_BASE_URL`, `STOCKTWITS_UNIVERSE_LOOKBACK_DAYS`, `STOCKTWITS_PREMARKET_GAP_THRESHOLD`, `STOCKTWITS_UNIVERSE_MAX_SIZE`.
- **New dependency**: `pandas_market_calendars>=4.3.0`.

## [3.0.3] - 2026-03-18

### Fixed
- **ETF Holdings `as_of_date` 정확도**: 요청 날짜에 가장 가까운 파일링을 반환하도록 수정
  - `_load_snapshot_holdings()`: DB 캐시 스냅샷과 `as_of_date` 간격이 120일 초과 시 stale 판단, SEC 신규 fetch 트리거
  - `_find_best_filing_and_xml()`: eligible 후보 필터에 365일 하한 추가 — target_date 기준 1년 이내 파일링 우선 탐색
  - 수정 전: `SPY?as_of_date=2021-06-30` → 2019-11-18 (19개월 stale)
  - 수정 후: `SPY?as_of_date=2021-06-30` → 2021-05-28 (정상)

## [3.0.2] - 2026-03-17

### Added
- **Attention 서브시스템**: 이벤트 중심 Wikipedia/GDELT 관심도 피처 (백테스팅용)
  - `GET /api/v1/attention/event/{ticker}?event_date=YYYY-MM-DD` — 이벤트 날짜 기준 관심도 피처 조회
  - `GET /api/v1/attention/entity/{ticker}` — 티커 → 캐노니컬 엔티티 정보 조회
  - `POST /api/v1/attention/admin/resolve/{ticker}` — 티커 → 캐노니컬 엔티티 해석
  - `POST /api/v1/attention/admin/collect/wiki/{ticker}` — Wikipedia 페이지뷰 수집
  - `POST /api/v1/attention/admin/collect/gdelt/{ticker}` — GDELT 뉴스 아티클 수집
  - Wikipedia 일별 페이지뷰 스파이크 / z-score 계산
  - GDELT GKG 아티클 수 기반 언론 노출 지표

## [3.0.1] - 2026-03-14

### Added
- **Filing Search `accepted_at` field**: `GET /filings/search/{ticker}` response now includes `accepted_at` (ISO 8601 timestamp) in each `FilingSummary`
  - Sourced from `acceptanceDateTime` in SEC EDGAR submissions JSON — no additional HTTP requests needed
  - `SECFiling` DB model gains a nullable `accepted_at TIMESTAMP` column
  - SQLite auto-migration: `ALTER TABLE sec_filings ADD COLUMN accepted_at TIMESTAMP` runs at startup (idempotent)
  - Existing rows have `accepted_at = NULL`; re-index with `force_refresh=true` to backfill
  - Enables downstream consumers (e.g. `filing_poller`) to populate `Document.accepted_at_utc` and `Event.filed_at_utc`

## [3.0.0] - 2026-03-12

### Added
- **Alpaca Market Data Adapter**: Complete integration with Alpaca Market Data API v2
  - `GET /alpaca/status` — Connection status and API key validation
  - `GET /alpaca/bars/{ticker}` — Raw bars from Alpaca (no DB)
  - `GET /alpaca/data/{ticker}` — OHLCV price data with DB storage (PriceDataResponse format)
  - `GET /alpaca/intraday/{ticker}` — Intraday candles (1m, 5m, 15m, 1h)
  - Built-in rate limiting (200 req/min token bucket) with retry + exponential backoff
  - VWAP (volume-weighted average price) and trade count per bar
  - Auto-pagination for large date ranges
  - Completely independent from Yahoo Finance — users choose their source

- **FINRA Short Sale Volume Adapter**: RegSHO daily short volume data
  - `GET /finra/short-volume/{symbol}` — Short volume data with auto-ingest
  - `GET /finra/short-ratio/{symbol}` — Aggregated short ratio history
  - `POST /finra/admin/ingest` — Manual ingest for single date or date range
  - Public FINRA CDN data (no API key required)
  - Pipe-delimited text parser for CNMS short volume files
  - Multi-market data (NYSE TRF, NASDAQ TRF, NYSE)
  - Auto-ingest on first query if data is missing

- **New DB Model**: `FinraShortVolume` with `(symbol, date, market)` unique constraint
- **New DataSource Enum**: `DataSource.ALPACA` added to schema
- **Configuration**: `ALPACA_API_KEY`, `ALPACA_SECRET_KEY`, `ALPACA_BASE_URL` settings
- **Docker**: Alpaca env vars passed to API container

### New Files
- `app/services/alpaca_client.py` — Alpaca HTTP client with httpx
- `app/services/alpaca_price_service.py` — Alpaca bars → PriceData conversion
- `app/api/v1/endpoints/alpaca.py` — Alpaca endpoints
- `app/models/finra_short_volume.py` — FINRA SQLAlchemy model
- `app/services/finra_short_volume_service.py` — FINRA fetch/parse/ingest service
- `app/schemas/finra.py` — FINRA Pydantic schemas
- `app/api/v1/endpoints/finra.py` — FINRA endpoints

## [2.1.0] - 2025-08-10

### Added
- **ETF Holdings API v2**: Complete rewrite with enhanced features
  - `availability` field in all error responses showing available date ranges
  - ETF launch date validation to prevent invalid historical requests
  - Automatic detection when ETF didn't exist on requested date
  - Fast performance optimization (<0.1s response time, down from 35s)
  - Enhanced error messages with actionable information

### Improved
- **Performance**: ETF date validation now uses cached launch dates for instant response
- **User Experience**: Clear error messages when ETF data is unavailable
- **Documentation**: Comprehensive API documentation with examples

### Fixed
- Historical date requests now correctly validate against ETF launch dates
- QQQM pre-launch date requests now return proper error instead of wrong data
- Response model now includes all fields (fixed Pydantic model filtering issue)

## [2.0.0] - 2025-07-01

### Added
- **ETF Holdings API**: New endpoint for ETF portfolio data
  - Support for 30+ major ETFs with pre-configured mappings
  - Automatic CIK to ticker conversion
  - Historical NPORT data from 2019 onwards
  - Support for both ticker symbols and CIK numbers

### Changed
- Simplified ETF API to single `/holdings/{ticker}` endpoint
- Removed redundant ETF endpoints

## [1.5.0] - 2024-12-01

### Added
- **Price Data API**: OHLCV data integration with yfinance
- **Bulk Data Support**: Batch requests for multiple tickers
- **Period Strings**: Convenient time period specification (1y, 6m, 3m, etc.)

### Improved
- Database caching strategy
- Error handling and logging
- API documentation

## [1.0.0] - 2024-10-01

### Initial Release
- **Financial Data API**: SEC filing data extraction
- **Metrics Calculation**: P/E, P/B, ROE, margins, etc.
- **Database Caching**: SQLite/PostgreSQL support
- **Docker Deployment**: Complete containerization
- **API Documentation**: Interactive Swagger/OpenAPI docs