1d087b54a4
feat: day_gainers 5분봉 수집 + News v2 ingest + overlay 정리
...
주요 변경사항:
gainer snapshot (신규)
- gainer_snapshots 테이블: 5분 단위 intraday top-100 day_gainers 스냅샷
- APScheduler CronTrigger (*/5 min, Mon-Fri ET) + 장중 guard (9:30~16:00)
- alembic migration p7g8h9i0j1k2
/stocks/gainers 엔드포인트 (신규)
- yfinance day_gainers preset, 실시간(캐시 없음)
- yfinance_plus screen() wrapper — 모든 screen() 호출에 세션 풀 + 브라우저 지문 우회 적용
News v2 ingest (신규)
- Alpaca News + StockTwits + Finnhub 수집 파이프라인
- news_headlines 테이블 + scheduler (5분 realtime poll, 일 1회 backfill)
Overlay 정리
- 미사용 서브시스템 제거: wikimedia, youtube, google_trends, feature_builder, overlay_scorer
- Yahoo RSS 어댑터 유지, 파이프라인 단순화
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
3 months ago
93a81fd35a
fix: Form 4 buy_only=P-only + 10분기 backfill 설정
...
- buy_only=true 필터를 P+A → P(open-market purchase) 단독으로 변경
- get_form4_pit / get_form4_by_date / get_form4_aggregate 3곳 동일 수정
- aggregate의 buy_dollar_total / cluster_size / csuite_count / avg_pct_of_holding
모두 P-only 기준으로 정확해짐 (A코드 awards 제외)
- _parse_transaction_element: purchase_pct_of_holding 계산도 P-only로 정합
- SEC_FORM4_BOOTSTRAP_QUARTERS: 8 → 10 (2024Q1~)
- Form4AggregateResponse docstring에 P-only 명시
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
4 months ago
db9abd7786
feat: SEC Form 4 PIT + SC 13D/G activist ownership API
...
## 신규 엔드포인트
- GET /insider/form4/{ticker} — PIT-safe Form 4 거래 목록 (as_of 필수)
- GET /insider/form4/by-date/{date} — 특정 공시일 cross-ticker 거래
- GET /insider/form4/aggregate/{ticker} — 내부자 매수 집계 (window_days)
- GET /ownership/13dg/{ticker} — SC 13D/G activist 이벤트 (as_of 필수)
- GET /ownership/13dg/active — 현재 활성 activist 포지션 (window function PIT)
## 데이터 인프라
- activist_ownership_events 테이블 (42,329행, 최근 2년 bootstrap 완료)
- insider_transactions: is_ceo/is_cfo/is_c_suite/purchase_pct_of_holding/owner_relationship 컬럼 추가
- SECFullIndexService: company.idx fixed-width 파서, form345.zip 파서
- ActivistOwnershipService: SGML header issuer 해석 + cover-page XML/HTML enrich
- SEC ingest scheduler: 매 영업일 09:00 ET daily ingest + 30분 enrich job
- scripts: bootstrap_form4_by_ticker.py, bootstrap_form4_2y.py, bootstrap_13dg.py
## 버그 수정 (enrich 품질)
- filing_url double edgar/ prefix 제거
- XML: classPercent + amountBeneficiallyOwned + reportingPerson... 태그 추가
- HTML: 태그 제거 후 regex 적용 (CSS 10pt → shares 오인식 방지)
- HTML: ownership_pct > 100 방어 로직
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
4 months ago
7efe6704b3
fix: DB 커넥션 풀 고갈 — pool_timeout 단축 + idle_in_transaction 자동 종료
...
외부 API 호출(yfinance, Alpaca) 중 DB 세션을 유지하는 구조로 인해
idle in transaction 커넥션이 쌓여 풀(50개) 고갈 → 서버 응답 불가.
- pool_timeout: 30s → 5s (풀 고갈 시 빠른 실패, 30s 행 방지)
- PostgreSQL idle_in_transaction_session_timeout: 90s 설정
(90초 이상 방치된 트랜잭션 자동 종료)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
4 months ago
e2cd16035a
feat: Form 4 내부자 거래 + Earnings Surprise + FINRA 확장
...
F1: SEC Form 4 내부자 거래 (insider transactions)
- GET /insider/transactions/{symbol} — Form 4 거래 내역 조회
- GET /insider/summary/{symbol} — 3/6/12개월 매수/매도 집계
- SEC EDGAR submissions JSON → Form 4 XML 파싱 → DB 저장
- 자동 인덱싱 (첫 조회 시 SEC에서 페치)
- joint filing, derivative/non-derivative 거래 모두 지원
F2: Earnings Surprise (Alpha Vantage)
- GET /earnings/surprise/{symbol} — 분기별 EPS surprise
- reported EPS vs estimated EPS, beat/miss streak 계산
- ALPHA_VANTAGE_API_KEY 환경변수 필요 (무료 tier: 25 req/day)
- DB 캐싱으로 반복 호출 시 API 절약
F3: FINRA Short Volume 확장
- days 파라미터: max 365 → 3650 (10년)
- limit 파라미터: max 1000 → 10000
- 5년치 백필 완료 반영
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
5 months ago
0752b29a63
perf: API 성능 개선 Round 2 — 9개 항목 (P6-P14)
...
Tier 1:
- P6: 미들웨어 request_info 지연 추출 (성공 경로에서 body/headers 읽기 제거)
- P7: GZip 압축 미들웨어 (minimum_size=1000, ~4x 압축)
- P8: Redis maxmemory 512MB + allkeys-lru 퇴거 정책
Tier 2:
- P9: _store_ticker_data() batch upsert 전환 (ON CONFLICT DO NOTHING)
- P10: DB 커넥션 풀 증가 (pool_size=20, max_overflow=30, 환경변수 설정)
- P11: Attention 엔드포인트 Redis 캐시 추가 (@with_cache ttl=3600)
Tier 3:
- P12: Dockerfile 멀티 워커 (--workers 4, dev는 --reload 오버라이드)
- P13: aiohttp ClientSession 싱글턴 공유 (4개 파일 7곳 TCP/TLS 재사용)
- P14: Pydantic model_validate → model_construct (bulk 경로 validation 스킵)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
5 months ago
d82ba624e2
chore: remove SQLite entirely, PostgreSQL only
...
SQLite was never used in production (Docker Compose always sets
DATABASE_URL to postgresql+asyncpg://...) but the fallback kept
creating confusing stock_oracle.db files locally.
- config.py: default DATABASE_URL fallback → PostgreSQL (localhost:15433)
- database.py: remove _is_sqlite conditional branch and NullPool import
- error_logs.py: replace strftime/dialect-check with pg to_char()
- request_logs.py: replace strftime with pg to_char()
- feature_builder.py: remove _ensure_utc() helper and all call sites
- overlay_pipeline.py: remove naive-datetime workaround for is_stale()
- requirements-api.txt / requirements-test.txt: drop aiosqlite
- stock_oracle.db: deleted
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
5 months ago
29fc870b37
Add Phase 5 Attention Overlay API and fix Redis port mismatch
...
- 12 REST endpoints: overlay score, bulk, top-movers, headlines, youtube, wiki, crowding, trends, history, admin (health/trigger/job-log)
- 10 services: entity_resolver, yahoo_rss_adapter, wikimedia_adapter, youtube_adapter, google_trends_adapter, finra_overlay_loader, feature_builder, overlay_scorer, overlay_pipeline, scheduler
- 10 DB tables across overlay_registry, overlay_raw_event, overlay_feature models
- APScheduler: collect @ 23:30 UTC + feature build @ 01:30 UTC weekdays
- Fix Redis port mismatch: config default 16379 → 16380 to match docker-compose external port
- 64 overlay tests covering cache utils, Redis config, all 12 endpoints, route ordering
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
5 months ago
bf6932ca2f
Add Alpaca-specific DB table and Redis caching for Alpaca/FINRA endpoints
...
Separate Alpaca price data into dedicated AlpacaPriceData table to avoid
UniqueConstraint('ticker', 'date') conflicts with Yahoo Finance PriceData.
Add Redis caching (build_cache_key/get_cached_response/set_cached_response)
to 3 Alpaca endpoints and 2 FINRA query endpoints with appropriate TTLs.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
5 months ago
355aeb19d3
Initial commit with full project improvements
...
Security: config-based CORS, default secret warnings, sort_by validation
Error handling: debug logging in cache silent failures
Architecture: shared resolve_time_parameters, deduplicated logger init, unified route structure
Database: conditional SQLite/PostgreSQL engine, in-memory test DB, dialect-aware date formatting, optimized stats query
Docker: .dockerignore, pinned yfinance_plus commit
Dependencies: removed duplicates, added version upper bounds, removed unused axios
Frontend: custom _document/_error pages, adminApi client, Layout standardization, ESLint version update
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
5 months ago
I Luk Kim