Stock Oracle 🔮

Investment Data Analysis API using SEC filings

Stock Oracle is a comprehensive investment analysis API that leverages SEC EDGAR filing data to provide detailed financial metrics and insights for informed investment decision-making.

🎯 Features

Core Investment Metrics

• Valuation Ratios: P/E, P/B, P/S, EV/EBITDA
• Profitability: ROE, ROA, Gross/Operating/Net Margins
• Growth Metrics: Revenue Growth, Earnings Growth YoY
• Financial Health: Debt-to-Equity, Market Cap
• Sector Analysis: Industry and sector categorization

Market Data Intelligence (NEW! 🆕)

• Most Active Stocks: Real-time ~170 most actively traded stocks (sub-5s response)
• 52-Week Gainers: 1,350+ top gaining stocks with intelligent rate limiting (5-90s)
• Index Constituents: S&P 500 / Nasdaq 100 구성 종목 조회 — Wikipedia 파싱, 24시간 캐시
• FRED Economic Data: Federal Reserve economic indicators with smart caching (1000/day limit)
• Advanced Web Scraping: curl_cffi + Chrome impersonation bypasses rate limits
• Smart Pagination: Configurable page limits (1-10 pages) for performance tuning
• Multi-Source News: Yahoo Finance + NewsAPI integration
• Social Media Analysis: Reddit sentiment and discussions
• Real-Time Updates: Fresh content aggregation with performance monitoring
• Sentiment Analysis: Automated content sentiment scoring
• Historical Context: Customizable time periods (1-30 days)

API Capabilities

• RESTful API: FastAPI-based with OpenAPI documentation
• Database Caching: Intelligent caching to avoid duplicate SEC parsing
• Date Range Queries: Flexible time period analysis
• Data Validation: Comprehensive request/response validation
• Error Handling: Detailed error categorization and reporting
• Migration Support: Database transfer capabilities

Attention Subsystem (NEW! 🆕)

• Event-Centric Features: Wikipedia pageviews + GDELT news coverage for backtesting
• Spike Detection: Daily pageview spike ratio and z-score vs. rolling baseline
• GDELT Integration: News article count from GDELT GKG (global knowledge graph)
• Entity Resolution: Ticker → canonical Wikipedia entity mapping
• On-Demand Collection: Collect data per event date without full precomputation

Alpaca Market Data (NEW! 🆕)

• Official REST API: Alpaca Market Data v2 with proper authentication
• OHLCV + VWAP: Daily/intraday bars with volume-weighted average price
• Independent Endpoints: Separate from Yahoo Finance — users choose their source
• DB Storage: Alpaca bars stored with data_source = "ALPACA" for distinction
• Rate Limiting: Built-in token bucket (200 req/min) with retry + backoff

FINRA Short Volume Data (NEW! 🆕)

• RegSHO Short Sale Volume: Daily short volume from FINRA public CDN
• No API Key Required: Public data, free access
• Auto-Ingest: Automatically downloads missing data on first query
• Short Ratio History: Aggregated short_ratio trends over time
• Bulk Ingest: Date range ingest for backfilling historical data

Technical Stack

• Backend: FastAPI, SQLAlchemy (async), Pydantic
• Frontend: Next.js 15, React 18, TypeScript, TailwindCSS
• Database: SQLite (development) / PostgreSQL (production)
• Caching: Redis for performance optimization
• Data Sources:
  • SEC EDGAR via edgartools (ETF holdings, financial data)
  • Yahoo Finance via yfinance_plus (news & prices)
  • Yahoo Finance via intelligent curl_cffi scraping (market data)
    • Most Active Stocks (~170 stocks)
    • 52-Week Gainers (~1,350 stocks with rate limiting)
  • Alpaca Market Data v2 (OHLCV bars, optional API key)
  • FINRA RegSHO (short sale volume, public CDN)
  • NewsAPI (news articles)
  • Reddit API (social media sentiment)
• Deployment: Docker with docker-compose
• Testing: Comprehensive test suite with pytest

🚀 Quick Start

Using Docker (Recommended)

# Clone and navigate
git clone <repo-url>
cd stock-oracle

# Start all services
docker-compose up -d

# Services will be available at:
# - API: http://localhost:18001
# - API Documentation: http://localhost:18001/docs
# - Frontend: http://localhost:18002
# - Error Log Viewer: http://localhost:18002/errors
# - PostgreSQL: localhost:15433
# - Redis: localhost:16380

Local Development

# Install dependencies
pip install -r requirements-api.txt

# Set up environment
cp .env.example .env
# Edit .env with your settings

# Start all services (first time)
docker-compose up -d

# Restart API after code changes (code is volume-mounted, no rebuild needed)
just run dev

Note: Always use just run dev to restart the API. The app/ directory is volume-mounted into the container, so code changes take effect on restart without rebuilding the image.

⚡ Quick Examples

Get Market Overview

# Trending stocks - best of both worlds (recommended, default: 500 total stocks)
curl "http://localhost:18001/api/v1/stocks/trending"

# Most active stocks (fast)
curl "http://localhost:18001/api/v1/stocks/most-active?limit=10"

# Top 52-week gainers (moderate)
curl "http://localhost:18001/api/v1/stocks/52-week-gainers?max_pages=1&limit=50"

# ETF holdings analysis
curl "http://localhost:18001/api/v1/etf/holdings/QQQ"

# Economic indicators from FRED (cached)
curl "http://localhost:18001/api/v1/fred/proxy/series?series_id=GDP"
curl "http://localhost:18001/api/v1/fred/proxy/series/observations?series_id=UNRATE&limit=12"

Performance Comparison

# Fast queries (< 15 seconds)
curl "http://localhost:18001/api/v1/stocks/trending?n=200"                # ~5-10s, 200 trending stocks (fast mode)
curl "http://localhost:18001/api/v1/stocks/most-active"                   # ~5s, 170 stocks
curl "http://localhost:18001/api/v1/stocks/52-week-gainers?max_pages=1"   # ~8s, 200 stocks

# Moderate queries (15-35 seconds) 
curl "http://localhost:18001/api/v1/stocks/trending"                      # ~15-30s, 500 trending stocks (default, recommended)
curl "http://localhost:18001/api/v1/stocks/52-week-gainers"               # ~25s, 600 stocks (default)

# Comprehensive queries (35+ seconds)
curl "http://localhost:18001/api/v1/stocks/trending?n=1000"               # ~30-60s, 1000 trending stocks (comprehensive)
curl "http://localhost:18001/api/v1/stocks/52-week-gainers?max_pages=7"   # ~75s, 1350 stocks (all)

📊 API Usage

Stock Market Data (NEW! 🔥)

Get Trending Stocks (🚀 Recommended)

# Get trending stocks (default: 500 total stocks)
curl "http://localhost:18001/api/v1/stocks/trending"

# Fast mode - 200 total stocks
curl "http://localhost:18001/api/v1/stocks/trending?n=200"

# Comprehensive mode - 1000 total stocks
curl "http://localhost:18001/api/v1/stocks/trending?n=1000"

# Custom mix - 50 most active + remaining gainers to reach 300 total
curl "http://localhost:18001/api/v1/stocks/trending?n=300&most_active_limit=50"

Trending Stocks Response Format

{
  "success": true,
  "message": "Retrieved 500 trending stocks (170 most active + 330 gainers) in 18.5s",
  "data": {
    "trending_stocks": [
      {
        "symbol": "NVDA",
        "company_name": "NVIDIA Corporation",
        "current_price": "181.96",
        "change_amount": "+0.42",
        "change_percent": "+0.23%",
        "volume": "45.2M",
        "avg_volume": "42.1M",
        "category": "most_active",
        "rank_in_category": 1,
        "scraped_at": "2025-01-14T18:30:15.123456"
      },
      {
        "symbol": "TSLA",
        "company_name": "Tesla Inc",
        "current_price": "248.50",
        "change_amount": "+12.30",
        "change_percent": "+125.50%",
        "volume": "2.1M",
        "high_52w": "250.00",
        "category": "52_week_gainer",
        "rank_in_category": 1,
        "scraped_at": "2025-01-14T18:30:15.123456"
      },
      {
        "symbol": "AAPL",
        "company_name": "Apple Inc",
        "current_price": "174.50",
        "change_amount": "+2.30",
        "change_percent": "+1.33%",
        "volume": "52.1M",
        "avg_volume": "45.2M",
        "high_52w": "199.62",
        "category": "both",
        "rank_in_category": 3,
        "gainer_rank": 15,
        "scraped_at": "2025-01-14T18:30:15.123456"
      }
    ],
    "summary": {
      "total_stocks": 500,
      "most_active_count": 170,
      "gainers_count": 330,
      "unique_symbols": 485,
      "overlap_count": 15
    },
    "performance": {
      "elapsed_time_seconds": 18.5,
      "most_active_time": 3.1,
      "gainers_time": 15.4,
      "parallel_execution": true
    },
    "scraped_at": "2025-01-14T18:30:28.987654"
  },
  "metadata": {
    "sources": [
      "finance.yahoo.com/markets/stocks/most-active/",
      "finance.yahoo.com/markets/stocks/52-week-gainers/"
    ],
    "method": "parallel_scraping_with_intelligent_rate_limiting",
    "categories": ["most_active", "52_week_gainer", "both"],
    "rate_limit_bypass": "curl_cffi_chrome_impersonation",
    "deduplication": "symbol_based_with_category_merge"
  }
}

Key Features

🔥 Trending Stocks (Recommended):

• Best of Both Worlds: Combines immediate market activity with long-term performance
• Smart Deduplication: Automatically merges overlapping stocks and marks as 'both'
• Parallel Execution: Fetches both datasets simultaneously for optimal performance
• Flexible Configuration: Customize limits for each category independently
• Performance Tracking: Real-time elapsed time and performance metrics

📊 Categories:

• most_active: High trading volume, immediate market attention
• 52_week_gainer: Strong long-term price performance (up to 52 weeks)
• both: Stocks appearing in both categories (high activity + strong gains)

⚡ Performance Modes:

• Fast Mode (n=200): ~5-10 seconds, 200 total stocks
• Default Mode (n=500): ~15-30 seconds, 500 total stocks (recommended)
• Comprehensive Mode (n=1000+): ~30-60 seconds, 1000+ total stocks

Get Real-Time Most Active Stocks

# Get top 10 most active stocks
curl "http://localhost:18001/api/v1/stocks/most-active?limit=10"

# Get all available most active stocks (no limit)
curl "http://localhost:18001/api/v1/stocks/most-active"

Get 52-Week Top Gainers

# Get top 100 52-week gainers (fast, 1 page)
curl "http://localhost:18001/api/v1/stocks/52-week-gainers?limit=100&max_pages=1"

# Get default set (~600 gainers, 3 pages, recommended)
curl "http://localhost:18001/api/v1/stocks/52-week-gainers"

# Get first 1000 gainers (5 pages, slower but comprehensive)  
curl "http://localhost:18001/api/v1/stocks/52-week-gainers?limit=1000&max_pages=5"

Most Active Stocks Response Format

{
  "success": true,
  "message": "Retrieved 3 most active stocks",
  "data": {
    "stocks": [
      {
        "symbol": "NVDA",
        "company_name": "NVIDIA Corporation",
        "current_price": "181.81",
        "price_change_raw": "181.81 +0.26 (+0.15%)",
        "change_amount": "+0.26",
        "change_percent": "+0.15%",
        "volume": "93.425M",
        "avg_volume": "184.951M",
        "scraped_at": "2025-01-14T18:12:28.931780"
      }
    ],
    "total_available": 171,
    "returned_count": 3,
    "scraped_at": "2025-01-14T18:12:28.934007"
  },
  "metadata": {
    "source": "finance.yahoo.com",
    "endpoint": "markets/stocks/most-active",
    "method": "web_scraping",
    "rate_limit_bypass": "curl_cffi_chrome_impersonation"
  }
}

52-Week Gainers Response Format

{
  "success": true,
  "message": "Retrieved all 400 52-week gaining stocks in 13.9s",
  "data": {
    "stocks": [
      {
        "symbol": "CLGPF",
        "company_name": "Clean Seed Capital Group Ltd.",
        "current_price": "0.1500",
        "price_change_raw": "0.1500 +0.0750 (+100.00%)",
        "change_amount": "+0.0750",
        "change_percent": "+100.00%",
        "volume": "25,000",
        "avg_volume": "942",
        "high_52w": "0.15",
        "scraped_at": "2025-01-14T18:30:15.123456"
      }
    ],
    "total_available": 1350,
    "returned_count": 400,
    "pages_fetched": 2,
    "scraped_at": "2025-01-14T18:30:28.987654",
    "elapsed_time_seconds": 13.9
  },
  "metadata": {
    "source": "finance.yahoo.com",
    "endpoint": "markets/stocks/52-week-gainers",
    "method": "intelligent_web_scraping",
    "rate_limit_bypass": "curl_cffi_chrome_impersonation_with_smart_delays",
    "requests_made": 5
  }
}

Key Features

🚀 Most Active Stocks:

• Real-Time Data: Scraped directly from Yahoo Finance markets page
• Complete Dataset: Access to all ~170 most actively traded stocks
• Fast Performance: Sub-5 second response time
• Rich Information: Price, change, volume, and company details

📈 52-Week Gainers:

• Comprehensive Data: Access to 1,350+ top gaining stocks
• Intelligent Rate Limiting: Advanced delays to prevent blocking
• Configurable Scope: Choose 1-10 pages based on needs
• Performance Metrics: Real-time elapsed time tracking
• Pagination Support: Automatic multi-page handling

🛡️ Rate Limiting Technology:

• curl_cffi + Chrome Impersonation: Bypass standard rate limits
• Smart Delays: 1-3s base + 5s batch delays every 3 requests
• Progressive Delays: Increased delays for later pages
• Session Management: 5-minute session rotation
• Error Recovery: Automatic retry with exponential backoff

Performance Benchmarks

Most Active Stocks:

• Response Time: 3-5 seconds
• Data Volume: ~170 stocks (2 pages)
• Success Rate: 99.9%
• Rate Limits: Virtually eliminated

52-Week Gainers:

Pages	Stocks	Time	Use Case
1 page	~200	5-8s	Quick overview
2 pages	~400	12-15s	Moderate analysis
3 pages	~600	20-30s	Recommended default
5 pages	~1000	35-50s	Comprehensive analysis
7 pages	~1350	60-90s	Complete dataset

Rate Limiting Strategy:

• Base Delay: 1-3 seconds (randomized)
• Batch Delay: 5+ seconds every 3 requests
• Progressive Delay: +0.5s per page after page 3
• Session Rotation: Every 5 minutes
• Success Rate: 99.5% even at scale

Alpaca Market Data (NEW! 🆕)

Check Connection

# Verify Alpaca API key validity
curl "http://localhost:18001/api/v1/alpaca/status"

Get Price Data (DB Storage)

# Daily bars with DB persistence (same PriceDataResponse format as /price)
curl "http://localhost:18001/api/v1/alpaca/data/AAPL?interval=1d&start_date=2025-01-01&end_date=2025-01-31"

Raw Bars (No DB)

# Fetch bars directly from Alpaca without DB storage
curl "http://localhost:18001/api/v1/alpaca/bars/AAPL?interval=1d&start_date=2025-01-02&end_date=2025-01-10"

Intraday Candles

# 1-hour candles from Alpaca
curl "http://localhost:18001/api/v1/alpaca/intraday/AAPL?interval=1h&start_date=2025-03-10&end_date=2025-03-11"

# 5-minute candles
curl "http://localhost:18001/api/v1/alpaca/intraday/TSLA?interval=5m&start_date=2025-03-10&end_date=2025-03-10"

Key Features

• Official REST API: Proper authentication with API key/secret
• VWAP Included: Volume-weighted average price in every bar
• Trade Count: Number of trades per bar (trade_count field)
• Rate Limited: Built-in 200 req/min token bucket with auto-wait
• Retry Logic: 3x retry with exponential backoff on 429/5xx errors
• Independent: Completely separate from Yahoo Finance endpoints

FINRA Short Volume Data (NEW! 🆕)

Get Short Volume

# Short volume for AAPL (last 30 days, auto-ingests missing data)
curl "http://localhost:18001/api/v1/finra/short-volume/AAPL?days=30"

# With custom limit
curl "http://localhost:18001/api/v1/finra/short-volume/TSLA?days=60&limit=20"

Short Ratio History

# Aggregated short ratio over 60 days
curl "http://localhost:18001/api/v1/finra/short-ratio/AAPL?days=60"

Manual Ingest

# Ingest a single date
curl -X POST "http://localhost:18001/api/v1/finra/admin/ingest?date=2025-03-10"

# Ingest a date range
curl -X POST "http://localhost:18001/api/v1/finra/admin/ingest?start_date=2025-03-01&end_date=2025-03-10"

# Force re-ingest
curl -X POST "http://localhost:18001/api/v1/finra/admin/ingest?date=2025-03-10&force=true"

FINRA Response Format

{
  "symbol": "AAPL",
  "entries": [
    {
      "date": "2025-03-11",
      "symbol": "AAPL",
      "short_volume": 3214614.5,
      "short_exempt_volume": 29855.0,
      "total_volume": 7471743.3,
      "market": "B,Q,N",
      "short_ratio": 0.430236
    }
  ],
  "total_count": 21,
  "metadata": {
    "days_requested": 30,
    "start_date": "2025-02-10",
    "end_date": "2025-03-12"
  }
}

Key Features

• No API Key Required: Public FINRA CDN data
• Auto-Ingest: Missing data automatically downloaded on query
• Multi-Market: Data from B (NYSE TRF), Q (NASDAQ TRF), N (NYSE) markets
• Short Ratio: Pre-calculated short_volume / total_volume per record
• Aggregated History: short-ratio endpoint aggregates across all markets per day
• Date Range Ingest: Bulk backfill for historical data (weekdays only)

FRED Economic Data (NEW! 🏦)

Universal FRED API Proxy (🚀 Recommended)

Access ALL FRED API endpoints through our pass-through proxy:

# Popular economic indicators
curl "http://localhost:18001/api/v1/fred/proxy/series?series_id=GDP"           # GDP
curl "http://localhost:18001/api/v1/fred/proxy/series?series_id=UNRATE"       # Unemployment Rate
curl "http://localhost:18001/api/v1/fred/proxy/series?series_id=FEDFUNDS"     # Fed Funds Rate
curl "http://localhost:18001/api/v1/fred/proxy/series?series_id=CPIAUCSL"     # Consumer Price Index

# Historical data with observations
curl "http://localhost:18001/api/v1/fred/proxy/series/observations?series_id=UNRATE&limit=12"
curl "http://localhost:18001/api/v1/fred/proxy/series/observations?series_id=GDP&observation_start=2020-01-01"

# Category data
curl "http://localhost:18001/api/v1/fred/proxy/category?category_id=125"
curl "http://localhost:18001/api/v1/fred/proxy/category/children?category_id=13"

# Release information
curl "http://localhost:18001/api/v1/fred/proxy/release?release_id=53"
curl "http://localhost:18001/api/v1/fred/proxy/releases"

# Search functionality
curl "http://localhost:18001/api/v1/fred/proxy/series/search?search_text=unemployment&limit=25"

# Sources and tags
curl "http://localhost:18001/api/v1/fred/proxy/sources"
curl "http://localhost:18001/api/v1/fred/proxy/tags?limit=100"

# System information
curl "http://localhost:18001/api/v1/fred/endpoints"        # Discover all available endpoints
curl "http://localhost:18001/api/v1/fred/stats/usage"      # Monitor API usage (1000/day limit)

FRED Response Format

{
  "success": true,
  "data": {
    "id": "GDP",
    "title": "Gross Domestic Product",
    "units": "Billions of Dollars",
    "frequency": "Quarterly",
    "last_updated": "2025-07-30T07:56:35",
    "cached": true,
    "cached_at": "2025-01-14T10:30:00"
  },
  "metadata": {
    "source": "fred.stlouisfed.org",
    "cache_duration_hours": 24,
    "daily_api_limit": 1000
  }
}

FRED Features

🚀 Universal Proxy Access (NEW!):

• Complete FRED API Coverage: Access to ALL FRED endpoints via proxy
• Pass-through Architecture: Direct forwarding with rate limiting
• Parameter Auto-mapping: Automatic parameter handling for all endpoints
• Enhanced Statistics: Endpoint-specific usage tracking

🏦 Smart Caching System:

• 24-hour cache duration for series and observations
• Database persistence with SQLite/PostgreSQL
• Automatic cache invalidation after expiry
• Cache-first strategy to minimize API calls

📊 Daily Limit Management:

• 1,000 API calls per day (FRED limitation)
• Usage tracking with detailed statistics
• Graceful degradation when limit reached
• Cache fallback for expired data when limit hit

⚡ Performance Optimization:

• Sub-second response for cached data
• 2-5 second response for fresh API calls
• Batch operations for multiple series
• Usage monitoring and optimization suggestions

🔧 Dual Access Methods:

• Direct Endpoints: Optimized for series and observations with caching
• Proxy Endpoints: Universal access to all FRED functionality
• Automatic Fallback: Seamless switching between methods

ETF Holdings Data

Get Current ETF Holdings

# Get latest holdings for QQQ
curl "http://localhost:18001/api/v1/etf/holdings/QQQ"

# Get holdings for specific date (returns closest prior NPORT-P filing)
curl "http://localhost:18001/api/v1/etf/holdings/SPY?as_of_date=2021-06-30"

# Without detailed holdings (metadata only)
curl "http://localhost:18001/api/v1/etf/holdings/SPY?include_holdings=false"

# Bypass Redis cache to force fresh fetch
curl "http://localhost:18001/api/v1/etf/holdings/SPY?as_of_date=2021-06-30&force_refresh=true"

Key Features

• Automatic CIK Lookup: No need to know CIK numbers - just use ticker symbols
• Historical Data Support: Access NPORT filings from 2019 onwards
• Accurate Date Matching: Returns the closest NPORT-P filing ≤ as_of_date within 365-day window
• Stale Cache Prevention: DB snapshots >120 days from requested date trigger fresh SEC fetch
• Date Validation: Automatically checks if ETF existed on requested date
• Availability Info: Returns available date ranges when data not found
• Fast Performance: <0.1s response time for cached results; Redis-backed 1-hour cache

Response with Availability Information

{
  "ticker": "QQQM",
  "as_of_date": "2020-01-01",
  "success": false,
  "error": "ETF QQQM did not exist on 2020-01-01. Launched on 2020-10-13",
  "availability": {
    "exists_for_date": false,
    "etf_launch_date": "2020-10-13",
    "first_nport_date": "2021-01-31",
    "available_date_range": {
      "start": "2021-01-31",
      "end": "present"
    }
  }
}

Supported ETFs

Major ETFs with pre-configured mappings:

• Invesco: QQQ, QQQM, XLG
• SPDR: SPY, XLF, XLE, XLK, XLV, XLI
• iShares: IWM, EFA, EEM, TLT, AGG, SLV, MTUM
• Vanguard: VTI, VOO, VEA, VWO, BND
• ARK: ARKK, ARKQ, ARKW, ARKG, ARKF
• And many more...

Get Financial Data

🔥 Three Ways to Specify Time Period:

1. Period String (NEW! Most convenient):

# Last 1 year of data (excludes today for data availability)
curl "http://localhost:18001/api/v1/financial/data/AAPL?period=1y"

# Using POST
curl -X POST "http://localhost:18001/api/v1/financial/data" \
  -H "Content-Type: application/json" \
  -d '{"ticker": "AAPL", "period": "1y"}'

2. Date Range (Traditional):

# Specific date range
curl "http://localhost:18001/api/v1/financial/data/AAPL?start_date=2023-01-01&end_date=2023-12-31"

# Using POST
curl -X POST "http://localhost:18001/api/v1/financial/data" \
  -H "Content-Type: application/json" \
  -d '{
    "ticker": "AAPL",
    "start_date": "2023-01-01",
    "end_date": "2023-12-31",
    "period_type": "quarterly",
    "include_metrics": true,
    "force_refresh": false
  }'

3. Quarters (Quarter-based):

# Using POST
curl -X POST "http://localhost:18001/api/v1/financial/data" \
  -H "Content-Type: application/json" \
  -d '{
    "ticker": "AAPL",
    "quarters": ["2024Q1", "2024Q2", "2024Q3"]
  }'

⚠️ IMPORTANT: Period parameters now use yesterday as end date to ensure data availability since today's data might not be available yet.

Get News & Social Media Data (NEW! 🆕)

Get comprehensive news and social media sentiment data for any ticker:

# Get complete news and social media data
curl "http://localhost:18001/api/v1/news/AAPL?days_back=7&max_articles=20&include_social=true"

# Get news only (faster response)
curl "http://localhost:18001/api/v1/news/TSLA/news-only?days_back=5&max_articles=30"

# Get social media only
curl "http://localhost:18001/api/v1/news/NVDA/social-only?days_back=3&max_social_posts=15"

Response Format

{
  "ticker": "AAPL",
  "retrieved_at": "2025-08-10T17:40:04.781906",
  "news": {
    "total_articles": 12,
    "sources": {"yahoo_finance": 6, "newsapi": 6},
    "articles": [
      {
        "title": "Apple Reports Strong Q3 Results",
        "summary": "Apple exceeded expectations...",
        "url": "https://finance.yahoo.com/...",
        "source": "Yahoo Finance",
        "published_at": "2025-08-10T14:30:00",
        "author": "John Smith"
      }
    ]
  },
  "social_media": {
    "total_posts": 8,
    "platforms": {"reddit": 8},
    "posts": [
      {
        "title": "$AAPL breakout incoming?",
        "content": "Technical analysis shows...",
        "url": "https://reddit.com/r/stocks/...",
        "platform": "Reddit",
        "author": "trader123",
        "score": 245,
        "comments_count": 67,
        "subreddit": "stocks"
      }
    ]
  },
  "summary": {
    "total_items": 20,
    "time_range_days": 7,
    "newest_item": "2025-08-10T14:30:00",
    "oldest_item": "2025-08-03T09:15:00"
  }
}

Query Parameters

• days_back: Number of days to look back (1-30, default: 7)
• max_articles: Maximum news articles to return (1-100, default: 20)
• max_social_posts: Maximum social posts to return (1-100, default: 15)
• include_social: Include social media data (true/false, default: true)

Available Endpoints

Stock Market Data (NEW! 🔥)

• GET /api/v1/stocks/trending - Trending stocks combining most active + 52-week gainers (🚀 Recommended)
• GET /api/v1/stocks/most-active - Most actively traded stocks (optional limit parameter)
• GET /api/v1/stocks/52-week-gainers - 52-week top gaining stocks with intelligent rate limiting
• GET /api/v1/stocks/index/{index_name} - S&P 500 / Nasdaq 100 constituents from Wikipedia (24h cache)

Alpaca Market Data (NEW! 🆕)

• GET /api/v1/alpaca/status - Alpaca connection status and API key validation
• GET /api/v1/alpaca/bars/{ticker} - Raw bars from Alpaca (no DB storage)
• GET /api/v1/alpaca/data/{ticker} - OHLCV price data via Alpaca (with DB storage)
• GET /api/v1/alpaca/intraday/{ticker} - Intraday candles from Alpaca

FINRA Short Volume (NEW! 🆕)

• GET /api/v1/finra/short-volume/{symbol} - Short sale volume data (auto-ingests if missing)
• GET /api/v1/finra/short-ratio/{symbol} - Short ratio history (aggregated across markets)
• POST /api/v1/finra/admin/ingest - Manually ingest FINRA data for a date or range

FRED Economic Data (NEW! 🏦)

• GET /api/v1/fred/proxy/{endpoint:path} - Universal FRED API proxy with caching (🚀 Recommended)
• GET /api/v1/fred/endpoints - List all supported FRED API endpoints
• GET /api/v1/fred/stats/usage - API usage statistics and daily limit monitoring

ETF Holdings

• GET /api/v1/etf/holdings/{ticker} - Get ETF holdings with date support (closest NPORT-P filing)
• POST /api/v1/etf/admin/refresh-maps - Refresh ETF CIK mappings

Attention (Backtesting Features)

• GET /api/v1/attention/event/{ticker} - Event-date attention features (Wikipedia pageviews + GDELT coverage)
• GET /api/v1/attention/entity/{ticker} - Canonical entity info for ticker
• POST /api/v1/attention/admin/resolve/{ticker} - Resolve ticker → canonical entity
• POST /api/v1/attention/admin/collect/wiki/{ticker} - Collect Wikipedia pageview data
• POST /api/v1/attention/admin/collect/gdelt/{ticker} - Collect GDELT news article data

Financial Data

• GET /api/v1/financial/data/{ticker} - Simple financial data with query parameters
• POST /api/v1/financial/data - Detailed financial data request
• POST /api/v1/financial/data/bulk - Bulk financial data for multiple tickers

Price Data (Yahoo Finance)

• GET /api/v1/price/data/{ticker} - Simple price data with query parameters
• POST /api/v1/price/data - Detailed price data request
• POST /api/v1/price/data/bulk - Bulk price data for multiple tickers

News & Social Media

• GET /api/v1/news/{ticker} - Complete news and social media data
• GET /api/v1/news/{ticker}/news-only - News articles only (faster)
• GET /api/v1/news/{ticker}/social-only - Social media posts only

System & Admin

• GET /api/v1/health - API health check
• GET /api/v1/metadata/catalog - Data field catalog
• POST /api/v1/admin/migrate - Database migration
• GET /api/v1/admin/errors/logs - Error log management
• GET /api/v1/admin/errors/stats - Error statistics

Frontend

• Frontend: http://localhost:18002 (when using Docker)
• Error Log Viewer: http://localhost:18002/errors

🐍 Python Client Usage

Installation

# Copy the client file to your project
# stock_oracle_client.py is included in the repository

Basic Usage

from stock_oracle_client import StockOracleClient

# Initialize client
client = StockOracleClient("http://localhost:18001")

# Check API health
health = client.get_health()
print("API Status:", health["status"])

# Get financial data using period (recommended)
data = client.get_financial_data("AAPL", period="1y")
print(f"Found {len(data['financial_data'])} quarters of data")

# Get financial data using date range
data = client.get_financial_data(
    "MSFT",
    start_date="2023-01-01",
    end_date="2023-12-31",
    period_type="quarterly"
)

# Get financial data using quarters
data = client.get_financial_data(
    "GOOGL",
    quarters=["2024Q1", "2024Q2", "2024Q3"]
)

# Get price data (period automatically excludes today's data)
prices = client.get_price_data("AAPL", period="30d", interval="1d")

# Get news and social media data (NEW!)
news_data = client.get_news_social_data(
    ticker="AAPL", 
    days_back=7, 
    max_articles=20, 
    include_social=True
)

# Get news only (faster response)
news_only = client.get_news_only("TSLA", days_back=5, max_articles=30)

# Get social media only
social_only = client.get_social_only("NVDA", days_back=3, max_social_posts=15)

# Bulk operations
bulk_data = client.get_bulk_financial_data(
    tickers=["AAPL", "MSFT", "GOOGL"],
    period="2y"
)

Period Options

• Days: "1d", "7d", "30d"
• Months: "1m", "3m", "6m"
• Years: "1y", "2y", "5y", "10y"

⚠️ Note: Period parameters automatically use yesterday as end date for data availability.

🔧 Configuration

Environment Variables

# Application
APP_NAME=Stock_Oracle
API_PREFIX=/api/v1

# Database
DATABASE_URL=sqlite+aiosqlite:///./stock_oracle.db

# SEC Data
SEC_EMAIL=your@email.com  # Required for SEC API access

# Alpaca Market Data (optional — leave blank to disable)
ALPACA_API_KEY=your_alpaca_api_key
ALPACA_SECRET_KEY=your_alpaca_secret_key

# Cache
REDIS_URL=redis://localhost:16379/0  # Use redis://redis:6379/0 in Docker
CACHE_TTL=3600                         # Response cache TTL in seconds (default: 3600)

# Server Ports
API_PORT=18000
DB_PORT=15432    # PostgreSQL (if used)
REDIS_PORT=16379 # Redis cache

Docker Ports

• API: 18001 (external) → 18000 (internal)
• Frontend: 18002 (external) → 3000 (internal)
• PostgreSQL: 15433 (external) → 5432 (internal)
• Redis: 16380 (external) → 6379 (internal)

📈 Supported Metrics

✅ Implemented (13/16)

• Market Cap
• P/E Ratio (Trailing)
• P/B Ratio
• Debt-to-Equity
• Return on Equity (ROE)
• Return on Assets (ROA)
• Revenue Growth (YoY)
• Earnings Growth (YoY)
• Gross Margins
• Operating Margins
• Profit Margins
• Sector Classification
• Industry Classification

❌ Requires External Data (3/16)

• Forward P/E (analyst estimates needed)
• PEG Ratio (growth estimates needed)
• Beta (market correlation data needed)

🧪 Testing

# Run all tests
python -m pytest tests/ -v

# Run simple functionality tests
python tests/test_simple.py

# Run specific test categories
python -m pytest tests/test_financial.py -v
python -m pytest tests/test_integration.py -v

⚡ Server-side Response Caching (NEW)

Stock Oracle now supports Redis-backed response caching for the most frequently used single-ticker endpoints.

Targets

• POST /api/v1/price/data
• GET /api/v1/price/data/{ticker} (internally uses the same logic)
• POST /api/v1/financial/data
• GET /api/v1/financial/data/{ticker} (internally uses the same logic)

Bulk endpoints are not cached.

Behavior

• Cache store: Redis (REDIS_URL)
• TTL: CACHE_TTL seconds
• Bypass/refresh: set force_refresh=true in the request body or query
• Response headers:
  • X-Cache: HIT or MISS
  • ETag: strong hash for the response body
  • Cache-Control: public, max-age={CACHE_TTL}
  • X-Data-Source: redis-cache (price endpoint에서 캐시 히트 시)

Quick checks

# 1) MISS (store in cache)
curl -s -X POST "http://localhost:18001/api/v1/price/data" \
 -H "Content-Type: application/json" \
 -d '{"ticker":"AAPL","period":"3m","interval":"1d"}' -i | grep -Ei 'x-cache|etag|cache-control|x-data-source'

# 2) HIT (served from cache)
curl -s -X POST "http://localhost:18001/api/v1/price/data" \
 -H "Content-Type: application/json" \
 -d '{"ticker":"AAPL","period":"3m","interval":"1d"}' -i | grep -Ei 'x-cache|etag|cache-control|x-data-source'

# Force fresh fetch, bypass cache
curl -s -X POST "http://localhost:18001/api/v1/price/data" \
 -H "Content-Type: application/json" \
 -d '{"ticker":"AAPL","period":"3m","interval":"1d","force_refresh":true}' -i | grep -Ei 'x-cache|etag|cache-control|x-data-source'

Notes:

• If Redis is unreachable, the API gracefully continues without caching.
• Adjust REDIS_URL appropriately (Docker: redis://redis:6379/0).

📁 Project Structure

stock-oracle/
├── app/
│   ├── api/v1/endpoints/    # API route handlers
│   ├── core/               # Configuration and database
│   ├── models/             # SQLAlchemy database models
│   ├── schemas/            # Pydantic data validation
│   ├── services/           # Business logic services
│   └── main.py            # FastAPI application entry
├── tests/                 # Comprehensive test suite
├── scripts/               # Database initialization
├── data/                  # SQLite database storage
├── docker-compose.yml     # Docker orchestration
├── Dockerfile            # Container definition
├── requirements-*.txt    # Python dependencies
├── stock_oracle_analyzer.py # Core analysis engine
└── .env                  # Environment configuration

🔐 Security & Production

Security Features

• Input validation and sanitization
• SQL injection prevention
• Rate limiting (configurable)
• Environment-based configuration
• Secure secret management

Production Deployment

1. Database: Switch to PostgreSQL for production
2. Secrets: Use proper secret management (not .env files)
3. Monitoring: Add application monitoring and logging
4. Scaling: Use container orchestration (Kubernetes, Docker Swarm)
5. SSL: Enable HTTPS with proper certificates

🤝 Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests for new functionality
5. Ensure all tests pass
6. Submit a pull request

📄 License

[Your License Here]

🆘 Support

• Documentation: Check /docs endpoint for interactive API docs
• Issues: Report bugs and feature requests in the issue tracker
• Email: [your-support-email]

---

Stock Oracle - Empowering investment decisions with comprehensive SEC data analysis 🔮📈