# Stock Oracle 🔮

**Investment Data Analysis API using SEC filings**

Stock Oracle is a comprehensive investment analysis API that leverages SEC EDGAR filing data to provide detailed financial metrics and insights for informed investment decision-making.

## 🎯 Features

### Core Investment Metrics
- **Valuation Ratios**: P/E, P/B, P/S, EV/EBITDA
- **Profitability**: ROE, ROA, Gross/Operating/Net Margins
- **Growth Metrics**: Revenue Growth, Earnings Growth YoY
- **Financial Health**: Debt-to-Equity, Market Cap
- **Sector Analysis**: Industry and sector categorization

### Market Data Intelligence (NEW! 🆕)
- **Most Active Stocks**: Real-time ~170 most actively traded stocks (sub-5s response)
- **52-Week Gainers**: 1,350+ top gaining stocks with intelligent rate limiting (5-90s)
- **FRED Economic Data**: Federal Reserve economic indicators with smart caching (1000/day limit)
- **Advanced Web Scraping**: curl_cffi + Chrome impersonation bypasses rate limits
- **Smart Pagination**: Configurable page limits (1-10 pages) for performance tuning
- **Multi-Source News**: Yahoo Finance + NewsAPI integration
- **Social Media Analysis**: Reddit sentiment and discussions
- **Real-Time Updates**: Fresh content aggregation with performance monitoring
- **Sentiment Analysis**: Automated content sentiment scoring
- **Historical Context**: Customizable time periods (1-30 days)

### API Capabilities
- **RESTful API**: FastAPI-based with OpenAPI documentation
- **Database Caching**: Intelligent caching to avoid duplicate SEC parsing
- **Date Range Queries**: Flexible time period analysis
- **Data Validation**: Comprehensive request/response validation
- **Error Handling**: Detailed error categorization and reporting
- **Migration Support**: Database transfer capabilities

### Alpaca Market Data (NEW! 🆕)
- **Official REST API**: Alpaca Market Data v2 with proper authentication
- **OHLCV + VWAP**: Daily/intraday bars with volume-weighted average price
- **Independent Endpoints**: Separate from Yahoo Finance — users choose their source
- **DB Storage**: Alpaca bars stored with `data_source = "ALPACA"` for distinction
- **Rate Limiting**: Built-in token bucket (200 req/min) with retry + backoff

### FINRA Short Volume Data (NEW! 🆕)
- **RegSHO Short Sale Volume**: Daily short volume from FINRA public CDN
- **No API Key Required**: Public data, free access
- **Auto-Ingest**: Automatically downloads missing data on first query
- **Short Ratio History**: Aggregated short_ratio trends over time
- **Bulk Ingest**: Date range ingest for backfilling historical data

### Technical Stack
- **Backend**: FastAPI, SQLAlchemy (async), Pydantic
- **Frontend**: Next.js 15, React 18, TypeScript, TailwindCSS
- **Database**: SQLite (development) / PostgreSQL (production)
- **Caching**: Redis for performance optimization
- **Data Sources**:
  - SEC EDGAR via edgartools (ETF holdings, financial data)
  - Yahoo Finance via yfinance_plus (news & prices)
  - Yahoo Finance via intelligent curl_cffi scraping (market data)
    - Most Active Stocks (~170 stocks)
    - 52-Week Gainers (~1,350 stocks with rate limiting)
  - Alpaca Market Data v2 (OHLCV bars, optional API key)
  - FINRA RegSHO (short sale volume, public CDN)
  - NewsAPI (news articles)
  - Reddit API (social media sentiment)
- **Deployment**: Docker with docker-compose
- **Testing**: Comprehensive test suite with pytest

## 🚀 Quick Start

### Using Docker (Recommended)
```bash
# Clone and navigate
git clone <repo-url>
cd stock-oracle

# Start all services
docker-compose up -d

# Services will be available at:
# - API: http://localhost:18001
# - API Documentation: http://localhost:18001/docs
# - Frontend: http://localhost:18002
# - Error Log Viewer: http://localhost:18002/errors
# - PostgreSQL: localhost:15433
# - Redis: localhost:16380
```

### Local Development
```bash
# Install dependencies
pip install -r requirements-api.txt

# Set up environment
cp .env.example .env
# Edit .env with your settings

# Run the API
python -m uvicorn app.main:app --host 0.0.0.0 --port 18000 --reload
```

## ⚡ Quick Examples

### Get Market Overview
```bash
# Trending stocks - best of both worlds (recommended, default: 500 total stocks)
curl "http://localhost:18001/api/v1/stocks/trending"

# Most active stocks (fast)
curl "http://localhost:18001/api/v1/stocks/most-active?limit=10"

# Top 52-week gainers (moderate)
curl "http://localhost:18001/api/v1/stocks/52-week-gainers?max_pages=1&limit=50"

# ETF holdings analysis
curl "http://localhost:18001/api/v1/etf/holdings/QQQ"

# Economic indicators from FRED (cached)
curl "http://localhost:18001/api/v1/fred/proxy/series?series_id=GDP"
curl "http://localhost:18001/api/v1/fred/proxy/series/observations?series_id=UNRATE&limit=12"
```

### Performance Comparison
```bash
# Fast queries (< 15 seconds)
curl "http://localhost:18001/api/v1/stocks/trending?n=200"                # ~5-10s, 200 trending stocks (fast mode)
curl "http://localhost:18001/api/v1/stocks/most-active"                   # ~5s, 170 stocks
curl "http://localhost:18001/api/v1/stocks/52-week-gainers?max_pages=1"   # ~8s, 200 stocks

# Moderate queries (15-35 seconds) 
curl "http://localhost:18001/api/v1/stocks/trending"                      # ~15-30s, 500 trending stocks (default, recommended)
curl "http://localhost:18001/api/v1/stocks/52-week-gainers"               # ~25s, 600 stocks (default)

# Comprehensive queries (35+ seconds)
curl "http://localhost:18001/api/v1/stocks/trending?n=1000"               # ~30-60s, 1000 trending stocks (comprehensive)
curl "http://localhost:18001/api/v1/stocks/52-week-gainers?max_pages=7"   # ~75s, 1350 stocks (all)
```

## 📊 API Usage

### Stock Market Data (NEW! 🔥)

#### Get Trending Stocks (🚀 Recommended)
```bash
# Get trending stocks (default: 500 total stocks)
curl "http://localhost:18001/api/v1/stocks/trending"

# Fast mode - 200 total stocks
curl "http://localhost:18001/api/v1/stocks/trending?n=200"

# Comprehensive mode - 1000 total stocks
curl "http://localhost:18001/api/v1/stocks/trending?n=1000"

# Custom mix - 50 most active + remaining gainers to reach 300 total
curl "http://localhost:18001/api/v1/stocks/trending?n=300&most_active_limit=50"
```

#### Trending Stocks Response Format
```json
{
  "success": true,
  "message": "Retrieved 500 trending stocks (170 most active + 330 gainers) in 18.5s",
  "data": {
    "trending_stocks": [
      {
        "symbol": "NVDA",
        "company_name": "NVIDIA Corporation",
        "current_price": "181.96",
        "change_amount": "+0.42",
        "change_percent": "+0.23%",
        "volume": "45.2M",
        "avg_volume": "42.1M",
        "category": "most_active",
        "rank_in_category": 1,
        "scraped_at": "2025-01-14T18:30:15.123456"
      },
      {
        "symbol": "TSLA",
        "company_name": "Tesla Inc",
        "current_price": "248.50",
        "change_amount": "+12.30",
        "change_percent": "+125.50%",
        "volume": "2.1M",
        "high_52w": "250.00",
        "category": "52_week_gainer",
        "rank_in_category": 1,
        "scraped_at": "2025-01-14T18:30:15.123456"
      },
      {
        "symbol": "AAPL",
        "company_name": "Apple Inc",
        "current_price": "174.50",
        "change_amount": "+2.30",
        "change_percent": "+1.33%",
        "volume": "52.1M",
        "avg_volume": "45.2M",
        "high_52w": "199.62",
        "category": "both",
        "rank_in_category": 3,
        "gainer_rank": 15,
        "scraped_at": "2025-01-14T18:30:15.123456"
      }
    ],
    "summary": {
      "total_stocks": 500,
      "most_active_count": 170,
      "gainers_count": 330,
      "unique_symbols": 485,
      "overlap_count": 15
    },
    "performance": {
      "elapsed_time_seconds": 18.5,
      "most_active_time": 3.1,
      "gainers_time": 15.4,
      "parallel_execution": true
    },
    "scraped_at": "2025-01-14T18:30:28.987654"
  },
  "metadata": {
    "sources": [
      "finance.yahoo.com/markets/stocks/most-active/",
      "finance.yahoo.com/markets/stocks/52-week-gainers/"
    ],
    "method": "parallel_scraping_with_intelligent_rate_limiting",
    "categories": ["most_active", "52_week_gainer", "both"],
    "rate_limit_bypass": "curl_cffi_chrome_impersonation",
    "deduplication": "symbol_based_with_category_merge"
  }
}
```

#### Key Features

**🔥 Trending Stocks (Recommended)**:
- **Best of Both Worlds**: Combines immediate market activity with long-term performance
- **Smart Deduplication**: Automatically merges overlapping stocks and marks as 'both' 
- **Parallel Execution**: Fetches both datasets simultaneously for optimal performance
- **Flexible Configuration**: Customize limits for each category independently
- **Performance Tracking**: Real-time elapsed time and performance metrics

**📊 Categories**:
- **`most_active`**: High trading volume, immediate market attention
- **`52_week_gainer`**: Strong long-term price performance (up to 52 weeks)
- **`both`**: Stocks appearing in both categories (high activity + strong gains)

**⚡ Performance Modes**:
- **Fast Mode** (n=200): ~5-10 seconds, 200 total stocks
- **Default Mode** (n=500): ~15-30 seconds, 500 total stocks (recommended)
- **Comprehensive Mode** (n=1000+): ~30-60 seconds, 1000+ total stocks

#### Get Real-Time Most Active Stocks
```bash
# Get top 10 most active stocks
curl "http://localhost:18001/api/v1/stocks/most-active?limit=10"

# Get all available most active stocks (no limit)
curl "http://localhost:18001/api/v1/stocks/most-active"
```

#### Get 52-Week Top Gainers 
```bash
# Get top 100 52-week gainers (fast, 1 page)
curl "http://localhost:18001/api/v1/stocks/52-week-gainers?limit=100&max_pages=1"

# Get default set (~600 gainers, 3 pages, recommended)
curl "http://localhost:18001/api/v1/stocks/52-week-gainers"

# Get first 1000 gainers (5 pages, slower but comprehensive)  
curl "http://localhost:18001/api/v1/stocks/52-week-gainers?limit=1000&max_pages=5"
```

#### Most Active Stocks Response Format
```json
{
  "success": true,
  "message": "Retrieved 3 most active stocks",
  "data": {
    "stocks": [
      {
        "symbol": "NVDA",
        "company_name": "NVIDIA Corporation",
        "current_price": "181.81",
        "price_change_raw": "181.81 +0.26 (+0.15%)",
        "change_amount": "+0.26",
        "change_percent": "+0.15%",
        "volume": "93.425M",
        "avg_volume": "184.951M",
        "scraped_at": "2025-01-14T18:12:28.931780"
      }
    ],
    "total_available": 171,
    "returned_count": 3,
    "scraped_at": "2025-01-14T18:12:28.934007"
  },
  "metadata": {
    "source": "finance.yahoo.com",
    "endpoint": "markets/stocks/most-active",
    "method": "web_scraping",
    "rate_limit_bypass": "curl_cffi_chrome_impersonation"
  }
}
```

#### 52-Week Gainers Response Format
```json
{
  "success": true,
  "message": "Retrieved all 400 52-week gaining stocks in 13.9s",
  "data": {
    "stocks": [
      {
        "symbol": "CLGPF",
        "company_name": "Clean Seed Capital Group Ltd.",
        "current_price": "0.1500",
        "price_change_raw": "0.1500 +0.0750 (+100.00%)",
        "change_amount": "+0.0750",
        "change_percent": "+100.00%",
        "volume": "25,000",
        "avg_volume": "942",
        "high_52w": "0.15",
        "scraped_at": "2025-01-14T18:30:15.123456"
      }
    ],
    "total_available": 1350,
    "returned_count": 400,
    "pages_fetched": 2,
    "scraped_at": "2025-01-14T18:30:28.987654",
    "elapsed_time_seconds": 13.9
  },
  "metadata": {
    "source": "finance.yahoo.com",
    "endpoint": "markets/stocks/52-week-gainers",
    "method": "intelligent_web_scraping",
    "rate_limit_bypass": "curl_cffi_chrome_impersonation_with_smart_delays",
    "requests_made": 5
  }
}
```

#### Key Features

**🚀 Most Active Stocks**:
- **Real-Time Data**: Scraped directly from Yahoo Finance markets page
- **Complete Dataset**: Access to all ~170 most actively traded stocks
- **Fast Performance**: Sub-5 second response time
- **Rich Information**: Price, change, volume, and company details

**📈 52-Week Gainers**:
- **Comprehensive Data**: Access to 1,350+ top gaining stocks
- **Intelligent Rate Limiting**: Advanced delays to prevent blocking
- **Configurable Scope**: Choose 1-10 pages based on needs
- **Performance Metrics**: Real-time elapsed time tracking
- **Pagination Support**: Automatic multi-page handling

**🛡️ Rate Limiting Technology**:
- **curl_cffi + Chrome Impersonation**: Bypass standard rate limits
- **Smart Delays**: 1-3s base + 5s batch delays every 3 requests
- **Progressive Delays**: Increased delays for later pages
- **Session Management**: 5-minute session rotation
- **Error Recovery**: Automatic retry with exponential backoff

#### Performance Benchmarks

**Most Active Stocks**:
- **Response Time**: 3-5 seconds
- **Data Volume**: ~170 stocks (2 pages)
- **Success Rate**: 99.9%
- **Rate Limits**: Virtually eliminated

**52-Week Gainers**:
| Pages | Stocks | Time | Use Case |
|-------|--------|------|----------|
| 1 page | ~200 | 5-8s | Quick overview |
| 2 pages | ~400 | 12-15s | Moderate analysis |
| 3 pages | ~600 | 20-30s | **Recommended default** |
| 5 pages | ~1000 | 35-50s | Comprehensive analysis |
| 7 pages | ~1350 | 60-90s | Complete dataset |

**Rate Limiting Strategy**:
- **Base Delay**: 1-3 seconds (randomized)
- **Batch Delay**: 5+ seconds every 3 requests
- **Progressive Delay**: +0.5s per page after page 3
- **Session Rotation**: Every 5 minutes
- **Success Rate**: 99.5% even at scale

### Alpaca Market Data (NEW! 🆕)

#### Check Connection
```bash
# Verify Alpaca API key validity
curl "http://localhost:18001/api/v1/alpaca/status"
```

#### Get Price Data (DB Storage)
```bash
# Daily bars with DB persistence (same PriceDataResponse format as /price)
curl "http://localhost:18001/api/v1/alpaca/data/AAPL?interval=1d&start_date=2025-01-01&end_date=2025-01-31"
```

#### Raw Bars (No DB)
```bash
# Fetch bars directly from Alpaca without DB storage
curl "http://localhost:18001/api/v1/alpaca/bars/AAPL?interval=1d&start_date=2025-01-02&end_date=2025-01-10"
```

#### Intraday Candles
```bash
# 1-hour candles from Alpaca
curl "http://localhost:18001/api/v1/alpaca/intraday/AAPL?interval=1h&start_date=2025-03-10&end_date=2025-03-11"

# 5-minute candles
curl "http://localhost:18001/api/v1/alpaca/intraday/TSLA?interval=5m&start_date=2025-03-10&end_date=2025-03-10"
```

#### Key Features
- **Official REST API**: Proper authentication with API key/secret
- **VWAP Included**: Volume-weighted average price in every bar
- **Trade Count**: Number of trades per bar (`trade_count` field)
- **Rate Limited**: Built-in 200 req/min token bucket with auto-wait
- **Retry Logic**: 3x retry with exponential backoff on 429/5xx errors
- **Independent**: Completely separate from Yahoo Finance endpoints

### FINRA Short Volume Data (NEW! 🆕)

#### Get Short Volume
```bash
# Short volume for AAPL (last 30 days, auto-ingests missing data)
curl "http://localhost:18001/api/v1/finra/short-volume/AAPL?days=30"

# With custom limit
curl "http://localhost:18001/api/v1/finra/short-volume/TSLA?days=60&limit=20"
```

#### Short Ratio History
```bash
# Aggregated short ratio over 60 days
curl "http://localhost:18001/api/v1/finra/short-ratio/AAPL?days=60"
```

#### Manual Ingest
```bash
# Ingest a single date
curl -X POST "http://localhost:18001/api/v1/finra/admin/ingest?date=2025-03-10"

# Ingest a date range
curl -X POST "http://localhost:18001/api/v1/finra/admin/ingest?start_date=2025-03-01&end_date=2025-03-10"

# Force re-ingest
curl -X POST "http://localhost:18001/api/v1/finra/admin/ingest?date=2025-03-10&force=true"
```

#### FINRA Response Format
```json
{
  "symbol": "AAPL",
  "entries": [
    {
      "date": "2025-03-11",
      "symbol": "AAPL",
      "short_volume": 3214614.5,
      "short_exempt_volume": 29855.0,
      "total_volume": 7471743.3,
      "market": "B,Q,N",
      "short_ratio": 0.430236
    }
  ],
  "total_count": 21,
  "metadata": {
    "days_requested": 30,
    "start_date": "2025-02-10",
    "end_date": "2025-03-12"
  }
}
```

#### Key Features
- **No API Key Required**: Public FINRA CDN data
- **Auto-Ingest**: Missing data automatically downloaded on query
- **Multi-Market**: Data from B (NYSE TRF), Q (NASDAQ TRF), N (NYSE) markets
- **Short Ratio**: Pre-calculated `short_volume / total_volume` per record
- **Aggregated History**: `short-ratio` endpoint aggregates across all markets per day
- **Date Range Ingest**: Bulk backfill for historical data (weekdays only)

### FRED Economic Data (NEW! 🏦)

#### Universal FRED API Proxy (🚀 Recommended)
Access **ALL** FRED API endpoints through our pass-through proxy:

```bash
# Popular economic indicators
curl "http://localhost:18001/api/v1/fred/proxy/series?series_id=GDP"           # GDP
curl "http://localhost:18001/api/v1/fred/proxy/series?series_id=UNRATE"       # Unemployment Rate
curl "http://localhost:18001/api/v1/fred/proxy/series?series_id=FEDFUNDS"     # Fed Funds Rate
curl "http://localhost:18001/api/v1/fred/proxy/series?series_id=CPIAUCSL"     # Consumer Price Index

# Historical data with observations
curl "http://localhost:18001/api/v1/fred/proxy/series/observations?series_id=UNRATE&limit=12"
curl "http://localhost:18001/api/v1/fred/proxy/series/observations?series_id=GDP&observation_start=2020-01-01"

# Category data
curl "http://localhost:18001/api/v1/fred/proxy/category?category_id=125"
curl "http://localhost:18001/api/v1/fred/proxy/category/children?category_id=13"

# Release information
curl "http://localhost:18001/api/v1/fred/proxy/release?release_id=53"
curl "http://localhost:18001/api/v1/fred/proxy/releases"

# Search functionality
curl "http://localhost:18001/api/v1/fred/proxy/series/search?search_text=unemployment&limit=25"

# Sources and tags
curl "http://localhost:18001/api/v1/fred/proxy/sources"
curl "http://localhost:18001/api/v1/fred/proxy/tags?limit=100"

# System information
curl "http://localhost:18001/api/v1/fred/endpoints"        # Discover all available endpoints
curl "http://localhost:18001/api/v1/fred/stats/usage"      # Monitor API usage (1000/day limit)
```

#### FRED Response Format
```json
{
  "success": true,
  "data": {
    "id": "GDP",
    "title": "Gross Domestic Product",
    "units": "Billions of Dollars",
    "frequency": "Quarterly",
    "last_updated": "2025-07-30T07:56:35",
    "cached": true,
    "cached_at": "2025-01-14T10:30:00"
  },
  "metadata": {
    "source": "fred.stlouisfed.org",
    "cache_duration_hours": 24,
    "daily_api_limit": 1000
  }
}
```

#### FRED Features
**🚀 Universal Proxy Access (NEW!)**:
- **Complete FRED API Coverage**: Access to ALL FRED endpoints via proxy
- **Pass-through Architecture**: Direct forwarding with rate limiting
- **Parameter Auto-mapping**: Automatic parameter handling for all endpoints
- **Enhanced Statistics**: Endpoint-specific usage tracking

**🏦 Smart Caching System**:
- **24-hour cache duration** for series and observations
- **Database persistence** with SQLite/PostgreSQL
- **Automatic cache invalidation** after expiry
- **Cache-first strategy** to minimize API calls

**📊 Daily Limit Management**:
- **1,000 API calls per day** (FRED limitation)
- **Usage tracking** with detailed statistics
- **Graceful degradation** when limit reached
- **Cache fallback** for expired data when limit hit

**⚡ Performance Optimization**:
- **Sub-second response** for cached data
- **2-5 second response** for fresh API calls  
- **Batch operations** for multiple series
- **Usage monitoring** and optimization suggestions

**🔧 Dual Access Methods**:
- **Direct Endpoints**: Optimized for series and observations with caching
- **Proxy Endpoints**: Universal access to all FRED functionality
- **Automatic Fallback**: Seamless switching between methods

### ETF Holdings Data

#### Get Current ETF Holdings
```bash
# Get latest holdings for QQQ
curl "http://localhost:18001/api/v1/etf/holdings/QQQ"

# Get holdings for specific date
curl "http://localhost:18001/api/v1/etf/holdings/QQQM?as_of_date=2024-01-01"

# Without detailed holdings (metadata only)
curl "http://localhost:18001/api/v1/etf/holdings/SPY?include_holdings=false"
```

#### Key Features
- **Automatic CIK Lookup**: No need to know CIK numbers - just use ticker symbols
- **Historical Data Support**: Access NPORT filings from 2019 onwards
- **Date Validation**: Automatically checks if ETF existed on requested date
- **Availability Info**: Returns available date ranges when data not found
- **Fast Performance**: <0.1s response time with launch date caching

#### Response with Availability Information
```json
{
  "ticker": "QQQM",
  "as_of_date": "2020-01-01",
  "success": false,
  "error": "ETF QQQM did not exist on 2020-01-01. Launched on 2020-10-13",
  "availability": {
    "exists_for_date": false,
    "etf_launch_date": "2020-10-13",
    "first_nport_date": "2021-01-31",
    "available_date_range": {
      "start": "2021-01-31",
      "end": "present"
    }
  }
}
```

#### Supported ETFs
Major ETFs with pre-configured mappings:
- **Invesco**: QQQ, QQQM, XLG
- **SPDR**: SPY, XLF, XLE, XLK, XLV, XLI
- **iShares**: IWM, EFA, EEM, TLT, AGG, SLV, MTUM
- **Vanguard**: VTI, VOO, VEA, VWO, BND
- **ARK**: ARKK, ARKQ, ARKW, ARKG, ARKF
- And many more...

### Get Financial Data

#### 🔥 Three Ways to Specify Time Period:

1. **Period String** (NEW! Most convenient):
```bash
# Last 1 year of data (excludes today for data availability)
curl "http://localhost:18001/api/v1/financial/data/AAPL?period=1y"

# Using POST
curl -X POST "http://localhost:18001/api/v1/financial/data" \
  -H "Content-Type: application/json" \
  -d '{"ticker": "AAPL", "period": "1y"}'
```

2. **Date Range** (Traditional):
```bash
# Specific date range
curl "http://localhost:18001/api/v1/financial/data/AAPL?start_date=2023-01-01&end_date=2023-12-31"

# Using POST
curl -X POST "http://localhost:18001/api/v1/financial/data" \
  -H "Content-Type: application/json" \
  -d '{
    "ticker": "AAPL",
    "start_date": "2023-01-01",
    "end_date": "2023-12-31",
    "period_type": "quarterly",
    "include_metrics": true,
    "force_refresh": false
  }'
```

3. **Quarters** (Quarter-based):
```bash
# Using POST
curl -X POST "http://localhost:18001/api/v1/financial/data" \
  -H "Content-Type: application/json" \
  -d '{
    "ticker": "AAPL",
    "quarters": ["2024Q1", "2024Q2", "2024Q3"]
  }'
```

⚠️ **IMPORTANT**: Period parameters now use **yesterday** as end date to ensure data availability since today's data might not be available yet.

### Get News & Social Media Data (NEW! 🆕)

Get comprehensive news and social media sentiment data for any ticker:

```bash
# Get complete news and social media data
curl "http://localhost:18001/api/v1/news/AAPL?days_back=7&max_articles=20&include_social=true"

# Get news only (faster response)
curl "http://localhost:18001/api/v1/news/TSLA/news-only?days_back=5&max_articles=30"

# Get social media only
curl "http://localhost:18001/api/v1/news/NVDA/social-only?days_back=3&max_social_posts=15"
```

#### Response Format
```json
{
  "ticker": "AAPL",
  "retrieved_at": "2025-08-10T17:40:04.781906",
  "news": {
    "total_articles": 12,
    "sources": {"yahoo_finance": 6, "newsapi": 6},
    "articles": [
      {
        "title": "Apple Reports Strong Q3 Results",
        "summary": "Apple exceeded expectations...",
        "url": "https://finance.yahoo.com/...",
        "source": "Yahoo Finance",
        "published_at": "2025-08-10T14:30:00",
        "author": "John Smith"
      }
    ]
  },
  "social_media": {
    "total_posts": 8,
    "platforms": {"reddit": 8},
    "posts": [
      {
        "title": "$AAPL breakout incoming?",
        "content": "Technical analysis shows...",
        "url": "https://reddit.com/r/stocks/...",
        "platform": "Reddit",
        "author": "trader123",
        "score": 245,
        "comments_count": 67,
        "subreddit": "stocks"
      }
    ]
  },
  "summary": {
    "total_items": 20,
    "time_range_days": 7,
    "newest_item": "2025-08-10T14:30:00",
    "oldest_item": "2025-08-03T09:15:00"
  }
}
```

#### Query Parameters
- `days_back`: Number of days to look back (1-30, default: 7)
- `max_articles`: Maximum news articles to return (1-100, default: 20)
- `max_social_posts`: Maximum social posts to return (1-100, default: 15)
- `include_social`: Include social media data (true/false, default: true)

### Available Endpoints

#### Stock Market Data (NEW! 🔥)
- `GET /api/v1/stocks/trending` - **Trending stocks combining most active + 52-week gainers** (🚀 Recommended)
- `GET /api/v1/stocks/most-active` - Most actively traded stocks (optional limit parameter)
- `GET /api/v1/stocks/52-week-gainers` - 52-week top gaining stocks with intelligent rate limiting

#### Alpaca Market Data (NEW! 🆕)
- `GET /api/v1/alpaca/status` - Alpaca connection status and API key validation
- `GET /api/v1/alpaca/bars/{ticker}` - Raw bars from Alpaca (no DB storage)
- `GET /api/v1/alpaca/data/{ticker}` - OHLCV price data via Alpaca (with DB storage)
- `GET /api/v1/alpaca/intraday/{ticker}` - Intraday candles from Alpaca

#### FINRA Short Volume (NEW! 🆕)
- `GET /api/v1/finra/short-volume/{symbol}` - Short sale volume data (auto-ingests if missing)
- `GET /api/v1/finra/short-ratio/{symbol}` - Short ratio history (aggregated across markets)
- `POST /api/v1/finra/admin/ingest` - Manually ingest FINRA data for a date or range

#### FRED Economic Data (NEW! 🏦)
- `GET /api/v1/fred/proxy/{endpoint:path}` - **Universal FRED API proxy with caching** (🚀 Recommended)
- `GET /api/v1/fred/endpoints` - List all supported FRED API endpoints
- `GET /api/v1/fred/stats/usage` - API usage statistics and daily limit monitoring

#### ETF Holdings
- `GET /api/v1/etf/holdings/{ticker}` - Get ETF holdings with date support
- `POST /api/v1/etf/admin/refresh-maps` - Refresh ETF CIK mappings

#### Financial Data
- `GET /api/v1/financial/data/{ticker}` - Simple financial data with query parameters
- `POST /api/v1/financial/data` - Detailed financial data request
- `POST /api/v1/financial/data/bulk` - Bulk financial data for multiple tickers

#### Price Data (Yahoo Finance)
- `GET /api/v1/price/data/{ticker}` - Simple price data with query parameters
- `POST /api/v1/price/data` - Detailed price data request
- `POST /api/v1/price/data/bulk` - Bulk price data for multiple tickers

#### News & Social Media
- `GET /api/v1/news/{ticker}` - Complete news and social media data
- `GET /api/v1/news/{ticker}/news-only` - News articles only (faster)
- `GET /api/v1/news/{ticker}/social-only` - Social media posts only

#### System & Admin
- `GET /api/v1/health` - API health check
- `GET /api/v1/metadata/catalog` - Data field catalog
- `POST /api/v1/admin/migrate` - Database migration
- `GET /api/v1/admin/errors/logs` - Error log management
- `GET /api/v1/admin/errors/stats` - Error statistics

#### Frontend
- Frontend: http://localhost:18002 (when using Docker)
- Error Log Viewer: http://localhost:18002/errors

## 🐍 Python Client Usage

### Installation
```python
# Copy the client file to your project
# stock_oracle_client.py is included in the repository
```

### Basic Usage
```python
from stock_oracle_client import StockOracleClient

# Initialize client
client = StockOracleClient("http://localhost:18001")

# Check API health
health = client.get_health()
print("API Status:", health["status"])

# Get financial data using period (recommended)
data = client.get_financial_data("AAPL", period="1y")
print(f"Found {len(data['financial_data'])} quarters of data")

# Get financial data using date range
data = client.get_financial_data(
    "MSFT",
    start_date="2023-01-01",
    end_date="2023-12-31",
    period_type="quarterly"
)

# Get financial data using quarters
data = client.get_financial_data(
    "GOOGL",
    quarters=["2024Q1", "2024Q2", "2024Q3"]
)

# Get price data (period automatically excludes today's data)
prices = client.get_price_data("AAPL", period="30d", interval="1d")

# Get news and social media data (NEW!)
news_data = client.get_news_social_data(
    ticker="AAPL", 
    days_back=7, 
    max_articles=20, 
    include_social=True
)

# Get news only (faster response)
news_only = client.get_news_only("TSLA", days_back=5, max_articles=30)

# Get social media only
social_only = client.get_social_only("NVDA", days_back=3, max_social_posts=15)

# Bulk operations
bulk_data = client.get_bulk_financial_data(
    tickers=["AAPL", "MSFT", "GOOGL"],
    period="2y"
)
```

### Period Options
- **Days**: "1d", "7d", "30d"
- **Months**: "1m", "3m", "6m"  
- **Years**: "1y", "2y", "5y", "10y"

⚠️ **Note**: Period parameters automatically use **yesterday** as end date for data availability.

## 🔧 Configuration

### Environment Variables
```bash
# Application
APP_NAME=Stock_Oracle
API_PREFIX=/api/v1

# Database
DATABASE_URL=sqlite+aiosqlite:///./stock_oracle.db

# SEC Data
SEC_EMAIL=your@email.com  # Required for SEC API access

# Alpaca Market Data (optional — leave blank to disable)
ALPACA_API_KEY=your_alpaca_api_key
ALPACA_SECRET_KEY=your_alpaca_secret_key

# Cache
REDIS_URL=redis://localhost:16379/0  # Use redis://redis:6379/0 in Docker
CACHE_TTL=3600                         # Response cache TTL in seconds (default: 3600)

# Server Ports
API_PORT=18000
DB_PORT=15432    # PostgreSQL (if used)
REDIS_PORT=16379 # Redis cache
```

### Docker Ports
- **API**: 18001 (external) → 18000 (internal)
- **Frontend**: 18002 (external) → 3000 (internal)
- **PostgreSQL**: 15433 (external) → 5432 (internal)
- **Redis**: 16380 (external) → 6379 (internal)

## 📈 Supported Metrics

### ✅ Implemented (13/16)
- Market Cap
- P/E Ratio (Trailing)
- P/B Ratio
- Debt-to-Equity
- Return on Equity (ROE)
- Return on Assets (ROA)
- Revenue Growth (YoY)
- Earnings Growth (YoY)
- Gross Margins
- Operating Margins
- Profit Margins
- Sector Classification
- Industry Classification

### ❌ Requires External Data (3/16)
- Forward P/E (analyst estimates needed)
- PEG Ratio (growth estimates needed)
- Beta (market correlation data needed)

## 🧪 Testing

```bash
# Run all tests
python -m pytest tests/ -v

# Run simple functionality tests
python tests/test_simple.py

# Run specific test categories
python -m pytest tests/test_financial.py -v
python -m pytest tests/test_integration.py -v
```

## ⚡ Server-side Response Caching (NEW)

Stock Oracle now supports Redis-backed response caching for the most frequently used single-ticker endpoints.

### Targets
- `POST /api/v1/price/data`
- `GET /api/v1/price/data/{ticker}` (internally uses the same logic)
- `POST /api/v1/financial/data`
- `GET /api/v1/financial/data/{ticker}` (internally uses the same logic)

Bulk endpoints are not cached.

### Behavior
- Cache store: Redis (`REDIS_URL`)
- TTL: `CACHE_TTL` seconds
- Bypass/refresh: set `force_refresh=true` in the request body or query
- Response headers:
  - `X-Cache`: `HIT` or `MISS`
  - `ETag`: strong hash for the response body
  - `Cache-Control`: `public, max-age={CACHE_TTL}`
  - `X-Data-Source`: `redis-cache` (price endpoint에서 캐시 히트 시)

### Quick checks
```bash
# 1) MISS (store in cache)
curl -s -X POST "http://localhost:18001/api/v1/price/data" \
 -H "Content-Type: application/json" \
 -d '{"ticker":"AAPL","period":"3m","interval":"1d"}' -i | grep -Ei 'x-cache|etag|cache-control|x-data-source'

# 2) HIT (served from cache)
curl -s -X POST "http://localhost:18001/api/v1/price/data" \
 -H "Content-Type: application/json" \
 -d '{"ticker":"AAPL","period":"3m","interval":"1d"}' -i | grep -Ei 'x-cache|etag|cache-control|x-data-source'

# Force fresh fetch, bypass cache
curl -s -X POST "http://localhost:18001/api/v1/price/data" \
 -H "Content-Type: application/json" \
 -d '{"ticker":"AAPL","period":"3m","interval":"1d","force_refresh":true}' -i | grep -Ei 'x-cache|etag|cache-control|x-data-source'
```

Notes:
- If Redis is unreachable, the API gracefully continues without caching.
- Adjust `REDIS_URL` appropriately (Docker: `redis://redis:6379/0`).

## 📁 Project Structure

```
stock-oracle/
├── app/
│   ├── api/v1/endpoints/    # API route handlers
│   ├── core/               # Configuration and database
│   ├── models/             # SQLAlchemy database models
│   ├── schemas/            # Pydantic data validation
│   ├── services/           # Business logic services
│   └── main.py            # FastAPI application entry
├── tests/                 # Comprehensive test suite
├── scripts/               # Database initialization
├── data/                  # SQLite database storage
├── docker-compose.yml     # Docker orchestration
├── Dockerfile            # Container definition
├── requirements-*.txt    # Python dependencies
├── stock_oracle_analyzer.py # Core analysis engine
└── .env                  # Environment configuration
```

## 🔐 Security & Production

### Security Features
- Input validation and sanitization
- SQL injection prevention
- Rate limiting (configurable)
- Environment-based configuration
- Secure secret management

### Production Deployment
1. **Database**: Switch to PostgreSQL for production
2. **Secrets**: Use proper secret management (not .env files)
3. **Monitoring**: Add application monitoring and logging
4. **Scaling**: Use container orchestration (Kubernetes, Docker Swarm)
5. **SSL**: Enable HTTPS with proper certificates

## 🤝 Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests for new functionality
5. Ensure all tests pass
6. Submit a pull request

## 📄 License

[Your License Here]

## 🆘 Support

- **Documentation**: Check `/docs` endpoint for interactive API docs
- **Issues**: Report bugs and feature requests in the issue tracker
- **Email**: [your-support-email]

---

**Stock Oracle** - Empowering investment decisions with comprehensive SEC data analysis 🔮📈