You cannot select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
1049 lines
35 KiB
Markdown
1049 lines
35 KiB
Markdown
# Stock Oracle 🔮
|
|
|
|
**Investment Data Analysis API using SEC filings**
|
|
|
|
Stock Oracle is a comprehensive investment analysis API that leverages SEC EDGAR filing data to provide detailed financial metrics and insights for informed investment decision-making.
|
|
|
|
## 🎯 Features
|
|
|
|
### Core Investment Metrics
|
|
- **Valuation Ratios**: P/E, P/B, P/S, EV/EBITDA
|
|
- **Profitability**: ROE, ROA, Gross/Operating/Net Margins
|
|
- **Growth Metrics**: Revenue Growth, Earnings Growth YoY
|
|
- **Financial Health**: Debt-to-Equity, Market Cap
|
|
- **Sector Analysis**: Industry and sector categorization
|
|
|
|
### Market Data Intelligence (NEW! 🆕)
|
|
- **Most Active Stocks**: Real-time ~170 most actively traded stocks (sub-5s response)
|
|
- **52-Week Gainers**: 1,350+ top gaining stocks with intelligent rate limiting (5-90s)
|
|
- **Index Constituents**: S&P 500 / Nasdaq 100 구성 종목 조회 — Wikipedia 파싱, 24시간 캐시
|
|
- **FRED Economic Data**: Federal Reserve economic indicators with smart caching (1000/day limit)
|
|
- **Advanced Web Scraping**: curl_cffi + Chrome impersonation bypasses rate limits
|
|
- **Smart Pagination**: Configurable page limits (1-10 pages) for performance tuning
|
|
- **Multi-Source News**: Yahoo Finance + NewsAPI integration
|
|
- **Social Media Analysis**: Reddit sentiment and discussions
|
|
- **Real-Time Updates**: Fresh content aggregation with performance monitoring
|
|
- **Sentiment Analysis**: Automated content sentiment scoring
|
|
- **Historical Context**: Customizable time periods (1-30 days)
|
|
|
|
### API Capabilities
|
|
- **RESTful API**: FastAPI-based with OpenAPI documentation
|
|
- **Database Caching**: Intelligent caching to avoid duplicate SEC parsing
|
|
- **Date Range Queries**: Flexible time period analysis
|
|
- **Data Validation**: Comprehensive request/response validation
|
|
- **Error Handling**: Detailed error categorization and reporting
|
|
- **Migration Support**: Database transfer capabilities
|
|
|
|
### Attention Subsystem (NEW! 🆕)
|
|
- **Event-Centric Features**: Wikipedia pageviews + GDELT news coverage for backtesting
|
|
- **Spike Detection**: Daily pageview spike ratio and z-score vs. rolling baseline
|
|
- **GDELT Integration**: News article count from GDELT GKG (global knowledge graph)
|
|
- **Entity Resolution**: Ticker → canonical Wikipedia entity mapping
|
|
- **On-Demand Collection**: Collect data per event date without full precomputation
|
|
|
|
### Alpaca Market Data (NEW! 🆕)
|
|
- **Official REST API**: Alpaca Market Data v2 with proper authentication
|
|
- **OHLCV + VWAP**: Daily/intraday bars with volume-weighted average price
|
|
- **Independent Endpoints**: Separate from Yahoo Finance — users choose their source
|
|
- **DB Storage**: Alpaca bars stored with `data_source = "ALPACA"` for distinction
|
|
- **Rate Limiting**: Built-in token bucket (200 req/min) with retry + backoff
|
|
|
|
### FINRA Short Volume Data (NEW! 🆕)
|
|
- **RegSHO Short Sale Volume**: Daily short volume from FINRA public CDN
|
|
- **No API Key Required**: Public data, free access
|
|
- **Auto-Ingest**: Automatically downloads missing data on first query
|
|
- **Short Ratio History**: Aggregated short_ratio trends over time
|
|
- **Bulk Ingest**: Date range ingest for backfilling historical data
|
|
|
|
### Technical Stack
|
|
- **Backend**: FastAPI, SQLAlchemy (async), Pydantic
|
|
- **Frontend**: Next.js 15, React 18, TypeScript, TailwindCSS
|
|
- **Database**: SQLite (development) / PostgreSQL (production)
|
|
- **Caching**: Redis for performance optimization
|
|
- **Data Sources**:
|
|
- SEC EDGAR via edgartools (ETF holdings, financial data)
|
|
- Yahoo Finance via yfinance_plus (news & prices)
|
|
- Yahoo Finance via intelligent curl_cffi scraping (market data)
|
|
- Most Active Stocks (~170 stocks)
|
|
- 52-Week Gainers (~1,350 stocks with rate limiting)
|
|
- Alpaca Market Data v2 (OHLCV bars, optional API key)
|
|
- FINRA RegSHO (short sale volume, public CDN)
|
|
- NewsAPI (news articles)
|
|
- Reddit API (social media sentiment)
|
|
- **Deployment**: Docker with docker-compose
|
|
- **Testing**: Comprehensive test suite with pytest
|
|
|
|
## 🚀 Quick Start
|
|
|
|
### Using Docker (Recommended)
|
|
```bash
|
|
# Clone and navigate
|
|
git clone <repo-url>
|
|
cd stock-oracle
|
|
|
|
# Start all services
|
|
docker-compose up -d
|
|
|
|
# Services will be available at:
|
|
# - API: http://localhost:18001
|
|
# - API Documentation: http://localhost:18001/docs
|
|
# - Frontend: http://localhost:18002
|
|
# - Error Log Viewer: http://localhost:18002/errors
|
|
# - PostgreSQL: localhost:15433
|
|
# - Redis: localhost:16380
|
|
```
|
|
|
|
### Local Development
|
|
```bash
|
|
# Install dependencies
|
|
pip install -r requirements-api.txt
|
|
|
|
# Set up environment
|
|
cp .env.example .env
|
|
# Edit .env with your settings
|
|
|
|
# Start all services (first time)
|
|
docker-compose up -d
|
|
|
|
# Restart API after code changes (code is volume-mounted, no rebuild needed)
|
|
just run dev
|
|
```
|
|
|
|
> **Note**: Always use `just run dev` to restart the API. The `app/` directory is volume-mounted into the container, so code changes take effect on restart without rebuilding the image.
|
|
|
|
## ⚡ Quick Examples
|
|
|
|
### Get Market Overview
|
|
```bash
|
|
# Trending stocks - best of both worlds (recommended, default: 500 total stocks)
|
|
curl "http://localhost:18001/api/v1/stocks/trending"
|
|
|
|
# Most active stocks (fast)
|
|
curl "http://localhost:18001/api/v1/stocks/most-active?limit=10"
|
|
|
|
# Top 52-week gainers (moderate)
|
|
curl "http://localhost:18001/api/v1/stocks/52-week-gainers?max_pages=1&limit=50"
|
|
|
|
# ETF holdings analysis
|
|
curl "http://localhost:18001/api/v1/etf/holdings/QQQ"
|
|
|
|
# Economic indicators from FRED (cached)
|
|
curl "http://localhost:18001/api/v1/fred/proxy/series?series_id=GDP"
|
|
curl "http://localhost:18001/api/v1/fred/proxy/series/observations?series_id=UNRATE&limit=12"
|
|
```
|
|
|
|
### Performance Comparison
|
|
```bash
|
|
# Fast queries (< 15 seconds)
|
|
curl "http://localhost:18001/api/v1/stocks/trending?n=200" # ~5-10s, 200 trending stocks (fast mode)
|
|
curl "http://localhost:18001/api/v1/stocks/most-active" # ~5s, 170 stocks
|
|
curl "http://localhost:18001/api/v1/stocks/52-week-gainers?max_pages=1" # ~8s, 200 stocks
|
|
|
|
# Moderate queries (15-35 seconds)
|
|
curl "http://localhost:18001/api/v1/stocks/trending" # ~15-30s, 500 trending stocks (default, recommended)
|
|
curl "http://localhost:18001/api/v1/stocks/52-week-gainers" # ~25s, 600 stocks (default)
|
|
|
|
# Comprehensive queries (35+ seconds)
|
|
curl "http://localhost:18001/api/v1/stocks/trending?n=1000" # ~30-60s, 1000 trending stocks (comprehensive)
|
|
curl "http://localhost:18001/api/v1/stocks/52-week-gainers?max_pages=7" # ~75s, 1350 stocks (all)
|
|
```
|
|
|
|
## 📊 API Usage
|
|
|
|
### Stock Market Data (NEW! 🔥)
|
|
|
|
#### Get Trending Stocks (🚀 Recommended)
|
|
```bash
|
|
# Get trending stocks (default: 500 total stocks)
|
|
curl "http://localhost:18001/api/v1/stocks/trending"
|
|
|
|
# Fast mode - 200 total stocks
|
|
curl "http://localhost:18001/api/v1/stocks/trending?n=200"
|
|
|
|
# Comprehensive mode - 1000 total stocks
|
|
curl "http://localhost:18001/api/v1/stocks/trending?n=1000"
|
|
|
|
# Custom mix - 50 most active + remaining gainers to reach 300 total
|
|
curl "http://localhost:18001/api/v1/stocks/trending?n=300&most_active_limit=50"
|
|
```
|
|
|
|
#### Trending Stocks Response Format
|
|
```json
|
|
{
|
|
"success": true,
|
|
"message": "Retrieved 500 trending stocks (170 most active + 330 gainers) in 18.5s",
|
|
"data": {
|
|
"trending_stocks": [
|
|
{
|
|
"symbol": "NVDA",
|
|
"company_name": "NVIDIA Corporation",
|
|
"current_price": "181.96",
|
|
"change_amount": "+0.42",
|
|
"change_percent": "+0.23%",
|
|
"volume": "45.2M",
|
|
"avg_volume": "42.1M",
|
|
"category": "most_active",
|
|
"rank_in_category": 1,
|
|
"scraped_at": "2025-01-14T18:30:15.123456"
|
|
},
|
|
{
|
|
"symbol": "TSLA",
|
|
"company_name": "Tesla Inc",
|
|
"current_price": "248.50",
|
|
"change_amount": "+12.30",
|
|
"change_percent": "+125.50%",
|
|
"volume": "2.1M",
|
|
"high_52w": "250.00",
|
|
"category": "52_week_gainer",
|
|
"rank_in_category": 1,
|
|
"scraped_at": "2025-01-14T18:30:15.123456"
|
|
},
|
|
{
|
|
"symbol": "AAPL",
|
|
"company_name": "Apple Inc",
|
|
"current_price": "174.50",
|
|
"change_amount": "+2.30",
|
|
"change_percent": "+1.33%",
|
|
"volume": "52.1M",
|
|
"avg_volume": "45.2M",
|
|
"high_52w": "199.62",
|
|
"category": "both",
|
|
"rank_in_category": 3,
|
|
"gainer_rank": 15,
|
|
"scraped_at": "2025-01-14T18:30:15.123456"
|
|
}
|
|
],
|
|
"summary": {
|
|
"total_stocks": 500,
|
|
"most_active_count": 170,
|
|
"gainers_count": 330,
|
|
"unique_symbols": 485,
|
|
"overlap_count": 15
|
|
},
|
|
"performance": {
|
|
"elapsed_time_seconds": 18.5,
|
|
"most_active_time": 3.1,
|
|
"gainers_time": 15.4,
|
|
"parallel_execution": true
|
|
},
|
|
"scraped_at": "2025-01-14T18:30:28.987654"
|
|
},
|
|
"metadata": {
|
|
"sources": [
|
|
"finance.yahoo.com/markets/stocks/most-active/",
|
|
"finance.yahoo.com/markets/stocks/52-week-gainers/"
|
|
],
|
|
"method": "parallel_scraping_with_intelligent_rate_limiting",
|
|
"categories": ["most_active", "52_week_gainer", "both"],
|
|
"rate_limit_bypass": "curl_cffi_chrome_impersonation",
|
|
"deduplication": "symbol_based_with_category_merge"
|
|
}
|
|
}
|
|
```
|
|
|
|
#### Key Features
|
|
|
|
**🔥 Trending Stocks (Recommended)**:
|
|
- **Best of Both Worlds**: Combines immediate market activity with long-term performance
|
|
- **Smart Deduplication**: Automatically merges overlapping stocks and marks as 'both'
|
|
- **Parallel Execution**: Fetches both datasets simultaneously for optimal performance
|
|
- **Flexible Configuration**: Customize limits for each category independently
|
|
- **Performance Tracking**: Real-time elapsed time and performance metrics
|
|
|
|
**📊 Categories**:
|
|
- **`most_active`**: High trading volume, immediate market attention
|
|
- **`52_week_gainer`**: Strong long-term price performance (up to 52 weeks)
|
|
- **`both`**: Stocks appearing in both categories (high activity + strong gains)
|
|
|
|
**⚡ Performance Modes**:
|
|
- **Fast Mode** (n=200): ~5-10 seconds, 200 total stocks
|
|
- **Default Mode** (n=500): ~15-30 seconds, 500 total stocks (recommended)
|
|
- **Comprehensive Mode** (n=1000+): ~30-60 seconds, 1000+ total stocks
|
|
|
|
#### Get Real-Time Most Active Stocks
|
|
```bash
|
|
# Get top 10 most active stocks
|
|
curl "http://localhost:18001/api/v1/stocks/most-active?limit=10"
|
|
|
|
# Get all available most active stocks (no limit)
|
|
curl "http://localhost:18001/api/v1/stocks/most-active"
|
|
```
|
|
|
|
#### Get 52-Week Top Gainers
|
|
```bash
|
|
# Get top 100 52-week gainers (fast, 1 page)
|
|
curl "http://localhost:18001/api/v1/stocks/52-week-gainers?limit=100&max_pages=1"
|
|
|
|
# Get default set (~600 gainers, 3 pages, recommended)
|
|
curl "http://localhost:18001/api/v1/stocks/52-week-gainers"
|
|
|
|
# Get first 1000 gainers (5 pages, slower but comprehensive)
|
|
curl "http://localhost:18001/api/v1/stocks/52-week-gainers?limit=1000&max_pages=5"
|
|
```
|
|
|
|
#### Most Active Stocks Response Format
|
|
```json
|
|
{
|
|
"success": true,
|
|
"message": "Retrieved 3 most active stocks",
|
|
"data": {
|
|
"stocks": [
|
|
{
|
|
"symbol": "NVDA",
|
|
"company_name": "NVIDIA Corporation",
|
|
"current_price": "181.81",
|
|
"price_change_raw": "181.81 +0.26 (+0.15%)",
|
|
"change_amount": "+0.26",
|
|
"change_percent": "+0.15%",
|
|
"volume": "93.425M",
|
|
"avg_volume": "184.951M",
|
|
"scraped_at": "2025-01-14T18:12:28.931780"
|
|
}
|
|
],
|
|
"total_available": 171,
|
|
"returned_count": 3,
|
|
"scraped_at": "2025-01-14T18:12:28.934007"
|
|
},
|
|
"metadata": {
|
|
"source": "finance.yahoo.com",
|
|
"endpoint": "markets/stocks/most-active",
|
|
"method": "web_scraping",
|
|
"rate_limit_bypass": "curl_cffi_chrome_impersonation"
|
|
}
|
|
}
|
|
```
|
|
|
|
#### 52-Week Gainers Response Format
|
|
```json
|
|
{
|
|
"success": true,
|
|
"message": "Retrieved all 400 52-week gaining stocks in 13.9s",
|
|
"data": {
|
|
"stocks": [
|
|
{
|
|
"symbol": "CLGPF",
|
|
"company_name": "Clean Seed Capital Group Ltd.",
|
|
"current_price": "0.1500",
|
|
"price_change_raw": "0.1500 +0.0750 (+100.00%)",
|
|
"change_amount": "+0.0750",
|
|
"change_percent": "+100.00%",
|
|
"volume": "25,000",
|
|
"avg_volume": "942",
|
|
"high_52w": "0.15",
|
|
"scraped_at": "2025-01-14T18:30:15.123456"
|
|
}
|
|
],
|
|
"total_available": 1350,
|
|
"returned_count": 400,
|
|
"pages_fetched": 2,
|
|
"scraped_at": "2025-01-14T18:30:28.987654",
|
|
"elapsed_time_seconds": 13.9
|
|
},
|
|
"metadata": {
|
|
"source": "finance.yahoo.com",
|
|
"endpoint": "markets/stocks/52-week-gainers",
|
|
"method": "intelligent_web_scraping",
|
|
"rate_limit_bypass": "curl_cffi_chrome_impersonation_with_smart_delays",
|
|
"requests_made": 5
|
|
}
|
|
}
|
|
```
|
|
|
|
#### Key Features
|
|
|
|
**🚀 Most Active Stocks**:
|
|
- **Real-Time Data**: Scraped directly from Yahoo Finance markets page
|
|
- **Complete Dataset**: Access to all ~170 most actively traded stocks
|
|
- **Fast Performance**: Sub-5 second response time
|
|
- **Rich Information**: Price, change, volume, and company details
|
|
|
|
**📈 52-Week Gainers**:
|
|
- **Comprehensive Data**: Access to 1,350+ top gaining stocks
|
|
- **Intelligent Rate Limiting**: Advanced delays to prevent blocking
|
|
- **Configurable Scope**: Choose 1-10 pages based on needs
|
|
- **Performance Metrics**: Real-time elapsed time tracking
|
|
- **Pagination Support**: Automatic multi-page handling
|
|
|
|
**🛡️ Rate Limiting Technology**:
|
|
- **curl_cffi + Chrome Impersonation**: Bypass standard rate limits
|
|
- **Smart Delays**: 1-3s base + 5s batch delays every 3 requests
|
|
- **Progressive Delays**: Increased delays for later pages
|
|
- **Session Management**: 5-minute session rotation
|
|
- **Error Recovery**: Automatic retry with exponential backoff
|
|
|
|
#### Performance Benchmarks
|
|
|
|
**Most Active Stocks**:
|
|
- **Response Time**: 3-5 seconds
|
|
- **Data Volume**: ~170 stocks (2 pages)
|
|
- **Success Rate**: 99.9%
|
|
- **Rate Limits**: Virtually eliminated
|
|
|
|
**52-Week Gainers**:
|
|
| Pages | Stocks | Time | Use Case |
|
|
|-------|--------|------|----------|
|
|
| 1 page | ~200 | 5-8s | Quick overview |
|
|
| 2 pages | ~400 | 12-15s | Moderate analysis |
|
|
| 3 pages | ~600 | 20-30s | **Recommended default** |
|
|
| 5 pages | ~1000 | 35-50s | Comprehensive analysis |
|
|
| 7 pages | ~1350 | 60-90s | Complete dataset |
|
|
|
|
**Rate Limiting Strategy**:
|
|
- **Base Delay**: 1-3 seconds (randomized)
|
|
- **Batch Delay**: 5+ seconds every 3 requests
|
|
- **Progressive Delay**: +0.5s per page after page 3
|
|
- **Session Rotation**: Every 5 minutes
|
|
- **Success Rate**: 99.5% even at scale
|
|
|
|
### Alpaca Market Data (NEW! 🆕)
|
|
|
|
#### Check Connection
|
|
```bash
|
|
# Verify Alpaca API key validity
|
|
curl "http://localhost:18001/api/v1/alpaca/status"
|
|
```
|
|
|
|
#### Get Price Data (DB Storage)
|
|
```bash
|
|
# Daily bars with DB persistence (same PriceDataResponse format as /price)
|
|
curl "http://localhost:18001/api/v1/alpaca/data/AAPL?interval=1d&start_date=2025-01-01&end_date=2025-01-31"
|
|
```
|
|
|
|
#### Raw Bars (No DB)
|
|
```bash
|
|
# Fetch bars directly from Alpaca without DB storage
|
|
curl "http://localhost:18001/api/v1/alpaca/bars/AAPL?interval=1d&start_date=2025-01-02&end_date=2025-01-10"
|
|
```
|
|
|
|
#### Intraday Candles
|
|
```bash
|
|
# 1-hour candles from Alpaca
|
|
curl "http://localhost:18001/api/v1/alpaca/intraday/AAPL?interval=1h&start_date=2025-03-10&end_date=2025-03-11"
|
|
|
|
# 5-minute candles
|
|
curl "http://localhost:18001/api/v1/alpaca/intraday/TSLA?interval=5m&start_date=2025-03-10&end_date=2025-03-10"
|
|
```
|
|
|
|
#### Key Features
|
|
- **Official REST API**: Proper authentication with API key/secret
|
|
- **VWAP Included**: Volume-weighted average price in every bar
|
|
- **Trade Count**: Number of trades per bar (`trade_count` field)
|
|
- **Rate Limited**: Built-in 200 req/min token bucket with auto-wait
|
|
- **Retry Logic**: 3x retry with exponential backoff on 429/5xx errors
|
|
- **Independent**: Completely separate from Yahoo Finance endpoints
|
|
|
|
### FINRA Short Volume Data (NEW! 🆕)
|
|
|
|
#### Get Short Volume
|
|
```bash
|
|
# Short volume for AAPL (last 30 days, auto-ingests missing data)
|
|
curl "http://localhost:18001/api/v1/finra/short-volume/AAPL?days=30"
|
|
|
|
# With custom limit
|
|
curl "http://localhost:18001/api/v1/finra/short-volume/TSLA?days=60&limit=20"
|
|
```
|
|
|
|
#### Short Ratio History
|
|
```bash
|
|
# Aggregated short ratio over 60 days
|
|
curl "http://localhost:18001/api/v1/finra/short-ratio/AAPL?days=60"
|
|
```
|
|
|
|
#### Manual Ingest
|
|
```bash
|
|
# Ingest a single date
|
|
curl -X POST "http://localhost:18001/api/v1/finra/admin/ingest?date=2025-03-10"
|
|
|
|
# Ingest a date range
|
|
curl -X POST "http://localhost:18001/api/v1/finra/admin/ingest?start_date=2025-03-01&end_date=2025-03-10"
|
|
|
|
# Force re-ingest
|
|
curl -X POST "http://localhost:18001/api/v1/finra/admin/ingest?date=2025-03-10&force=true"
|
|
```
|
|
|
|
#### FINRA Response Format
|
|
```json
|
|
{
|
|
"symbol": "AAPL",
|
|
"entries": [
|
|
{
|
|
"date": "2025-03-11",
|
|
"symbol": "AAPL",
|
|
"short_volume": 3214614.5,
|
|
"short_exempt_volume": 29855.0,
|
|
"total_volume": 7471743.3,
|
|
"market": "B,Q,N",
|
|
"short_ratio": 0.430236
|
|
}
|
|
],
|
|
"total_count": 21,
|
|
"metadata": {
|
|
"days_requested": 30,
|
|
"start_date": "2025-02-10",
|
|
"end_date": "2025-03-12"
|
|
}
|
|
}
|
|
```
|
|
|
|
#### Key Features
|
|
- **No API Key Required**: Public FINRA CDN data
|
|
- **Auto-Ingest**: Missing data automatically downloaded on query
|
|
- **Multi-Market**: Data from B (NYSE TRF), Q (NASDAQ TRF), N (NYSE) markets
|
|
- **Short Ratio**: Pre-calculated `short_volume / total_volume` per record
|
|
- **Aggregated History**: `short-ratio` endpoint aggregates across all markets per day
|
|
- **Date Range Ingest**: Bulk backfill for historical data (weekdays only)
|
|
|
|
### FRED Economic Data (NEW! 🏦)
|
|
|
|
#### Universal FRED API Proxy (🚀 Recommended)
|
|
Access **ALL** FRED API endpoints through our pass-through proxy:
|
|
|
|
```bash
|
|
# Popular economic indicators
|
|
curl "http://localhost:18001/api/v1/fred/proxy/series?series_id=GDP" # GDP
|
|
curl "http://localhost:18001/api/v1/fred/proxy/series?series_id=UNRATE" # Unemployment Rate
|
|
curl "http://localhost:18001/api/v1/fred/proxy/series?series_id=FEDFUNDS" # Fed Funds Rate
|
|
curl "http://localhost:18001/api/v1/fred/proxy/series?series_id=CPIAUCSL" # Consumer Price Index
|
|
|
|
# Historical data with observations
|
|
curl "http://localhost:18001/api/v1/fred/proxy/series/observations?series_id=UNRATE&limit=12"
|
|
curl "http://localhost:18001/api/v1/fred/proxy/series/observations?series_id=GDP&observation_start=2020-01-01"
|
|
|
|
# Category data
|
|
curl "http://localhost:18001/api/v1/fred/proxy/category?category_id=125"
|
|
curl "http://localhost:18001/api/v1/fred/proxy/category/children?category_id=13"
|
|
|
|
# Release information
|
|
curl "http://localhost:18001/api/v1/fred/proxy/release?release_id=53"
|
|
curl "http://localhost:18001/api/v1/fred/proxy/releases"
|
|
|
|
# Search functionality
|
|
curl "http://localhost:18001/api/v1/fred/proxy/series/search?search_text=unemployment&limit=25"
|
|
|
|
# Sources and tags
|
|
curl "http://localhost:18001/api/v1/fred/proxy/sources"
|
|
curl "http://localhost:18001/api/v1/fred/proxy/tags?limit=100"
|
|
|
|
# System information
|
|
curl "http://localhost:18001/api/v1/fred/endpoints" # Discover all available endpoints
|
|
curl "http://localhost:18001/api/v1/fred/stats/usage" # Monitor API usage (1000/day limit)
|
|
```
|
|
|
|
#### FRED Response Format
|
|
```json
|
|
{
|
|
"success": true,
|
|
"data": {
|
|
"id": "GDP",
|
|
"title": "Gross Domestic Product",
|
|
"units": "Billions of Dollars",
|
|
"frequency": "Quarterly",
|
|
"last_updated": "2025-07-30T07:56:35",
|
|
"cached": true,
|
|
"cached_at": "2025-01-14T10:30:00"
|
|
},
|
|
"metadata": {
|
|
"source": "fred.stlouisfed.org",
|
|
"cache_duration_hours": 24,
|
|
"daily_api_limit": 1000
|
|
}
|
|
}
|
|
```
|
|
|
|
#### FRED Features
|
|
**🚀 Universal Proxy Access (NEW!)**:
|
|
- **Complete FRED API Coverage**: Access to ALL FRED endpoints via proxy
|
|
- **Pass-through Architecture**: Direct forwarding with rate limiting
|
|
- **Parameter Auto-mapping**: Automatic parameter handling for all endpoints
|
|
- **Enhanced Statistics**: Endpoint-specific usage tracking
|
|
|
|
**🏦 Smart Caching System**:
|
|
- **24-hour cache duration** for series and observations
|
|
- **Database persistence** with SQLite/PostgreSQL
|
|
- **Automatic cache invalidation** after expiry
|
|
- **Cache-first strategy** to minimize API calls
|
|
|
|
**📊 Daily Limit Management**:
|
|
- **1,000 API calls per day** (FRED limitation)
|
|
- **Usage tracking** with detailed statistics
|
|
- **Graceful degradation** when limit reached
|
|
- **Cache fallback** for expired data when limit hit
|
|
|
|
**⚡ Performance Optimization**:
|
|
- **Sub-second response** for cached data
|
|
- **2-5 second response** for fresh API calls
|
|
- **Batch operations** for multiple series
|
|
- **Usage monitoring** and optimization suggestions
|
|
|
|
**🔧 Dual Access Methods**:
|
|
- **Direct Endpoints**: Optimized for series and observations with caching
|
|
- **Proxy Endpoints**: Universal access to all FRED functionality
|
|
- **Automatic Fallback**: Seamless switching between methods
|
|
|
|
### ETF Holdings Data
|
|
|
|
#### Get Current ETF Holdings
|
|
```bash
|
|
# Get latest holdings for QQQ
|
|
curl "http://localhost:18001/api/v1/etf/holdings/QQQ"
|
|
|
|
# Get holdings for specific date (returns closest prior NPORT-P filing)
|
|
curl "http://localhost:18001/api/v1/etf/holdings/SPY?as_of_date=2021-06-30"
|
|
|
|
# Without detailed holdings (metadata only)
|
|
curl "http://localhost:18001/api/v1/etf/holdings/SPY?include_holdings=false"
|
|
|
|
# Bypass Redis cache to force fresh fetch
|
|
curl "http://localhost:18001/api/v1/etf/holdings/SPY?as_of_date=2021-06-30&force_refresh=true"
|
|
```
|
|
|
|
#### Key Features
|
|
- **Automatic CIK Lookup**: No need to know CIK numbers - just use ticker symbols
|
|
- **Historical Data Support**: Access NPORT filings from 2019 onwards
|
|
- **Accurate Date Matching**: Returns the closest NPORT-P filing ≤ `as_of_date` within 365-day window
|
|
- **Stale Cache Prevention**: DB snapshots >120 days from requested date trigger fresh SEC fetch
|
|
- **Date Validation**: Automatically checks if ETF existed on requested date
|
|
- **Availability Info**: Returns available date ranges when data not found
|
|
- **Fast Performance**: <0.1s response time for cached results; Redis-backed 1-hour cache
|
|
|
|
#### Response with Availability Information
|
|
```json
|
|
{
|
|
"ticker": "QQQM",
|
|
"as_of_date": "2020-01-01",
|
|
"success": false,
|
|
"error": "ETF QQQM did not exist on 2020-01-01. Launched on 2020-10-13",
|
|
"availability": {
|
|
"exists_for_date": false,
|
|
"etf_launch_date": "2020-10-13",
|
|
"first_nport_date": "2021-01-31",
|
|
"available_date_range": {
|
|
"start": "2021-01-31",
|
|
"end": "present"
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
#### Supported ETFs
|
|
Major ETFs with pre-configured mappings:
|
|
- **Invesco**: QQQ, QQQM, XLG
|
|
- **SPDR**: SPY, XLF, XLE, XLK, XLV, XLI
|
|
- **iShares**: IWM, EFA, EEM, TLT, AGG, SLV, MTUM
|
|
- **Vanguard**: VTI, VOO, VEA, VWO, BND
|
|
- **ARK**: ARKK, ARKQ, ARKW, ARKG, ARKF
|
|
- And many more...
|
|
|
|
### Get Financial Data
|
|
|
|
#### 🔥 Three Ways to Specify Time Period:
|
|
|
|
1. **Period String** (NEW! Most convenient):
|
|
```bash
|
|
# Last 1 year of data (excludes today for data availability)
|
|
curl "http://localhost:18001/api/v1/financial/data/AAPL?period=1y"
|
|
|
|
# Using POST
|
|
curl -X POST "http://localhost:18001/api/v1/financial/data" \
|
|
-H "Content-Type: application/json" \
|
|
-d '{"ticker": "AAPL", "period": "1y"}'
|
|
```
|
|
|
|
2. **Date Range** (Traditional):
|
|
```bash
|
|
# Specific date range
|
|
curl "http://localhost:18001/api/v1/financial/data/AAPL?start_date=2023-01-01&end_date=2023-12-31"
|
|
|
|
# Using POST
|
|
curl -X POST "http://localhost:18001/api/v1/financial/data" \
|
|
-H "Content-Type: application/json" \
|
|
-d '{
|
|
"ticker": "AAPL",
|
|
"start_date": "2023-01-01",
|
|
"end_date": "2023-12-31",
|
|
"period_type": "quarterly",
|
|
"include_metrics": true,
|
|
"force_refresh": false
|
|
}'
|
|
```
|
|
|
|
3. **Quarters** (Quarter-based):
|
|
```bash
|
|
# Using POST
|
|
curl -X POST "http://localhost:18001/api/v1/financial/data" \
|
|
-H "Content-Type: application/json" \
|
|
-d '{
|
|
"ticker": "AAPL",
|
|
"quarters": ["2024Q1", "2024Q2", "2024Q3"]
|
|
}'
|
|
```
|
|
|
|
⚠️ **IMPORTANT**: Period parameters now use **yesterday** as end date to ensure data availability since today's data might not be available yet.
|
|
|
|
### Get News & Social Media Data (NEW! 🆕)
|
|
|
|
Get comprehensive news and social media sentiment data for any ticker:
|
|
|
|
```bash
|
|
# Get complete news and social media data
|
|
curl "http://localhost:18001/api/v1/news/AAPL?days_back=7&max_articles=20&include_social=true"
|
|
|
|
# Get news only (faster response)
|
|
curl "http://localhost:18001/api/v1/news/TSLA/news-only?days_back=5&max_articles=30"
|
|
|
|
# Get social media only
|
|
curl "http://localhost:18001/api/v1/news/NVDA/social-only?days_back=3&max_social_posts=15"
|
|
```
|
|
|
|
#### Response Format
|
|
```json
|
|
{
|
|
"ticker": "AAPL",
|
|
"retrieved_at": "2025-08-10T17:40:04.781906",
|
|
"news": {
|
|
"total_articles": 12,
|
|
"sources": {"yahoo_finance": 6, "newsapi": 6},
|
|
"articles": [
|
|
{
|
|
"title": "Apple Reports Strong Q3 Results",
|
|
"summary": "Apple exceeded expectations...",
|
|
"url": "https://finance.yahoo.com/...",
|
|
"source": "Yahoo Finance",
|
|
"published_at": "2025-08-10T14:30:00",
|
|
"author": "John Smith"
|
|
}
|
|
]
|
|
},
|
|
"social_media": {
|
|
"total_posts": 8,
|
|
"platforms": {"reddit": 8},
|
|
"posts": [
|
|
{
|
|
"title": "$AAPL breakout incoming?",
|
|
"content": "Technical analysis shows...",
|
|
"url": "https://reddit.com/r/stocks/...",
|
|
"platform": "Reddit",
|
|
"author": "trader123",
|
|
"score": 245,
|
|
"comments_count": 67,
|
|
"subreddit": "stocks"
|
|
}
|
|
]
|
|
},
|
|
"summary": {
|
|
"total_items": 20,
|
|
"time_range_days": 7,
|
|
"newest_item": "2025-08-10T14:30:00",
|
|
"oldest_item": "2025-08-03T09:15:00"
|
|
}
|
|
}
|
|
```
|
|
|
|
#### Query Parameters
|
|
- `days_back`: Number of days to look back (1-30, default: 7)
|
|
- `max_articles`: Maximum news articles to return (1-100, default: 20)
|
|
- `max_social_posts`: Maximum social posts to return (1-100, default: 15)
|
|
- `include_social`: Include social media data (true/false, default: true)
|
|
|
|
### Available Endpoints
|
|
|
|
#### Stock Market Data (NEW! 🔥)
|
|
- `GET /api/v1/stocks/trending` - **Trending stocks combining most active + 52-week gainers** (🚀 Recommended)
|
|
- `GET /api/v1/stocks/most-active` - Most actively traded stocks (optional limit parameter)
|
|
- `GET /api/v1/stocks/52-week-gainers` - 52-week top gaining stocks with intelligent rate limiting
|
|
- `GET /api/v1/stocks/index/{index_name}` - S&P 500 / Nasdaq 100 constituents from Wikipedia (24h cache)
|
|
|
|
#### Alpaca Market Data (NEW! 🆕)
|
|
- `GET /api/v1/alpaca/status` - Alpaca connection status and API key validation
|
|
- `GET /api/v1/alpaca/bars/{ticker}` - Raw bars from Alpaca (no DB storage)
|
|
- `GET /api/v1/alpaca/data/{ticker}` - OHLCV price data via Alpaca (with DB storage)
|
|
- `GET /api/v1/alpaca/intraday/{ticker}` - Intraday candles from Alpaca
|
|
|
|
#### FINRA Short Volume (NEW! 🆕)
|
|
- `GET /api/v1/finra/short-volume/{symbol}` - Short sale volume data (auto-ingests if missing)
|
|
- `GET /api/v1/finra/short-ratio/{symbol}` - Short ratio history (aggregated across markets)
|
|
- `POST /api/v1/finra/admin/ingest` - Manually ingest FINRA data for a date or range
|
|
|
|
#### FRED Economic Data (NEW! 🏦)
|
|
- `GET /api/v1/fred/proxy/{endpoint:path}` - **Universal FRED API proxy with caching** (🚀 Recommended)
|
|
- `GET /api/v1/fred/endpoints` - List all supported FRED API endpoints
|
|
- `GET /api/v1/fred/stats/usage` - API usage statistics and daily limit monitoring
|
|
|
|
#### ETF Holdings
|
|
- `GET /api/v1/etf/holdings/{ticker}` - Get ETF holdings with date support (closest NPORT-P filing)
|
|
- `POST /api/v1/etf/admin/refresh-maps` - Refresh ETF CIK mappings
|
|
|
|
#### Attention (Backtesting Features)
|
|
- `GET /api/v1/attention/event/{ticker}` - Event-date attention features (Wikipedia pageviews + GDELT coverage)
|
|
- `GET /api/v1/attention/entity/{ticker}` - Canonical entity info for ticker
|
|
- `POST /api/v1/attention/admin/resolve/{ticker}` - Resolve ticker → canonical entity
|
|
- `POST /api/v1/attention/admin/collect/wiki/{ticker}` - Collect Wikipedia pageview data
|
|
- `POST /api/v1/attention/admin/collect/gdelt/{ticker}` - Collect GDELT news article data
|
|
|
|
#### Financial Data
|
|
- `GET /api/v1/financial/data/{ticker}` - Simple financial data with query parameters
|
|
- `POST /api/v1/financial/data` - Detailed financial data request
|
|
- `POST /api/v1/financial/data/bulk` - Bulk financial data for multiple tickers
|
|
|
|
#### Price Data (Yahoo Finance)
|
|
- `GET /api/v1/price/data/{ticker}` - Simple price data with query parameters
|
|
- `POST /api/v1/price/data` - Detailed price data request
|
|
- `POST /api/v1/price/data/bulk` - Bulk price data for multiple tickers
|
|
|
|
#### News & Social Media
|
|
- `GET /api/v1/news/{ticker}` - Complete news and social media data
|
|
- `GET /api/v1/news/{ticker}/news-only` - News articles only (faster)
|
|
- `GET /api/v1/news/{ticker}/social-only` - Social media posts only
|
|
|
|
#### System & Admin
|
|
- `GET /api/v1/health` - API health check
|
|
- `GET /api/v1/metadata/catalog` - Data field catalog
|
|
- `POST /api/v1/admin/migrate` - Database migration
|
|
- `GET /api/v1/admin/errors/logs` - Error log management
|
|
- `GET /api/v1/admin/errors/stats` - Error statistics
|
|
|
|
#### Frontend
|
|
- Frontend: http://localhost:18002 (when using Docker)
|
|
- Error Log Viewer: http://localhost:18002/errors
|
|
|
|
## 🐍 Python Client Usage
|
|
|
|
### Installation
|
|
```python
|
|
# Copy the client file to your project
|
|
# stock_oracle_client.py is included in the repository
|
|
```
|
|
|
|
### Basic Usage
|
|
```python
|
|
from stock_oracle_client import StockOracleClient
|
|
|
|
# Initialize client
|
|
client = StockOracleClient("http://localhost:18001")
|
|
|
|
# Check API health
|
|
health = client.get_health()
|
|
print("API Status:", health["status"])
|
|
|
|
# Get financial data using period (recommended)
|
|
data = client.get_financial_data("AAPL", period="1y")
|
|
print(f"Found {len(data['financial_data'])} quarters of data")
|
|
|
|
# Get financial data using date range
|
|
data = client.get_financial_data(
|
|
"MSFT",
|
|
start_date="2023-01-01",
|
|
end_date="2023-12-31",
|
|
period_type="quarterly"
|
|
)
|
|
|
|
# Get financial data using quarters
|
|
data = client.get_financial_data(
|
|
"GOOGL",
|
|
quarters=["2024Q1", "2024Q2", "2024Q3"]
|
|
)
|
|
|
|
# Get price data (period automatically excludes today's data)
|
|
prices = client.get_price_data("AAPL", period="30d", interval="1d")
|
|
|
|
# Get news and social media data (NEW!)
|
|
news_data = client.get_news_social_data(
|
|
ticker="AAPL",
|
|
days_back=7,
|
|
max_articles=20,
|
|
include_social=True
|
|
)
|
|
|
|
# Get news only (faster response)
|
|
news_only = client.get_news_only("TSLA", days_back=5, max_articles=30)
|
|
|
|
# Get social media only
|
|
social_only = client.get_social_only("NVDA", days_back=3, max_social_posts=15)
|
|
|
|
# Bulk operations
|
|
bulk_data = client.get_bulk_financial_data(
|
|
tickers=["AAPL", "MSFT", "GOOGL"],
|
|
period="2y"
|
|
)
|
|
```
|
|
|
|
### Period Options
|
|
- **Days**: "1d", "7d", "30d"
|
|
- **Months**: "1m", "3m", "6m"
|
|
- **Years**: "1y", "2y", "5y", "10y"
|
|
|
|
⚠️ **Note**: Period parameters automatically use **yesterday** as end date for data availability.
|
|
|
|
## 🔧 Configuration
|
|
|
|
### Environment Variables
|
|
```bash
|
|
# Application
|
|
APP_NAME=Stock_Oracle
|
|
API_PREFIX=/api/v1
|
|
|
|
# Database
|
|
DATABASE_URL=sqlite+aiosqlite:///./stock_oracle.db
|
|
|
|
# SEC Data
|
|
SEC_EMAIL=your@email.com # Required for SEC API access
|
|
|
|
# Alpaca Market Data (optional — leave blank to disable)
|
|
ALPACA_API_KEY=your_alpaca_api_key
|
|
ALPACA_SECRET_KEY=your_alpaca_secret_key
|
|
|
|
# Cache
|
|
REDIS_URL=redis://localhost:16379/0 # Use redis://redis:6379/0 in Docker
|
|
CACHE_TTL=3600 # Response cache TTL in seconds (default: 3600)
|
|
|
|
# Server Ports
|
|
API_PORT=18000
|
|
DB_PORT=15432 # PostgreSQL (if used)
|
|
REDIS_PORT=16379 # Redis cache
|
|
```
|
|
|
|
### Docker Ports
|
|
- **API**: 18001 (external) → 18000 (internal)
|
|
- **Frontend**: 18002 (external) → 3000 (internal)
|
|
- **PostgreSQL**: 15433 (external) → 5432 (internal)
|
|
- **Redis**: 16380 (external) → 6379 (internal)
|
|
|
|
## 📈 Supported Metrics
|
|
|
|
### ✅ Implemented (13/16)
|
|
- Market Cap
|
|
- P/E Ratio (Trailing)
|
|
- P/B Ratio
|
|
- Debt-to-Equity
|
|
- Return on Equity (ROE)
|
|
- Return on Assets (ROA)
|
|
- Revenue Growth (YoY)
|
|
- Earnings Growth (YoY)
|
|
- Gross Margins
|
|
- Operating Margins
|
|
- Profit Margins
|
|
- Sector Classification
|
|
- Industry Classification
|
|
|
|
### ❌ Requires External Data (3/16)
|
|
- Forward P/E (analyst estimates needed)
|
|
- PEG Ratio (growth estimates needed)
|
|
- Beta (market correlation data needed)
|
|
|
|
## 🧪 Testing
|
|
|
|
```bash
|
|
# Run all tests
|
|
python -m pytest tests/ -v
|
|
|
|
# Run simple functionality tests
|
|
python tests/test_simple.py
|
|
|
|
# Run specific test categories
|
|
python -m pytest tests/test_financial.py -v
|
|
python -m pytest tests/test_integration.py -v
|
|
```
|
|
|
|
## ⚡ Server-side Response Caching (NEW)
|
|
|
|
Stock Oracle now supports Redis-backed response caching for the most frequently used single-ticker endpoints.
|
|
|
|
### Targets
|
|
- `POST /api/v1/price/data`
|
|
- `GET /api/v1/price/data/{ticker}` (internally uses the same logic)
|
|
- `POST /api/v1/financial/data`
|
|
- `GET /api/v1/financial/data/{ticker}` (internally uses the same logic)
|
|
|
|
Bulk endpoints are not cached.
|
|
|
|
### Behavior
|
|
- Cache store: Redis (`REDIS_URL`)
|
|
- TTL: `CACHE_TTL` seconds
|
|
- Bypass/refresh: set `force_refresh=true` in the request body or query
|
|
- Response headers:
|
|
- `X-Cache`: `HIT` or `MISS`
|
|
- `ETag`: strong hash for the response body
|
|
- `Cache-Control`: `public, max-age={CACHE_TTL}`
|
|
- `X-Data-Source`: `redis-cache` (price endpoint에서 캐시 히트 시)
|
|
|
|
### Quick checks
|
|
```bash
|
|
# 1) MISS (store in cache)
|
|
curl -s -X POST "http://localhost:18001/api/v1/price/data" \
|
|
-H "Content-Type: application/json" \
|
|
-d '{"ticker":"AAPL","period":"3m","interval":"1d"}' -i | grep -Ei 'x-cache|etag|cache-control|x-data-source'
|
|
|
|
# 2) HIT (served from cache)
|
|
curl -s -X POST "http://localhost:18001/api/v1/price/data" \
|
|
-H "Content-Type: application/json" \
|
|
-d '{"ticker":"AAPL","period":"3m","interval":"1d"}' -i | grep -Ei 'x-cache|etag|cache-control|x-data-source'
|
|
|
|
# Force fresh fetch, bypass cache
|
|
curl -s -X POST "http://localhost:18001/api/v1/price/data" \
|
|
-H "Content-Type: application/json" \
|
|
-d '{"ticker":"AAPL","period":"3m","interval":"1d","force_refresh":true}' -i | grep -Ei 'x-cache|etag|cache-control|x-data-source'
|
|
```
|
|
|
|
Notes:
|
|
- If Redis is unreachable, the API gracefully continues without caching.
|
|
- Adjust `REDIS_URL` appropriately (Docker: `redis://redis:6379/0`).
|
|
|
|
## 📁 Project Structure
|
|
|
|
```
|
|
stock-oracle/
|
|
├── app/
|
|
│ ├── api/v1/endpoints/ # API route handlers
|
|
│ ├── core/ # Configuration and database
|
|
│ ├── models/ # SQLAlchemy database models
|
|
│ ├── schemas/ # Pydantic data validation
|
|
│ ├── services/ # Business logic services
|
|
│ └── main.py # FastAPI application entry
|
|
├── tests/ # Comprehensive test suite
|
|
├── scripts/ # Database initialization
|
|
├── data/ # SQLite database storage
|
|
├── docker-compose.yml # Docker orchestration
|
|
├── Dockerfile # Container definition
|
|
├── requirements-*.txt # Python dependencies
|
|
├── stock_oracle_analyzer.py # Core analysis engine
|
|
└── .env # Environment configuration
|
|
```
|
|
|
|
## 🔐 Security & Production
|
|
|
|
### Security Features
|
|
- Input validation and sanitization
|
|
- SQL injection prevention
|
|
- Rate limiting (configurable)
|
|
- Environment-based configuration
|
|
- Secure secret management
|
|
|
|
### Production Deployment
|
|
1. **Database**: Switch to PostgreSQL for production
|
|
2. **Secrets**: Use proper secret management (not .env files)
|
|
3. **Monitoring**: Add application monitoring and logging
|
|
4. **Scaling**: Use container orchestration (Kubernetes, Docker Swarm)
|
|
5. **SSL**: Enable HTTPS with proper certificates
|
|
|
|
## 🤝 Contributing
|
|
|
|
1. Fork the repository
|
|
2. Create a feature branch
|
|
3. Make your changes
|
|
4. Add tests for new functionality
|
|
5. Ensure all tests pass
|
|
6. Submit a pull request
|
|
|
|
## 📄 License
|
|
|
|
[Your License Here]
|
|
|
|
## 🆘 Support
|
|
|
|
- **Documentation**: Check `/docs` endpoint for interactive API docs
|
|
- **Issues**: Report bugs and feature requests in the issue tracker
|
|
- **Email**: [your-support-email]
|
|
|
|
---
|
|
|
|
**Stock Oracle** - Empowering investment decisions with comprehensive SEC data analysis 🔮📈 |