yirugi
/
stock-oracle
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
main
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from '8c9abe3e31'
${ noResults }
stock-oracle/README.md

1030 lines
34 KiB
Markdown
Raw Blame History

# Stock Oracle 🔮
**Investment Data Analysis API using SEC filings**
Stock Oracle is a comprehensive investment analysis API that leverages SEC EDGAR filing data to provide detailed financial metrics and insights for informed investment decision-making.
## 🎯 Features
### Core Investment Metrics
- **Valuation Ratios**: P/E, P/B, P/S, EV/EBITDA
- **Profitability**: ROE, ROA, Gross/Operating/Net Margins
- **Growth Metrics**: Revenue Growth, Earnings Growth YoY
- **Financial Health**: Debt-to-Equity, Market Cap
- **Sector Analysis**: Industry and sector categorization
### Market Data Intelligence (NEW! 🆕)
- **Most Active Stocks**: Real-time ~170 most actively traded stocks (sub-5s response)
- **52-Week Gainers**: 1,350+ top gaining stocks with intelligent rate limiting (5-90s)
- **Index Constituents**: S&P 500 / Nasdaq 100 구성 종목 조회 — Wikipedia 파싱, 24시간 캐시
- **FRED Economic Data**: Federal Reserve economic indicators with smart caching (1000/day limit)
- **Advanced Web Scraping**: curl_cffi + Chrome impersonation bypasses rate limits
- **Smart Pagination**: Configurable page limits (1-10 pages) for performance tuning
- **Multi-Source News**: Yahoo Finance + NewsAPI integration
- **Social Media Analysis**: Reddit sentiment and discussions
- **Real-Time Updates**: Fresh content aggregation with performance monitoring
- **Sentiment Analysis**: Automated content sentiment scoring
- **Historical Context**: Customizable time periods (1-30 days)
### API Capabilities
- **RESTful API**: FastAPI-based with OpenAPI documentation
- **Database Caching**: Intelligent caching to avoid duplicate SEC parsing
- **Date Range Queries**: Flexible time period analysis
- **Data Validation**: Comprehensive request/response validation
- **Error Handling**: Detailed error categorization and reporting
- **Migration Support**: Database transfer capabilities
### Alpaca Market Data (NEW! 🆕)
- **Official REST API**: Alpaca Market Data v2 with proper authentication
- **OHLCV + VWAP**: Daily/intraday bars with volume-weighted average price
- **Independent Endpoints**: Separate from Yahoo Finance — users choose their source
- **DB Storage**: Alpaca bars stored with `data_source = "ALPACA"` for distinction
- **Rate Limiting**: Built-in token bucket (200 req/min) with retry + backoff
### FINRA Short Volume Data (NEW! 🆕)
- **RegSHO Short Sale Volume**: Daily short volume from FINRA public CDN
- **No API Key Required**: Public data, free access
- **Auto-Ingest**: Automatically downloads missing data on first query
- **Short Ratio History**: Aggregated short_ratio trends over time
- **Bulk Ingest**: Date range ingest for backfilling historical data
### Technical Stack
- **Backend**: FastAPI, SQLAlchemy (async), Pydantic
- **Frontend**: Next.js 15, React 18, TypeScript, TailwindCSS
- **Database**: SQLite (development) / PostgreSQL (production)
- **Caching**: Redis for performance optimization
- **Data Sources**:
- SEC EDGAR via edgartools (ETF holdings, financial data)
- Yahoo Finance via yfinance_plus (news & prices)
- Yahoo Finance via intelligent curl_cffi scraping (market data)
- Most Active Stocks (~170 stocks)
- 52-Week Gainers (~1,350 stocks with rate limiting)
- Alpaca Market Data v2 (OHLCV bars, optional API key)
- FINRA RegSHO (short sale volume, public CDN)
- NewsAPI (news articles)
- Reddit API (social media sentiment)
- **Deployment**: Docker with docker-compose
- **Testing**: Comprehensive test suite with pytest
## 🚀 Quick Start
### Using Docker (Recommended)
```bash
# Clone and navigate
git clone <repo-url>
cd stock-oracle
# Start all services
docker-compose up -d
# Services will be available at:
# - API: http://localhost:18001
# - API Documentation: http://localhost:18001/docs
# - Frontend: http://localhost:18002
# - Error Log Viewer: http://localhost:18002/errors
# - PostgreSQL: localhost:15433
# - Redis: localhost:16380
```
### Local Development
```bash
# Install dependencies
pip install -r requirements-api.txt
# Set up environment
cp .env.example .env
# Edit .env with your settings
# Start all services (first time)
docker-compose up -d
# Restart API after code changes (code is volume-mounted, no rebuild needed)
just run dev
```
> **Note**: Always use `just run dev` to restart the API. The `app/` directory is volume-mounted into the container, so code changes take effect on restart without rebuilding the image.
## ⚡ Quick Examples
### Get Market Overview
```bash
# Trending stocks - best of both worlds (recommended, default: 500 total stocks)
curl "http://localhost:18001/api/v1/stocks/trending"
# Most active stocks (fast)
curl "http://localhost:18001/api/v1/stocks/most-active?limit=10"
# Top 52-week gainers (moderate)
curl "http://localhost:18001/api/v1/stocks/52-week-gainers?max_pages=1&limit=50"
# ETF holdings analysis
curl "http://localhost:18001/api/v1/etf/holdings/QQQ"
# Economic indicators from FRED (cached)
curl "http://localhost:18001/api/v1/fred/proxy/series?series_id=GDP"
curl "http://localhost:18001/api/v1/fred/proxy/series/observations?series_id=UNRATE&limit=12"
```
### Performance Comparison
```bash
# Fast queries (< 15 seconds)
curl "http://localhost:18001/api/v1/stocks/trending?n=200" # ~5-10s, 200 trending stocks (fast mode)
curl "http://localhost:18001/api/v1/stocks/most-active" # ~5s, 170 stocks
curl "http://localhost:18001/api/v1/stocks/52-week-gainers?max_pages=1" # ~8s, 200 stocks
# Moderate queries (15-35 seconds)
curl "http://localhost:18001/api/v1/stocks/trending" # ~15-30s, 500 trending stocks (default, recommended)
curl "http://localhost:18001/api/v1/stocks/52-week-gainers" # ~25s, 600 stocks (default)
# Comprehensive queries (35+ seconds)
curl "http://localhost:18001/api/v1/stocks/trending?n=1000" # ~30-60s, 1000 trending stocks (comprehensive)
curl "http://localhost:18001/api/v1/stocks/52-week-gainers?max_pages=7" # ~75s, 1350 stocks (all)
```
## 📊 API Usage
### Stock Market Data (NEW! 🔥)
#### Get Trending Stocks (🚀 Recommended)
```bash
# Get trending stocks (default: 500 total stocks)
curl "http://localhost:18001/api/v1/stocks/trending"
# Fast mode - 200 total stocks
curl "http://localhost:18001/api/v1/stocks/trending?n=200"
# Comprehensive mode - 1000 total stocks
curl "http://localhost:18001/api/v1/stocks/trending?n=1000"
# Custom mix - 50 most active + remaining gainers to reach 300 total
curl "http://localhost:18001/api/v1/stocks/trending?n=300&most_active_limit=50"
```
#### Trending Stocks Response Format
```json
{
"success": true,
"message": "Retrieved 500 trending stocks (170 most active + 330 gainers) in 18.5s",
"data": {
"trending_stocks": [
{
"symbol": "NVDA",
"company_name": "NVIDIA Corporation",
"current_price": "181.96",
"change_amount": "+0.42",
"change_percent": "+0.23%",
"volume": "45.2M",
"avg_volume": "42.1M",
"category": "most_active",
"rank_in_category": 1,
"scraped_at": "2025-01-14T18:30:15.123456"
},
{
"symbol": "TSLA",
"company_name": "Tesla Inc",
"current_price": "248.50",
"change_amount": "+12.30",
"change_percent": "+125.50%",
"volume": "2.1M",
"high_52w": "250.00",
"category": "52_week_gainer",
"rank_in_category": 1,
"scraped_at": "2025-01-14T18:30:15.123456"
},
{
"symbol": "AAPL",
"company_name": "Apple Inc",
"current_price": "174.50",
"change_amount": "+2.30",
"change_percent": "+1.33%",
"volume": "52.1M",
"avg_volume": "45.2M",
"high_52w": "199.62",
"category": "both",
"rank_in_category": 3,
"gainer_rank": 15,
"scraped_at": "2025-01-14T18:30:15.123456"
}
],
"summary": {
"total_stocks": 500,
"most_active_count": 170,
"gainers_count": 330,
"unique_symbols": 485,
"overlap_count": 15
},
"performance": {
"elapsed_time_seconds": 18.5,
"most_active_time": 3.1,
"gainers_time": 15.4,
"parallel_execution": true
},
"scraped_at": "2025-01-14T18:30:28.987654"
},
"metadata": {
"sources": [
"finance.yahoo.com/markets/stocks/most-active/",
"finance.yahoo.com/markets/stocks/52-week-gainers/"
],
"method": "parallel_scraping_with_intelligent_rate_limiting",
"categories": ["most_active", "52_week_gainer", "both"],
"rate_limit_bypass": "curl_cffi_chrome_impersonation",
"deduplication": "symbol_based_with_category_merge"
}
}
```
#### Key Features
**🔥 Trending Stocks (Recommended)**:
- **Best of Both Worlds**: Combines immediate market activity with long-term performance
- **Smart Deduplication**: Automatically merges overlapping stocks and marks as 'both'
- **Parallel Execution**: Fetches both datasets simultaneously for optimal performance
- **Flexible Configuration**: Customize limits for each category independently
- **Performance Tracking**: Real-time elapsed time and performance metrics
**📊 Categories**:
- **`most_active`**: High trading volume, immediate market attention
- **`52_week_gainer`**: Strong long-term price performance (up to 52 weeks)
- **`both`**: Stocks appearing in both categories (high activity + strong gains)
**⚡ Performance Modes**:
- **Fast Mode** (n=200): ~5-10 seconds, 200 total stocks
- **Default Mode** (n=500): ~15-30 seconds, 500 total stocks (recommended)
- **Comprehensive Mode** (n=1000+): ~30-60 seconds, 1000+ total stocks
#### Get Real-Time Most Active Stocks
```bash
# Get top 10 most active stocks
curl "http://localhost:18001/api/v1/stocks/most-active?limit=10"
# Get all available most active stocks (no limit)
curl "http://localhost:18001/api/v1/stocks/most-active"
```
#### Get 52-Week Top Gainers
```bash
# Get top 100 52-week gainers (fast, 1 page)
curl "http://localhost:18001/api/v1/stocks/52-week-gainers?limit=100&max_pages=1"
# Get default set (~600 gainers, 3 pages, recommended)
curl "http://localhost:18001/api/v1/stocks/52-week-gainers"
# Get first 1000 gainers (5 pages, slower but comprehensive)
curl "http://localhost:18001/api/v1/stocks/52-week-gainers?limit=1000&max_pages=5"
```
#### Most Active Stocks Response Format
```json
{
"success": true,
"message": "Retrieved 3 most active stocks",
"data": {
"stocks": [
{
"symbol": "NVDA",
"company_name": "NVIDIA Corporation",
"current_price": "181.81",
"price_change_raw": "181.81 +0.26 (+0.15%)",
"change_amount": "+0.26",
"change_percent": "+0.15%",
"volume": "93.425M",
"avg_volume": "184.951M",
"scraped_at": "2025-01-14T18:12:28.931780"
}
],
"total_available": 171,
"returned_count": 3,
"scraped_at": "2025-01-14T18:12:28.934007"
},
"metadata": {
"source": "finance.yahoo.com",
"endpoint": "markets/stocks/most-active",
"method": "web_scraping",
"rate_limit_bypass": "curl_cffi_chrome_impersonation"
}
}
```
#### 52-Week Gainers Response Format
```json
{
"success": true,
"message": "Retrieved all 400 52-week gaining stocks in 13.9s",
"data": {
"stocks": [
{
"symbol": "CLGPF",
"company_name": "Clean Seed Capital Group Ltd.",
"current_price": "0.1500",
"price_change_raw": "0.1500 +0.0750 (+100.00%)",
"change_amount": "+0.0750",
"change_percent": "+100.00%",
"volume": "25,000",
"avg_volume": "942",
"high_52w": "0.15",
"scraped_at": "2025-01-14T18:30:15.123456"
}
],
"total_available": 1350,
"returned_count": 400,
"pages_fetched": 2,
"scraped_at": "2025-01-14T18:30:28.987654",
"elapsed_time_seconds": 13.9
},
"metadata": {
"source": "finance.yahoo.com",
"endpoint": "markets/stocks/52-week-gainers",
"method": "intelligent_web_scraping",
"rate_limit_bypass": "curl_cffi_chrome_impersonation_with_smart_delays",
"requests_made": 5
}
}
```
#### Key Features
**🚀 Most Active Stocks**:
- **Real-Time Data**: Scraped directly from Yahoo Finance markets page
- **Complete Dataset**: Access to all ~170 most actively traded stocks
- **Fast Performance**: Sub-5 second response time
- **Rich Information**: Price, change, volume, and company details
**📈 52-Week Gainers**:
- **Comprehensive Data**: Access to 1,350+ top gaining stocks
- **Intelligent Rate Limiting**: Advanced delays to prevent blocking
- **Configurable Scope**: Choose 1-10 pages based on needs
- **Performance Metrics**: Real-time elapsed time tracking
- **Pagination Support**: Automatic multi-page handling
**🛡️ Rate Limiting Technology**:
- **curl_cffi + Chrome Impersonation**: Bypass standard rate limits
- **Smart Delays**: 1-3s base + 5s batch delays every 3 requests
- **Progressive Delays**: Increased delays for later pages
- **Session Management**: 5-minute session rotation
- **Error Recovery**: Automatic retry with exponential backoff
#### Performance Benchmarks
**Most Active Stocks**:
- **Response Time**: 3-5 seconds
- **Data Volume**: ~170 stocks (2 pages)
- **Success Rate**: 99.9%
- **Rate Limits**: Virtually eliminated
**52-Week Gainers**:
| Pages | Stocks | Time | Use Case |
|-------|--------|------|----------|
| 1 page | ~200 | 5-8s | Quick overview |
| 2 pages | ~400 | 12-15s | Moderate analysis |
| 3 pages | ~600 | 20-30s | **Recommended default** |
| 5 pages | ~1000 | 35-50s | Comprehensive analysis |
| 7 pages | ~1350 | 60-90s | Complete dataset |
**Rate Limiting Strategy**:
- **Base Delay**: 1-3 seconds (randomized)
- **Batch Delay**: 5+ seconds every 3 requests
- **Progressive Delay**: +0.5s per page after page 3
- **Session Rotation**: Every 5 minutes
- **Success Rate**: 99.5% even at scale
### Alpaca Market Data (NEW! 🆕)
#### Check Connection
```bash
# Verify Alpaca API key validity
curl "http://localhost:18001/api/v1/alpaca/status"
```
#### Get Price Data (DB Storage)
```bash
# Daily bars with DB persistence (same PriceDataResponse format as /price)
curl "http://localhost:18001/api/v1/alpaca/data/AAPL?interval=1d&start_date=2025-01-01&end_date=2025-01-31"
```
#### Raw Bars (No DB)
```bash
# Fetch bars directly from Alpaca without DB storage
curl "http://localhost:18001/api/v1/alpaca/bars/AAPL?interval=1d&start_date=2025-01-02&end_date=2025-01-10"
```
#### Intraday Candles
```bash
# 1-hour candles from Alpaca
curl "http://localhost:18001/api/v1/alpaca/intraday/AAPL?interval=1h&start_date=2025-03-10&end_date=2025-03-11"
# 5-minute candles
curl "http://localhost:18001/api/v1/alpaca/intraday/TSLA?interval=5m&start_date=2025-03-10&end_date=2025-03-10"
```
#### Key Features
- **Official REST API**: Proper authentication with API key/secret
- **VWAP Included**: Volume-weighted average price in every bar
- **Trade Count**: Number of trades per bar (`trade_count` field)
- **Rate Limited**: Built-in 200 req/min token bucket with auto-wait
- **Retry Logic**: 3x retry with exponential backoff on 429/5xx errors
- **Independent**: Completely separate from Yahoo Finance endpoints
### FINRA Short Volume Data (NEW! 🆕)
#### Get Short Volume
```bash
# Short volume for AAPL (last 30 days, auto-ingests missing data)
curl "http://localhost:18001/api/v1/finra/short-volume/AAPL?days=30"
# With custom limit
curl "http://localhost:18001/api/v1/finra/short-volume/TSLA?days=60&limit=20"
```
#### Short Ratio History
```bash
# Aggregated short ratio over 60 days
curl "http://localhost:18001/api/v1/finra/short-ratio/AAPL?days=60"
```
#### Manual Ingest
```bash
# Ingest a single date
curl -X POST "http://localhost:18001/api/v1/finra/admin/ingest?date=2025-03-10"
# Ingest a date range
curl -X POST "http://localhost:18001/api/v1/finra/admin/ingest?start_date=2025-03-01&end_date=2025-03-10"
# Force re-ingest
curl -X POST "http://localhost:18001/api/v1/finra/admin/ingest?date=2025-03-10&force=true"
```
#### FINRA Response Format
```json
{
"symbol": "AAPL",
"entries": [
{
"date": "2025-03-11",
"symbol": "AAPL",
"short_volume": 3214614.5,
"short_exempt_volume": 29855.0,
"total_volume": 7471743.3,
"market": "B,Q,N",
"short_ratio": 0.430236
}
],
"total_count": 21,
"metadata": {
"days_requested": 30,
"start_date": "2025-02-10",
"end_date": "2025-03-12"
}
}
```
#### Key Features
- **No API Key Required**: Public FINRA CDN data
- **Auto-Ingest**: Missing data automatically downloaded on query
- **Multi-Market**: Data from B (NYSE TRF), Q (NASDAQ TRF), N (NYSE) markets
- **Short Ratio**: Pre-calculated `short_volume / total_volume` per record
- **Aggregated History**: `short-ratio` endpoint aggregates across all markets per day
- **Date Range Ingest**: Bulk backfill for historical data (weekdays only)
### FRED Economic Data (NEW! 🏦)
#### Universal FRED API Proxy (🚀 Recommended)
Access **ALL** FRED API endpoints through our pass-through proxy:
```bash
# Popular economic indicators
curl "http://localhost:18001/api/v1/fred/proxy/series?series_id=GDP" # GDP
curl "http://localhost:18001/api/v1/fred/proxy/series?series_id=UNRATE" # Unemployment Rate
curl "http://localhost:18001/api/v1/fred/proxy/series?series_id=FEDFUNDS" # Fed Funds Rate
curl "http://localhost:18001/api/v1/fred/proxy/series?series_id=CPIAUCSL" # Consumer Price Index
# Historical data with observations
curl "http://localhost:18001/api/v1/fred/proxy/series/observations?series_id=UNRATE&limit=12"
curl "http://localhost:18001/api/v1/fred/proxy/series/observations?series_id=GDP&observation_start=2020-01-01"
# Category data
curl "http://localhost:18001/api/v1/fred/proxy/category?category_id=125"
curl "http://localhost:18001/api/v1/fred/proxy/category/children?category_id=13"
# Release information
curl "http://localhost:18001/api/v1/fred/proxy/release?release_id=53"
curl "http://localhost:18001/api/v1/fred/proxy/releases"
# Search functionality
curl "http://localhost:18001/api/v1/fred/proxy/series/search?search_text=unemployment&limit=25"
# Sources and tags
curl "http://localhost:18001/api/v1/fred/proxy/sources"
curl "http://localhost:18001/api/v1/fred/proxy/tags?limit=100"
# System information
curl "http://localhost:18001/api/v1/fred/endpoints" # Discover all available endpoints
curl "http://localhost:18001/api/v1/fred/stats/usage" # Monitor API usage (1000/day limit)
```
#### FRED Response Format
```json
{
"success": true,
"data": {
"id": "GDP",
"title": "Gross Domestic Product",
"units": "Billions of Dollars",
"frequency": "Quarterly",
"last_updated": "2025-07-30T07:56:35",
"cached": true,
"cached_at": "2025-01-14T10:30:00"
},
"metadata": {
"source": "fred.stlouisfed.org",
"cache_duration_hours": 24,
"daily_api_limit": 1000
}
}
```
#### FRED Features
**🚀 Universal Proxy Access (NEW!)**:
- **Complete FRED API Coverage**: Access to ALL FRED endpoints via proxy
- **Pass-through Architecture**: Direct forwarding with rate limiting
- **Parameter Auto-mapping**: Automatic parameter handling for all endpoints
- **Enhanced Statistics**: Endpoint-specific usage tracking
**🏦 Smart Caching System**:
- **24-hour cache duration** for series and observations
- **Database persistence** with SQLite/PostgreSQL
- **Automatic cache invalidation** after expiry
- **Cache-first strategy** to minimize API calls
**📊 Daily Limit Management**:
- **1,000 API calls per day** (FRED limitation)
- **Usage tracking** with detailed statistics
- **Graceful degradation** when limit reached
- **Cache fallback** for expired data when limit hit
**⚡ Performance Optimization**:
- **Sub-second response** for cached data
- **2-5 second response** for fresh API calls
- **Batch operations** for multiple series
- **Usage monitoring** and optimization suggestions
**🔧 Dual Access Methods**:
- **Direct Endpoints**: Optimized for series and observations with caching
- **Proxy Endpoints**: Universal access to all FRED functionality
- **Automatic Fallback**: Seamless switching between methods
### ETF Holdings Data
#### Get Current ETF Holdings
```bash
# Get latest holdings for QQQ
curl "http://localhost:18001/api/v1/etf/holdings/QQQ"
# Get holdings for specific date
curl "http://localhost:18001/api/v1/etf/holdings/QQQM?as_of_date=2024-01-01"
# Without detailed holdings (metadata only)
curl "http://localhost:18001/api/v1/etf/holdings/SPY?include_holdings=false"
```
#### Key Features
- **Automatic CIK Lookup**: No need to know CIK numbers - just use ticker symbols
- **Historical Data Support**: Access NPORT filings from 2019 onwards
- **Date Validation**: Automatically checks if ETF existed on requested date
- **Availability Info**: Returns available date ranges when data not found
- **Fast Performance**: <0.1s response time with launch date caching
#### Response with Availability Information
```json
{
"ticker": "QQQM",
"as_of_date": "2020-01-01",
"success": false,
"error": "ETF QQQM did not exist on 2020-01-01. Launched on 2020-10-13",
"availability": {
"exists_for_date": false,
"etf_launch_date": "2020-10-13",
"first_nport_date": "2021-01-31",
"available_date_range": {
"start": "2021-01-31",
"end": "present"
}
}
}
```
#### Supported ETFs
Major ETFs with pre-configured mappings:
- **Invesco**: QQQ, QQQM, XLG
- **SPDR**: SPY, XLF, XLE, XLK, XLV, XLI
- **iShares**: IWM, EFA, EEM, TLT, AGG, SLV, MTUM
- **Vanguard**: VTI, VOO, VEA, VWO, BND
- **ARK**: ARKK, ARKQ, ARKW, ARKG, ARKF
- And many more...
### Get Financial Data
#### 🔥 Three Ways to Specify Time Period:
1. **Period String** (NEW! Most convenient):
```bash
# Last 1 year of data (excludes today for data availability)
curl "http://localhost:18001/api/v1/financial/data/AAPL?period=1y"
# Using POST
curl -X POST "http://localhost:18001/api/v1/financial/data" \
-H "Content-Type: application/json" \
-d '{"ticker": "AAPL", "period": "1y"}'
```
2. **Date Range** (Traditional):
```bash
# Specific date range
curl "http://localhost:18001/api/v1/financial/data/AAPL?start_date=2023-01-01&end_date=2023-12-31"
# Using POST
curl -X POST "http://localhost:18001/api/v1/financial/data" \
-H "Content-Type: application/json" \
-d '{
"ticker": "AAPL",
"start_date": "2023-01-01",
"end_date": "2023-12-31",
"period_type": "quarterly",
"include_metrics": true,
"force_refresh": false
}'
```
3. **Quarters** (Quarter-based):
```bash
# Using POST
curl -X POST "http://localhost:18001/api/v1/financial/data" \
-H "Content-Type: application/json" \
-d '{
"ticker": "AAPL",
"quarters": ["2024Q1", "2024Q2", "2024Q3"]
}'
```
⚠️ **IMPORTANT**: Period parameters now use **yesterday** as end date to ensure data availability since today's data might not be available yet.
### Get News & Social Media Data (NEW! 🆕)
Get comprehensive news and social media sentiment data for any ticker:
```bash
# Get complete news and social media data
curl "http://localhost:18001/api/v1/news/AAPL?days_back=7&max_articles=20&include_social=true"
# Get news only (faster response)
curl "http://localhost:18001/api/v1/news/TSLA/news-only?days_back=5&max_articles=30"
# Get social media only
curl "http://localhost:18001/api/v1/news/NVDA/social-only?days_back=3&max_social_posts=15"
```
#### Response Format
```json
{
"ticker": "AAPL",
"retrieved_at": "2025-08-10T17:40:04.781906",
"news": {
"total_articles": 12,
"sources": {"yahoo_finance": 6, "newsapi": 6},
"articles": [
{
"title": "Apple Reports Strong Q3 Results",
"summary": "Apple exceeded expectations...",
"url": "https://finance.yahoo.com/...",
"source": "Yahoo Finance",
"published_at": "2025-08-10T14:30:00",
"author": "John Smith"
}
]
},
"social_media": {
"total_posts": 8,
"platforms": {"reddit": 8},
"posts": [
{
"title": "$AAPL breakout incoming?",
"content": "Technical analysis shows...",
"url": "https://reddit.com/r/stocks/...",
"platform": "Reddit",
"author": "trader123",
"score": 245,
"comments_count": 67,
"subreddit": "stocks"
}
]
},
"summary": {
"total_items": 20,
"time_range_days": 7,
"newest_item": "2025-08-10T14:30:00",
"oldest_item": "2025-08-03T09:15:00"
}
}
```
#### Query Parameters
- `days_back`: Number of days to look back (1-30, default: 7)
- `max_articles`: Maximum news articles to return (1-100, default: 20)
- `max_social_posts`: Maximum social posts to return (1-100, default: 15)
- `include_social`: Include social media data (true/false, default: true)
### Available Endpoints
#### Stock Market Data (NEW! 🔥)
- `GET /api/v1/stocks/trending` - **Trending stocks combining most active + 52-week gainers** (🚀 Recommended)
- `GET /api/v1/stocks/most-active` - Most actively traded stocks (optional limit parameter)
- `GET /api/v1/stocks/52-week-gainers` - 52-week top gaining stocks with intelligent rate limiting
- `GET /api/v1/stocks/index/{index_name}` - S&P 500 / Nasdaq 100 constituents from Wikipedia (24h cache)
#### Alpaca Market Data (NEW! 🆕)
- `GET /api/v1/alpaca/status` - Alpaca connection status and API key validation
- `GET /api/v1/alpaca/bars/{ticker}` - Raw bars from Alpaca (no DB storage)
- `GET /api/v1/alpaca/data/{ticker}` - OHLCV price data via Alpaca (with DB storage)
- `GET /api/v1/alpaca/intraday/{ticker}` - Intraday candles from Alpaca
#### FINRA Short Volume (NEW! 🆕)
- `GET /api/v1/finra/short-volume/{symbol}` - Short sale volume data (auto-ingests if missing)
- `GET /api/v1/finra/short-ratio/{symbol}` - Short ratio history (aggregated across markets)
- `POST /api/v1/finra/admin/ingest` - Manually ingest FINRA data for a date or range
#### FRED Economic Data (NEW! 🏦)
- `GET /api/v1/fred/proxy/{endpoint:path}` - **Universal FRED API proxy with caching** (🚀 Recommended)
- `GET /api/v1/fred/endpoints` - List all supported FRED API endpoints
- `GET /api/v1/fred/stats/usage` - API usage statistics and daily limit monitoring
#### ETF Holdings
- `GET /api/v1/etf/holdings/{ticker}` - Get ETF holdings with date support
- `POST /api/v1/etf/admin/refresh-maps` - Refresh ETF CIK mappings
#### Financial Data
- `GET /api/v1/financial/data/{ticker}` - Simple financial data with query parameters
- `POST /api/v1/financial/data` - Detailed financial data request
- `POST /api/v1/financial/data/bulk` - Bulk financial data for multiple tickers
#### Price Data (Yahoo Finance)
- `GET /api/v1/price/data/{ticker}` - Simple price data with query parameters
- `POST /api/v1/price/data` - Detailed price data request
- `POST /api/v1/price/data/bulk` - Bulk price data for multiple tickers
#### News & Social Media
- `GET /api/v1/news/{ticker}` - Complete news and social media data
- `GET /api/v1/news/{ticker}/news-only` - News articles only (faster)
- `GET /api/v1/news/{ticker}/social-only` - Social media posts only
#### System & Admin
- `GET /api/v1/health` - API health check
- `GET /api/v1/metadata/catalog` - Data field catalog
- `POST /api/v1/admin/migrate` - Database migration
- `GET /api/v1/admin/errors/logs` - Error log management
- `GET /api/v1/admin/errors/stats` - Error statistics
#### Frontend
- Frontend: http://localhost:18002 (when using Docker)
- Error Log Viewer: http://localhost:18002/errors
## 🐍 Python Client Usage
### Installation
```python
# Copy the client file to your project
# stock_oracle_client.py is included in the repository
```
### Basic Usage
```python
from stock_oracle_client import StockOracleClient
# Initialize client
client = StockOracleClient("http://localhost:18001")
# Check API health
health = client.get_health()
print("API Status:", health["status"])
# Get financial data using period (recommended)
data = client.get_financial_data("AAPL", period="1y")
print(f"Found {len(data['financial_data'])} quarters of data")
# Get financial data using date range
data = client.get_financial_data(
"MSFT",
start_date="2023-01-01",
end_date="2023-12-31",
period_type="quarterly"
)
# Get financial data using quarters
data = client.get_financial_data(
"GOOGL",
quarters=["2024Q1", "2024Q2", "2024Q3"]
)
# Get price data (period automatically excludes today's data)
prices = client.get_price_data("AAPL", period="30d", interval="1d")
# Get news and social media data (NEW!)
news_data = client.get_news_social_data(
ticker="AAPL",
days_back=7,
max_articles=20,
include_social=True
)
# Get news only (faster response)
news_only = client.get_news_only("TSLA", days_back=5, max_articles=30)
# Get social media only
social_only = client.get_social_only("NVDA", days_back=3, max_social_posts=15)
# Bulk operations
bulk_data = client.get_bulk_financial_data(
tickers=["AAPL", "MSFT", "GOOGL"],
period="2y"
)
```
### Period Options
- **Days**: "1d", "7d", "30d"
- **Months**: "1m", "3m", "6m"
- **Years**: "1y", "2y", "5y", "10y"
⚠️ **Note**: Period parameters automatically use **yesterday** as end date for data availability.
## 🔧 Configuration
### Environment Variables
```bash
# Application
APP_NAME=Stock_Oracle
API_PREFIX=/api/v1
# Database
DATABASE_URL=sqlite+aiosqlite:///./stock_oracle.db
# SEC Data
SEC_EMAIL=your@email.com # Required for SEC API access
# Alpaca Market Data (optional — leave blank to disable)
ALPACA_API_KEY=your_alpaca_api_key
ALPACA_SECRET_KEY=your_alpaca_secret_key
# Cache
REDIS_URL=redis://localhost:16379/0 # Use redis://redis:6379/0 in Docker
CACHE_TTL=3600 # Response cache TTL in seconds (default: 3600)
# Server Ports
API_PORT=18000
DB_PORT=15432 # PostgreSQL (if used)
REDIS_PORT=16379 # Redis cache
```
### Docker Ports
- **API**: 18001 (external) → 18000 (internal)
- **Frontend**: 18002 (external) → 3000 (internal)
- **PostgreSQL**: 15433 (external) → 5432 (internal)
- **Redis**: 16380 (external) → 6379 (internal)
## 📈 Supported Metrics
### ✅ Implemented (13/16)
- Market Cap
- P/E Ratio (Trailing)
- P/B Ratio
- Debt-to-Equity
- Return on Equity (ROE)
- Return on Assets (ROA)
- Revenue Growth (YoY)
- Earnings Growth (YoY)
- Gross Margins
- Operating Margins
- Profit Margins
- Sector Classification
- Industry Classification
### ❌ Requires External Data (3/16)
- Forward P/E (analyst estimates needed)
- PEG Ratio (growth estimates needed)
- Beta (market correlation data needed)
## 🧪 Testing
```bash
# Run all tests
python -m pytest tests/ -v
# Run simple functionality tests
python tests/test_simple.py
# Run specific test categories
python -m pytest tests/test_financial.py -v
python -m pytest tests/test_integration.py -v
```
## ⚡ Server-side Response Caching (NEW)
Stock Oracle now supports Redis-backed response caching for the most frequently used single-ticker endpoints.
### Targets
- `POST /api/v1/price/data`
- `GET /api/v1/price/data/{ticker}` (internally uses the same logic)
- `POST /api/v1/financial/data`
- `GET /api/v1/financial/data/{ticker}` (internally uses the same logic)
Bulk endpoints are not cached.
### Behavior
- Cache store: Redis (`REDIS_URL`)
- TTL: `CACHE_TTL` seconds
- Bypass/refresh: set `force_refresh=true` in the request body or query
- Response headers:
- `X-Cache`: `HIT` or `MISS`
- `ETag`: strong hash for the response body
- `Cache-Control`: `public, max-age={CACHE_TTL}`
- `X-Data-Source`: `redis-cache` (price endpoint에서 캐시 히트 시)
### Quick checks
```bash
# 1) MISS (store in cache)
curl -s -X POST "http://localhost:18001/api/v1/price/data" \
-H "Content-Type: application/json" \
-d '{"ticker":"AAPL","period":"3m","interval":"1d"}' -i | grep -Ei 'x-cache|etag|cache-control|x-data-source'
# 2) HIT (served from cache)
curl -s -X POST "http://localhost:18001/api/v1/price/data" \
-H "Content-Type: application/json" \
-d '{"ticker":"AAPL","period":"3m","interval":"1d"}' -i | grep -Ei 'x-cache|etag|cache-control|x-data-source'
# Force fresh fetch, bypass cache
curl -s -X POST "http://localhost:18001/api/v1/price/data" \
-H "Content-Type: application/json" \
-d '{"ticker":"AAPL","period":"3m","interval":"1d","force_refresh":true}' -i | grep -Ei 'x-cache|etag|cache-control|x-data-source'
```
Notes:
- If Redis is unreachable, the API gracefully continues without caching.
- Adjust `REDIS_URL` appropriately (Docker: `redis://redis:6379/0`).
## 📁 Project Structure
```
stock-oracle/
├── app/
│ ├── api/v1/endpoints/ # API route handlers
│ ├── core/ # Configuration and database
│ ├── models/ # SQLAlchemy database models
│ ├── schemas/ # Pydantic data validation
│ ├── services/ # Business logic services
│ └── main.py # FastAPI application entry
├── tests/ # Comprehensive test suite
├── scripts/ # Database initialization
├── data/ # SQLite database storage
├── docker-compose.yml # Docker orchestration
├── Dockerfile # Container definition
├── requirements-*.txt # Python dependencies
├── stock_oracle_analyzer.py # Core analysis engine
└── .env # Environment configuration
```
## 🔐 Security & Production
### Security Features
- Input validation and sanitization
- SQL injection prevention
- Rate limiting (configurable)
- Environment-based configuration
- Secure secret management
### Production Deployment
1. **Database**: Switch to PostgreSQL for production
2. **Secrets**: Use proper secret management (not .env files)
3. **Monitoring**: Add application monitoring and logging
4. **Scaling**: Use container orchestration (Kubernetes, Docker Swarm)
5. **SSL**: Enable HTTPS with proper certificates
## 🤝 Contributing
1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests for new functionality
5. Ensure all tests pass
6. Submit a pull request
## 📄 License
[Your License Here]
## 🆘 Support
- **Documentation**: Check `/docs` endpoint for interactive API docs
- **Issues**: Report bugs and feature requests in the issue tracker
- **Email**: [your-support-email]
---
**Stock Oracle** - Empowering investment decisions with comprehensive SEC data analysis 🔮📈
Reference in New Issue View Git Blame Copy Permalink