yirugi
/
stock-oracle
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

fix: 서버 안정성 종합 — BaseHTTPMiddleware 제거 + gc.collect() + Semaphore(5)

Browse Source 
반복적인 서버 응답불능의 근본 원인 2가지를 해결:

1. BaseHTTPMiddleware → Pure ASGI 미들웨어 (app/middleware/error_logger.py)
   - BaseHTTPMiddleware.call_next()가 요청당 asyncio task 2개 생성
   - 20 동시연결 × 2 = 40 tasks → event loop scheduler 포화 → health check 타임아웃
   - __call__(scope, receive, send) + send_wrapper 패턴으로 교체
   - 요청당 단일 task, X-Request-ID 헤더 주입, 에러 응답 body 캡처 유지
   - 불필요한 의존성 제거: BaseHTTPMiddleware, Request, Callable, get_db, AsyncSession

2. gc.collect() 추가 + --limit-max-requests 제거 (alpaca.py, docker-compose.yml)
   - --limit-max-requests 500: 500요청 후 단일 worker 종료 → 서비스 gap 발생
   - 대신 intraday 요청 처리 후 gc.collect()로 Python heap 명시적 회수
   - 장시간 백필 중 메모리 누적 방지, worker 재시작 없이 안정 운영

3. Semaphore(3 → 5): BaseHTTPMiddleware 제거로 task 수 절반 → 처리량 복원

유지: mem_limit 2g, --limit-concurrency 20, Phase 4 경량 쿼리, request_logs 7일 retention

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
main
I Luk Kim 4 months ago
parent 253086eaad
commit 8d1216aa81
3 changed files with 174 additions and 209 deletions
Show Stats Download Patch File Download Diff File

11
app/api/v1/endpoints/alpaca.py
Unescape Escape View File

@ -3,6 +3,7 @@ Alpaca Market Data endpoints — standalone price data via Alpaca API
""" """
import asyncio import asyncio
import gc
from datetime import date, datetime, timezone, timedelta from datetime import date, datetime, timezone, timedelta
from typing import Optional from typing import Optional
from zoneinfo import ZoneInfo from zoneinfo import ZoneInfo
@ -15,7 +16,8 @@ _MARKET_CLOSE_HOUR = 16 # 4:00 PM ET
# Limit concurrent Alpaca intraday processing to prevent event-loop saturation # Limit concurrent Alpaca intraday processing to prevent event-loop saturation
# under bulk backfill workloads. Callers beyond this limit wait on the semaphore # under bulk backfill workloads. Callers beyond this limit wait on the semaphore
# (cheap asyncio wait) rather than flooding httpx connections and DB sessions. # (cheap asyncio wait) rather than flooding httpx connections and DB sessions.
_INTRADAY_SEMAPHORE = asyncio.Semaphore(3) # Semaphore(5): BaseHTTPMiddleware removed → task count halved → safe to allow 5.
_INTRADAY_SEMAPHORE = asyncio.Semaphore(5)
def _market_closed_for(d: date) -> bool: def _market_closed_for(d: date) -> bool:
@ -143,6 +145,11 @@ async def get_alpaca_intraday_multi(
finally: finally:
await svc.client.close() await svc.client.close()
# Reclaim Alpaca HTTP buffers, intermediate bar dicts, and DB row objects
# before building the serialised response. Prevents Python heap from growing
# unboundedly across thousands of backfill requests in a long-running worker.
gc.collect()
bars = { bars = {
ticker: [ ticker: [
{ {
@ -217,6 +224,8 @@ async def get_alpaca_intraday_today(
finally: finally:
await svc.client.close() await svc.client.close()
gc.collect()
bars = { bars = {
ticker: [ ticker: [
{ {

350
app/middleware/error_logger.py
Unescape Escape View File

@ -2,13 +2,19 @@
Error logging middleware for capturing and storing API errors. Error logging middleware for capturing and storing API errors.
Request logs are buffered in an in-memory async queue and flushed in batch Request logs are buffered in an in-memory async queue and flushed in batch
every second (or when the buffer reaches 100 entries) by a background task every second (or when the buffer reaches 500 entries) by a background task
started at app startup via start_request_log_flusher(). This avoids one DB started at app startup via start_request_log_flusher(). This avoids one DB
write per request at high throughput (1 M+ req/day) and eliminates DB lock write per request at high throughput (1 M+ req/day) and eliminates DB lock
contention as a contributor to p95 tail latency. contention as a contributor to p95 tail latency.
Error logs (4xx/5xx) are still written synchronously because they are Error logs (4xx/5xx) are still written synchronously because they are
relatively rare and need accurate timing. relatively rare and need accurate timing.
Pure ASGI implementation — does NOT subclass BaseHTTPMiddleware.
BaseHTTPMiddleware spawns an extra asyncio task per request via call_next(),
doubling event-loop task counts under load and causing health-check timeouts.
A raw ASGI __call__ wraps the downstream send callable instead, staying in
the same task throughout.
""" """
import asyncio import asyncio
@ -17,15 +23,12 @@ import time
import traceback import traceback
import uuid import uuid
from datetime import datetime, timedelta, timezone from datetime import datetime, timedelta, timezone
from typing import Callable, Optional from typing import Optional
from urllib.parse import parse_qs
from fastapi import Request, Response from starlette.responses import JSONResponse
from fastapi.responses import JSONResponse from starlette.types import ASGIApp, Receive, Scope, Send, Message
from sqlalchemy.ext.asyncio import AsyncSession
from starlette.middleware.base import BaseHTTPMiddleware
from starlette.types import ASGIApp
from app.core.database import get_db
from app.models.error_log import ErrorLog from app.models.error_log import ErrorLog
from app.models.request_log import RequestLog from app.models.request_log import RequestLog
import logging import logging
@ -112,174 +115,147 @@ def start_request_log_flusher() -> None:
logger.info("Request log flusher background task started") logger.info("Request log flusher background task started")
class ErrorLoggingMiddleware(BaseHTTPMiddleware): class ErrorLoggingMiddleware:
"""Middleware to log all API requests and errors to database""" """Pure ASGI middleware: logs all requests and 4xx/5xx errors to the database.
def __init__(self, app: ASGIApp): Unlike a BaseHTTPMiddleware subclass, this implementation wraps the ASGI
super().__init__(app) `send` callable directly. Every request is handled in a single asyncio
task — no call_next() extra task, no event-loop task doubling.
"""
async def dispatch(self, request: Request, call_next: Callable) -> Response: def __init__(self, app: ASGIApp) -> None:
"""Process request and log any errors that occur""" self.app = app
# Generate unique request ID async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
request_id = str(uuid.uuid4())[:8] # Pass non-HTTP scopes (lifespan, websocket) straight through.
request.state.request_id = request_id if scope["type"] != "http":
await self.app(scope, receive, send)
return
# Track request start time request_id = str(uuid.uuid4())[:8]
start_time = time.time() start_time = time.time()
request_info = self._extract_request_info(scope)
# Store request details for potential error logging
request_info = self._extract_request_info_minimal(request) # Mutable state shared with the send wrapper (nonlocal in closure).
status_code: list[int] = [200]
logger.debug(f"Processing request {request_id}: {request.method} {request.url.path}") response_headers: list = []
body_chunks: list[bytes] = []
is_error: list[bool] = [False]
async def send_wrapper(message: Message) -> None:
if message["type"] == "http.response.start":
status_code[0] = message["status"]
is_error[0] = status_code[0] >= 400
# Inject X-Request-ID into the response headers.
headers = list(message.get("headers", []))
headers.append((b"x-request-id", request_id.encode()))
response_headers.extend(headers)
await send({**message, "headers": headers})
elif message["type"] == "http.response.body":
if is_error[0]:
body_chunks.append(message.get("body", b""))
await send(message)
else:
await send(message)
try: try:
# Process the request await self.app(scope, receive, send_wrapper)
response = await call_next(request) except Exception as exc:
# Calculate response time
response_time_ms = (time.time() - start_time) * 1000 response_time_ms = (time.time() - start_time) * 1000
# Store response body for error cases
response_body = b""
error_detail = None
# Check if response indicates an error (4xx or 5xx)
if response.status_code >= 400:
logger.debug(f"Error response detected: {response.status_code} for request {request_id}")
# Full extraction for error logging (body + headers)
request_info = await self._extract_request_info(request)
# Try to capture response body for errors
# We need to consume the response body and recreate it
from starlette.responses import Response
# Collect response body chunks
body_chunks = []
async for chunk in response.body_iterator:
body_chunks.append(chunk)
response_body = b''.join(body_chunks)
# Try to parse as JSON
try:
if response_body:
error_detail = json.loads(response_body.decode('utf-8'))
except Exception as e:
logger.warning(f"Could not parse error response body as JSON: {e}")
# Store raw text if not JSON
try:
error_detail = {"raw_response": response_body.decode('utf-8')}
except:
error_detail = {"raw_response": str(response_body)}
# Log the error with response body
logger.debug(f"Logging error for request {request_id}")
await self._log_error(
request_id=request_id,
request_info=request_info,
status_code=response.status_code,
error_detail=error_detail,
response_time_ms=response_time_ms
)
# Recreate response with the same body
response = Response(
content=response_body,
status_code=response.status_code,
headers=dict(response.headers),
media_type=response.media_type
)
# Log all requests (not just errors)
# Add data source info to headers for successful responses
data_source = response.headers.get("X-Data-Source", None)
await self._log_request(
request_id=request_id,
request_info=request_info,
status_code=response.status_code,
response_time_ms=response_time_ms,
response_size=len(response_body) if response_body else None,
data_source=data_source
)
# Add request ID to response headers
response.headers["X-Request-ID"] = request_id
return response
except Exception as e:
# Log unexpected errors
response_time_ms = (time.time() - start_time) * 1000
# Full extraction for error logging
request_info = await self._extract_request_info(request)
await self._log_error( await self._log_error(
request_id=request_id, request_id=request_id,
request_info=request_info, request_info=request_info,
status_code=500, status_code=500,
error_type="INTERNAL_SERVER_ERROR", error_type="INTERNAL_SERVER_ERROR",
error_message=str(e), error_message=str(exc),
stack_trace=traceback.format_exc(), stack_trace=traceback.format_exc(),
response_time_ms=response_time_ms response_time_ms=response_time_ms,
) )
error_response = JSONResponse(
# Return error response
return JSONResponse(
status_code=500, status_code=500,
content={ content={
"error_type": "INTERNAL_SERVER_ERROR", "error_type": "INTERNAL_SERVER_ERROR",
"message": "An unexpected error occurred", "message": "An unexpected error occurred",
"request_id": request_id, "request_id": request_id,
"timestamp": datetime.now(timezone.utc).isoformat() "timestamp": datetime.now(timezone.utc).isoformat(),
}, },
headers={"X-Request-ID": request_id} headers={"X-Request-ID": request_id},
) )
await error_response(scope, receive, send)
return
def _extract_request_info_minimal(self, request: Request) -> dict: # --- Post-response logging (runs after all body chunks are sent) ---
"""Extract minimal request info (sync, no body read) for success-path logging.""" response_time_ms = (time.time() - start_time) * 1000
return {
"endpoint": str(request.url.path),
"method": request.method,
"path": str(request.url),
"query_params": dict(request.query_params) if request.query_params else None,
"request_body": None,
"headers": None,
"user_agent": request.headers.get("user-agent"),
"client_ip": request.client.host if request.client else None,
}
async def _extract_request_info(self, request: Request) -> dict:
"""Extract full request information for error logging"""
# Get request body if present if is_error[0]:
body = None response_body = b"".join(body_chunks)
if request.method in ["POST", "PUT", "PATCH"]: error_detail = None
try: try:
body_bytes = await request.body() if response_body:
if body_bytes: error_detail = json.loads(response_body.decode("utf-8"))
body = json.loads(body_bytes.decode('utf-8')) except Exception:
# Store body for later use in request processing try:
request._body = body_bytes error_detail = {"raw_response": response_body.decode("utf-8")}
except Exception as e: except Exception:
logger.warning(f"Could not parse request body: {e}") error_detail = {"raw_response": str(response_body)}
# Extract headers (remove sensitive ones) await self._log_error(
headers = dict(request.headers) request_id=request_id,
sensitive_headers = ['authorization', 'api-key', 'x-api-key', 'cookie'] request_info=request_info,
for header in sensitive_headers: status_code=status_code[0],
if header in headers: error_detail=error_detail,
headers[header] = '***REDACTED***' response_time_ms=response_time_ms,
)
# Extract X-Data-Source from response headers if present.
data_source = None
for name, value in response_headers:
if name == b"x-data-source":
data_source = value.decode()
break
response_size = sum(len(c) for c in body_chunks) if body_chunks else None
await self._log_request(
request_id=request_id,
request_info=request_info,
status_code=status_code[0],
response_time_ms=response_time_ms,
response_size=response_size,
data_source=data_source,
)
# ------------------------------------------------------------------
# Helpers
# ------------------------------------------------------------------
def _extract_request_info(self, scope: Scope) -> dict:
"""Extract minimal request info from the ASGI scope (sync, no body read)."""
headers_raw = dict(scope.get("headers", []))
user_agent = headers_raw.get(b"user-agent", b"").decode("latin-1") or None
path = scope.get("path", "")
qs_bytes = scope.get("query_string", b"")
qs = qs_bytes.decode("latin-1")
full_url = f"{path}?{qs}" if qs else path
query_params = None
if qs:
parsed = parse_qs(qs)
query_params = {k: v[0] if len(v) == 1 else v for k, v in parsed.items()}
client = scope.get("client")
client_ip = client[0] if client else None
return { return {
"endpoint": str(request.url.path), "endpoint": path,
"method": request.method, "method": scope.get("method", ""),
"path": str(request.url), "path": full_url,
"query_params": dict(request.query_params) if request.query_params else None, "query_params": query_params,
"request_body": body, "request_body": None,
"headers": headers, "headers": None,
"user_agent": headers.get("user-agent"), "user_agent": user_agent,
"client_ip": request.client.host if request.client else None "client_ip": client_ip,
} }
async def _log_error( async def _log_error(
@ -291,48 +267,37 @@ class ErrorLoggingMiddleware(BaseHTTPMiddleware):
error_message: Optional[str] = None, error_message: Optional[str] = None,
error_detail: Optional[dict] = None, error_detail: Optional[dict] = None,
stack_trace: Optional[str] = None, stack_trace: Optional[str] = None,
response_time_ms: Optional[float] = None response_time_ms: Optional[float] = None,
): ) -> None:
"""Log error to database""" """Log error to database."""
# Skip logging errors for log deletion endpoints to avoid logging the deletion of logs
endpoint = request_info["endpoint"] endpoint = request_info["endpoint"]
method = request_info["method"] method = request_info["method"]
# Don't log errors for DELETE requests to log management endpoints # Don't log errors for DELETE requests to log management endpoints.
if (method == "DELETE" and if method == "DELETE" and (
(endpoint.startswith("/api/v1/admin/requests/logs") or endpoint.startswith("/api/v1/admin/requests/logs")
endpoint.startswith("/api/v1/admin/errors/logs"))): or endpoint.startswith("/api/v1/admin/errors/logs")
):
return return
try: try:
# Get database session
from app.core.database import AsyncSessionLocal from app.core.database import AsyncSessionLocal
async with AsyncSessionLocal() as db: async with AsyncSessionLocal() as db:
# Extract error information from detail if available
if error_detail and isinstance(error_detail, dict): if error_detail and isinstance(error_detail, dict):
if not error_type: if not error_type:
error_type = error_detail.get("error_type", "UNKNOWN_ERROR") error_type = error_detail.get("error_type", "UNKNOWN_ERROR")
if not error_message: if not error_message:
error_message = error_detail.get("message", "Unknown error occurred") error_message = error_detail.get("message", "Unknown error occurred")
# Set defaults
if not error_type: if not error_type:
if status_code >= 500: error_type = "INTERNAL_SERVER_ERROR" if status_code >= 500 else "CLIENT_ERROR"
error_type = "INTERNAL_SERVER_ERROR"
elif status_code >= 400:
error_type = "CLIENT_ERROR"
else:
error_type = "UNKNOWN_ERROR"
if not error_message: if not error_message:
error_message = f"HTTP {status_code} error" error_message = f"HTTP {status_code} error"
# Create error log entry db.add(ErrorLog(
error_log = ErrorLog(
request_id=request_id, request_id=request_id,
endpoint=request_info["endpoint"], endpoint=endpoint,
method=request_info["method"], method=method,
path=request_info["path"], path=request_info["path"],
query_params=request_info["query_params"], query_params=request_info["query_params"],
request_body=request_info["request_body"], request_body=request_info["request_body"],
@ -344,26 +309,20 @@ class ErrorLoggingMiddleware(BaseHTTPMiddleware):
stack_trace=stack_trace, stack_trace=stack_trace,
user_agent=request_info["user_agent"], user_agent=request_info["user_agent"],
client_ip=request_info["client_ip"], client_ip=request_info["client_ip"],
response_time_ms=response_time_ms response_time_ms=response_time_ms,
) ))
db.add(error_log)
await db.commit() await db.commit()
logger.error( logger.error(
f"Error logged - Request ID: {request_id}, " f"Error logged - Request ID: {request_id}, "
f"Endpoint: {request_info['endpoint']}, " f"Endpoint: {endpoint}, Status: {status_code}, "
f"Status: {status_code}, " f"Error: {error_type} - {error_message}"
f"Error: {error_type} - {error_message}" )
)
except Exception as e: except Exception as e:
# If we can't log to database, at least log to file
logger.error(f"Failed to log error to database: {e}") logger.error(f"Failed to log error to database: {e}")
logger.error( logger.error(
f"Original error - Request ID: {request_id}, " f"Original error - Request ID: {request_id}, "
f"Status: {status_code}, " f"Status: {status_code}, Error: {error_type} - {error_message}"
f"Error: {error_type} - {error_message}"
) )
async def _log_request( async def _log_request(
@ -373,28 +332,26 @@ class ErrorLoggingMiddleware(BaseHTTPMiddleware):
status_code: int, status_code: int,
response_time_ms: Optional[float] = None, response_time_ms: Optional[float] = None,
response_size: Optional[int] = None, response_size: Optional[int] = None,
data_source: Optional[str] = None data_source: Optional[str] = None,
): ) -> None:
"""Enqueue a request log entry for async batch flushing. """Enqueue a request log entry for async batch flushing.
Non-blocking: the entry is placed in _REQUEST_LOG_QUEUE and written to Non-blocking: the entry is placed in _REQUEST_LOG_QUEUE and written to
the database in bulk by the background flusher, avoiding a synchronous the database in bulk by the background flusher, avoiding a synchronous
DB round-trip on every request. DB round-trip on every request.
""" """
# Skip logging for log deletion endpoints
endpoint = request_info["endpoint"] endpoint = request_info["endpoint"]
method = request_info["method"] method = request_info["method"]
if (method == "DELETE" and if method == "DELETE" and (
(endpoint.startswith("/api/v1/admin/requests/logs") or endpoint.startswith("/api/v1/admin/requests/logs")
endpoint.startswith("/api/v1/admin/errors/logs"))): or endpoint.startswith("/api/v1/admin/errors/logs")
):
return return
# Augment headers with data source if provided headers: dict = {}
headers = request_info.get("headers") or {}
if data_source: if data_source:
headers = {**headers, "X-Data-Source": data_source} headers["X-Data-Source"] = data_source
entry = { entry = {
"request_id": request_id, "request_id": request_id,
@ -403,7 +360,7 @@ class ErrorLoggingMiddleware(BaseHTTPMiddleware):
"path": request_info["path"], "path": request_info["path"],
"query_params": request_info["query_params"], "query_params": request_info["query_params"],
"request_body": request_info["request_body"], "request_body": request_info["request_body"],
"headers": headers, "headers": headers or None,
"status_code": status_code, "status_code": status_code,
"response_size": response_size, "response_size": response_size,
"user_agent": request_info["user_agent"], "user_agent": request_info["user_agent"],
@ -414,7 +371,6 @@ class ErrorLoggingMiddleware(BaseHTTPMiddleware):
try: try:
_REQUEST_LOG_QUEUE.put_nowait(entry) _REQUEST_LOG_QUEUE.put_nowait(entry)
except asyncio.QueueFull: except asyncio.QueueFull:
# Queue is at capacity — fall back to a direct write so we don't drop the log
logger.warning("Request log queue full — writing synchronously") logger.warning("Request log queue full — writing synchronously")
try: try:
from app.core.database import AsyncSessionLocal from app.core.database import AsyncSessionLocal

2
docker-compose.yml
Unescape Escape View File

@ -68,7 +68,7 @@ services:
mem_limit: 2g mem_limit: 2g
memswap_limit: 2g memswap_limit: 2g
restart: unless-stopped restart: unless-stopped
command: ["python", "-m", "uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "18000", "--reload", "--limit-concurrency", "20", "--limit-max-requests", "500"] command: ["python", "-m", "uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "18000", "--reload", "--limit-concurrency", "20"]
# Frontend Application # Frontend Application
frontend: frontend:

Write Preview
Loading…
Cancel
Save