01
Overview

High-Level System Architecture

How the major components of StockVisionz interact. Each color represents a layer.

Frontend External Database Backend ML
FRONTEND EXTERNAL SERVICES DATABASE PYTHON BACKEND 🖥️ UI ⚡ API Routes 📈 Market Data 🔐 Clerk Auth ☁️ S3/R2 📊 stocks 🧪 models Neon PostgreSQL + TimescaleDB 📥 Ingestion 📐 Indicators 🤖 ML Worker 🧠 Model Pipeline 📈 Backtest ⚡ vectorbt queues job fetches writes OHLCV polls jobs writes results saves artifacts
02
Data Layer

Data Ingestion & Indicator Pipeline

Trace the flow when a new ticker is ingested into the system.

👤 User 📥 Ingestion 📈 Market API 💾 Database 📐 Indicators Search "AAPL" Check if ticker exists NEW TICKER Insert instrument metadata Fetch OHLCV 2016-present OHLCV DataFrame Bulk insert stocks.ohlcv_bar Trigger indicator computation Read OHLCV Write indicators POST-PROCESSING Refresh stocks.ml_feature_matrix Ingestion Complete
03
Intelligence

ML Training Pipeline

Walk-forward validation: how models are trained and validated with strict temporal separation.

🤖 Worker 🧠 Pipeline 💾 Database 🔄 Walk-Forward ☁️ Storage run_model() Extract ml_feature_matrix X, y, dates Initialize (252d train, 21d test) Verify leakage checks EACH FOLD Train Data (fold_i) Fit Scaler, Train Model Test Data (fold_i+1) Transform, Predict OOS OOS Predictions Aggregated OOS Results Compute metrics Upload model.pkl / .pt Log experiment + metrics Done
04
Orchestration

ML Job Lifecycle

Frontend POST → Worker picks up → Pipeline trains → Results streamed back via SSE

POST /api/jobs ⏳ QUEUED ML Worker Polls (15s) FOR UPDATE SKIP LOCKED 🔄 PICKED UP Pipeline init 🏋️ TRAINING Success! ✅ COMPLETED Error ❌ FAILED SSE Done SSE Error Stored in: models.ml_job table Multiple workers safe — SKIP LOCKED prevents double-pickup. Frontend polls via SSE at /api/jobs/[jobId]/stream