🎬 AI Recommendation & Streaming Lakehouse

title	Movie Recommendation System
emoji	🎬
colorFrom	blue
colorTo	indigo
sdk	docker
pinned	false

🎬 AI Recommendation & Streaming Lakehouse

Distributed Databricks PySpark Delta Lakehouse, 10-Shard Neon Serverless pgvector HNSW & Real-Time Deep Learning Platform (21M+ Records)

A production-grade recommendation intelligence platform and distributed Databricks Lakehouse serving sub-5ms personalized movie recommendations over 21 Million+ records.

Engineered as an enterprise production-grade blueprint for large-scale distributed data engineering, Lakehouse architecture, and sub-5ms neural vector serving over 21 Million+ records. Bridges large-scale distributed batch/stream data engineering (Databricks Serverless, Apache Spark 4.x, Delta Lake 4.x Medallion Architecture, SCD Type 2 dimension versioning, Auto Loader streaming, and data quality quarantine gates) with real-time online AI serving (10-Shard Neon PostgreSQL pgvector HNSW cluster, sub-5ms vector search, Redis caching, and an autonomous Multi-Agent AI reasoning mesh).

💡 Why I Built This & Executive Architecture Summary

"Most recommendation projects stop at small toy datasets inside isolated Jupyter notebooks. I engineered this platform from the ground up as a production-hardened blueprint solving the full end-to-end data engineering lifecycle: ingesting and processing over 21 Million real-world records (1M+ TMDB movies and 20M+ MovieLens user interaction ratings).

The architecture uses a Databricks Serverless Medallion Lakehouse with PySpark and Delta Lake for distributed ETL, SCD Type 2 dimension versioning, and Liquid Clustering. We decouple long-term OLAP Lakehouse storage from low-latency OLTP vector serving by hash-partitioning and exporting Gold records across a 10-shard Neon Serverless PostgreSQL cluster using pgvector HNSW indexes (<5ms query latency).

Heavy deep learning embeddings are generated via SentenceTransformers with zero-copy Apache Arrow serialization and Kaggle GPU clusters. 100% of pipeline orchestration is automated via Databricks Workflows DAGs, 17 GitHub Actions CI/CD workflows, and centralized Doppler secret management."

📸 Production Previews (Captured from Live Hugging Face Space)

Authentic screenshots captured directly from the live deployment on Hugging Face Spaces:

🎬 1. Modern Cinema Workstation (Personalized Discovery Feed)

Real-time personalized movie carousels with dynamic ambient backdrop lighting, AI similarity badges, and instant trailer playback.

🔍 2. Interactive Movie Details & Neural Match Scores

Clicking any title opens the deep inspection modal featuring real-time similarity matching, cast bios, trailer embeds, and similar recommendations.

🌊 3. Lakehouse Data Platform & MLOps Dashboard

Live visibility into the 21M+ record Databricks Medallion pipelines, 10-shard Neon vector clusters, and sub-5ms P99 SLO latency tracking.

🌊 Databricks Medallion & 10-Shard Neon Cluster	📊 Real-Time MLOps Latency & SLO Tracking

🚀 Real Usage Examples (Ready to Run)

No placeholder API keys required. You can query the live public endpoints right now from your terminal, Python script, or frontend application:

1. cURL (Instant CLI Query)

Get real-time personalized recommendations for any movie (e.g., Inception, ID 27205):

# ✨ Pretty-printed JSON in terminal (Windows CMD/PowerShell, macOS, Linux):
curl -s "https://pavanbadempet-movie-rec-api.hf.space/v1/recommendations/id/27205?n=3" | python -m json.tool

# Or using the native ?pretty=true server flag:
curl -s "https://pavanbadempet-movie-rec-api.hf.space/v1/recommendations/id/27205?n=3&pretty=true"

# Or with jq (if installed):
curl -s "https://pavanbadempet-movie-rec-api.hf.space/v1/recommendations/id/27205?n=3" | jq .

Real JSON Response (Live Output):

{
  "status": "success",
  "request_id": "req_27205_1789665706623",
  "query_id": 27205,
  "query_movie": {
    "id": 27205,
    "title": "Inception",
    "genres": ["Action", "Science Fiction", "Adventure"],
    "vote_average": 8.4
  },
  "recommendations": [
    {
      "id": 157336,
      "title": "Interstellar",
      "genres": ["Adventure", "Drama", "Science Fiction"],
      "similarity_score": 0.892,
      "vote_average": 8.4,
      "poster_path": "/gEU2QniE6E77NI6lCU6MxlNBvIx.jpg"
    },
    {
      "id": 348,
      "title": "Alien",
      "genres": ["Horror", "Science Fiction"],
      "similarity_score": 0.814,
      "vote_average": 8.1,
      "poster_path": "/vfrQk5IPloGg1v9Rzbh2Eg3VGyM.jpg"
    }
  ]
}

2. Natural Language Semantic Vector Search

Query the 10-shard Neon pgvector HNSW database using conversational natural language:

curl -s "https://pavanbadempet-movie-rec-api.hf.space/v1/search/ai?q=space+exploration+and+wormholes&limit=2" | python -m json.tool

Real JSON Response:

[
  {
    "id": 157336,
    "title": "Interstellar",
    "overview": "The adventures of a group of explorers who make use of a newly discovered wormhole...",
    "genres": ["Adventure", "Drama", "Science Fiction"],
    "vote_average": 8.4,
    "similarity_score": 0.898,
    "trailer_key": "zSWdZVtXT7E"
  }
]

3. Python Client (Zero Dependencies besides `requests`)

Copy and paste this into quickstart.py to run immediately:

import requests

API_HOST = "https://pavanbadempet-movie-rec-api.hf.space"

# 1. Check system readiness
health = requests.get(f"{API_HOST}/health").json()
print(f"✅ System Status: {health['status']} (v{health['version']})")

# 2. Fetch top-3 recommendations for Inception
response = requests.get(f"{API_HOST}/v1/recommendations/id/27205", params={"n": 3})
data = response.json()

print(f"\n🎬 Recommendations based on '{data['query_movie']['title']}':")
for movie in data["recommendations"]:
    match_pct = movie["similarity_score"] * 100
    print(f"  • {movie['title']} ({movie['vote_average']} ⭐) — {match_pct:.1f}% match")

4. TypeScript / React (Frontend Integration)

Plug real-time recommendations into any web application in just 6 lines:

export async function getMovieRecommendations(movieId: number, count = 5) {
  const url = `https://pavanbadempet-movie-rec-api.hf.space/v1/recommendations/id/${movieId}?n=${count}`;
  const response = await fetch(url, { headers: { Accept: "application/json" } });

  if (!response.ok) throw new Error(`API Error: ${response.status}`);
  const data = await response.json();
  return data.recommendations;
}

// Usage in any React component:
// const recs = await getMovieRecommendations(27205, 5);

⚡ Quickstart & Installation (Zero Guesswork)

Choose the setup that fits your workflow. Zero cloud credentials required for local execution.

Option A: 1-Click Docker Compose (Fastest & Recommended)

Requirements: Docker Desktop or Docker Engine.

# 1. Clone the repository
git clone https://github.com/pavanbadempet/AI-Recommendation-System.git
cd AI-Recommendation-System

# 2. Start the unified container stack (Backend + UI + SQLite pre-warmed cache)
docker compose up -d

# 3. Open your browser:
# ➡️ http://localhost:7860

Option B: Local Step-by-Step (Python & Node / Bun)

Requirements: Python 3.11+ and Node.js 18+ (or Bun).

# 1. Clone the repo
git clone https://github.com/pavanbadempet/AI-Recommendation-System.git
cd AI-Recommendation-System

# 2. Create and activate a virtual environment
python -m venv .venv
# On Windows (PowerShell):
.venv\Scripts\Activate.ps1
# On macOS / Linux:
source .venv/bin/activate

# 3. Install Python dependencies
pip install -r requirements.txt

# 4. Start the FastAPI / Axum backend server
uvicorn backend.main:app --host 127.0.0.1 --port 8000 --reload

# 5. In a separate terminal, launch the React 19 cinema frontend
cd frontend
npm install       # or: bun install
npm run dev       # or: bun run dev

# ➡️ Backend API: http://127.0.0.1:8000/docs
# ➡️ Cinema Portal: http://localhost:5173

Option C: 1-Click Cloud Workspace (GitHub Codespaces)

No installation required on your machine. Click below to launch a fully configured Linux environment in your browser:

🎯 Core Platform Specifications & Tech Stack

⚡ Latest Production Stack & Ecosystem Matrix (2026 Standards)

The platform is standardized on the latest 2026 enterprise open-source and cloud-native versions, ensuring zero legacy debt and adherence to modern security/performance standards:

Infrastructure Layer	Latest Version (2026)	Framework & Protocol	Architectural Role & Modern Capability
Distributed Compute	PySpark 4.3+	Apache Spark 4.3.0 / Databricks DBR 17.x	Native Apache Arrow vectorization, Catalyst SIMD codegen, Adaptive Query Execution (AQE), Spark Declarative Pipelines.
Storage & Serialization	Parquet 2.0	Apache Parquet Format v2 / PyArrow 23.x	Nanosecond timestamp precision (INT64), column-level Bloom filters, dictionary-encoded nested structs.
Lakehouse Storage	Delta Lake 4.0+	Delta 4.x ACID Transaction Log	Dynamic Liquid Clustering (clusterBy), SCD Type 2 temporal versioning, Auto Loader streaming.
Vector Database	PostgreSQL 17	pgvector 0.8+	10-shard hash-partitioned HNSW graph indexing (
ector_cosine_ops), covering indexes, sub-5ms lookups.
Deep Learning	PyTorch 2.6+	TorchDynamo / CUDA 12.6+	6-Model ensemble (SASRec, KAN, LightGCN, Neural ODE, Poincaré, Latent Diffusion) with DR-IPS debiasing.
Serving Gateway	FastAPI 0.136+	Starlette 1.3+ / Uvicorn 0.46+	High-throughput asynchronous ASGI gateway, native Pydantic v2 schemas, zero-copy orjson serialization.
Edge SIMD Core	Rust 2024 Edition	Axum 0.8+ / PyO3 0.23+ / Maturin 1.8+	Sub-0.3ms AVX2/NEON vector math, zero Python GIL bottleneck, memory-safe standalone binary.
Client Workstation	React 19.2+	TypeScript 5.8+ / Vite 8.0+	Glassmorphic cinema SPA, concurrent rendering, dynamic trailer player, WebGL 3D vector galaxy.
Runtime Toolchain	Bun 1.2+	Native JavaScript/TypeScript Engine	30x faster package installation, native test runner (�un test), instant compilation.
Secret Mesh & MLOps	Doppler	OpenTelemetry & Doppler CLI	Zero-sprawl centralized secret injection, automated Kaggle GPU triggers, 17 GitHub Actions CI DAGs.
Data Contracts	Pandera 0.31+	PyArrow 23.0+ & Polars 1.41+	Schema enforcement, dead-letter quarantine validation, type-safe feature pipelines.

Category	Core Specs & Technologies (Search Optimization Keywords)
💡 Core Purpose	Enterprise Distributed Lakehouse & Real-Time Deep Learning Recommendation Engine
🌐 Multi-Cloud Mesh	Databricks Serverless Unity Catalog + Delta Lake 4.x + 10-Shard Neon Serverless + Hugging Face Spaces + GitHub Pages
🤖 Foundation ML	6-Model PyTorch Ensemble (SASRec, KAN B-Splines, LightGCN, Neural ODE, Poincaré Hyperbolic, Latent Diffusion)
📊 Evaluation	HR@10 = 0.785, NDCG@10 = 0.542 (+4.3% lift over best individual model); Doubly Robust IPS offline debiasing
🏛️ Vector Sharding	Hash-partitioned writes across 10 Neon PostgreSQL shards with `pgvector` HNSW cosine similarity graphs (<5ms latency)
📁 Lakehouse ETL	Databricks Medallion (Bronze/Silver/Gold), Delta Lake SCD Type 2 temporal versioning, Liquid Clustering (`clusterBy`), Auto Loader
⚡ Serving Tiers	Adaptive 3-Tier Hardware Detection: Tier 1 (NVIDIA GPU CUDA), Tier 2 (INT8 ONNX CPU), Tier 3 (Rust-SIMD `turbovec` <4ms)
🦀 Edge & Gateway	Pure-Rust Axum Serving Core / PyO3 SIMD bindings (<0.3ms), FastAPI ASGI gateway, gRPC Protobuf streaming (Port 50051)
🤖 Agentic AI	Autonomous multi-agent pipeline (`RetrievalAgent`, `RecommenderAgent`, `RankingAgent`, `ExplanationAgent`) with LLM reasoning
🔐 Privacy & Security	Differential Privacy Laplace mechanism ($\epsilon=5.0$), JWT authentication, Doppler zero-sprawl secrets, PII masking
💻 Client Surface	React 19 SPA, TypeScript 5.8, Tailwind CSS, Lucide icons, Bun 1.2 runtime toolchain, Vite 8 build pipeline
🧪 Quality Gates	100% CI pass rate across 17 GitHub Actions workflows, k6 SLO load testing, property-based tests (Hypothesis), mutation testing

🏛️ Data Engineering & Distributed Systems Deep-Dive

1. System Design & DDIA Principles (Designing Data-Intensive Applications)

A. Reliability (Fault Tolerance & Data Integrity)

ACID Transactions: Delta Lake's _delta_log provides Snapshot Isolation and Serializable Writes, ensuring concurrent batch & streaming writes never corrupt table state.
Idempotent Ingestion & MERGE: The SCD Type 2 MERGE INTO operation is deterministic and idempotent. Re-running the ETL produces identical output without duplicating records.
Fault-Tolerant Quality Gates: Corrupted raw data is handled via expr("try_cast(...)"), preventing pipeline crashes while logging bad records into apex.default.corrupted_data_quarantine.

B. Scalability (Volume & Throughput Growth)

Shared-Nothing Distributed Execution: PySpark partitions computation across independent worker nodes, scaling linearly from 10K to 100M+ records.
Dynamic Liquid Clustering (clusterBy("id")): Replaces static Hive-style partitioning to eliminate data skew and partition explosion without manual partition management.
Decoupled Storage & Compute: Storage resides in Unity Catalog Volumes/S3 while compute scales down to zero when idle, eliminating 24/7 cluster costs.

C. Maintainability (Operability, Simplicity, & Evolvability)

Operability & Auditability: Every record carries full data provenance (_source_file, _ingested_at), and Delta Time Travel enables point-in-time auditing and instant rollback.
Evolvability & Unbundling: Analytical processing (Delta Lake OLAP) is cleanly decoupled from real-time vector serving (Neon PostgreSQL Vector DB), allowing the UI and serving tier to evolve independently of the ETL core.

2. Enterprise Tradeoff & Edge Case Matrix

Architectural Pattern	Chosen Implementation	Why We Chose It (Pros)	Alternative Rejected	Why Rejected (Cons / Tradeoffs)	Edge Cases Handled
Data Quality Gates	`expr("try_cast(...)")`	Converts malformed strings to `NULL`, allowing valid rows to pass while routing bad rows to quarantine.	Strict `.cast("double")`	Throws `CAST_INVALID_INPUT` and crashes the entire pipeline if a single row has shifted text.	Multiline plot overviews containing quotes/newlines that shift columns into ratings.
AI Embeddings	PySpark `@pandas_udf` + Arrow	Distributed Hugging Face inference across worker nodes using zero-copy Apache Arrow memory transfer.	Standard Python UDF / Loops	High pickling serialization overhead, single-node bottleneck, and Out-Of-Memory (`OOM`) crashes.	Worker memory pressure handled via Arrow memory pooling and batched encoding (`batch_size=32`).
State & CDC Tracking	SCD Type 2 MERGE	Preserves full historical auditability (`is_current`, `effective_start_at`, `effective_end_at`).	SCD Type 1 (Overwrite)	Destroys historical data, preventing point-in-time ML model backtesting and audit compliance.	Out-of-order batch dumps, duplicate movie updates, and unchanged record skips via tag hashing.
Data Layout	Liquid Clustering (`clusterBy`)	Dynamic, incremental data clustering as writes occur; 10x faster query pruning.	Legacy Z-Ordering (`ZORDER BY`)	Requires expensive full-table rewrites on every update; does not scale incrementally.	Skewed primary keys and high-concurrency writes without write amplification.
Serving Sync	PySpark Native `to_json`	Pre-formats 768-D dense vectors in parallel across Spark workers before PostgreSQL export.	Python `apply(json.dumps)`	Single-threaded driver bottleneck that slows down large dataset exports.	Postgres string array format incompatibilities during vector bulk export.

3. Core Technical Architecture

flowchart TD
    subgraph Ingestion_Layer["1. RAW DATA INGESTION & BRONZE LAYER"]
        K1["1M+ TMDB Movie Records\n(Kaggle API / Doppler Secrets)"]
        K2["20M+ MovieLens User Ratings\n(Batch CSV / JSON Feeds)"]
        K3["Real-Time Interaction Stream\n(FastAPI /v1/events & WebSockets)"]
    end

    subgraph Lakehouse_Layer["2. DATABRICKS SERVERLESS MEDALLION LAKEHOUSE"]
        Bronze["workspace.movie_recs_bronze.raw_movies\n(Append-Only Raw Parquet / Lineage Stamped)"]
        QualityGate["try_cast() Quality Gates\n& Dead-Letter Quarantine"]
        Silver["workspace.movie_recs_silver.dim_movies\n(SCD Type 2 MERGE INTO & Liquid Clustering)"]
        Gold["workspace.movie_recs_gold.movie_features\n(768-D SBERT Embeddings & Star Schema)"]
    end

    subgraph Vector_Serving["3. DISTRIBUTED VECTOR SERVING TIER"]
        Hasher["Hash Partitioner: pmod(spark_hash(id), 10)"]
        Neon["10-Shard Neon Serverless PostgreSQL Cluster\n(pgvector HNSW Graphs & Covering B-Tree Indexes)"]
        Redis["Redis Feature & Session Sequence Cache\n(TTL 60s / Sub-1ms Lookups)"]
    end

    subgraph ML_Mesh["4. 6-MODEL DEEP LEARNING & AGENTIC AI MESH"]
        Agent["Autonomous Agentic Orchestrator\n(Retrieval -> Recommender -> Ranking -> Explainer)"]
        Ensemble["6-Model PyTorch Ensemble\n(SASRec + KAN + LightGCN + Neural ODE + Poincaré + Diffusion)"]
        Causal["Doubly Robust IPS Weighting\n& Differential Privacy (eps=5.0)"]
    end

    subgraph Client_Surface["5. EDGE GATEWAY & CLIENT PLATFORM"]
        Gateway["FastAPI Gateway & Rust Axum Serving Core\n(Sub-2ms Serving / gRPC Port 50051)"]
        UI["React 19 Cinema Workstation\n(Bun 1.2 / TypeScript / Vite / Tailwind)"]
    end

    Ingestion_Layer --> Bronze
    Bronze --> QualityGate
    QualityGate --> Silver
    Silver --> Gold
    Gold --> Hasher
    Hasher --> Neon
    Neon --> Redis
    Redis --> Gateway
    Gateway --> Agent
    Agent --> Ensemble
    Ensemble --> Causal
    Causal --> Gateway
    Gateway --> UI

4. Databricks PySpark Medallion Lakehouse Notebooks

The distributed data backbone is structured into 5 production-grade Databricks notebooks (databricks_notebooks/):

Notebook	Lakehouse Layer	Operational Responsibilities	Source Reference
`00_kaggle_download.py`	Bronze (Raw)	Authenticates via Doppler Kaggle API tokens; downloads 1M+ TMDB movies and 20M+ MovieLens ratings to Unity Catalog Volumes; performs fast single-pass ingestion (`inferSchema=false`) with `_source_file` and `_ingested_at` audit lineage.	00_kaggle_download.py
`01_pyspark_etl.py`	Silver / Gold	Executes data quality gates (`try_cast`); performs multi-table relational joins & rating aggregations; builds star schema fact/dimension tables (`dim_movies`, `dim_genres`, `fact_genre_top_movies`); enforces SCD Type 2 temporal history with Delta `MERGE INTO`; applies Liquid Clustering (`clusterBy("id")`).	01_pyspark_etl.py
`01b_streaming_events.py`	Streaming Silver	Real-time clickstream ingestion using Databricks Auto Loader (`cloudFiles`) with persistent checkpoint offsets for exactly-once processing guarantees.	01b_streaming_events.py
`01c_gpu_embeddings.py`	Gold AI Layer	Generates 768-D semantic dense vectors using SentenceTransformers (`all-mpnet-base-v2`) via PySpark `@pandas_udf` with Apache Arrow zero-copy memory transfer.	01c_gpu_embeddings.py
`02_export_to_neon.py`	Serving Export	Serializes vectors via PySpark native `to_json()`; hash-partitions records across a 10-shard Neon PostgreSQL cluster; builds clustered primary keys, covering indexes, and deferred `pgvector` HNSW indexes.	02_export_to_neon.py

Real PySpark Transformations & Joins

# 1. Multi-Table Relational Join: Joins 20M MovieLens ratings with MovieLens Links on 'movieId'
ratings_with_tmdb = ratings_raw.join(links_raw, "movieId", "inner")

# 2. Aggregates average rating and review counts per tmdbId
user_ratings_agg = ratings_with_tmdb.groupBy("tmdbId").agg(
    avg(col("rating").cast("double")).alias("movielens_avg_rating"),
    count("userId").alias("movielens_user_review_count")
).withColumnRenamed("tmdbId", "movie_id")

# 3. Enriches TMDB metadata via Left Outer Join
dim_movies_enriched = dim_movies.join(user_ratings_agg, "movie_id", "left")

# 4. Fault-Tolerant Data Quality Gate with Dead-Letter Quarantine
corrupted_df = incoming_df.filter(
    col("id").isNull() |
    (expr("try_cast(id as long)").isNull()) |
    (expr("try_cast(vote_average as double)").isNull()) |
    (expr("try_cast(vote_average as double)") < 0.0) |
    (expr("try_cast(vote_average as double)") > 10.0)
)
corrupted_df.write.format("delta").mode("append").saveAsTable("apex.default.corrupted_data_quarantine")

5. 10-Shard Neon Serverless Vector Serving Tier

To eliminate the cost of keeping large Databricks clusters running 24/7, the Gold layer exports pre-computed vector embeddings to a 10-Shard Neon PostgreSQL cluster:

# Hash-partitioning across N active shards in databricks_notebooks/02_export_to_neon.py
if num_shards > 1:
    df_shard = df_spark.filter(pmod(spark_hash(col("id")), num_shards) == shard_idx)
else:
    df_shard = df_spark

High-Performance Post-Sync Indexing DDL

-- 1. Clustered Primary Key
ALTER TABLE movies ADD PRIMARY KEY (id);

-- 2. Covering Index for Zero-Heap Metadata Scans
CREATE INDEX idx_movies_serving_covering ON movies (id)
INCLUDE (title, genres, vote_average, vote_count, release_date);

-- 3. Deferred HNSW Cosine Similarity Graph Index
CREATE EXTENSION IF NOT EXISTS vector;
ALTER TABLE movies ALTER COLUMN embedding TYPE vector(768) USING embedding::vector(768);
CREATE INDEX IF NOT EXISTS idx_movies_embedding_hnsw ON movies
USING hnsw (embedding vector_cosine_ops)
WITH (m = 16, ef_construction = 64);

6. Autonomous Agentic AI Suite

The system incorporates an Agentic Multi-Agent Orchestration Architecture (backend/agents/multi_agent_orchestrator.py) that coordinates autonomous AI sub-agents to serve recommendations with natural language context and causal reasoning:

sequenceDiagram
    autonumber
    actor User as User / Client App
    participant Orchestrator as Multi-Agent Orchestrator
    participant Retrieval as RetrievalAgent
    participant Ranking as Recommender & RankingAgent
    participant Explainer as ExplanationAgent

    User->>Orchestrator: GET /v1/recommendations/user/{user_id}
    Orchestrator->>Retrieval: Retrieve Candidate Pool (pgvector HNSW / FAISS)
    Retrieval-->>Orchestrator: Top-200 Candidate Vectors
    Orchestrator->>Ranking: Score & Fuse (6-Model PyTorch Ensemble + KAN)
    Ranking-->>Orchestrator: Top-10 Ranked Items (Debiased via DR-IPS)
    Orchestrator->>Explainer: Generate Natural Language Rationale
    Explainer-->>Orchestrator: "Recommended because you recently watched Nolan sci-fi thrillers..."
    Orchestrator-->>User: JSON Response (Top-10 Movies + Agent Reasoning + Confidence)

Agent Name	Module Implementation	Primary Operational Responsibilities
`RetrievalAgent`	`backend/agents/multi_agent_orchestrator.py`	Queries 10-shard `pgvector` HNSW indexes or in-memory FAISS indices to generate top-200 candidate pools.
`RecommenderAgent`	`backend/agents/multi_agent_orchestrator.py`	Executes multi-model neural forward passes across the 6 PyTorch architectures.
`RankingAgent`	`backend/agents/multi_agent_orchestrator.py`	Applies Kolmogorov-Arnold B-Spline calibration and Doubly Robust IPS propensity score debiasing.
`ExplanationAgent`	`backend/agents/multi_agent_orchestrator.py`	Synthesizes personalized natural language explanations justifying recommendations based on user history.

7. 6-Model PyTorch Deep Learning Ensemble & Model Registry

                            ┌────────────────────────────────────────┐
                            │           USER & ITEM INPUTS           │
                            └───────────────────┬────────────────────┘
                                                │
         ┌──────────────────┬───────────────────┼───────────────────┬──────────────────┐
         ▼                  ▼                   ▼                   ▼                  ▼
  ┌──────────────┐   ┌──────────────┐   ┌──────────────┐   ┌──────────────┐   ┌──────────────┐
  │   SASRec     │   │     KAN      │   │   LightGCN   │   │  Neural ODE  │   │  Poincaré    │
  │ Transformer  │   │ B-Spline Net │   │ Graph Conv   │   │ Fluid Dynamics│   │  Hyperbolic  │
  └──────┬───────┘   └──────┬───────┘   └──────┬───────┘   └──────┬───────┘   └──────┬───────┘
         │                  │                   │                   │                  │
         └──────────────────┴─────────┬─────────┴───────────────────┴──────────────────┘
                                      ▼
                        ┌───────────────────────────┐
                        │   LATENT DIFFUSION FUSION │
                        └─────────────┬─────────────┘
                                      ▼
                        ┌───────────────────────────┐
                        │ CAUSAL IPS DEBIASED SCORE │
                        └─────────────┬─────────────┘

Model	Task	Paradigm	HR@10	NDCG@10	DR-Optimized Blend Weight	Parameter Count
APEX Ensemble	Top-$K$ Ranking	Multi-Model Weighted Fusion	0.785	0.542	—	1.85M
SASRec	Sequential Rec	Causal Self-Attention Transformer	0.761	0.520	0.659	524K
KAN Ranker	Non-Linear Ranking	Kolmogorov-Arnold B-Spline Net	0.694	0.439	0.298	128K
LightGCN	Collaborative Filter	3-Layer Graph Convolution	0.672	0.411	0.005	380K
Latent Diffusion	Candidate Refinement	Continuous Score-Based Diffusion	0.521	0.309	0.024	412K
Quantum-Fluid	Temporal Drift	Neural ODE + Complex Embeddings	0.583	0.354	0.010	256K
Poincaré Ball	Taxonomy Embedding	Riemannian Hyperbolic Geometry	0.498	0.287	0.004	148K

Note: Ensemble lift over the best individual model (SASRec) is +4.3% NDCG@10.

8. Advanced Mathematical & Algorithmic Foundations

A. Kolmogorov-Arnold Network (KAN) B-Splines

Instead of standard Multi-Layer Perceptrons (MLPs) with fixed node activation functions $\sigma(w^T x + b)$, KAN replaces fixed weights with learnable univariate splines on network edges based on the Kolmogorov-Arnold representation theorem:

$$ f(x) = \sum_{q=1}^{2n+1} \Phi_q \left( \sum_{p=1}^n \phi_{q,p}(x_p) \right) $$

Where each activation function $\phi(x)$ is parameterized as a linear combination of B-spline basis functions $B_i(x)$ with grid size $G$:

$$ \phi(x) = w_b b(x) + w_s \sum_{i=1}^{G+k} c_i B_i(x), \quad b(x) = \text{silu}(x) $$

This allows the ranker to learn arbitrary non-linear feature interactions with drastically fewer parameters and zero catastrophic forgetting during online updates.

B. Neural ODE Continuous Temporal Drift

User preference trajectories evolve continuously over time. Rather than bucketing interactions into discrete time windows, the temporal state $h(t)$ is modeled as a continuous dynamical system:

$$ \frac{dh(t)}{dt} = f_\theta(h(t), t) \implies h(t_1) = h(t_0) + \int_{t_0}^{t_1} f_\theta(h(t), t) , dt $$

Solved using adaptive Runge-Kutta 4th-order numerical integration (torchdiffeq), allowing the system to extrapolate preference drift across arbitrary time gaps $\Delta t = t_1 - t_0$.

C. Poincaré Ball Hyperbolic Metric

Hierarchical relationships (e.g., Cinema $\rightarrow$ Sci-Fi $\rightarrow$ Cyberpunk) cannot be embedded in Euclidean space without exponential dimensional distortion. In the Poincaré ball model $\mathbb{D}^n = { x \in \mathbb{R}^n : |x| < 1 }$, the Riemannian distance between user $u$ and item $v$ is computed as:

$$ d_{\mathbb{D}}(u, v) = \text{arcosh}\left(1 + 2 \frac{|u - v|^2}{(1 - |u|^2)(1 - |v|^2)}\right) $$

The metric tree capacity grows exponentially with radius, matching the natural exponential branching of genre taxonomy trees.

D. Doubly Robust Inverse Propensity Scoring (DR-IPS)

To eliminate presentation and popularity bias from logged clickstreams, the offline evaluation metric estimates the counterfactual policy value $\hat{V}_{\text{DR}}$:

$$ \hat{V}_{\text{DR}}(\pi) = \frac{1}{|U|} \sum_{u \in U} \left[ \hat{r}(u, \pi(u)) + \frac{Y_{u, i} - \hat{r}(u, i)}{P(O_{u, i} = 1)} \cdot \mathbb{I}(i = \pi(u)) \right] $$

Where $P(O_{u, i} = 1)$ is the observation propensity score estimated via Laplace-smoothed logistic regression, and $\hat{r}(u, i)$ is the baseline imputation model.

E. Differential Privacy Laplace Mechanism

To guarantee privacy for user interaction logs during online federated learning, noise is injected into aggregated gradient updates $g$:

$$ \tilde{g} = g + \text{Lap}\left(0, \frac{\Delta S}{\epsilon}\right), \quad \Delta S = \max |g_1 - g_2|_1 $$

With privacy budget $\epsilon = 5.0$ and clipping threshold $C = 1.0$, satisfying strict privacy audits.

9. Adaptive 3-Tier Hardware Serving Architecture

The startup tier detector (backend/serving/serving_tier.py) profiles available hardware at boot and binds the highest-performance serving tier available:

Operational Tier	Hardware Trigger	Runtime Execution Engine	Cold-Start	Typical Request Latency
Tier 1 (Enterprise GPU)	NVIDIA GPU (CUDA $\ge$ 8GB VRAM) + $\ge$16GB RAM	Full PyTorch 6-Model Ensemble	`~3.5s`	`12.5 ms`
Tier 2 (Recommended CPU)	Standard CPU-only + $\ge$8GB RAM	INT8 Quantized ONNX Runtime Blend	`~1.8s`	`24.8 ms`
Tier 3 (Ultra-Light SIMD)	Memory-constrained ($<$8GB RAM) / Free Tier	SIMD Vector Indexing (`turbovec`)	`<50ms`	`< 4.2 ms`

10. Architecture Decision Records (ADR) Summary

Key engineering decisions are documented in detail in docs/ARCHITECTURE_DECISIONS.md:

Record	Architectural Decision	Context & Technical Rationale	Engineering Impact
ADR-001	LightGCN Component	Collaborative filtering requires multi-hop graph message passing without heavy MLP non-linearities.	Highest collaborative NDCG on sparse users; initialized via PySpark ALS.
ADR-002	Neural ODE Dynamics	User preferences drift continuously over irregular calendar intervals rather than discrete buckets.	Models temporal decay and genre fatigue with zero time-bin discretization.
ADR-003	SASRec Sequential	Multi-item viewing sessions exhibit short-term momentum not captured by static matrix factorization.	Dominant model in DR-IPS optimization (weight = 0.659) for active sessions.
ADR-004	Conditional Retainment	KAN, Hyperbolic, and Diffusion models have low offline weights but provide critical niche lift.	Models are conditionally activated during cold-start or heavy taxonomy queries.
ADR-005	3-Tier Hardware Serving	Must run on free cloud instances (512MB RAM) up to multi-GPU enterprise Kubernetes clusters.	Eliminates out-of-memory crashes; auto-detects hardware with zero configuration.
ADR-006	Pipeline Decomposition	Decoupled 3-phase flow: Candidate Retrieval (10K $\rightarrow$ 200) $\rightarrow$ Ranking $\rightarrow$ Reranking.	Bounds latency to $<25\text{ms}$ even under high concurrent traffic spikes.
ADR-007	Doubly Robust IPS	Historical training logs suffer from severe presentation and popularity bias.	Unbiased offline evaluation; +4.3% true counterfactual NDCG lift.
ADR-008	Online Learning Loop	User interactions must update model parameters without requiring full multi-hour batch retraining.	Dedicated daemon thread processes clicks and ratings with bounded queues.
ADR-009	Differential Privacy	User viewing histories contain sensitive behavioral information requiring privacy controls.	Bounded gradient clipping with Laplace noise satisfies privacy regulations.
ADR-010	Uncertainty Gating	Recommender must detect out-of-distribution users and fall back to safe popularity priors.	Prevents wild hallucinated recommendations on cold-start accounts.

🗄 Database Layer Schema & Star Schema

The Gold Layer and PostgreSQL serving tier are organized into an optimized Star Schema Dimensional Model:

erDiagram
    fact_ratings {
        bigint rating_id PK
        bigint user_id FK
        bigint movie_id FK
        float rating
        timestamp interaction_time
    }
    dim_movies {
        bigint movie_id PK
        string title
        string overview
        float vote_average
        int vote_count
        date release_date
        vector embedding_768d
        boolean is_current
        timestamp valid_from
        timestamp valid_to
    }
    dim_genres {
        int genre_id PK
        string genre_name
    }
    fact_movie_genres {
        bigint movie_id FK
        int genre_id FK
    }
    fact_events {
        bigint event_id PK
        bigint user_id
        bigint movie_id
        string event_type
        float weight
        timestamp created_at
    }

    dim_movies ||--o{ fact_ratings : "receives"
    dim_movies ||--o{ fact_movie_genres : "categorized_by"
    dim_genres ||--o{ fact_movie_genres : "contains"
    dim_movies ||--o{ fact_events : "logged_in"

🛡️ High-Concurrency Backend System Design & Distributed Serving

Serving low-latency neural vector recommendations and real-time behavioral streams under enterprise traffic requires solving core distributed systems challenges: noisy neighbor isolation, sudden traffic spikes, cascading microservice failures, cache invalidation stampedes, and zero-downtime rollouts. The platform implements production backend patterns across the serving lifecycle.

┌──────────────────────────────────────────────────────────────────────────────────────────────────┐
│                                    TRAFFIC INGRESS & EDGE                                        │
│   Cloudflare CDN / Anycast DNS  ──►  Layer 4 eBPF NLB (TCP/TLS)  ──►  Layer 7 Envoy / NGINX      │
│   • TLS 1.3 Termination         • Syn Flood & DDoS Filtering     • Path Routing & Header Rules   │
└────────────────────────────────────────────────┬─────────────────────────────────────────────────┘
                                                 │
                                                 ▼
┌──────────────────────────────────────────────────────────────────────────────────────────────────┐
│                             API GATEWAY & TRAFFIC SHAPING LAYER                                  │
│   Distributed Redis Token Bucket  ──►  Consistent Hashing Ring  ──►  Circuit Breaker & Fallback  │
│   • Multi-Tenant Quotas (TPS)         • Virtual Nodes (Ketama)      • 50% Failure Window Trip     │
│   • RFC 6585 (HTTP 429 Retry-After)   • L1/L2 Cache Affinity (>94%) • Jittered Exponential Backoff│
└───────────────────────┬──────────────────────────────────────────────────┬───────────────────────┘
                        │                                                  │
         [Read Path: Vector Serving]                        [Write Path: Telemetry & Ingestion]
                        ▼                                                  ▼
┌──────────────────────────────────────────────┐   ┌──────────────────────────────────────────────┐
│       ISOLATED BULKHEAD SERVING PODS         │   │       STREAMING TELEMETRY PIPELINE           │
│  • Tokio Async Rayon SIMD Worker Pool        │   │  • Apache Kafka Partitioned Topic (64 parts) │
│  • Multi-Tier Cache: L1 Moka -> L2 Redis     │   │  • Redis Stream Sliding Buffer               │
│  • XFetch Probabilistic Cache Stampede Guard │   │  • Asynchronous Background Event Worker      │
│  • Sub-5ms Vector Similarity Retrieval       │   │  • Delta Lake Bronze Medallion Appends       │
└───────────────────────┬──────────────────────┘   └──────────────────────┬───────────────────────┘
                        │                                                 │
                        ▼                                                 ▼
┌──────────────────────────────────────────────┐   ┌──────────────────────────────────────────────┐
│          STORAGE & VECTOR PERSISTENCE        │   │           KUBERNETES LIFECYCLE               │
│  • 10-Shard Neon Serverless pgvector HNSW    │   │  • Liveness Probe: /health (Process Alive)   │
│  • Delta Lake 4.x Medallion Lakehouse        │   │  • Readiness Probe: /v1/platform/readiness   │
│  • Read-Replicas with Connection Pooling     │   │  • SIGTERM Connection Draining (15s grace)   │
└──────────────────────────────────────────────┘   └──────────────────────────────────────────────┘

1. Distributed Rate Limiting & Multi-Tenant Quota Management

To prevent denial-of-service degradation and protect downstream vector databases, incoming requests pass through a Distributed Token Bucket & Sliding Window Rate Limiter implemented in backend/middleware/rate_limiter.py.

Incoming Request
       │
       ▼
Resolve Tenant Context ──► [dim_api_key Hash Check]
       │
       ├─► Anonymous User ──► IP Keyed (`rate_limit:ip:{client_ip}:{window}`) ──► Limit: 10 RPS
       ├─► Pro Tenant     ──► Tenant Keyed (`rate_limit:{tenant_id}:{window}`) ──► Limit: 100 RPS
       └─► Enterprise     ──► Tenant Keyed (`rate_limit:{tenant_id}:{window}`) ──► Limit: 5,000 RPS
       │
       ▼
Atomic Redis Execution:
  INCR rate_limit:{id}:{epoch_second}
  EXPIRE rate_limit:{id}:{epoch_second} 5
       │
       ├─► count <= limit ──► Forward to Rust SIMD / ONNX Inference Engine (HTTP 200)
       └─► count >  limit ──► Reject Early with HTTP 429 Too Many Requests

Algorithm Comparison & Selection:
- Fixed Window Counter: Suffers from boundary burst vulnerabilities (allowing $2\times$ traffic at the boundary).
- Sliding Window Log: High memory footprint ($O(N)$ stored timestamps per user).
- Redis Sliding Window Counter: Our chosen approach. Buckets timestamps to 1-second epochs with atomic Redis INCR and an aggressive 5-second TTL. Ensures $O(1)$ memory consumption and sub-millisecond evaluation latency.

RFC 6585 Standardized HTTP 429 Response Headers:

HTTP/1.1 429 Too Many Requests
Retry-After: 1
X-RateLimit-Limit: 100
X-RateLimit-Remaining: 0
X-RateLimit-Reset: 1789667240
Content-Type: application/json

{"error": "SLA Quota Exceeded. Please upgrade to a higher Enterprise Tier for more TPS."}

High-Availability Fail-Open Architecture: Rate limiters must never become a single point of failure. If the Redis cluster experiences network partitions or response times exceed 15ms, the middleware logs an operational alert and fails open (logger.error("Redis Rate Limiter Error. Failing open.")), ensuring 99.99% recommendation uptime.

2. Layer 4 vs. Layer 7 Load Balancing & Consistent Hashing

High-throughput recommendation platforms cannot rely on basic round-robin routing due to cache-affinity requirements.

Layer 4 vs. Layer 7 Responsibilities:
- Layer 4 (eBPF / AWS NLB): Operates at the transport layer (TCP/UDP), terminating raw connections and balancing raw packets across ingress gateway nodes without inspecting HTTP headers. Achieves millions of packets per second with minimal CPU overhead.
- Layer 7 (Envoy / NGINX / Cloudflare Edge): Inspects HTTP path prefixes and headers. Routes /v1/recommendations/* directly to high-performance Rust SIMD worker pods while dispatching /v1/events to streaming Kafka buffers and /v1/billing to PostgreSQL services.
Consistent Hashing with Virtual Nodes (Ketama Algorithm):
- In distributed vector recommendation, worker nodes maintain warm in-memory L1 vector caches (catalog embeddings and candidate pools).
- The Modulo Hashing Flaw: Standard modulo routing ($\text{hash}(\text{user_id}) \pmod K$) causes 100% cache invalidation whenever a worker node crashes or scales out ($K \rightarrow K+1$), triggering disastrous database cold-starts.
- Our Solution: We map worker replicas onto a Consistent Hash Ring using 256 virtual nodes per server: $$\theta_{\text{node}} = \text{Murmur3}(\text{node_ip} + \text{replica_id}) \pmod{2^{32}}$$ Incoming user sessions and movie IDs map to the nearest clockwise node. Adding or removing a pod redistributes strictly $\frac{1}{K}$ of the keyspace, preserving a 94%+ vector cache hit rate and eliminating cold-cache thrashing.
Client-Side Hedged Requests (Tail-at-Scale Mitigation):
- The frontend Cinema SPA incorporates Google’s Tail-at-Scale pattern via Promise.any([primary_hf_space, backup_render_gateway]). Requests are raced concurrently, and the first valid response completes the UI render, driving P99 client latency down to parity with P50.

3. Resilience Engineering: Circuit Breakers, Bulkheads & Jittered Backoff

Cascading failures occur when an upstream service slows down, causing downstream thread pools to exhaust their connection queues.

                  ┌──────────────────────────────┐
                  │    CLOSED (Normal State)     │
                  │   All requests executed      │
                  └──────────────┬───────────────┘
                                 │ Failure Rate > 50%
                                 │ over 10s Window
                                 ▼
                  ┌──────────────────────────────┐
       ┌─────────►│     OPEN (Tripped State)     │
       │          │   Fast-Fail to Fallback      │
       │          │   (Popularity Cache / 0ms)   │
       │          └──────────────┬───────────────┘
       │                         │ Cooldown Period Expired
       │ Probe Fails             │ (30 Seconds)
       │                         ▼
       │          ┌──────────────────────────────┐
       └──────────┤   HALF-OPEN (Testing State)  │
                  │   Limited Probe Requests Sent│
                  └──────────────┬───────────────┘
                                 │ Probe Succeeds (100% OK)
                                 ▼
                         Return to CLOSED

Circuit Breaker Finite State Machine:
- Integrated across external dependencies (Neon Database, Redis Feature Store, TMDB metadata).
- Trip Condition: If error rates exceed 50% or P99 response times exceed 2,000ms over a rolling 10-second window, the circuit trips to OPEN.
- Fast Fallback: While OPEN, requests bypass the failing upstream service entirely, immediately returning pre-computed popularity rankings in < 0.2ms.
- Half-Open Recovery: After a 30-second cooldown, a canary probe (5% of requests) tests upstream health before fully closing the circuit.
Thundering Herd Elimination with Full Jitter:
- When upstream nodes recover, synchronized linear retries cause secondary traffic spikes (Retry Storms).
- We enforce exponential backoff with Full Jitter: $$T_{\text{sleep}} = \min(T_{\max}, T_0 \cdot 2^{\text{attempt}}) \times \text{Uniform}(0, 1)$$ Spreading retry intervals evenly across the time domain flattens the retry distribution into uniform, manageable traffic.
Bulkhead Isolation:
- Fast vector retrieval operations (Tokio Rayon SIMD kernel, < 1ms) are strictly segregated from expensive generative explanation tasks (LLM attributions, > 250ms) using dedicated worker pools and independent Semaphore permits, preventing thread starvation.

4. Distributed Multi-Tier Caching & Cache Stampede Defense

Recommendations exhibit high power-law popularity distributions (Pareto 80/20 rule). A single trending blockbuster movie can receive 50,000 requests/sec.

3-Tier Cache Topology:
1. L1 (In-Process LRU): Moka concurrent cache in Rust / functools.lru_cache in Python. Sub-microsecond (< 0.05ms) access for the top 5,000 hottest query IDs.
2. L2 (Distributed Redis 7 Cluster): Shared multi-node Redis instances storing vectorized recommendation responses with a 60-second TTL. Sub-millisecond (0.8ms) latency across worker pods.
3. L3 (Neon Database NVMe): PostgreSQL covering B-Tree indexes and 10-shard pgvector HNSW indexes (< 5ms).
Probabilistic Early Expiration (XFetch Algorithm):
- The Cache Stampede Threat: When a hot movie's cache entry expires at $t = \text{TTL}$, hundreds of concurrent requests simultaneously observe a cache miss and blast the database and neural models at the exact same millisecond.
- The Solution: We implement the XFetch Algorithm (Vattani et al.): $$t - \beta \cdot \delta \cdot \ln(U) > \text{expiry}$$
  - $t$: Current timestamp
  - $\delta$: Computation delta (time taken to compute recommendations, e.g. 15ms)
  - $\beta > 0$: Aggressiveness constant (default $\beta = 1.0$)
  - $U \sim \text{Uniform}(0, 1)$: Random variable
- As the key approaches expiration, read requests probabilistically trigger an asynchronous background recomputation before the key expires. The hot cache is refreshed transparently without a single user thread blocking.
Singleflight Mutex (Request Coalescing):
- For completely cold keys, a distributed mutex ensures only one worker thread computes the vector embedding while all other concurrent requests for the same key await the first worker's result.

5. Zero-Downtime Deployment, Kubernetes Probes & Graceful Shutdown

Production serving clusters must support continuous rollouts without dropping in-flight HTTP connections.

Dual Kubernetes Health Probes:
- Liveness Probe (GET /health): Verifies that the operating system process and async runtime event loop are unblocked. Failure triggers a container restart.
- Readiness Probe (GET /v1/platform/readiness): Crucial for ML workloads. Checks that dense vector embeddings, TF-IDF feature vocabulary, and SQLite/Postgres connection pools have finished warming up in RAM. Ingress load balancers refuse to route user traffic to the pod until this endpoint returns HTTP 200.
Graceful Termination & Connection Draining:
- Upon receiving a SIGTERM signal from the Kubernetes kubelet:
  1. The pod immediately transitions readiness probe to HTTP 503, prompting the Ingress controller to remove the pod from the active endpoint pool.
  2. The HTTP server enters connection-draining mode with a 15-second grace period, allowing all active recommendation requests to finish streaming their responses.
  3. In-memory clickstream telemetry buffers flush their remaining micro-batches to PostgreSQL and Kafka.
  4. Database connection pools close cleanly before process exit.
Horizontal Pod Autoscaling (HPA):
- Automated horizontal pod autoscaling scales the deployment from 2 to 20 replicas based on dual metrics: $$\text{Target CPU Utilization} \ge 70% \quad \text{OR} \quad \text{P95 Request Duration} \ge 25\text{ms}$$

🌊 Architecture Deep-Dive: Scaling & Technical Decisions

🚀 1. Scaling Architecture: Scaling from 21 Million to 21 Billion Records

Architectural Evolution Strategy (Compute $\rightarrow$ Storage $\rightarrow$ Ingestion $\rightarrow$ Serving):

Compute & Partitioning:

At 21B rows, file ingestion cannot rely on directory globbing. Migrate raw ingestion from Volume files to a distributed event streaming backbone like Apache Kafka or AWS Kinesis with 64+ partitions.

Tune PySpark shuffle partitions: increase spark.sql.shuffle.partitions from 200 to 2,000–5,000 based on the 128MB partition rule of thumb.

Delta Lake Optimization:

Switch from micro-batch availableNow=True to continuous Spark Structured Streaming with watermarking (e.g. withWatermark("timestamp", "2 hours")) to drop duplicate late-arriving events from state stores.

Implement Delta Lake Partition Pruning by date(ingested_at) combined with Liquid Clustering on high-cardinality join keys (user_id, movie_id).

Serving Tier:

10 PostgreSQL shards would become a bottleneck at 21B vectors. Migrate online vector serving to a distributed vector search cluster like Milvus, Qdrant, or Pinecone with Product Quantization (PQ) to compress vectors by 90% and store index graphs across Kubernetes worker nodes.

⏱️ 2. Temporal State Fidelity: Why Delta Lake SCD Type 2 for Feature Stores

Engineering Decision: "In production Machine Learning and recommendation systems, feature store integrity is critical. If a movie's genres, description, or rating stats change over time and we overwrite the record (SCD Type 1), we create data leakage and prevent historical backtesting. When evaluating how a recommendation model performed on historical user interactions from 6 months ago, the model must see the exact movie metadata that existed at that point in time. SCD Type 2 provides temporal auditability through is_current, effective_start_at, and effective_end_at, allowing point-in-time ML training and compliance auditing."

⚡ 3. Storage Optimization: Liquid Clustering vs. Legacy Z-Ordering

Engineering Decision: "Z-Ordering maps multi-dimensional data into a one-dimensional space along a Space-Filling Z-Curve to achieve data skipping. However, Z-Ordering is a static clustering technique that requires a full table rewrite every time OPTIMIZE ... ZORDER BY is called. For tables receiving frequent streaming appends or CDC updates, Z-Ordering causes severe write-amplification.

Liquid Clustering (clusterBy) replaces fixed partition hierarchies and Z-Ordering with dynamic, incremental clustering. As new micro-batches arrive, Spark clusters only the newly written data without rewriting historical partitions. It allows cluster keys to be redefined without rebuilding the table and speeds up query pruning by up to 10x for concurrent workloads."

🛡️ 4. Fault Tolerance & Idempotency: Handling Worker Failures

Engineering Decision: "Two mechanisms guarantee end-to-end fault tolerance and idempotence:

Delta Lake ACID Log (_delta_log): PySpark writes data files as temporary parquet files. A commit is only recorded if the entire batch completes. If an executor dies, Spark's DAG scheduler retries the failed task. If the entire job fails, no commit is written to _delta_log, and future readers simply ignore the uncommitted parquet files.

Idempotent Upserts: The Gold layer ETL uses deterministic MERGE INTO logic. Re-running the pipeline on the same source batch produces the exact same table state without creating duplicate active records.

Serving Layer Export: The Neon PostgreSQL sync writes in deterministic shard batches using explicit database transactions. If a shard sync fails, the transaction rolls back cleanly."*

🔍 5. Vector Indexing Trade-offs: Why HNSW over IVF for Online Serving

Engineering Decision: "IVF (Inverted File Index) clusters vectors into Voronoi cells using K-Means and searches only the nearest centroids. While IVF has a small memory footprint and fast build time, its recall drops significantly if query vectors lie near cluster boundaries, and it requires periodic retraining as the dataset distribution shifts.

HNSW (Hierarchical Navigable Small World) constructs a multi-layer geometric graph structure. Top layers allow fast logarithmic skipping across vector space ($O(\log N)$), while bottom layers navigate fine-grained local neighborhoods. HNSW offers superior recall (>98%), sub-5ms query latency, and does not require periodic centroid retraining, making it the ideal choice for high-precision real-time recommendation retrieval."

💥 6. Memory Efficiency: Zero-OOM Catalyst Vector Serialization

Engineering Decision: "In PySpark, running standard Python UDFs like df.apply(lambda row: json.dumps(row.embedding)) causes severe JVM-to-Python pickling overhead, bottlenecks on the single driver node, and triggers driver OOM crashes.

We solved this with two techniques:

PySpark Native to_json(): We serialized the ArrayType(FloatType) column into JSON strings using Spark's native C++ Catalyst engine (to_json(col("embedding"))), parallelizing serialization across all worker executors with zero Python inter-process overhead.

Batch JDBC Streaming: We wrote to PostgreSQL in chunked micro-batches (batchsize=5000) rather than a single massive transaction, keeping memory usage constant regardless of dataset size."*

📁 Repository Project Structure

AI-Recommendation-System/
├── .github/workflows/               # 17 Automated CI/CD & MLOps GitHub Actions Workflows
│   ├── ci.yml                       # Full testing matrix (lint, type-check, unit, integration)
│   ├── data-refresh.yml             # Automated Daily Databricks/Kaggle ETL pipeline trigger
│   ├── frontend-pages.yml           # Automated React 19 deployment to GitHub Pages
│   ├── secrets-scan.yml             # TruffleHog & Gitleaks credential scanning
│   ├── sync-hf.yml                  # Bi-directional sync with Hugging Face Space repository
│   └── keep-alive.yml               # Anti-spin down scheduler for free cloud tiers
├── backend/                         # Core FastAPI Serving Gateway & AI Infrastructure
│   ├── agents/                      # Autonomous Multi-Agent Orchestrator & Reasoning Agents
│   │   └── multi_agent_orchestrator.py # Retrieval, Recommender, Ranking, Explanation agents
│   ├── models/                      # 6 PyTorch Deep Learning Recommendation Architectures
│   │   ├── sasrec.py                # Causal Transformer Sequential Recommendation
│   │   ├── kan_ranker.py            # Kolmogorov-Arnold Network with learnable B-Splines
│   │   ├── lightgcn.py              # 3-Layer Graph Convolutional Collaborative Filtering
│   │   ├── neural_ode_recommender.py# Continuous dynamical preference drift model
│   │   ├── hyperbolic_recommender.py# Poincaré Ball Riemannian manifold embeddings
│   │   └── diffusion_recommender.py # Generative score-based latent diffusion ranker
│   ├── pipeline/                    # Recommender orchestration, caching, and rerankers
│   ├── proto/                       # High-throughput gRPC Protobuf definitions & stubs
│   │   └── recommendation.proto     # Protobuf service contract for RPC streaming
│   ├── rust_core/                   # Pure Rust Axum Serving Core (sub-millisecond gateway)
│   │   ├── src/main.rs              # Rust Axum entry point and SIMD vector search
│   │   └── Cargo.toml               # Rust dependencies and compiler optimizations
│   ├── routes/                      # Modular FastAPI REST API endpoints
│   ├── serving/                     # 3-Tier hardware detector and model warmup engine
│   └── main.py                      # FastAPI ASGI application initialization & middleware
├── databricks_notebooks/            # PySpark Medallion Lakehouse ETL Notebooks (21M+ Rows)
│   ├── 00_kaggle_download.py        # Raw TMDB & MovieLens automated Bronze ingestion
│   ├── 01_pyspark_etl.py            # Data Quality Gates, SCD2 MERGE, Liquid Clustering
│   ├── 01b_streaming_events.py      # Auto Loader streaming ingestion with checkpoints
│   ├── 01c_gpu_embeddings.py        # GPU SentenceTransformers @pandas_udf vector generator
│   └── 02_export_to_neon.py         # 10-Shard Neon Serverless PostgreSQL HNSW export
├── docs/                            # Deep Architectural Specs, ADRs, and Engineering Guides
│   ├── ARCHITECTURE.md              # Definitive system design and pipeline specification
│   ├── ARCHITECTURE_DECISIONS.md    # Full ADR records (ADR-001 through ADR-010)
│   ├── MODEL_CARDS.md               # Model cards for the 6 ensemble architectures
│   └── DATA_CONTRACTS.md            # Schema definitions, Pandera checks, and validation rules
├── frontend/                        # Modern React 19 + TypeScript + Bun 1.2 Cinema Portal
│   ├── src/                         # React UI source tree (Components, Pages, Hooks)
│   ├── package.json                 # Frontend manifests and scripts
│   └── vite.config.ts               # Vite 8 compilation and compression configuration
├── interview_prep/                  # Systems Design & Data Engineering In-Depth Guide
│   └── master_faang_de_interview_guide.md # Comprehensive technical systems architecture guide
├── k8s/                             # Kubernetes Deployment Manifests & Helm Charts
├── load-tests/                      # k6 Performance & SLO Verification Scripts
│   └── recommendation_slo.js        # p95/p99 latency and throughput load harness
├── scripts/                         # Kaggle GPU runners, benchmarks, and maintenance CLI
├── sql/                             # PostgreSQL DDL migrations, indexes, and schemas
└── tests/                           # Exhaustive Test Suite (~90 test files, unit/PBT/integration)

📚 Repository Documentation Index

The repository contains extensive documentation covering system design, mathematical proofs, and operational runbooks:

📂 Documentation Catalog (Click to Collapse)

1. System Architecture & Engineering

ARCHITECTURE.md — Comprehensive technical architecture reference.
ARCHITECTURE_DECISIONS.md — Detail records for ADR-001 through ADR-010.
AGENTIC_AI.md — Multi-agent orchestration, agent state machines, and LLM reasoning.
DATA_ENGINEERING_SYSTEM_DESIGN.md — Deep dive on distributed Lakehouse design.
MODERN_STACK_ARCHITECTURE.md — Modern stack decomposition and latency optimization.

2. Data Engineering & Lakehouse

README_DATA_ENGINEERING.md — Canonical Data Engineering & Lakehouse Blueprint (PySpark 4.x, Delta Lake Medallion, SCD-2, Liquid Clustering, 10-Shard Neon pgvector sync).
DELTA_LAKE_IMPLEMENTATION.md — SCD Type 2 MERGE, Liquid Clustering, and Auto Loader.
DATA_CONTRACTS.md — Schema contracts, Pandera expectations, and quarantine logic.
SPARK_42_FEATURES.md — PySpark 4.x features, Arrow vectorization, and Catalyst optimization.
LINEAGE.md — OpenLineage metadata tracking and data provenance graphs.
BACKFILL_AND_RECOVERY.md — Disaster recovery runbooks and historical re-indexing.

3. Machine Learning & MLOps

MODEL_CARDS.md — Model cards, training parameters, evaluation metrics, and limits.
ONLINE_LEARNING.md — Real-time gradient updates, event buffering, and checkpointing.
MODEL_HOSTING.md — Hugging Face Hub integration and ONNX quantization guides.

4. Technical System Design & Deep-Dive Guides

master_faang_de_interview_guide.md — Comprehensive technical systems design, trade-offs, and architectural scenario handbook.

⚙ Environment Configuration Reference

Create a .env file in the project root based on the following reference table:

Variable Name	Type	Default Value	Description / Operational Purpose
`ENVIRONMENT`	string	`development`	Runtime environment (`development`, `staging`, `production`).
`PORT`	int	`8000`	Port for the FastAPI application gateway.
`DATABASE_URL`	string	`sqlite:///./data/dev.db`	PostgreSQL or SQLite connection string.
`REDIS_URL`	string	`redis://localhost:6379/0`	Connection URL for Redis vector and session cache.
`JWT_SECRET_KEY`	string	—	Cryptographic secret for signing JWT auth tokens.
`NOVA_SERVING_PROFILE`	string	`lite`	Serving profile: `lite` (low memory), `full` (deep learning).
`NOVA_SERVING_TIER`	string	`tier3`	Hardware tier: `tier1` (GPU), `tier2` (ONNX CPU), `tier3` (SIMD).
`KAGGLE_USERNAME`	string	—	Kaggle API credential for automated dataset downloads.
`KAGGLE_KEY`	string	—	Kaggle API key for GPU worker orchestration.
`HF_TOKEN`	string	—	Hugging Face token for synchronizing model weights and Spaces.
`DOPPLER_TOKEN`	string	—	Doppler service token for zero-sprawl secret management.

📡 Complete REST & gRPC API Contracts

REST API Specifications

Endpoint	Method	Purpose & Contract Description
`/health`	`GET`	Health check, system diagnostics, and active hardware serving tier.
`/v1/recommendations/user/{user_id}`	`GET`	Retrieve personalized top-$K$ recommendations for a specific user ID.
`/v1/recommendations/id/{movie_id}`	`GET`	Retrieve item-to-item collaborative and content recommendations.
`/v1/recommendations/visually-similar/{id}`	`GET`	Retrieve visually similar titles using 512-D CLIP poster embeddings.
`/v1/search/ai`	`GET`	Real-time semantic natural language search over 1M+ movie vectors.
`/v1/events`	`POST`	Ingest real-time clickstream events (clicks, ratings, duration) for online learning.
`/v1/admin/reload_models`	`POST`	Zero-downtime hot-reload of model weights and vector indexes in memory.

gRPC Protobuf Service Contract (`backend/proto/recommendation.proto`)

syntax = "proto3";

package recommendation;

service RecommendationService {
  rpc GetUserRecommendations (UserRequest) returns (RecommendationResponse);
  rpc GetItemRecommendations (ItemRequest) returns (RecommendationResponse);
  rpc StreamEvents (stream EventRequest) returns (EventResponse);
}

message UserRequest {
  string user_id = 1;
  int32 top_k = 2;
  repeated string preferred_genres = 3;
}

message ItemRequest {
  string movie_id = 1;
  int32 top_k = 2;
}

message RecommendationResponse {
  repeated RecommendedMovie movies = 1;
  string model_version = 2;
  double latency_ms = 3;
  string rationale = 4;
}

message RecommendedMovie {
  string movie_id = 1;
  string title = 2;
  double score = 3;
  repeated string genres = 4;
}

🧪 Verification & Quality Gate Suite

The platform enforces strict code health, property-based testing, and automated security scans across 17 GitHub Actions workflows:

# 1. Run backend unit, integration, and ML benchmark tests
python -m pytest tests/ -v

# 2. Run frontend React unit tests (Native Bun runner)
bun --cwd frontend run test

# 3. Run pre-commit code formatting, linting, and security audits
python -m pre_commit run --all-files

# 4. Run k6 recommendation latency SLO load test
k6 run load-tests/recommendation_slo.js

🗺 Roadmap & Milestones

Milestone 1: 21M+ Record Databricks Serverless Medallion Lakehouse (Bronze/Silver/Gold).
Milestone 2: 10-Shard Neon Serverless pgvector HNSW partition cluster.
Milestone 3: 6-Model PyTorch Deep Learning Ensemble with Doubly Robust IPS debiasing.
Milestone 4: Adaptive 3-Tier Hardware Serving Engine (GPU, ONNX INT8, Rust SIMD).
Milestone 5: Autonomous Agentic Multi-Agent AI Orchestration with natural language rationale.
Milestone 6: Pure Bun 1.2 + React 19 + Vite 8 Cinema Portal deployed to GitHub Pages.
Milestone 7: WebAssembly (Wasm) client-side neural vector reranking running directly inside the browser.
Milestone 8: Distributed Ray Core integration for real-time online multi-armed bandit exploration.

📖 Research, Academic References & Acknowledgements

The algorithmic architectures implemented in this repository are based on the following seminal research papers:

SASRec: Kang, W. C., & McAuley, J. (2018). Self-Attentive Sequential Recommendation. IEEE International Conference on Data Mining (ICDM). arXiv:1808.09781.
LightGCN: He, X., et al. (2020). LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation. ACM SIGIR. arXiv:2002.02126.
KAN: Liu, Z., et al. (2024). KAN: Kolmogorov-Arnold Networks. arXiv:2404.19756.
Neural ODE: Chen, R. T. Q., et al. (2018). Neural Ordinary Differential Equations. NeurIPS. arXiv:1806.07366.
Poincaré Embeddings: Nickel, M., & Kiela, D. (2017). Poincaré Embeddings for Learning Hierarchical Representations. NeurIPS. arXiv:1705.08039.
Doubly Robust IPS: Dudík, M., et al. (2011). Doubly Robust Policy Evaluation and Learning. ICML.

❓ Frequently Asked Questions (FAQ)

Q: Do I need an expensive GPU or cloud account to run this locally?

No. While the models were trained on distributed GPU clusters, the serving engine features an automatic Adaptive 3-Tier Hardware Detector. When running on a laptop or free cloud instance (like Hugging Face Spaces), the engine automatically engages Tier 2 (Quantized INT8 ONNX CPU) or Tier 3 (Rust SIMD turbovec), delivering sub-5ms cosine similarity vector lookups on standard CPUs with <500MB RAM.

Q: Where does the dataset come from, and is it real data?

Yes, 100% real-world data provenance. The pipeline ingests over 21 Million records:

1,000,000+ TMDB movies (synopsis, genres, release dates, poster paths, budgets, cast).
20,000,000+ MovieLens user interaction ratings (user IDs, timestamps, 0.5–5.0 star ratings). These datasets are joined, deduplicated, and enriched across Databricks PySpark Delta Lake Medallion tables (bronze, silver, gold).

Q: How does this differ from basic TF-IDF or simple cosine similarity?

Basic cosine similarity over text fails to capture sequential user intent, collaborative user patterns, and deep hierarchical categories. Our engine combines 6 distinct neural paradigms:

LightGCN captures collaborative filtering patterns (what similar users enjoyed).
SASRec captures sequential momentum (what you watched 10 minutes ago vs. 2 weeks ago).
Poincaré Hyperbolic geometry captures hierarchical taxonomy trees (e.g. Cinema $\rightarrow$ Sci-Fi $\rightarrow$ Cyberpunk).
Doubly Robust IPS strips out popularity bias so niche gems aren't buried by blockbusters.

Q: Can I use this for my own catalog (e-commerce products, music, articles)?

Yes. The data ingestion, embedding generation (01c_gpu_embeddings.py), and Neon vector export (02_export_to_neon.py) pipelines are domain-agnostic. To adapt for e-commerce or music:

Map your item catalog schema to id, title, description, category.
Run the SentenceTransformer embedding generator to create 768-D vectors.
Sync to Neon pgvector or local SQLite/FAISS to immediately start serving recommendations.

Q: How does the 10-shard Neon vector architecture work?

In databricks_notebooks/02_export_to_neon.py, gold embeddings are hash-partitioned across 10 Neon serverless PostgreSQL databases using pmod(spark_hash(col("id")), 10). Each shard maintains an independent pgvector HNSW graph (m=16, ef_construction=64). The FastAPI gateway queries shards in parallel and merges candidates in-memory via SIMD cosine distance.

Q: Is this project open source? Can I contribute?

Yes! The project is licensed under the GNU AGPLv3. Issues labeled good first issue are open for community contributions. Check out CONTRIBUTING.md for guidelines!

🤝 Contributing & Community

We welcome contributions of all kinds!

📘 Contributing Guide (CONTRIBUTING.md) — Detailed environment setup, coding patterns, and testing guidelines.
🏛️ Project Governance (GOVERNANCE.md) — Contributor progression ladder and RFC process.
🤝 Code of Conduct (CODE_OF_CONDUCT.md) — Community standards and pledge.
🛡️ Security Policy (SECURITY.md) — Vulnerability disclosure guidelines.

📄 License

Distributed under the GNU Affero General Public License v3.0 (AGPL-3.0). See LICENSE for details.

For an in-depth systems design and scenario breakdown based on this architecture, see interview_prep/master_faang_de_interview_guide.md.

_{Built with ❤️ by Pavan Badempet. Star ⭐ this repository if you find it helpful!}

Name		Name	Last commit message	Last commit date
.github		.github
.gitlab		.gitlab
.kiro/specs		.kiro/specs
airflow/dags		airflow/dags
backend		backend
book		book
config		config
contracts		contracts
data		data
databricks_notebooks		databricks_notebooks
docs		docs
etl		etl
frontend		frontend
interview_prep		interview_prep
k8s		k8s
load-tests		load-tests
logs		logs
models		models
recommender		recommender
scripts		scripts
services/engine-rs		services/engine-rs
sql		sql
tests		tests
.claudecoderules		.claudecoderules
.cursorignore		.cursorignore
.cursorrules		.cursorrules
.deepsource.toml		.deepsource.toml
.dockerignore		.dockerignore
.env.example		.env.example
.gitattributes		.gitattributes
.gitignore		.gitignore
.gitlab-ci.yml		.gitlab-ci.yml
.node-version		.node-version
.nvmrc		.nvmrc
.pre-commit-config.yaml		.pre-commit-config.yaml
.windsurfrules		.windsurfrules
AGENTS.md		AGENTS.md
ARCHITECTURE_SHOWCASE.md		ARCHITECTURE_SHOWCASE.md
AUTOMATION_GUIDE.md		AUTOMATION_GUIDE.md
CHANGELOG.md		CHANGELOG.md
CITATION.cff		CITATION.cff
CODE_OF_CONDUCT.md		CODE_OF_CONDUCT.md
CONTRIBUTING.md		CONTRIBUTING.md
CONTRIBUTION_QUICKSTART.md		CONTRIBUTION_QUICKSTART.md
DEPLOYMENT.md		DEPLOYMENT.md
Dockerfile		Dockerfile
GOVERNANCE.md		GOVERNANCE.md
LICENSE		LICENSE
README.md		README.md
README_DATA_ENGINEERING.md		README_DATA_ENGINEERING.md
SECURITY.md		SECURITY.md
TOKEN_OPTIMIZATION_GUIDE.md		TOKEN_OPTIMIZATION_GUIDE.md
docker-compose.kafka-cluster.yml		docker-compose.kafka-cluster.yml
docker-compose.override.yml		docker-compose.override.yml
docker-compose.yml		docker-compose.yml
manage.py		manage.py
package-lock.json		package-lock.json
package.json		package.json
prometheus.rules.yml		prometheus.rules.yml
prometheus.yml		prometheus.yml
pyproject.toml		pyproject.toml
render.yaml		render.yaml
requirements-dev.txt		requirements-dev.txt
requirements-etl.txt		requirements-etl.txt
requirements.txt		requirements.txt
start.bat		start.bat
stop.bat		stop.bat
uv.lock		uv.lock
wrangler.toml		wrangler.toml

Latest commit

History

Folders and files

Repository files navigation

🎬 AI Recommendation & Streaming Lakehouse

Distributed Databricks PySpark Delta Lakehouse, 10-Shard Neon Serverless pgvector HNSW & Real-Time Deep Learning Platform (21M+ Records)

💡 Why I Built This & Executive Architecture Summary

📸 Production Previews (Captured from Live Hugging Face Space)

🎬 1. Modern Cinema Workstation (Personalized Discovery Feed)

🔍 2. Interactive Movie Details & Neural Match Scores

🌊 3. Lakehouse Data Platform & MLOps Dashboard

🚀 Real Usage Examples (Ready to Run)

1. cURL (Instant CLI Query)

2. Natural Language Semantic Vector Search

3. Python Client (Zero Dependencies besides requests)

4. TypeScript / React (Frontend Integration)

⚡ Quickstart & Installation (Zero Guesswork)

Option A: 1-Click Docker Compose (Fastest & Recommended)

Option B: Local Step-by-Step (Python & Node / Bun)

Option C: 1-Click Cloud Workspace (GitHub Codespaces)

🎯 Core Platform Specifications & Tech Stack

⚡ Latest Production Stack & Ecosystem Matrix (2026 Standards)

🏛️ Data Engineering & Distributed Systems Deep-Dive

1. System Design & DDIA Principles (Designing Data-Intensive Applications)

A. Reliability (Fault Tolerance & Data Integrity)

B. Scalability (Volume & Throughput Growth)

C. Maintainability (Operability, Simplicity, & Evolvability)

2. Enterprise Tradeoff & Edge Case Matrix

3. Core Technical Architecture

4. Databricks PySpark Medallion Lakehouse Notebooks

Real PySpark Transformations & Joins

5. 10-Shard Neon Serverless Vector Serving Tier

High-Performance Post-Sync Indexing DDL

6. Autonomous Agentic AI Suite

7. 6-Model PyTorch Deep Learning Ensemble & Model Registry

8. Advanced Mathematical & Algorithmic Foundations

A. Kolmogorov-Arnold Network (KAN) B-Splines

B. Neural ODE Continuous Temporal Drift

C. Poincaré Ball Hyperbolic Metric

D. Doubly Robust Inverse Propensity Scoring (DR-IPS)

E. Differential Privacy Laplace Mechanism

9. Adaptive 3-Tier Hardware Serving Architecture

10. Architecture Decision Records (ADR) Summary

🗄 Database Layer Schema & Star Schema

🛡️ High-Concurrency Backend System Design & Distributed Serving

1. Distributed Rate Limiting & Multi-Tenant Quota Management

2. Layer 4 vs. Layer 7 Load Balancing & Consistent Hashing

3. Resilience Engineering: Circuit Breakers, Bulkheads & Jittered Backoff

4. Distributed Multi-Tier Caching & Cache Stampede Defense

5. Zero-Downtime Deployment, Kubernetes Probes & Graceful Shutdown

🌊 Architecture Deep-Dive: Scaling & Technical Decisions

🚀 1. Scaling Architecture: Scaling from 21 Million to 21 Billion Records

⏱️ 2. Temporal State Fidelity: Why Delta Lake SCD Type 2 for Feature Stores

⚡ 3. Storage Optimization: Liquid Clustering vs. Legacy Z-Ordering

🛡️ 4. Fault Tolerance & Idempotency: Handling Worker Failures

🔍 5. Vector Indexing Trade-offs: Why HNSW over IVF for Online Serving

💥 6. Memory Efficiency: Zero-OOM Catalyst Vector Serialization

📁 Repository Project Structure

📚 Repository Documentation Index

1. System Architecture & Engineering

2. Data Engineering & Lakehouse

3. Machine Learning & MLOps

4. Technical System Design & Deep-Dive Guides

⚙ Environment Configuration Reference

📡 Complete REST & gRPC API Contracts

REST API Specifications

gRPC Protobuf Service Contract (backend/proto/recommendation.proto)

🧪 Verification & Quality Gate Suite

🗺 Roadmap & Milestones

📖 Research, Academic References & Acknowledgements

❓ Frequently Asked Questions (FAQ)

🤝 Contributing & Community

📄 License

About

Topics

Resources

Code of conduct

Contributing

Security policy

Accessibility

3. Python Client (Zero Dependencies besides `requests`)

gRPC Protobuf Service Contract (`backend/proto/recommendation.proto`)