Recommendation-Feed Algorithm Reimplementation (Rust)
Overview
A from-scratch build reimplementation of a large social platform’s “For You” recommendation feed, built as a Rust workspace. It models the full pipeline, in-network and out-of-network candidate retrieval, transformer-based ranking, and filtering, alongside a Python news-recommendation build on a public dataset.
Why It Exists
To deeply understand modern feed-ranking architecture by rebuilding it: how a system mixes followed-account content with ML-retrieved out-of-network content, scores it with a transformer, and assembles a ranked timeline. An ambitious learning/R&D effort, not an affiliated or production system.
What We Built
A Cargo workspace with four crates, proto (gRPC contracts), candidate-pipeline (retrieval), thunder (in-network candidate source), and home-mixer (the orchestration/mixing stage), plus phoenix (out-of-network retrieval) and a transformer-ranking layer ported from the open Grok-1 release. Services communicate over gRPC (Tonic/Prost) on a Tokio async runtime, with Kafka (rdkafka) for streaming. A separate news-recsys-build/ holds a Python pipeline over the EBNeRD news dataset (Parquet articles, an embeddings DB, Dockerized backend) for offline iterative development.
Technologies & Approach
Rust for the high-throughput serving path, Tokio, Tonic/Prost gRPC, dashmap, rdkafka, structured as independent microservice crates that mirror real feed-ranking stages (retrieve → mix → rank → filter). Python with the EBNeRD corpus handles the recsys research side. The transformer ranker reuses the open Grok-1 implementation adapted for recommendation.
Outcome / Impact
A substantial build that validated an end-to-end, microservice-based recommendation-feed architecture in Rust and exercised transformer-based ranking and embedding retrieval, proving the studio can design and reason about web-scale recsys systems.
Capabilities Demonstrated
- Designing a multi-stage recommendation/ranking pipeline
- High-performance gRPC microservices in Rust (Tokio/Tonic)
- Embedding-based candidate retrieval over a large corpus
- Transformer ranking model integration
- Offline recsys iterative development on public datasets (EBNeRD)