Engagement-Based News Recommendation System

Overview

A build personalized news recommendation system for the platform’s content aggregator, inspired by X’s (Twitter’s) open-source recommendation algorithm. Instead of relying on manual category tags, it learns user preferences from engagement patterns, clicks, reads, saves, shares, and negative signals, over a corpus of millions of articles.

The Challenge

Editorial category tagging is brittle and labour-intensive, and it fails to capture how individual readers actually behave. The build set out to validate whether an engagement-driven, embedding-based recommender could surface relevant content automatically and adapt to each reader’s implicit interests.

What We Built

A polyglot, multi-service system documented in a six-part architecture spec. The Python backend (FastAPI + SQLAlchemy over PostgreSQL holding 5M+ articles) handles article sync, embedding sync, recommendation, and topic-diversity logic, and talks over gRPC to a “Phoenix” ML service (with generated home_mixer, phoenix, and thunder protobuf stubs) plus dedicated Rust services for performance-critical ranking. A React + Vite UI provides Feed, Explore, and History pages with article/post/profile cards, a user selector, and engagement-capturing hooks (useDwellTime, useInfiniteScroll). Training pipelines (train_ebnerd.py, train_mind.py, EB-NeRD/MIND datasets with checkpointing) back the model, and the whole thing is containerized via docker-compose with per-service Dockerfiles.

Technologies & Approach

Python/FastAPI/SQLAlchemy for the API and data layer, Rust for low-latency ranking services, gRPC for inter-service ML calls, Kafka for the ingestion/event layer, PostgreSQL for storage, and a React/Vite front-end. The design deliberately mirrors X’s open-sourced “home mixer” philosophy: preferences emerge from behavior, with negative signals actively filtering content.

Outcome / Impact

As a build it validated an end-to-end architecture for engagement-based recommendations, ingestion, embedding, ranking, diversity, and a behavior-capturing UI, proving the team could stand up a sophisticated, multi-language recommender modeled on industry-leading open-source systems. Dwell-time and infinite-scroll instrumentation demonstrated how implicit engagement signals feed back into ranking.

Capabilities Demonstrated

• Designing an engagement-driven recommendation engine (no manual tagging)
• Polyglot microservice architecture: Python, Rust, and gRPC interop
• Embedding-based content ranking with topic-diversity controls
• Training pipelines over public news datasets (EB-NeRD, MIND) with checkpointing
• Instrumented React/Vite UI that captures implicit engagement signals
• Containerized multi-service delivery with Kafka and PostgreSQL at article-scale