Back

Project 07

RAG Document Assistant + GPT-from-Scratch

Two projects in one, on purpose: a RAG pipeline over 2,000 ML paper abstracts, and a GPT built from raw PyTorch with a full SFT-to-DPO alignment pass, so the RAG system and the thing it's built on top of are both understood, not just one of them.

The Retrieval Pipeline

Retrieval combines dense embedding search and BM25 keyword search via reciprocal rank fusion, measured against a synthetic ground truth built so each question is answerable only from its source chunk. BM25-only (NDCG@5 0.905) actually beat dense-only (0.794) here, because questions generated directly from a chunk's own text share exact vocabulary with it, favoring lexical match, a real and explainable bias in synthetic-query eval sets, not a bug. Hybrid RRF (0.921) still won overall, and cross-encoder reranking recovered dense retrieval's gap further, to 0.960.

Context Precision came out at 0.28, lower than the other three RAGAS metrics, and was reported as measured rather than tuned until it looked better: a system can have high recall and faithfulness while dragging in a few tangential chunks, and being able to explain that tradeoff is the point, not a red flag to hide.

GPT-from-Scratch, then SFT to DPO

The GPT half implements a byte-pair-encoding tokenizer, causal self-attention, and a transformer stack directly in PyTorch, no nn.MultiheadAttention, no pretrained tokenizer, trained on TinyShakespeare from random initialization down to a val loss of 3.18. On top of that, Qwen2.5-0.5B goes through QLoRA-SFT, then DPO alignment. A bias-controlled pairwise win-rate eval showed SFT alone winning more often than DPO, 35% to 20%, reported as-is rather than re-tuned into a more flattering number.

The Honest Parts

The live demo runs real hybrid retrieval only, the reranker (2.3GB) and a 1.5B-parameter generator don't fit a 512MB free-tier host, with a separate local FastAPI service and 16 tests covering the full pipeline, including attention-shape, causal-masking, and BPE-tokenizer-roundtrip unit tests for the from-scratch GPT.

What I Learned

  • A negative result, reported honestly, beats an inflated oneDPO's training-time reward signal looked clean, but the bias-controlled win rate showed SFT winning more often. Re-tuning hyperparameters until the number looked better would have been the exact kind of post-hoc tuning that makes an eval meaningless. Reporting it as measured demonstrates knowing that LLM-judge win rates need bias controls to mean anything in the first place.
  • Bias control is what makes a win-rate number trustworthyAn independent judge (not the DPO teacher model) plus swapping presentation order and reconciling disagreements is what separates a real measurement from a number that just reflects which model the judge happened to see first, or was trained to prefer.
  • Synthetic-query eval sets carry a known lexical-overlap bias, worth naming explicitlyGenerating a question directly from a chunk's own text gives real ground truth without manual labeling, but it also biases toward lexical match, which is exactly why BM25-only outscored dense-only here. Naming that bias is more useful than a number presented without it.
  • Deploy scope should be an honest tradeoff, not a silent downgradeThe live demo shows retrieval only because the reranker and generator genuinely don't fit a free host's RAM, documented on the page itself rather than silently swapped for something smaller and weaker without saying so.

Tech Stack

PyTorchHuggingFaceFAISSBM25Qwen2.5LoRA/QLoRADPO (trl)FastAPIGradio

Links

LiveGitHub