AI Engineering · RAG
Index Methodology RAG Assistant
Answering “what's the cap rule for this index?” means searching hundreds of pages of methodology PDFs. It's slow, and a wrong answer ends up in a rebalancing file.
- My role
- Solo project: parsing, retrieval, evaluation sets and web UI.
- When
- 2026
81.9%
held-out accuracy (95% CI 73–88%)
14/14
unanswerable questions correctly refused
1,082
pages from 7 index rulebooks
4 GB
laptop GPU, fully local
Approach
Page-level PDF parsing with OCR, hybrid retrieval (BM25 + bge-m3 embeddings) with LLM query rewriting, and a local model that answers only from the retrieved pages, citing the exact page or replying NOT_FOUND.
Results
Run once on a frozen system, the assistant answers 81.9% of 94 held-out questions correctly (95% CI 73–88%) and refuses all 14 questions the documents don't answer. Held-out accuracy matches the 126-question dev set it was tuned on (81.7%), so tuning didn't overfit. Testing paraphrased questions showed that keyword search only looked good on questions worded like the documents (98% vs 42% retrieval), and swapping the answer model added 12 points.
Technical highlights
- 01Two question sets: a dev set for every decision, a held-out set run once at the end
- 02Held-out labels reviewed blind by a person, 95% confidence intervals on every number
- 03Hybrid retrieval with LLM query rewriting; paraphrased questions tracked separately
- 04Deterministic scoring with accepted wordings and forbidden facts, no LLM judge
Stack
Related case studies
AI Engineering · Agents
AI Agents Toolkit for Index Operations
46
reusable skills across 5 agents, in daily use
AI Engineering · Agents
Aloi, a Market Research Agent that Checks its Figures
27/28
eval questions passed with no figure missing from the fetched data (one run)
AI Engineering · Developer Tools
AgentDeck, One Place to Answer Your Coding Agents
1 place
to answer Claude Code and Codex, on the PC or from a phone