AI Engineering · Evals
LLM Evaluation Toolkit
One accuracy number hides what matters: are the labels right, how precise is the number, and is version B actually better than version A or just luckier on this test set?
- My role
- Solo project, extracted and generalised from the methodology-rag evaluation harness.
- When
- 2026
1 of 9
changes distinguishable from noise (p < 0.05)
+11.9
points from the model swap, p = 0.003
0
dependencies: Python standard library only
3
Python versions passing CI (3.10, 3.12, 3.13)
Approach
Deterministic fact-based scoring (accepted wordings, forbidden facts, refusals, page-level citations), Wilson confidence intervals, exact McNemar tests for A/B comparisons, label checks against the source pages, and a held-out ledger that flags a test set once it has been reused.
Results
Re-analysing the saved runs of methodology-rag showed that only 1 of 9 design decisions was a statistically significant improvement: swapping the answer model (+11.9 points, 19 questions gained vs 4 lost, p = 0.003). Two “better” prompts lost 8–9 points, but even that was borderline (p ≈ 0.05–0.06), and retrieval changes of 3–7 points were not distinguishable from chance on about 118 questions. Proving a 5-point gain would take a few hundred questions.
Technical highlights
- 01Scoring with accepted wordings and forbidden facts, no LLM judge
- 02Wilson intervals on every rate, exact McNemar test for A/B comparisons
- 03Label check: every expected fact must appear on its gold page
- 04Held-out ledger warns when a test set has already influenced a decision
Stack
Related case studies
AI Engineering · Agents
AI Agents Toolkit for Index Operations
46
reusable skills across 5 agents, in daily use
AI Engineering · Agents
Aloi, a Market Research Agent that Checks its Figures
27/28
eval questions passed with no figure missing from the fetched data (one run)
AI Engineering · Developer Tools
AgentDeck, One Place to Answer Your Coding Agents
1 place
to answer Claude Code and Codex, on the PC or from a phone