Eval-driven development
Evals
The archive's answer quality is measured, not asserted. A versioned golden question set runs weekly against the live corpus; every scorecard below — including the failures — is published automatically.
Latest run
2026-10-26 · 11 questions · model gemini-2.5-flash · 3cfafa6
Retrieval hits
73%
expected post surfaced
Valid citations
100%
every citation was actually retrieved
Judge quality
4.8/5
LLM-judged faithfulness
Failed
0
errors or unjudgeable
Trend across runs
Honest failures — the lowest-scored answers of the latest run
What did I save about 18th-century French naval strategy?
[negative] · retrieval miss · judge 4/5
The system correctly states it has no direct information on the specific topic. However, it then provides tangential information about general strategic methodologies that is irrelevant to the specific historical and geographical context of the question, diluting the otherwise perfect decline.
Pulling together everything I've read, what does building agent-first products actually require?
[synthesis] · retrieval hit · judge 4/5
The answer provides a comprehensive and useful response to the question, drawing from multiple relevant sources. However, some points citing the 'Trading Margin for Moat: Why the Forward Deployed Engineer Is the Hottest Job in Startups' source seem to apply principles from human forward-deployed engineers directly to AI agents, which is a loose use of citation for direct grounding about AI agents.
Summarize my notes on competitive figure skating scoring rules.
[negative] · retrieval miss · judge 5/5
The system correctly identified that no relevant information on competitive figure skating scoring rules was available in the provided sources, earning a perfect score for a correct decline.
Methodology
The golden set lives in the repository (evals/golden.json): single-hop questions with a known correct source, multi-hop questions spanning posts, synthesis questions, and negative questions the archive should decline — declining honestly scores a 5, hallucinating scores a 1.
Three metrics per question. Retrieval hit rate: did the expected post appear in the retrieved set? Citation validity: is every [^slug] marker in the answer a post that was actually retrieved (no hallucinated citations)? Answer quality: a 1–5 grade from an LLM judge scoring faithfulness to the retrieved sources.
Limitations, stated plainly: the judge is the same model family as the generator (self-grading bias), the set is small and curated by the person it grades, and a weekly cadence smooths over regressions between runs. It still catches the failures that matter — retrieval drift, citation hallucination, and decline-vs-hallucinate behavior.
What the archive couldn't answer
Questions that retrieved zero sources are logged automatically and reviewed before they appear here. They become the reading list: the feedback loop that decides what gets saved next.
No reviewed questions yet. A question the archive couldn't answer shows up here once it has been reviewed.