Case Study
Ask the Claude Docs
An evaluated RAG chatbot with honest refusals
A RAG assistant over the live Claude Code documentation. Every answer cites the exact doc passage it came from; when nothing clears a calibrated confidence threshold, it declines instead of guessing, and shows its receipts. Quality is measured, not claimed: a CI eval harness scores retrieval and answer quality on every change and publishes the results, failures included.
Overview
A production RAG pipeline end to end: a nightly GitHub Action ingests the live Claude Code docs, chunks them along heading boundaries, and embeds only what changed (an unchanged corpus re-syncs in 26 seconds with zero embedding calls). Questions are answered by Claude Haiku from retrieved passages only, streamed with the sources arriving before the first token, with real cost printed on every receipt. The distinctive part is what happens when it can't answer: a server-side gate calibrated from measured score distributions refuses before the model is ever called, and a public /evals page shows retrieval accuracy, LLM-judged answer quality with a noise margin, and the refusal tests, all re-run by CI so a regression cannot merge. The entire application was built by Claude Code executing twelve specification documents I wrote, under a three-tier decision authority that escalated ten real engineering conflicts to me for rulings. Hand-built editorial design system in three typefaces, no UI framework, zero accessibility violations.
Key Features
- 01Server-side refusal gate: below the calibrated similarity threshold the model is never called; a decline costs 30x less than an answer and says so on its receipt
- 02Passage-level citations streamed sources-first over SSE; every [n] marker deep-links to the exact heading on code.claude.com
- 03Public eval scoreboard: retrieval hit@5 and MRR plus a four-check LLM judge with a measured noise margin, re-run by CI on every change; regressions cannot merge
- 04Self-maintaining corpus: a daily Action hash-diffs the live docs and embeds only deltas; first ingest 8 minutes, unchanged re-sync 26 seconds
- 05Spend defenses with asymmetric failure postures: per-IP rate limiting fails open, the global daily spend cap fails closed; every receipt computes cost from real API usage
- 06Built by an AI agent under governance: twelve frozen spec documents, three-tier decision authority, phase gates with self-audits, ten escalated rulings, all logged
- 07Hand-built design system: three typefaces as three voices, token palette with measured WCAG contrast, no UI framework, zero axe violations
- 08Hardened CI: Playwright e2e with axe, Lighthouse budgets, payload assertions, secret greps and CVE gates that caught three real advisories mid-build