Measure it.
Don't just claim it.
Groundtruth is an eval lab for the Atlas assistant. Sixteen frozen cases with deterministic assertions, plus a retrieval ablation that recomputes in the browser — no model calls, no rate limit.
- 16 cases
- cases
- 20 chunks · 6 docs
- corpus
- 1.000
- hit-rate@4
// 01
The golden set.
16 cases · frozen 2026-09-19| id | question | expects |
|---|---|---|
| eval-numbers | What does the Atlas Eval Report say the retrieval recall at k=4 is? | eval-report |
| eval-rubric | How does the eval rubric score an answer? | eval-report |
| eval-block | What does the Atlas Eval Report say about the deploy threshold? | eval-report |
| v03-title-bonus | What did v0.3 change? | changelog |
| v03-score | How much did the eval score improve in v0.3? | changelog |
| admit-rule | What do the Atlas Design Notes say about admitting nothing? | design-notes |
| cold-start | Why is cold start under a second? | design-notes, runbook |
| zero-chunk-retry | What should happen when retrieval returns zero chunks? | runbook |
| restart-steps | How do I restart the service? | runbook |
| tool-contract | What does `search_knowledge` return? | api-reference |
| empty-query | What does the tool do with a query of only stopwords? | api-reference |
| out-of-corpus-topic | What does the FAQ say the corpus does *not* cover? | faq |
| refuse-pizza | What is the CEO's favourite pizza? | — |
| refuse-aws | Which AWS region does Atlas run in? | — |
| refuse-pricing | How much does Atlas cost per month? | — |
| refuse-hiring | How many engineers work on Atlas? | — |
demo corpus — not the production system
// 02
Live run.
against /api/atlas — 1 model call per runSelect a case from above to run it against /api/atlas.
The scorecard below updates after the answer finishes streaming.
Pick a case to score it. Free-typed questions are not scored.
10 requests / minute shared across the whole site (interview chat, store support, and the Atlas live run). The ablation below is free.
// 03
The ablation.
client-side · no model calls| case | default rank | this config | hit |
|---|---|---|---|
| eval-numbers | 1 | 1 | ✓ |
| eval-rubric | 1 | 1 | ✓ |
| eval-block | 1 | 1 | ✓ |
| v03-title-bonus | 1 | 1 | ✓ |
| v03-score | 1 | 1 | ✓ |
| admit-rule | 1 | 1 | ✓ |
| cold-start | 1 | 1 | ✓ |
| zero-chunk-retry | 1 | 1 | ✓ |
| restart-steps | 1 | 1 | ✓ |
| tool-contract | 1 | 1 | ✓ |
| empty-query | 1 | 1 | ✓ |
| out-of-corpus-topic | 1 | 1 | ✓ |
// 04
Method & limits.
what this measures — and what it can'tWhat it does: The ablation runs rank() over the 20 shipped chunks with 12 grounded cases at configurable title-bonus, k, and stopword settings. Every number is computed client-side in milliseconds — zero gateway calls.
Two sets: The corpus's fictional recall@4 of 0.88 comes from 48 golden questions in Atlas Changelog(the corpus's own fiction). This lab measures its own 12 grounded cases against the shipped retriever. Both are honest; they are not the same set.
What it does not measure: semantic correctness, paraphrase quality, hallucination beyond the named assertions, or latency. Sixteen cases against a 20-chunk corpus is a demonstration harness, not an eval platform.
Assertions are brittle: scoring uses exact string matching on phrases the corpus fixes verbatim (version numbers, scores, doc titles). The model may phrase things differently across runs.
The unconfirmed claim: the Atlas Changelog says its remaining retrieval misses are changelog queries. This harness reproduces the title-bonus claim (name-first cases rise with the bonus) but does not reproduce the changelog-miss claim — no changelog case misses at any config. The harness says so rather than implying otherwise.