The Coroner’s Reports
18 questions · 4 witnesses · graded blind · frozen 2026-09-27
Every answer the migration agent gave in its evaluation, with every query it ran. Nothing to install.
How to read this
- No tools. The model alone, answering from memory.
- Keyword search. BM25 over the same 446 documents: top 8 passages pasted in, one pass, references not followed.
- Sanity Context · GROQ. The agent with the GROQ endpoint: schema, joins, Portable Text.
- GROQ + Knowledge Base. The agent with both endpoints.
- Answers were shuffled and relabelled before an independent model graded them against a fixed rubric; citations were checked by script. Verdict: reached the expected conclusion. Grounded: no claim beyond retrieved evidence. Cited: cited every required publisher.
- Models: the two Sanity conditions ran on the smaller model; the two baselines ran on the larger model. Grader: the larger model. The DEV posts name both models.
Totals
No tools10/18verdicts · grounded 0/18 · cited 11/16
Keyword search14/18verdicts · grounded 16/18 · cited 4/16
Sanity Context · GROQ17/18verdicts · grounded 18/18 · cited 12/16
GROQ + Knowledge Base17/18verdicts · grounded 18/18 · cited 12/16
The cases
Cases q04 and q13 were re-run after a fix; open them to see both runs. Each cell: verdict · grounded · cited. Hover for the grader’s note. Raw runs: agent/eval.
Added after the freeze
How-to questions written after the frozen run to test the Knowledge Base on the prose it holds. Same conditions and models; not counted in the totals above.
| # | Question | No tools | Keyword search | Sanity Context · GROQ | GROQ + Knowledge Base |
|---|---|---|---|---|---|
| q19 | We're moving a 19c Enterprise Edition database from IBM AIX to Oracle Database@Azure (Exadata) with a 2-hour outage window. Which migration method should we use, and in what order should we rule the options out? howto | ✗✗✗ | ✗✓✗ | ✓✓✗ | ✓✓✓ |
| q20 | Our Azure standby is in sync and the on-premises primary is healthy. For the cutover, should we do a Data Guard switchover or a failover, and what do we check first? howto | ✓✗✗ | ✓✓✓ | ✓✓✓ | ✓✓✓ |