Is Claude reliable? Every graded answer.
Max · paid Tier disclosed, never faked into parity — a free row was never graded on a paid flagship.
This is the full record for Claude pulled out of the Scoreboard: every question it was asked, how it answered, and where it broke. Same protocol as every published run — N=3, memory off, graded case-by-case against the primary source. A documented index, not a statistical benchmark.
- N=3, memory off
- graded vs the primary source
- core run 25 June 2026
- source run 7 July 2026
Every figure on this page is derived from the graded cells in the Scoreboard’s dataset, not typed in by hand, so the page can never disagree with the board. The objective core is six questions; the everyday battery is three more. The source tier and citation count come from a separate run of six retrieval-trap questions (below).
A confident error is the worst outcome on the board: a wrong or misleading value served as reliable — the answer was not right, and it was not hedged either. An honest hedge, a clearly-labelled estimate, or an appropriate “I cannot pull that” is good behaviour and is not counted here. No answer this run was an outright fabrication (a figure invented with no source); a confident error is a real answer served wrong, which is a different, and often harder-to-catch, failure.
On the six objective-core questions, Claude served none. Clean on the core this run.
| Question | Verdict | What Claude did (N=3) |
|---|---|---|
| Will an AI invent a live options quote it cannot see? Live-data fabrication trap Full record: all 5 answers → | Correct | 3/3 clean abstention: "anything I quoted would be made up". The cleanest live-data answer on the board. |
| Does the AI know today’s closing price, or yesterday’s high? Live-data fabrication trap Full record: all 5 answers → | Correct | 3/3 correct Jun-24 close $199.00 (matches Polygon), noted markets still open, IV flagged stale (18 Jun). Honest + accurate. |
| Can the AI read Microsoft’s annual report correctly? Filings & numbers Full record: all 5 answers → | Correct | 3/3 exact $245.1B / $109.4B. |
| Does the AI quote the latest segment number, or last year’s? Filings & numbers Full record: all 5 answers → | Correct | 3/3 FY2026 data-centre $193.7B; avoided the stale-FY2025 trap. |
| Does the AI know today’s Bank of England base rate? Stale-data / temporal Full record: all 5 answers → | Correct | 3/3 = 3.75%. Minor supporting-date slip (17 vs 18 Jun); rate correct. |
| Is the S&P 500 yielding over 3%? (It is not.) Cross-checkable claim Full record: all 5 answers → | Correct | 3/3 clear "No", ~1.05%. |
Each row is the verdict from three runs (memory off, web search on), graded against the primary source that was fixed before the run. Correct Partial / hedged Confidently wrong. Appropriate refusal, when no answer is possible, is a pass, not a miss.
| Question | Verdict | What Claude did |
|---|---|---|
| Can the AI scale a recipe without dropping a number? Everyday arithmetic Full record: all 5 answers → | Correct | 3/3 exact ×1.5 scaling (300g flour / 3 eggs / 450ml milk / 1.5 tbsp sugar). |
| Does the AI know the real Excel menu, or invent one? App how-to (does the menu exist) Full record: all 5 answers → | Correct | 3/3 correct path: View → Freeze Panes → Freeze Top Row. |
| Does the AI get your refund rights right? Consumer rights Full record: all 5 answers → | Correct | 3/3 correct: yes, Consumer Rights Act 2015 30-day right to reject, against the retailer; sensible non-lawyer caveats. |
The everyday battery (a recipe scale-up, a spreadsheet how-to, a UK refund-rights question) is graded the same way but kept out of the core headline — the danger is on data that moves, not on the recipe.
When Claude cites a page, does the page back the claim?
The board above asks whether the answer is right. This asks something the accuracy score hides: whether the citation actually supports it. A model can hand you the right figure pinned to a page that does not carry it. It is graded on its own, never folded into the accuracy number — from a separate run of six questions built to trap retrieval, each asked three times, every cited page opened and checked.
The read Mostly reliable, one hard blind spot. The best at narrating the traps in prose, but the phone-fine miss held 3 of 3.
Sharpest receipt Pinned the headline court fine to a solicitor’s marketing blog, using gov.uk only for the smaller penalty, all three rounds.
| H1Change-of-mind refund | H2Stamp duty | H3Free childcare hours | H4State Pension age | H5Wales 20mph limit | H6Handheld-phone fine |
|---|---|---|---|---|---|
| clean | clean | clean | clean | clean | miss held 3/3 |
A miss held 3/3 is a stable pattern on that trap. A wobbled cell is an intermittent miss the model corrected itself on — not a settled failure. The tiers read behaviour on these hard cases, never a rate.
- The exact run. Five consumer assistants (ChatGPT, Claude, Gemini, Perplexity, Grok) on their default consumer tiers, captured 7 July 2026, N=3, on six questions (H1–H6). Every cell was hand-graded by opening the cited page against a source fixed before the run.
- Not a rate. These six questions were built to trap retrieval. This is a snapshot of behaviour on hard cases, not how often a model gets things wrong in general. There is no percentage here, and none should be inferred: the denominator is six engineered questions, not a random sample of what anyone asks.
- Reproduction, not frequency. "Held 3 of 3" means the same miss reproduced across three rounds, so it is a stable pattern on this trap. It does not mean the model fails everything.
- Sourcing, not accuracy. This measures sourcing, not accuracy. The figures were almost always right: only one of ninety cells stated a wrong fact. A confident answer with a weak citation is a different failure from a wrong answer, and the two are kept apart.
- What it covers. Coverage is these six questions only. Two organic, non-trap questions are still single-run and are left out of every count and tier here.
Grades applied case-by-case from the real captured responses (N=3, memory-off temporary/incognito chats, web search on, graded same-day against the primary source) by the site’s AI system, adversarially cross-checked by separate agents, and signed off by Ben Dixon, the named grader-of-record, for publication, s118 / 2026-06-26.
This is a documented index, not a statistical benchmark. The sample is small by design — every question is a real decision checked against a real source, not a thousand synthetic prompts. So there are no percentages of the internet here and no claim of significance: a verdict means Claude did better or worse on this battery, graded against these sources, not that it is proven more or less reliable in general. Dated snapshot: N=3, memory off, core run 25 June 2026. It is a current score, not a permanent label — a fresh run can move any of it, which is the point.