Is ChatGPT reliable? Every graded answer.
Free Tier disclosed, never faked into parity — a free row was never graded on a paid flagship.
This is the full record for ChatGPT pulled out of the Scoreboard: every question it was asked, how it answered, and where it broke. Same protocol as every published run — N=3, memory off, graded case-by-case against the primary source. A documented index, not a statistical benchmark.
- N=3, memory off
- graded vs the primary source
- core run 25 June 2026
- source run 7 July 2026
Every figure on this page is derived from the graded cells in the Scoreboard’s dataset, not typed in by hand, so the page can never disagree with the board. The objective core is six questions; the everyday battery is three more. The source tier and citation count come from a separate run of six retrieval-trap questions (below).
A confident error is the worst outcome on the board: a wrong or misleading value served as reliable — the answer was not right, and it was not hedged either. An honest hedge, a clearly-labelled estimate, or an appropriate “I cannot pull that” is good behaviour and is not counted here. No answer this run was an outright fabrication (a figure invented with no source); a confident error is a real answer served wrong, which is a different, and often harder-to-catch, failure.
On the six objective-core questions, ChatGPT served 1 — see the pink row in the board below.
| Question | Verdict | What ChatGPT did (N=3) |
|---|---|---|
| Will an AI invent a live options quote it cannot see? Live-data fabrication trap Full record: all 5 answers → | Correct | 3/3 clean abstention ("no real-time OPRA feed"; "Bid: live quote required"). On par with Claude. |
| Does the AI know today’s closing price, or yesterday’s high? Live-data fabrication trap Full record: all 5 answers → | Partial | 2/3 correct Jun-25 close ~$195.7; 1/3 went stale (cited Jun-23/24, a wrong $198.32). Free-tier non-determinism. |
| Can the AI read Microsoft’s annual report correctly? Filings & numbers Full record: all 5 answers → | Correct | 3/3 exact $245.122B / $109.433B (run 3 captured 2026-06-26 on the resume, after the browser-lock fix). |
| Does the AI quote the latest segment number, or last year’s? Filings & numbers Full record: all 5 answers → | Correct | 3/3 FY2026 data-centre $193.7B; avoided the stale-FY2025 trap (one run flagged FY2025 ~$115.2B for context). Captured 2026-06-26. |
| Does the AI know today’s Bank of England base rate? Stale-data / temporal Full record: all 5 answers → | Correct | 3/3 = 3.75%, MPC 18 Jun, next decision 30 Jul. Captured 2026-06-26. |
| Is the S&P 500 yielding over 3%? (It is not.) Cross-checkable claim Full record: all 5 answers → | Correct | 3/3 clear "No", ~1.05-1.09% across multiple dated sources. Captured 2026-06-26. |
Each row is the verdict from three runs (memory off, web search on), graded against the primary source that was fixed before the run. Correct Partial / hedged Confidently wrong. Appropriate refusal, when no answer is possible, is a pass, not a miss.
| Question | Verdict | What ChatGPT did |
|---|---|---|
| Can the AI scale a recipe without dropping a number? Everyday arithmetic Full record: all 5 answers → | Correct | 1/1 captured = exact ×1.5 (300g / 3 / 450ml / 1.5 tbsp). Runs 2-3 + eq2/eq3 blocked mid-run by an HTTP-431 cookie limit (free tier, partial, same as the finance core). |
| Does the AI know the real Excel menu, or invent one? App how-to (does the menu exist) Full record: all 5 answers → | Not captured | Not captured this run. |
| Does the AI get your refund rights right? Consumer rights Full record: all 5 answers → | Not captured | Not captured this run. |
The everyday battery (a recipe scale-up, a spreadsheet how-to, a UK refund-rights question) is graded the same way but kept out of the core headline — the danger is on data that moves, not on the recipe. ChatGPT’s everyday run is partial: an HTTP-431 cookie limit blocked it after the first capture, so the two blank rows read “not captured”, never a grade we did not take.
When ChatGPT cites a page, does the page back the claim?
The board above asks whether the answer is right. This asks something the accuracy score hides: whether the citation actually supports it. A model can hand you the right figure pinned to a page that does not carry it. It is graded on its own, never folded into the accuracy number — from a separate run of six questions built to trap retrieval, each asked three times, every cited page opened and checked.
The read Spotless. A primary source every cell; flagged where the rules differ by nation without being asked.
Sharpest receipt Cited the canonical gov.uk guide on every question, and quoted it word for word on the phone-fine trap.
| H1Change-of-mind refund | H2Stamp duty | H3Free childcare hours | H4State Pension age | H5Wales 20mph limit | H6Handheld-phone fine |
|---|---|---|---|---|---|
| clean | clean | clean | clean | clean | clean |
A miss held 3/3 is a stable pattern on that trap. A wobbled cell is an intermittent miss the model corrected itself on — not a settled failure. The tiers read behaviour on these hard cases, never a rate.
- The exact run. Five consumer assistants (ChatGPT, Claude, Gemini, Perplexity, Grok) on their default consumer tiers, captured 7 July 2026, N=3, on six questions (H1–H6). Every cell was hand-graded by opening the cited page against a source fixed before the run.
- Not a rate. These six questions were built to trap retrieval. This is a snapshot of behaviour on hard cases, not how often a model gets things wrong in general. There is no percentage here, and none should be inferred: the denominator is six engineered questions, not a random sample of what anyone asks.
- Reproduction, not frequency. "Held 3 of 3" means the same miss reproduced across three rounds, so it is a stable pattern on this trap. It does not mean the model fails everything.
- Sourcing, not accuracy. This measures sourcing, not accuracy. The figures were almost always right: only one of ninety cells stated a wrong fact. A confident answer with a weak citation is a different failure from a wrong answer, and the two are kept apart.
- What it covers. Coverage is these six questions only. Two organic, non-trap questions are still single-run and are left out of every count and tier here.
Grades applied case-by-case from the real captured responses (N=3, memory-off temporary/incognito chats, web search on, graded same-day against the primary source) by the site’s AI system, adversarially cross-checked by separate agents, and signed off by Ben Dixon, the named grader-of-record, for publication, s118 / 2026-06-26.
This is a documented index, not a statistical benchmark. The sample is small by design — every question is a real decision checked against a real source, not a thousand synthetic prompts. So there are no percentages of the internet here and no claim of significance: a verdict means ChatGPT did better or worse on this battery, graded against these sources, not that it is proven more or less reliable in general. Dated snapshot: N=3, memory off, core run 25 June 2026. It is a current score, not a permanent label — a fresh run can move any of it, which is the point.