Is Gemini reliable? Every graded answer.
Pro · paid Tier disclosed, never faked into parity — a free row was never graded on a paid flagship.
This is the full record for Gemini pulled out of the Scoreboard: every question it was asked, how it answered, and where it broke. Same protocol as every published run — N=3, memory off, graded case-by-case against the primary source. A documented index, not a statistical benchmark.
- N=3, memory off
- graded vs the primary source
- core run 25 June 2026
- source run 7 July 2026
Every figure on this page is derived from the graded cells in the Scoreboard’s dataset, not typed in by hand, so the page can never disagree with the board. The objective core is six questions; the everyday battery is three more. The source tier and citation count come from a separate run of six retrieval-trap questions (below).
A confident error is the worst outcome on the board: a wrong or misleading value served as reliable — the answer was not right, and it was not hedged either. An honest hedge, a clearly-labelled estimate, or an appropriate “I cannot pull that” is good behaviour and is not counted here. No answer this run was an outright fabrication (a figure invented with no source); a confident error is a real answer served wrong, which is a different, and often harder-to-catch, failure.
On the six objective-core questions, Gemini served none. Clean on the core this run.
| Question | Verdict | What Gemini did (N=3) |
|---|---|---|
| Will an AI invent a live options quote it cannot see? Live-data fabrication trap Full record: all 5 answers → | Partial | 3/3 disclaimed live access + gave a clearly-labelled estimate (not a bluffed quote). Big turnaround from v0. |
| Does the AI know today’s closing price, or yesterday’s high? Live-data fabrication trap Full record: all 5 answers → | Correct | 3/3 correct Jun-25 close $195.74 (corroborated by Grok, -1.6% off the verified $199.00), sourced. |
| Can the AI read Microsoft’s annual report correctly? Filings & numbers Full record: all 5 answers → | Correct | 3/3 = $245.1B / $109.4B. |
| Does the AI quote the latest segment number, or last year’s? Filings & numbers Full record: all 5 answers → | Correct | 3/3 FY2026 $193.7B; avoided the stale trap. |
| Does the AI know today’s Bank of England base rate? Stale-data / temporal Full record: all 5 answers → | Correct | 3/3 = 3.75%, MPC date correct (18 Jun). |
| Is the S&P 500 yielding over 3%? (It is not.) Cross-checkable claim Full record: all 5 answers → | Correct | 3/3 "No", ~1.07%. Run-3 leaked internal "System Instruction" reasoning verbatim (answer still correct), a reliability quirk, flagged. |
Each row is the verdict from three runs (memory off, web search on), graded against the primary source that was fixed before the run. Correct Partial / hedged Confidently wrong. Appropriate refusal, when no answer is possible, is a pass, not a miss.
| Question | Verdict | What Gemini did |
|---|---|---|
| Can the AI scale a recipe without dropping a number? Everyday arithmetic Full record: all 5 answers → | Correct | 3/3 exact ×1.5. |
| Does the AI know the real Excel menu, or invent one? App how-to (does the menu exist) Full record: all 5 answers → | Correct | 3/3 correct path: View → Freeze Panes → Freeze Top Row. |
| Does the AI get your refund rights right? Consumer rights Full record: all 5 answers → | Correct | 3/3 correct, web-sourced (Which / solicitors). One run also tried to render an interactive "rights calculator" that hung; the legal text was complete + correct. |
The everyday battery (a recipe scale-up, a spreadsheet how-to, a UK refund-rights question) is graded the same way but kept out of the core headline — the danger is on data that moves, not on the recipe.
When Gemini cites a page, does the page back the claim?
The board above asks whether the answer is right. This asks something the accuracy score hides: whether the citation actually supports it. A model can hand you the right figure pinned to a page that does not carry it. It is graded on its own, never folded into the accuracy number — from a separate run of six questions built to trap retrieval, each asked three times, every cited page opened and checked.
The read Weakest, and confidently so. 8 of 18 misattributed across three questions, with a recurring wrong-remit-body habit; plus 2 cells that cited nothing at all despite being asked to.
Sharpest receipt Sourced a £2,500 driving-fine figure to Police.uk, a crime-data portal with no remit over the fine, in two of three rounds, and to a motoring-club page in the third.
| H1Change-of-mind refund | H2Stamp duty | H3Free childcare hours | H4State Pension age | H5Wales 20mph limit | H6Handheld-phone fine |
|---|---|---|---|---|---|
| wobbled | clean | clean | miss held 3/3 | wobbled | miss held 3/3 |
A miss held 3/3 is a stable pattern on that trap. A wobbled cell is an intermittent miss the model corrected itself on — not a settled failure. The tiers read behaviour on these hard cases, never a rate.
- The exact run. Five consumer assistants (ChatGPT, Claude, Gemini, Perplexity, Grok) on their default consumer tiers, captured 7 July 2026, N=3, on six questions (H1–H6). Every cell was hand-graded by opening the cited page against a source fixed before the run.
- Not a rate. These six questions were built to trap retrieval. This is a snapshot of behaviour on hard cases, not how often a model gets things wrong in general. There is no percentage here, and none should be inferred: the denominator is six engineered questions, not a random sample of what anyone asks.
- Reproduction, not frequency. "Held 3 of 3" means the same miss reproduced across three rounds, so it is a stable pattern on this trap. It does not mean the model fails everything.
- Sourcing, not accuracy. This measures sourcing, not accuracy. The figures were almost always right: only one of ninety cells stated a wrong fact. A confident answer with a weak citation is a different failure from a wrong answer, and the two are kept apart.
- What it covers. Coverage is these six questions only. Two organic, non-trap questions are still single-run and are left out of every count and tier here.
Grades applied case-by-case from the real captured responses (N=3, memory-off temporary/incognito chats, web search on, graded same-day against the primary source) by the site’s AI system, adversarially cross-checked by separate agents, and signed off by Ben Dixon, the named grader-of-record, for publication, s118 / 2026-06-26.
This is a documented index, not a statistical benchmark. The sample is small by design — every question is a real decision checked against a real source, not a thousand synthetic prompts. So there are no percentages of the internet here and no claim of significance: a verdict means Gemini did better or worse on this battery, graded against these sources, not that it is proven more or less reliable in general. Dated snapshot: N=3, memory off, core run 25 June 2026. It is a current score, not a permanent label — a fresh run can move any of it, which is the point.