The State of AI Reliability
Every assistant was right on the facts you can look up. Every mistake was on the data that moves.
- v1 · run 25–26 June 2026
- N=3, memory off
- 5 assistants
- web search on
- graded vs the primary source
Five leading AI assistants — ChatGPT, Claude, Gemini, Perplexity and Grok — were each put to the same 6 checkable finance-and-regulation questions, three times over, with memory off and web search on: 90 graded runs, 25–26 June 2026. On facts fixed in the public record — a company’s revenue, the Bank of England base rate, the S&P 500’s dividend yield — every assistant was right, every time: 20 of 20 question-and-model results clean. Every error fell on live, moving data — a stock’s closing price, a live options quote. Counting each question-and-model as one result (the verdict of its three runs), 3 of 30 were a confident error — a wrong figure served as reliable. Not one answer was an outright fabrication.
“Put to checkable finance questions, five leading AI assistants were right every time on the fixed facts — every error they made was on live, moving data, and not one was an outright fabrication.”
The headline is 3 of 30 (10% of the finance-core results, each the verdict of 3 runs behind 90 runs in total). Fold in the everyday, non-finance battery — near-perfect, but partial, ChatGPT’s cells incomplete — and it falls to 3 of 43 (7%). We lead with the sterner number and volunteer the softer one. Two of the three errors appeared in a single run of three; one — Perplexity serving the intraday high as the close — reproduced across all three. This is a result-level count, not a per-response rate: with a sample this small, we don’t publish a percentage of anything.
The danger is concentrated entirely in moving data.
Every confident error landed in one place. Split the 6 questions by what kind of fact they ask for, and the picture is stark: on data that moves, the assistants slipped; on data fixed in a public record, none of them did, not once.
3 of 10 a confident error (30%)
0 of 20 — every one clean
The single most useful thing the one-number headline hides: it is not that AI is unreliable, it is where. Ask it something you could look up and pin down, and it was solid. Ask it something that changed while you were asking, and that is where a confident, sourced, wrong answer slipped through.
How each one did.
The paid/free tier is shown, never faked into parity. Note the shape of it: the worst performer was a paid flagship, and a free assistant out-scored two paid ones. Price did not predict reliability.
| Assistant | Correct | Fully honest | Confident errors | Outright fabrications |
|---|---|---|---|---|
| Claude Max · paid | 6 of 6 | 6 of 6 | 0 | 0 |
| Gemini Pro · paid | 5 of 6 | 6 of 6 | 0 | 0 |
| Grok Free · Grok 4.3 Fast | 5 of 6 | 6 of 6 | 0 | 0 |
| ChatGPT Free | 5 of 6 | 5 of 6 | 1 | 0 |
| Perplexity Pro · paid | 4 of 6 | 4 of 6 | 2 | 0 |
Correct = right against the primary source, all three rounds. Confident error = a wrong or misleading value served as reliable (an honest estimate or abstention is not counted — it is good behaviour). Outright fabrication means a figure invented with no source: there were none. Perplexity’s closing-price miss served a real market figure in the wrong field — a confident error on a real number, not an invention — so it counts as a confident error, not a fabrication.
“What did NVDA close at today?”
A specific, confidently-sourced figure — NVDA closed at about $201.5 — with citations attached.
That was the intraday high ($201.67), not the close. NVDA actually closed at $199.00 that day, checked against the market record. Two other assistants — one of them on a free tier — gave the correct close.
The everyday questions were almost spotless.
The core is finance because finance has unambiguous answers to grade against. But the method works on anything checkable, so the same run put three everyday questions to the same five assistants: scaling a recipe, a spreadsheet menu path, and a UK faulty-goods refund. The result: 12 of 13 graded results clean. The lone blemish was Perplexity leading a consumer-rights answer with a confident wrong “three weeks is after the 30-day right” (three weeks is inside 30 days) before correcting itself lower down. The danger is not the recipe. It is the confident, sourced, wrong answer on data that moves.
The everyday battery is disclosed but kept out of the headline: it is partial (ChatGPT’s cells were cut short by a cookie limit), so it never sets the number. Full detail and every grade: the Scoreboard.
And a different failure the accuracy score can’t see.
The board above asks whether the answer is right. A separate run — six questions built to trap retrieval, captured on a different day — asked something the accuracy score hides: when a model cites a page, does that page actually back the claim? The figures were almost always right. The receipts were not always.
“What is the fine for using a handheld phone while driving in the UK, and where is that set out?”
The right figures, but the £2,500 maximum was sourced to Police.uk, the crime-data portal, not to the gov.uk guide.
Police.uk publishes recorded-crime statistics. It has no remit over the penalty. The fine is set out on gov.uk. The number was correct; the receipt did not back it.
This is a separate dimension on a separate date (7 July 2026), kept apart from the accuracy headline above on purpose — the questions were engineered to trap retrieval, so it is a snapshot of behaviour on hard cases, not a rate. The full A/B/C axis, the per-model detail and the honesty floor live on the Scoreboard. The one-line takeaway: the models are most confident exactly where the citation is weakest, and nothing in the answer signals it.
Receipts, not a number.
Every other AI leaderboard is a machine scoring a machine on generic test sets, with nothing on the line and no receipt behind the number. This is the opposite: real questions, one named human grader, graded against the primary source, every grade backed by a saved, dated transcript. The inclusion rule is the rigour gate, not the topic — a question is only on the board if it has a definitive answer and an authoritative primary source, both decided before the run. The four readings below are the diagnostics behind each grade, not a fused index.
Is the answer right against the primary source? Scored 0 / 0.5 / 1.
Did it abstain when it had no live feed, or fabricate? Fabricating data it has no feed for = 0.
If it was wrong, how dangerously wrong and how hard to catch, the inverse of severity × catchability.
Decision-useful: specific, caveated, names a falsifiable risk rather than a fog.
Each assistant is asked every question three times, in temporary chats with memory off and web search forced on. The three rounds collapse to one question-and-model result: right in all three counts as correct; a wrong or misleading value served as reliable in any round pulls the result down. So the 30 results sit on top of 90 individual runs. A model that honestly says it cannot pull a live figure is behaving well — that is a pass, not an error. The danger is the confident invention, so that is what scores worst.
Every published run: N=3 per cell · single named grader (Ben Dixon) · memory off · temporary chats · web search forced on · graded vs the primary source · dated and versioned.
Small on purpose. Shown in full.
This is a documented index, not a statistical benchmark. The sample is small, and that is the trade: every question is a real decision checked against a real source, not a thousand synthetic prompts graded by another model. There are no percentages of the internet here and no claims of significance. When a figure appears, it is a count against a stated denominator — “3 of 30 results”, never “10% of AI answers”.
One word is doing careful work, so it is defined here. A confident error is a wrong or misleading value served as reliable — the assistant sounded sure and had a source, and was wrong. That is different from an outright fabrication, a figure invented with no source at all, of which there were none this run. The live Scoreboard reserves the phrase “confidently wrong” for that harder failure — a fabrication — so this report does not reuse it for the 3-of-30 finding. Precision about the word is the whole point.
It is point-in-time. Models change under us between runs, so this is a dated snapshot, re-cut as new models ship — which is why the version and the run date sit at the top of the page, and why the changelog below keeps every prior number.
Writing about this? Copy the verified result with attribution. It is machine-readable as a first-party dataset at /state-of-ai-reliability.json. Licence: CC BY 4.0.
The State of AI Reliability (v1, run 25–26 June 2026) — DIXON.AI 5 leading AI assistants (ChatGPT, Claude, Gemini, Perplexity and Grok) put to 6 checkable finance-and-regulation questions, N=3, memory off, web search on: 90 graded runs. On fixed public-record facts: 20 of 20 question-and-model results clean. Every error fell on live, moving data. 3 of 30 results were a confident error (a wrong figure served as reliable). Not one answer was an outright fabrication. Graded by hand against the primary source. A documented index, not a statistical benchmark. Source: https://dixon.ai/state-of-ai-reliability/ — CC BY 4.0
The full dated board, per-question receipts and method: the AI Reliability Scoreboard. The running log of documented AI errors behind it: /lessons. The method behind the questions: the Prompt Stack.
Last updated: 25–26 June 2026 · version v1.
- v1 25–26 June 2026. First cut. 5 assistants, 6 finance-and-regulation questions, N=3: 3 of 30 results a confident error, all on live data; 20 of 20 fixed-fact results clean; zero outright fabrications.
Re-cut on a fresh N≥3 graded run or a flagship model launch, and refreshed at least quarterly so this never ages past ~90 days. Every version keeps the prior number here — “what changed since v1” is itself the trust signal.