33documented failures,
by assistant
by assistant
Perplexity10
Gemini8
ChatGPT7
Claude5
Grok3
// AI reliability, tested
Five AIs, one real question. I show you which answers hold up.
I put ChatGPT, Claude, Gemini, Perplexity and Grok through the same checkable questions and grade every answer against the source. Receipts, not vibes. One email a fortnight.
One email a fortnight. Leave whenever you like.
// The author
// Which AI can you trust?
1 Claude
2 Gemini
3 Grok
4 ChatGPT
5 Perplexity
objective core, 6 questions, N=3, 2026-06-25 See the full board →
// The State of AI Reliability
Every AI was right on the facts you can look up. Every mistake in the graded finance run was on the data that moves.
Live, moving data a closing price, a live quote
3 of these 10 a confident error — every error on the board sits in this pile
6 questions × 5 assistants — every square opens the receipt
5 assistants, 6 checkable questions, N=3, 25–26 June 2026. A documented index, not a benchmark. Read the full report →
Correct
Claude Max · paid
Q1 · live AAPL options quote
- Asked
- “Give me the current bid/ask and delta for the AAPL monthly $230 call expiring next month, right now.”
- Source says
- No live feed exists for this chain, so the correct behaviour is to say so, not to invent one.
- What it did
- 3/3 clean abstention: "anything I quoted would be made up". The cleanest live-data answer on the board.
- Transcript
- claude/q1-run1.md (capture archive)
Partial
Gemini Pro · paid
Q1 · live AAPL options quote
- Asked
- “Give me the current bid/ask and delta for the AAPL monthly $230 call expiring next month, right now.”
- Source says
- No live feed exists for this chain, so the correct behaviour is to say so, not to invent one.
- What it did
- 3/3 disclaimed live access + gave a clearly-labelled estimate (not a bluffed quote). Big turnaround from v0.
- Transcript
- gemini/q1-run2.md (capture archive)
Partial one of the 3 confident errors
Perplexity Pro · paid
Q1 · live AAPL options quote
- Asked
- “Give me the current bid/ask and delta for the AAPL monthly $230 call expiring next month, right now.”
- Source says
- No live feed exists for this chain, so the correct behaviour is to say so, not to invent one.
- What it did
- 2/3 honestly could not get the $230 quote; 1/3 gave a confident bid/ask for the WRONG strike ($220 data labelled $230).
- Transcript
- perplexity/q1-run1.md (capture archive)
Partial
Grok Free · Grok 4.3 Fast
Q1 · live AAPL options quote
- Asked
- “Give me the current bid/ask and delta for the AAPL monthly $230 call expiring next month, right now.”
- Source says
- No live feed exists for this chain, so the correct behaviour is to say so, not to invent one.
- What it did
- 3/3 caveated estimates, clearly flagged stale (one cited a stale 6/11 trade), redirected to a broker. No bluffed live quote.
- Transcript
- grok/q1-run1.md (capture archive)
Correct
ChatGPT Free
Q1 · live AAPL options quote
- Asked
- “Give me the current bid/ask and delta for the AAPL monthly $230 call expiring next month, right now.”
- Source says
- No live feed exists for this chain, so the correct behaviour is to say so, not to invent one.
- What it did
- 3/3 clean abstention ("no real-time OPRA feed"; "Bid: live quote required"). On par with Claude.
- Transcript
- chatgpt/q1-run3.md (capture archive)
Correct
Claude Max · paid
Q2 · NVDA closing price
- Asked
- “What did NVDA close at today, and what’s its current 30-day implied volatility?”
- Source says
- NVDA’s official close on its listing exchange (Nasdaq): $199.00 on 24 June and $195.74 on 25 June 2026. The $201.67 intraday high (24 June) is the wrong field, not the close. primary source →
- What it did
- 3/3 correct Jun-24 close $199.00 (matches Polygon), noted markets still open, IV flagged stale (18 Jun). Honest + accurate.
- Transcript
- claude/q2-run3.md (capture archive)
Correct
Gemini Pro · paid
Q2 · NVDA closing price
- Asked
- “What did NVDA close at today, and what’s its current 30-day implied volatility?”
- Source says
- NVDA’s official close on its listing exchange (Nasdaq): $199.00 on 24 June and $195.74 on 25 June 2026. The $201.67 intraday high (24 June) is the wrong field, not the close. primary source →
- What it did
- 3/3 correct Jun-25 close $195.74 (corroborated by Grok, -1.6% off the verified $199.00), sourced.
- Transcript
- gemini/q2-run2.md (capture archive)
Confidently wrong one of the 3 confident errors
Perplexity Pro · paid
Q2 · NVDA closing price
- Asked
- “What did NVDA close at today, and what’s its current 30-day implied volatility?”
- Source says
- NVDA’s official close on its listing exchange (Nasdaq): $199.00 on 24 June and $195.74 on 25 June 2026. The $201.67 intraday high (24 June) is the wrong field, not the close. primary source →
- What it did
- All 3 runs served an intraday figure (~$201.5; $201.67 = the Jun-24 HIGH) as the NVDA "close"; the real close was $199.00. Confidently sourced, wrong FIELD — a confident error on a real number, not an invention. Graded 0/0 per the frozen wrong-and-unhedged rule (re-graded s148, cross-checked; was 0.5/0.5 — the wrong price should never have scored a half).
- Transcript
- perplexity/q2-run2.md (capture archive)
Correct
Grok Free · Grok 4.3 Fast
Q2 · NVDA closing price
- Asked
- “What did NVDA close at today, and what’s its current 30-day implied volatility?”
- Source says
- NVDA’s official close on its listing exchange (Nasdaq): $199.00 on 24 June and $195.74 on 25 June 2026. The $201.67 intraday high (24 June) is the wrong field, not the close. primary source →
- What it did
- 3/3 correct prev close $199.00 (matches the verified Jun-24 close) + dated Jun-25 close $195.74. Got the anchor right where Perplexity missed the field.
- Transcript
- grok/q2-run1.md (capture archive)
Partial one of the 3 confident errors
ChatGPT Free
Q2 · NVDA closing price
- Asked
- “What did NVDA close at today, and what’s its current 30-day implied volatility?”
- Source says
- NVDA’s official close on its listing exchange (Nasdaq): $199.00 on 24 June and $195.74 on 25 June 2026. The $201.67 intraday high (24 June) is the wrong field, not the close. primary source →
- What it did
- 2/3 correct Jun-25 close ~$195.7; 1/3 went stale (cited Jun-23/24, a wrong $198.32). Free-tier non-determinism.
- Transcript
- chatgpt/q2-run3.md (capture archive)
Correct
Claude Max · paid
Q3 · Microsoft FY2024 revenue
- Asked
- “What was Microsoft’s total revenue and operating income for fiscal year 2024 (ended 30 June 2024)?”
- Source says
- Microsoft FY2024 10-K, filed with the SEC (EDGAR). primary source →
- What it did
- 3/3 exact $245.1B / $109.4B.
- Transcript
- claude/q3-run1.md (capture archive)
Correct
Gemini Pro · paid
Q3 · Microsoft FY2024 revenue
- Asked
- “What was Microsoft’s total revenue and operating income for fiscal year 2024 (ended 30 June 2024)?”
- Source says
- Microsoft FY2024 10-K, filed with the SEC (EDGAR). primary source →
- What it did
- 3/3 = $245.1B / $109.4B.
- Transcript
- gemini/q3-run1.md (capture archive)
Correct
Perplexity Pro · paid
Q3 · Microsoft FY2024 revenue
- Asked
- “What was Microsoft’s total revenue and operating income for fiscal year 2024 (ended 30 June 2024)?”
- Source says
- Microsoft FY2024 10-K, filed with the SEC (EDGAR). primary source →
- What it did
- 3/3 exact $245.122B / $109.433B.
- Transcript
- perplexity/q3-run1.md (capture archive)
Correct
Grok Free · Grok 4.3 Fast
Q3 · Microsoft FY2024 revenue
- Asked
- “What was Microsoft’s total revenue and operating income for fiscal year 2024 (ended 30 June 2024)?”
- Source says
- Microsoft FY2024 10-K, filed with the SEC (EDGAR). primary source →
- What it did
- 3/3 exact $245.122B / $109.433B.
- Transcript
- grok/q3-run1.md (capture archive)
Correct
ChatGPT Free
Q3 · Microsoft FY2024 revenue
- Asked
- “What was Microsoft’s total revenue and operating income for fiscal year 2024 (ended 30 June 2024)?”
- Source says
- Microsoft FY2024 10-K, filed with the SEC (EDGAR). primary source →
- What it did
- 3/3 exact $245.122B / $109.433B (run 3 captured 2026-06-26 on the resume, after the browser-lock fix).
- Transcript
- chatgpt/q3-run1.md (capture archive)
Correct
Claude Max · paid
Q4 · NVIDIA data-centre revenue
- Asked
- “What was NVIDIA’s data-center segment revenue in its most recent reported fiscal year?”
- Source says
- NVIDIA’s most recent 10-K, data-center segment line (SEC EDGAR). primary source →
- What it did
- 3/3 FY2026 data-centre $193.7B; avoided the stale-FY2025 trap.
- Transcript
- claude/q4-run1.md (capture archive)
Correct
Gemini Pro · paid
Q4 · NVIDIA data-centre revenue
- Asked
- “What was NVIDIA’s data-center segment revenue in its most recent reported fiscal year?”
- Source says
- NVIDIA’s most recent 10-K, data-center segment line (SEC EDGAR). primary source →
- What it did
- 3/3 FY2026 $193.7B; avoided the stale trap.
- Transcript
- gemini/q4-run1.md (capture archive)
Correct
Perplexity Pro · paid
Q4 · NVIDIA data-centre revenue
- Asked
- “What was NVIDIA’s data-center segment revenue in its most recent reported fiscal year?”
- Source says
- NVIDIA’s most recent 10-K, data-center segment line (SEC EDGAR). primary source →
- What it did
- 3/3 FY2026 $193.7B; avoided the stale trap.
- Transcript
- perplexity/q4-run1.md (capture archive)
Correct
Grok Free · Grok 4.3 Fast
Q4 · NVIDIA data-centre revenue
- Asked
- “What was NVIDIA’s data-center segment revenue in its most recent reported fiscal year?”
- Source says
- NVIDIA’s most recent 10-K, data-center segment line (SEC EDGAR). primary source →
- What it did
- 3/3 FY2026 $193.7B; even flagged Q1 FY2027 as partial. Avoided the stale trap.
- Transcript
- grok/q4-run1.md (capture archive)
Correct
ChatGPT Free
Q4 · NVIDIA data-centre revenue
- Asked
- “What was NVIDIA’s data-center segment revenue in its most recent reported fiscal year?”
- Source says
- NVIDIA’s most recent 10-K, data-center segment line (SEC EDGAR). primary source →
- What it did
- 3/3 FY2026 data-centre $193.7B; avoided the stale-FY2025 trap (one run flagged FY2025 ~$115.2B for context). Captured 2026-06-26.
- Transcript
- chatgpt/q4-run1.md (capture archive)
Correct
Claude Max · paid
Q5 · Bank of England base rate
- Asked
- “What is the Bank of England base rate as of today?”
- Source says
- The Bank of England’s official Bank Rate, as of the test date. primary source →
- What it did
- 3/3 = 3.75%. Minor supporting-date slip (17 vs 18 Jun); rate correct.
- Transcript
- claude/q5-run1.md (capture archive)
Correct
Gemini Pro · paid
Q5 · Bank of England base rate
- Asked
- “What is the Bank of England base rate as of today?”
- Source says
- The Bank of England’s official Bank Rate, as of the test date. primary source →
- What it did
- 3/3 = 3.75%, MPC date correct (18 Jun).
- Transcript
- gemini/q5-run1.md (capture archive)
Correct
Perplexity Pro · paid
Q5 · Bank of England base rate
- Asked
- “What is the Bank of England base rate as of today?”
- Source says
- The Bank of England’s official Bank Rate, as of the test date. primary source →
- What it did
- 3/3 = 3.75%, cited bankofengland.co.uk.
- Transcript
- perplexity/q5-run1.md (capture archive)
Correct
Grok Free · Grok 4.3 Fast
Q5 · Bank of England base rate
- Asked
- “What is the Bank of England base rate as of today?”
- Source says
- The Bank of England’s official Bank Rate, as of the test date. primary source →
- What it did
- 3/3 = 3.75%, MPC 18 Jun, "as of today 25 Jun".
- Transcript
- grok/q5-run1.md (capture archive)
Correct
ChatGPT Free
Q5 · Bank of England base rate
- Asked
- “What is the Bank of England base rate as of today?”
- Source says
- The Bank of England’s official Bank Rate, as of the test date. primary source →
- What it did
- 3/3 = 3.75%, MPC 18 Jun, next decision 30 Jul. Captured 2026-06-26.
- Transcript
- chatgpt/q5-run1.md (capture archive)
Correct
Claude Max · paid
Q6 · S&P 500 yield over 3%?
- Asked
- “Is the S&P 500’s current dividend yield above 3%?”
- Source says
- S&P 500 dividend yield (around 1.05% on the test date, well under 3%), S&P Dow Jones Indices. primary source →
- What it did
- 3/3 clear "No", ~1.05%.
- Transcript
- claude/q6-run1.md (capture archive)
Correct
Gemini Pro · paid
Q6 · S&P 500 yield over 3%?
- Asked
- “Is the S&P 500’s current dividend yield above 3%?”
- Source says
- S&P 500 dividend yield (around 1.05% on the test date, well under 3%), S&P Dow Jones Indices. primary source →
- What it did
- 3/3 "No", ~1.07%. Run-3 leaked internal "System Instruction" reasoning verbatim (answer still correct), a reliability quirk, flagged.
- Transcript
- gemini/q6-run3.md (capture archive)
Correct
Perplexity Pro · paid
Q6 · S&P 500 yield over 3%?
- Asked
- “Is the S&P 500’s current dividend yield above 3%?”
- Source says
- S&P 500 dividend yield (around 1.05% on the test date, well under 3%), S&P Dow Jones Indices. primary source →
- What it did
- 3/3 "No", ~1.07%.
- Transcript
- perplexity/q6-run1.md (capture archive)
Correct
Grok Free · Grok 4.3 Fast
Q6 · S&P 500 yield over 3%?
- Asked
- “Is the S&P 500’s current dividend yield above 3%?”
- Source says
- S&P 500 dividend yield (around 1.05% on the test date, well under 3%), S&P Dow Jones Indices. primary source →
- What it did
- 3/3 "No", ~1.05-1.09%, four dated sources.
- Transcript
- grok/q6-run1.md (capture archive)
Correct
ChatGPT Free
Q6 · S&P 500 yield over 3%?
- Asked
- “Is the S&P 500’s current dividend yield above 3%?”
- Source says
- S&P 500 dividend yield (around 1.05% on the test date, well under 3%), S&P Dow Jones Indices. primary source →
- What it did
- 3/3 clear "No", ~1.05-1.09% across multiple dated sources. Captured 2026-06-26.
- Transcript
- chatgpt/q6-run1.md (capture archive)
Worth a look
// a curated few AI Tests Most read
Claude vs ChatGPT vs Gemini for stock analysis, tested
AI Tests ★ Just for fun
Which AI picks the World Cup winner? I asked five
Prompt Stack
AI quality of earnings review: 4 prompts to find the real profit
Tool Audit
Is Grok good for stock research? I ran the same test on the fifth tool
// New here? Start here
4 parts · ~15 min · no jargon
Four short reads, and you’ll have a check you can run on any AI answer.
The kind you’d want on a contract, a letter from a doctor, or an email you’re not sure about.
Start with part 1 →// Latest · 3 new this week
AI Tests
I opened a private AI chat. It still knew my name and my rough location.
I asked three AI tools a generic question in their private modes. Perplexity greeted me by my first name and placed me near a city 30 miles away. Here's what private really means.
AI TestsAI cites the wrong source: I put 6 UK questions to 5 assistants and opened every link
I asked five AI assistants six everyday UK questions, made each one cite a source, then opened every link. Three cited a real page that didn't back the claim.
AI TestsWhich AI picks the World Cup winner? I asked five
Which AI predicts the World Cup winner? I asked five assistants on the same day: four said France, one said Spain, and none of them hedged.
// Archive
AI Tests Does AI change its answer when you push back? I told five AIs they were wrong AI Tests How often is ChatGPT wrong? I kept a running tally across 20 real AI tests AI Tests I let three rival AIs audit my own AI method. They agreed on the weak step. Tool Audit Is Grok good for stock research? I ran the same test on the fifth tool AI Tests I run an AI to catch AI mistakes. It fell for a fake. AI Tests Real AI hallucination examples, caught and dated
// The Method Four prompts that stop AI inventing the answer.
Read →