How are the AIs holding up?
Deliberately hard questions, the kind known to trip AI up.
The full board, every receipt// AI reliability, tested
Six AIs, one real question. I show you which answers hold up.
I put ChatGPT, Claude, Gemini, Perplexity, Grok and Copilot through the same checkable questions and grade every answer against the source. Subscribe and the Bluff Filter is yours, free: a paste-in prompt that makes any AI flag its guesses before you act.
One email a fortnight. Leave whenever you like.
// The author
// The State of AI Reliability
Every AI was right on the facts you can look up. Every mistake in the graded finance run was on the data that moves.
2 confident errors, 4 partial, all on live data 24 of 24 fixed public-record facts clean 0 answers made up outright
Data that moves a price, a quote 2 confident errors, 4 partial
Facts you can look up a filing, a rate 24 of 24 all clean
6 assistants, 6 checkable questions, N=3, 25–26 June 2026 (Copilot: 18 July, its join run). A documented index, not a benchmark. See every receipt in the full report →
// Meet the six
The six assistants, by the numbers
Ranked cleanest-first, every answer graded against the source. Tap any card for its full record.
0Confident errors on the core
ASource tier: Reliable citer
Full record 0Confident errors on the core
ASource tier: Reliable citer
Full record 0Confident errors on the core
BSource tier: Mostly reliable, one held blind spot
Full record 0Confident errors on the core
CSource tier: Confident misattributor
Full record 1Confident error on the core
BSource tier: Mostly reliable, one held blind spot
Full record 1Confident error on the core
CSource tier: Confident misattributor
Full record Worth a look
// a curated few AI Tests Flagship
Claude vs ChatGPT: level on reliability, split on character
AI Tests ★ Just for fun
Which AI picks the World Cup winner? I asked five
Prompt Stack
AI quality of earnings review: 4 prompts to find the real profit
Tool Audit
Is Grok good for stock research? I ran the same test on the fifth tool
// New? Start here
4 parts · ~15 min · no jargon
Four short reads, and you’ll have a check you can run on any AI answer.
The kind you’d want on a contract, a letter from a doctor, or an email you’re not sure about.
Start with part 1 →// Latest · 4 new this week
AI Tests
Gemini vs ChatGPT: which one can you actually trust?
Which is better, Gemini or ChatGPT? I tested both. They're level on getting facts right, but one kept pointing me to sources that didn't back its answer.
GuardrailsHow to check if a ChatGPT citation is real or fake
The dangerous AI citation isn't the dead link. It's the working link to a real page that doesn't back the claim. Here's the ten-second check that catches it.
AI TestsClaude vs ChatGPT: level on reliability, split on character
Is Claude better than ChatGPT? I tested both against the source. They're level on reliability, both 9 of 9 on accuracy. The gap is character, not trust.
// Archive
AI Tests How accurate is Google Gemini? Right on the number, wrong on the source AI Tests I opened a private AI chat. It still knew my name and my rough location. AI Tests AI cites the wrong source: I put 6 UK questions to 5 assistants and opened every link AI Tests Which AI picks the World Cup winner? I asked five AI Tests Does AI change its answer when you push back? I told five AIs they were wrong AI Tests How often is ChatGPT wrong? I kept a running tally across 20 real AI tests
// The Method Four prompts that stop AI inventing the answer.
Read →// Newsletter
You should subscribe to my newsletter, Confidently Wrong
Free when you subscribe
The Bluff Filter
A paste-in prompt that makes any AI flag what it’s guessing before you act on it. Learn it once, use it on every answer.
Your email stays private, leave whenever you like.
Or follow via RSS: /rss.xml (main) · /lessons (AI failures) · /catches (AI catches).
