Skip to content
Newsletter →
// AI reliability, tested

AI sounds exactly as sure when it’s wrong as when it’s right. I test which answers hold up.

I run the big AI assistants through real, checkable questions and show you which ones you can trust. Receipts, not vibes — one email a fortnight.

// The Lessons22 documented failures // Latest test:
  1. Other·unit-denomination

    Grok (free, 'Fast'): asked for BitMine Immersion's (BMNR) most recent full-year revenue, it returned '$6,095' (about $6K) on one…

  2. Other·constraint-disregard

    Grok (free, 'Fast'): given a covered-call question with the explicit instruction 'without access to a live options chain', it…

  3. ChatGPT·misattributed-source

    Asked for the UK ISA partial-transfer rule with a source, ChatGPT (free, web search on) cited…

All evidence
Ben Dixon
// The author

I started testing AI because I wanted to know which tools could be trusted with a real decision: my own investing first, where I needed the answer to be right. The verdicts here are what I found, including the times a tool simply wasn’t worth the trust. More on why I do this →

22 AI errors logged 15 AI catches 14 trades documented May 2026 tracking since
// New here? Start here
4 parts · ~14 min · no jargon

A short, plain-English path to trusting what AI tells you.

Four short reads. By the end you’ll have a simple check you can run on any answer that matters: a contract, a letter from a doctor, an email you’re not sure about.

Start with part 1 →
  1. 01 How to tell if an AI answer is true in 30 seconds
  2. 02 The whole method, in four questions
  3. 03 Make it show its working
  4. 04 Where I actually use this
// See all four parts →
// From the field · 5 new this week
Field Notes

I let three rival AIs audit my own AI method. They agreed on the weak step.

I asked three frontier models from three different labs to tear apart the method I use to keep AI honest. All three flagged the same step, and they were right.

New · · 4 min read
Tool Audit

Is Grok good for stock research? I ran the same test on the fifth tool

Is Grok good for stock research? I ran four dimensions of my AI comparison on its free tier: strong reasoning, one unit slip, a constraint it would not keep.

New · · 14 min read
Field Notes

I run an AI to catch AI mistakes. It fell for a fake.

The automated radar that watches this site for AI-reliability failures logged a satirical incident report as a real, documented one. Here's what caught it.

New · · 6 min read
// Archive
Field Notes Real AI hallucination examples, caught and dated New · 14 min Field Notes AI stock picker: I asked three models if I should buy NVDA, and watched the methodology break New · 12 min Field Notes Does ChatGPT make up sources? I checked two finance claims against the actual pages · 11 min Field Notes Does web search make AI more accurate? I ran the same questions both ways · 10 min Field Notes AI ISA advice: I tested four tools on the questions people get wrong · 12 min Field Notes I asked 4 AIs to scale a recipe. Two got the maths wrong. · 9 min
Browse all →
// By topic Prompt StackTool AuditDecision ProcessField Notes

// The Method Four prompts that stop AI inventing the answer.

Read →