Asked for the UK ISA partial-transfer rule with a source, ChatGPT (free, web search on) cited…
Asked how long cooked chicken keeps in the fridge by a stated UK (Newcastle) user, Perplexity (web search on) led with US food…
Asked whether this year's ISA contributions can be partially transferred, Perplexity said they must be transferred in full — the…
// Honest AI tool & model comparisons
I test whether AI tools actually do the work — on my own investing first, where being wrong costs real money.
The work is ongoing. One email a week: what worked, what didn’t, and the prompts I’d run again.
No spam. Unsubscribe any time.
// The author
// How this site is run
I don’t just write about AI from the outside. The agent that drafts, fact-checks and publishes everything here is one I direct and keep reliable enough to trust with the work — across roughly a hundred sessions. And whenever I catch an AI inventing something in the testing, it goes in the public log, in full.
30 posts shipped · every AI failure I’ve caught testing, logged in the open → // New here? Start here. 4 parts · ~14 min · no jargon
A short, plain-English path to trusting what AI tells you.
Four short reads. By the end you’ll have a simple check you can run on any answer that matters — a contract, a letter from a doctor, an email you’re not sure about.
Start with part 1 →// From the field · 10 new this week
Field Notes
Does ChatGPT make up sources? I checked two finance claims against the actual pages
Does ChatGPT make up sources? Mostly no — but I opened every link on two finance questions and found a real gov.uk page that didn't back the claim.
Field NotesDoes web search make AI more accurate? I ran the same questions both ways
Does web search make AI more accurate? I ran the same questions both ways. It didn't make the answers more reliable — it moved where the errors hide.
Field NotesAI ISA advice: I tested four tools on the questions people get wrong
I asked four AI tools for ISA advice on the questions people get wrong. All four aced the basics — then two gave a rule abolished in April 2024.
// Archive
Field Notes I asked 4 AIs to scale a recipe. Two got the maths wrong. Field Notes 9 types of AI hallucinations, named from real tests Tool Audit AI stock research tools tested: what each one got wrong Tool Audit ChatGPT vs Claude for earnings call analysis: which one reads what management didn't say Tool Audit Is Perplexity good for investment research? An honest, scored audit Prompt Stack AI quality of earnings review: 4 prompts to find the real profit
// The Method Four prompts that stop AI inventing the answer.
Read →