Skip to content
Newsletter →
Decision Process

Make it show its working

An AI hallucination is a model mixing what it knows with what it's making up, in the same smooth tone. One prompt makes it sort the two into separate lists — so you can see the invented parts.

Ben Dixon Ben Dixon 3 min read
// TL;DR
Problem
AI blends what it knows with what it's guessing in one even tone, so you can't tell them apart.
Fix
make it sort its answer into two lists — what the document says, and what it's adding itself.
Payoff
every guess lands in plain view, where you can decide what to trust.

An AI doesn’t tell you which parts of its answer it’s sure about. It hands you the facts and the guesses in the same even voice, packed into the same tidy paragraph, and leaves you to work out which is which. Usually you can’t. The guesses wear the same face as the facts. When one of those guesses is wrong, the technical word for it is a hallucination — though “confident guess” describes it better.

So I stop it before it gives me an opinion and make it do one thing first: sort what it just read into two lists. List A is only what the document says. List B is everything it’s adding on top — the assumptions, the filled-in gaps, the bits it’s inferring. Paste this in with whatever you’re checking:

// Paste this before you ask the AI what it thinks

Before you give me any opinion or summary, sort your answer into two lists. List A: only what this document says, in its own words. List B: anything you are adding, assuming, inferring or guessing that the document does not state outright. If you are unsure which list something belongs in, put it in List B.

Why two lists beats one summary

This is the one move I’d keep if I had to drop everything else. A summary mixes everything together and smooths the joins. Two lists pull the joins apart. The second you can see List B on its own, the guesses stop hiding — they’re sitting in a column with a label on it.

Take a letter from your doctor. You paste it in, ask what it means, and the AI says, calmly, “this is nothing to worry about.” Reassuring. Also not in the letter. Run the two-list prompt and that line lands in List B, where it belongs — the comfort it invented, sitting in the guesses column with a label on it, next to everything else it added to be kind. The letter never said don’t worry. The AI did, because it sounded like the helpful thing to say.

That’s the whole value of the move. List A you can check against the document, line by line. List B is where you slow down — some of it will be fair inference, some will be the AI being agreeable, and now you can tell the two apart instead of swallowing both.

It works on anything you’d paste in: a contract, a job offer, an email from your child’s school, a holiday booking. List B is the part that’s a question wearing the clothes of an answer.

Next: where I use all of this → Where I actually use this

Ben Dixon
// Written by Ben Dixon

Ben tests ways of getting reliable answers from AI on his own investing — documenting what each model got wrong, what each one caught, and the prompts that survived the cuts. About Ben →

// Keep reading
Decision Process

How to tell if an AI answer is true in 30 seconds

Two questions — and a prompt you can copy — that catch a confident, wrong AI answer before you act on it. No jargon, no AI knowledge needed.

 Decision Process

The whole method, in four questions

The four questions I run any AI answer through before I trust it — shown end to end on one everyday example, with a prompt you can copy. No jargon.

 Decision Process

Where I actually use this

The two-question check works on any decision that matters. Here's where I push it hardest — and where I wrote down what AI got wrong as well as right.
// New here?

The site runs AI on real investing decisions. Start with the Prompt Stack for the four-stage framework, or the Field Guide PDF for the condensed version — free, no email.

← All posts More in Decision Process →