Telling AI to be sceptical: three rivals audited my method

For the better part of a year I’ve run the same four-step method before I trust anything an AI tells me on a decision that matters. Step one was called ROLE: tell the model to take a sceptical stance, to hunt for reasons the answer is wrong. It always felt like the softest step in the set. I kept it anyway, because it was tidy and it sounded sensible.

This week I stopped trusting that hunch and did the one thing the method itself is built on: get a second opinion, then a third. I handed the whole method to three frontier models from three different labs, GLM, DeepSeek and Qwen, and asked each of them, separately, to pull it apart. I gave no hint about which step I doubted.

Three labs matters here. They don’t share a training run, so they don’t share the same blind spots. When three models that have never met land in the same place, that’s a signal. It’s the same reason the AI Reliability Scoreboard grades every assistant on the board.

All three went for the same step

They didn’t hedge, and they didn’t spread their fire. All three went straight for ROLE, and independently made the same point: telling an AI to “be sceptical” changes its tone, not its accuracy. It writes you a more cautious-sounding answer that’s no more correct. A made-up figure delivered in a careful, sceptical register is still a made-up figure.

GLM, the model from Z.ai, put it most plainly:

Telling an LLM to “be sceptical” mostly changes tone, not epistemics. It’ll sound more cautious while still hallucinating.

I had half-known that. The honest version is that I had known it for months and kept the step because pulling it meant admitting the method had a soft spot. Hearing it from three machines with no stake in my feelings was the push I had been avoiding.

So ROLE is gone. Meet SCOPE.

So I replaced it with SCOPE. It sets a boundary. I now tell the model exactly what it’s allowed to use, and have it say plainly when a question is past what it can verify. A looser prompt lets it paper over that gap with something fluent. SCOPE removes the room to invent. That is the part that moves the error rate. Telling it to be sceptical never did.

The second step, FILTER, got sharper in the same pass. It used to ask the model to separate fact from guesswork. Now it has to label every claim as sourced, inferred or guessed, so you can see at a glance which parts to check first. The guessed ones, usually.

The method got better the moment I stopped exempting it from its own rule

There’s a neatness to this I cannot pretend I planned. This whole site exists to make one argument: a single confident AI answer needs checking against something else before you trust it. The method is that argument turned into steps. And the method only improved once I ran the rule on the rule, and checked my own work against a different intelligence. The different intelligence was right.

The revised method is live and open. The four steps are at the Prompt Stack, and the whole thing is on GitHub under a licence that lets you take it, fork it or teach it. If you run something similar, the cheapest upgrade you can make is the one I resisted for too long: before you trust the answer, put the same question to a model from a different lab, and go looking for where the two of them disagree. That gap is where the work is.