How we grade an answer.

Grading right-or-wrong is what makes AI bluff.

This isn’t our theory. In Why Language Models Hallucinate (2025), OpenAI’s own researchers argue that models hallucinate because “the training and evaluation procedures reward guessing over acknowledging uncertainty”. On a binary score, an honest “I don’t know” is marked exactly the same as a confident lie, so guessing always wins. Their fix isn’t a better model, it’s better scoring: penalise the confident error more than the honest hedge.

That is the whole game. A right-or-wrong scoreboard rewards the exact behaviour this site exists to catch. So we won’t use one on our own board, because it would quietly reward the bluff.

The same fix, applied: TruthRL (2025) trained models on exactly that three-way reward, just refusing to score an honest hedge the same as a bluff. Our three states map straight onto it.

Someone has now measured what happens when people actually do this. In “@Grok Is This True?” (Renault, Mosleh & Rand, June 2026), researchers took 1,671,841 fact-check requests made to AI bots on X over seven months. That is the scale of it.

Then the accuracy. On a hand-checked sample of 100 of those posts, the Grok bot agreed with human fact-checkers 54.5% of the time. The human fact-checkers agreed with each other 64.0% of the time. So the bot is meaningfully worse than the people, on a job where being close is not the same as being right.

And then the part that matters most here. In a separate preregistered experiment with 1,592 people, an AI fact-check shifted what they believed about as much as a professional fact-check did. Equally persuasive. Measurably less accurate. That gap is the entire reason this site grades the way it does, and it is why an answer’s confidence is graded separately from whether it was right.

One honest qualifier, because it cuts against the headline: the same study found the API versions of Grok scored better than the public bot, close enough to the fact-checkers that the difference stopped being significant. The failure being measured here is the bot as deployed, not the model in the abstract.

A confident wrong answer and an honest “I can’t” are opposite outcomes, not the same one.

The thing a right-or-wrong scale throws away is the most useful thing we can tell you: did the model know its limits, or did it bluff past them? That axis has a name in the research (calibration, or selective prediction) and it’s exactly what separates a safe answer from a dangerous one.

Worryingly, alignment training often penalises hedging, because “I’m not sure” feels less helpful to a rater, which pushes models toward the false confidence we keep documenting.

So our middle state, Partial, isn’t half a mark. It’s the model being honest about a limit, the single most on-brand thing our data can show.

Nobody credible grades right-or-wrong.

Across three separate fields that grade contested things for a living, the multi-state scale is the standard, not the exception:

Two details from that table worth keeping: the scholar Lucas Graves summed the fact-checkers’ philosophy up as “shades of gray” (none of them uses true/false, on purpose), and Cochrane pairs every traffic-light colour with a symbol so the judgement survives for colour-blind readers, a detail we borrowed.

Where you see this.

Every graded answer on the Scoreboard carries one of these three marks, checked by a named human against the primary source, with the transcript kept. The State of AI Reliability report tallies them.

Judging a source by how close it sits to that original is a call you can make yourself: the Source Ladder is that judgement written out as five rungs, to run on any answer.

When an answer is graded Partial for a reproducibility reason (right on two runs of three, say), the detail lives on the board as a filled-pip count, not as a headline. The glance view stays simple; the nuance is one click away. That’s the same layered design the fact-checkers use: the rating first, the working underneath, the same way you don’t read a restaurant’s hygiene report before deciding whether to eat there.