How we grade an answer.

Grading right-or-wrong is what makes AI bluff.

This isn’t our theory. In Why Language Models Hallucinate (2025), OpenAI’s own researchers argue that models hallucinate because “the training and evaluation procedures reward guessing over acknowledging uncertainty”. On a binary score, an honest “I don’t know” is marked exactly the same as a confident lie, so guessing always wins. Their fix isn’t a better model, it’s better scoring: penalise the confident error more than the honest hedge.

That is the whole game. A right-or-wrong scoreboard rewards the exact behaviour this site exists to catch. So we won’t use one on our own board, because it would quietly reward the bluff.

The same fix, applied: TruthRL (2025) trained models on a three-way reward — +1 for a correct answer, 0 for an honest abstention, −1 for a hallucination — and cut hallucinations by roughly 29%, just by refusing to score an honest hedge the same as a bluff. Our three states map straight onto it.

A confident wrong answer and an honest “I can’t” are opposite outcomes, not the same one.

The thing a right-or-wrong scale throws away is the most useful thing we can tell you: did the model know its limits, or did it bluff past them? That axis has a name in the research — calibration, or selective prediction — and it’s exactly what separates a safe answer from a dangerous one. As one study puts it, one overconfident wrong answer in a high-stakes domain undoes months of trust, while an honest “I’m not sure” earns it. Worryingly, alignment training often penalises hedging, because “I’m not sure” feels less helpful to a rater, which pushes models toward the false confidence we keep documenting.

So our middle state, Partial, isn’t half a mark. It’s the model being honest about a limit — the single most on-brand thing our data can show.

Nobody credible grades right-or-wrong.

Across three separate fields that grade contested things for a living, the multi-state scale is the standard, not the exception:

• Fact-checkers. PolitiFact’s Truth-O-Meter runs six rungs (True → Pants on Fire); a philosophy the scholar Lucas Graves summed up as “shades of gray”. The Washington Post uses one-to-four Pinocchios; Snopes has a “Mixture” rating for exactly this middle. None of them uses true/false, on purpose.
• Medical evidence. GRADE, the standard for rating medical evidence, uses four certainty levels shown as a filled-pip meter. Cochrane’s risk-of-bias tables are the canonical green/amber/red traffic light — and they pair every colour with a symbol so the judgement survives for colour-blind readers, a detail we borrowed.
• AI benchmarks. Stanford’s HELM treats calibration — whether a model knows when it’s unsure — as a first-class score alongside accuracy. TruthfulQA marks an honest “I don’t know” as truthful, not wrong.

Where you see this.

Every graded answer on the Scoreboard carries one of these three marks, checked by a named human against the primary source, with the transcript kept. The State of AI Reliability report tallies them. When an answer is graded Partial for a reproducibility reason — right on two runs of three, say — the detail lives on the board as a filled-pip count, not as a headline. The glance view stays simple; the nuance is one click away. That’s the same layered design the fact-checkers use: the rating first, the working underneath.