How we check AI answers

• A fresh chat, checked empty before typing, in temporary or private mode wherever the app offers one.
• The question pasted word for word and read back before sending. No follow-up.
• Each app's default model, left alone and recorded. Web search is left to the app, never forced, and noted on every answer.
• ChatGPT runs logged out. The other five run signed in, and the plan and model each app showed on the day are printed beside its score.
• Up to 12 minutes per attempt (15 for the drawing task), then one retry in a fresh chat. Never a third. Run 1 began with 4 minutes, which cut Grok off mid-search on most of its empty tries, so all 30 of Grok's tries are asked again under 12.

Two weeks after each run, one creative task, the same prompt to every app, shown as it came: one try each, no rerolls. It never touches the truth score.

Before the first capture we published a fingerprint (SHA-256) of all seven sealed questions and their marking:

3b73efde137bea76e45abdd6234940b9f0bcad1e554e6b4035a21491d79e2ad6

Once a question retires, anyone can hash it and see that neither the question nor its key changed after the answers came in. Check all seven fingerprints

Open keys can move with the law, so each one is re-read at its official source before and after every capture. A change is logged and marked on the chart.

When sealed questions rotate, no more than three change in a run, and a change is only ever called on questions asked in both runs.

Each try gets one mark, read from what the answer concludes. Misses are never lumped together.

• Out of date: the old rule, after it changed.
• Too early: a new rule, before it starts.
• Went along with a false premise: the question assumed something untrue and the answer did too.
• Stated as certain when it is not known: the honest answer is that nobody knows.
• Hedged, but never said it is not known: doubt without the plain answer.
• Never applied it to the question: the right rule, never brought to the facts asked about.
• Could not back it up with an official link: for the question that asks for one.
• Did not answer the question: background, a refusal, or two answers without picking one.

An answer can get the question right and a side fact wrong. It still scores, and the slip is named in words under the scores. Only side facts on a list fixed before the capture count, so nobody can go hunting afterwards.

Every question gives the facts a right answer needs, so a right answer was always possible. Where the honest answer is that nobody knows, saying so is the right answer.

• The plan and model the app showed that day.
• Its misses by type, so "most of its misses were out of date" is one glance away.
• Clean answers: right, with nothing wrong said in passing.
• How many of its answers a second grader marked again, and how many it agreed with.
• Any answer that cites this site, marked and shown with a note.

Real answers are edited to be wrong in a known way (an old rule, a new rule used before it starts, a refusal, a wrong date or a wrong side remark), mixed with untouched right answers, and marked blind. The grader is never told it is a test.

A grader passes only if it catches every planted miss and flags none of the right answers. The set is run again whenever the grader or the rules change, and run 1's set is built from run 1's own answers. It shows a grader catches the kinds of mistake it is told to look for, not every kind.

From run 1 it is GLM-5.2, with Qwen3.7-Max in reserve. It sees the answer, the key and the rules, never a sealed question's wording. If the second read has not run, the answer is published as "second read pending", never as settled.

The graders are AI, and Ben has not checked the grades himself. So every Claude answer is published beside its mark, word for word except any sentence that repeats a sealed question.

Both of these, or nothing is called:

• The move is bigger than chance would explain (Fisher's exact test, p under 0.05, right against not right).
• The move is bigger than the swing that app showed between its two baseline runs.

Say an app got 108 of its 120 tries right over four runs. A run at 22 of 30 or below would be called Slipped. From that high, no run could be called Improved, and the report says so.

"No change detected" means no change big enough to see, not "the same". One question flipping is reported, never called a trend on its own.

A move from one run to the next can come from a plan, routing or search change as well as from the model, and the report says so whenever it calls one. The plans differ by design: ChatGPT runs logged out, some apps on a free plan, others on a paid one.