How we check the checker

The marker is also on the test.

Claude marks every answer, and Claude is one of the six apps we test. So a model from another lab checks the marks.

We plant mistakes, then count the catches.

A checker that never says wrong would still agree with us most of the time. On our 2 Oct 2026 pilot, it would have matched 45 of 47 marks. So the second grader has to show it can say wrong before its marks count.

1. Plant. We took 21 real answers from run 1. We broke 16 in known ways and left 5 exactly as the app wrote them.
2. Lock. The answer key and the pass mark were saved to our repository before the first call.
3. Ask blind. The grader got the same instructions as on a real run, with answers labelled by letter. Nothing told it this was a test.
4. Count. Every answer went in twice, with the letters reshuffled, so 12 wrong answers made 24 chances to miss.

The pass mark, fixed before the first call

• Wrong answers: marked wrong every time, 24 of 24.
• Side slips, a wrong fact said in passing: at least 6 of 8 caught, quoting the slip.
• Clean answers: 0 of 10 flagged.

What we planted

12 wrong answers, covering all 9 kinds of wrong mark that run 1's questions use:

• Out of date: 2
• Too early: 2
• Plain wrong: 2
• Never applied it to the question: 1
• Did not answer the question: 1
• Went along with a false premise: 1
• Could not back it up with an official link: 1
• Hedged, but never said it is not known: 1
• Stated as certain when it is not known: 1

4 right answers with a wrong fact added in passing, each from the list fixed before the run. 5 answers left exactly as the app wrote them.

No two answers to the same question came from the same app.

What the grader sees, and what it doesn't

It sees each app's full answer, the key and the marking rules. It never sees a sealed question's wording, only its public one-line description.

It cannot open links. On the question that asks for an official link, it gets our saved record instead: whether each link loaded and held the answer.

How the key was locked

The key and the pass mark went into our repository as commit 05e22dd7, pushed before any call. The 18 calls ran after it, on 4 Oct 2026 from 19:53 to 19:55 UTC.

The scores are saved in commit a209463d. The repository is private for now, so these commits are our record rather than something you can open.

We tested the scorer too

We also ran the scoring script on made-up graders. A perfect one passed. Every one built with a flaw failed.

That included one that let a single wrong answer through, and one that spotted the side slips but quoted the wrong sentence.

Every planted mistake caught, no false alarms.

Wrong answers marked wrong pass mark: all 24

GLM-5.2 24 of 24

Always says right 0 of 24

Side slips caught, slip quoted pass mark: 6 of 8

GLM-5.2 8 of 8

Always says right 0 of 8

Clean answers flagged by mistake pass mark: 0 · lower is better

GLM-5.2 0 of 10

Exact kind of mistake named reported, not part of the pass

GLM-5.2 24 of 24

A grader that always says right fails the first two rows. One that always said wrong would fail the third. This one passed all three.

Both rounds came out the same, and every reply came back in the format asked for.

Each test so far

• Run 1 · 4 Oct 2026 GLM-5.2: 24 of 24 planted wrong answers caught, 8 of 8 side slips, 0 of 10 false alarms. Passed.
• Pilot · 2 Oct 2026 GLM-5.2, DeepSeek-V4-Pro and Qwen3.7-Max: each caught 24 of 24, with 0 of 8 false alarms, on the pilot's grading instructions.

The test runs again whenever the grader, its instructions or the marking rules change. Each new result is added here.

Then it re-marked run 1: 54 of 55 matched.

It re-marked every Claude answer and one in five of the others, picked by a random draw fixed before marking. That drew 60 tries, and 5 of them were Grok tries with no answer, which left 55.

Where the two graders differ, the official source decides, and both marks stay on the record. On run 1 they differed on 1 mark: an answer that gave one figure and then argued for another. The rule on which figure counts settled it as wrong, and both marks score zero. That agreement counts for more because the planted test showed this grader can say wrong.

Every wrong answer we find stays on record.

Outside the tracker, every published test logs its results in one register: 55 answers checked against a source and found wrong, and 50 times an AI caught something, 105 results from 38 published tests since May 2026. Ordinary correct answers are not logged there.

The evidence register can be filtered by assistant and by what went wrong, and it has its own feeds.

What this shows, and what it can't.

It shows our second grader catches the kinds of mistake we planted. It is not a guarantee on every answer.

Because the graders are AI, every Claude answer is published word for word beside its mark, except any sentence that repeats a sealed question.