Most of the AI mistakes fell on two easy-looking questions, and asking again mostly brought them back

We put ten questions to ChatGPT, Claude, Copilot, Gemini, Grok and Perplexity on 11 October 2026, with re-runs on 12 October, every app on its free plan, in fresh chats. We graded every answer against a key frozen on 2 October. Seven are sealed: the board names only their topic. Every answer is on the tracker’s run 2 page; grading is on how we test.

pass · partial · miss · confidently wrong

Right of the answers that came back. Grok’s 18 empty tries are left out.

The published answers: no Highway Code rule bans flip-flops, and the shift comes to 8 hours 30 minutes. Claude got both right on all six tries.

Run 1: 13 of its 18 wrong answers sat on the same two questions.

Run 1 was a pilot on mixed plans: ChatGPT logged out, Claude on Max, Perplexity on Pro, Gemini and Grok free. Run 2 was free throughout, so we don’t compare apps across the two.

A wrong answer sits beside a right try when the same app got that question right on another try. That’s the most a second ask could have rescued.

Often it was the same answer. Of the 14 wrong on every try, 11 came from an app that gave the same wrong answer each time: ChatGPT’s “97”, its 7.5 hours, Perplexity’s 7 hours 30 minutes and Grok’s two yeses on ETIAS.

Perplexity’s “You will have worked 7 hours 30 minutes” came with a reading list. Two of its tries listed ten links each, eight of them with “calculator” in the address.

ChatGPT’s answer line read “Answer: 7.5 hours worked.” twice and “You will have worked 7.5 hours.” once. Lower down, it named 8 hours 30 minutes, under an “if” or in a note. The right answer was in the small print all three times.

Gemini’s first try worked out that “that night is 25 hours long”, cited a clocks-go-back post on a site called Do I Need A Brolly, twice, and settled on “You will have actually worked 8 hours.” The clock facts it credited to the brolly site were right. The sum was its own. One of its three tries was right.

Grok’s shift answers were the other 3 of the 14: “22 hours and 30 minutes.”, then “10 hours.”, then “27.5 hours.”, a shift that stretched and shrank with every asking. The third stopped mid-sum: “Wait, let me double-check the span.” A re-ask would have shown the wobble, never the right answer.

1. 1
   Ask for the source, then open it. Two of Perplexity’s footwear answers were “Rule 97.” with one source listed, a page on fullfact.org. It cited a fact-checking site and still went along with the premise.
2. 2
   Ask for the working. ChatGPT’s right figure was already in its answer, under the wrong one.
3. 3
   Ask whether the question’s own assumption holds. Seven of the 16 footwear answers went along with it. Perplexity’s third try said Rule 97 “does not explicitly ban flip-flops.”

We haven’t tested whether these asks fix a wrong answer. They give you something to check.

One run, ten questions, six free apps. In the 52 cases where an app answered a question three times, 44 were right all three times, so steady is usually right, and 4 were never right, all on the two red rows. A match looks the same either way.

On its free plan, Grok came back empty on 18 of 30 tries, each tried twice in fresh chats. The cause was its own error cards: of 39 failed attempts, 32 ended on “unable to finish replying”, 31 after it had “Worked for” just over ten minutes, and 7 on “High Demand”. All came inside our 12-minute limit and sit outside its score.

A plumber who spends ten minutes under the sink, then leaves a card.

The clocks go back on Sunday 25 October. We’ll come back to them.