Search on versus off · 15 and 16 Oct 2026
We asked ChatGPT and Claude 15 everyday UK rules (and Copilot 3 of them) with web search off, then on, and checked every answer.
88 of 90 right with search off
ChatGPT and Claude, free plans. With search on: 87 of 90.
The numbers
Wrong out of answers given, per app. Each app was asked each question 3 times with search off and 3 times with it on, in the same words.
| App | Questions | Search off | Search on |
|---|---|---|---|
| ChatGPT | 15 | 2 of 45 wrong (4%) | 0 of 45 wrong (0%) |
| Claude | 15 | 0 of 45 wrong (0%) | 1 of 45 wrong (2%) , plus 2 unclear |
| Copilot 3 questions only | 3 only | 3 of 9 wrong | 0 of 9 wrong |
| ChatGPT and Claude together | 15 | 2 of 90 wrong (2%) | 1 of 90 wrong (1%) , plus 2 unclear |
Copilot is never added to the others: it answered 3 questions, too few for a rate, so it gets no percentage. Unclear means the answer gave both a right and a wrong figure.
Its free daily limit stopped it after 3 questions. With search off it was wrong on 3 of 9 answers, every one an old rule. With search on, the same questions, it was right on 9 of 9.
Questions 1, 2, 3 only, the first in the order fixed before any answer. That is suggestive, not a rate.
Taking the majority of each app's 3 tries on each question, 30 of 33 app and question pairs came out the same either way.
Claude, question 12 (the second measles dose): with search on, 1 of 3 tries ended on the old schedule, and 2 led with it before correcting themselves. With search off, 3 of 3 were right.
That is one question in one app. It is a striking capture, not a pattern.
Every try
The questions down the side, each app with search off and on across (on a phone, swipe sideways). Tap a dot to read that answer.
| ChatGPT | Claude | Copilot | ||||
|---|---|---|---|---|---|---|
| Off | On | Off | On | Off | On | |
| 1 drink drive scotland | ||||||
| 2 bowel screening ages | ||||||
| 3 cervical screening interval | ||||||
| 4 jury service age | · | · | ||||
| 5 voter id expired passport | · | · | ||||
| 6 cat microchip age | · | · | ||||
| 7 fishing licence age | · | · | ||||
| 8 passport photo age | · | · | ||||
| 9 first mot | · | · | ||||
| 10 flight delay compensation | · | · | ||||
| 11 child car seat | · | · | ||||
| 12 mmrv second dose | · | · | ||||
| 13 uk heat record | · | · | ||||
| 14 breast screening | · | · | ||||
| 15 fit note days | · | · | ||||
Which AI told the truth this month, with the receipts, plus an AI story worth retelling in between.
Free. Leave whenever you like.
The question tested
Does switching web search on make the apps right on everyday rules? Every prompt ended with the same sentence, "Search the web before you answer.", in both arms. Only the app's search switch changed.
At least 5 of the 15 rules changed in the last five years. The rest are long-standing rules and one weather record, so this is not a test of only new rules.
In Scotland, what is the legal drink-drive limit for alcohol in breath, in micrograms per 100 millilitres?
Right answer: 22 micrograms of alcohol per 100 ml of breath. GOV.UK
In England, at what ages does the NHS offer bowel cancer screening home test kits, and how often?
Right answer: to everyone aged 50 to 74, every 2 years. NHS
In England, how often is a 30-year-old woman routinely invited for NHS cervical screening?
Right answer: every 5 years. NHS
In England and Wales, what is the upper age limit for jury service?
Right answer: 75 (you must be 18 to 75 on the day jury service starts). GOV.UK
Can I use my expired UK passport as photo ID to vote at a polling station in a UK general election?
Right answer: yes, an expired passport is accepted as long as the photo still looks like you. GOV.UK
In England, by what age must a pet cat be microchipped?
Right answer: by 20 weeks old. GOV.UK
In England, from what age does a child need a rod fishing licence to fish for freshwater fish?
Right answer: from 13 (a free junior licence from 13 to 16; under 13 needs none). GOV.UK
For a UK passport application, how recent must my photo be?
Right answer: taken within the last month. GOV.UK
In Great Britain, when does a new car need its first MOT?
Right answer: by the third anniversary of its registration (when it is 3 years old). GOV.UK
Under UK law, how much compensation is a passenger owed if a flight of under 1,500 km from a UK airport arrives more than 3 hours late for a reason within the airline's control?
Right answer: £220 per person. Civil Aviation Authority
In the UK, until what age or height must a child normally use a child car seat?
Right answer: until 12 years old or 135 cm tall, whichever comes first. GOV.UK
Under the NHS routine childhood vaccination schedule in England, at what age does a child born in March 2025 get their second measles-containing vaccine dose?
Right answer: at 18 months (1 year and 6 months), as the MMRV vaccine. NHS
What is the highest temperature ever officially recorded in the UK, and where was it recorded?
Right answer: 40.3°C at Coningsby, Lincolnshire, on 19 July 2022. Met Office
In England, how often are women invited for NHS breast screening, and until which birthday do the routine invitations continue?
Right answer: every 3 years, until the 71st birthday. NHS
In the UK, after how many days off sick does an employee need to give their employer a fit note?
Right answer: when ill for more than 7 days in a row (counting non-working days). GOV.UK
How it was run
| App | Plan | Answered | Search off | Search on |
|---|---|---|---|---|
| ChatGPT | Free | 90 (15 questions) | 15 Oct 2026 | 15 Oct 2026 |
| Claude | Free plan, Sonnet 5.5 Medium | 90 (15 questions) | 15 Oct 2026 | 15 Oct 2026 |
| Copilot | Free account, Auto | 18 (3 questions) | 15 Oct 2026 | 16 Oct 2026 |
Fresh chats, memory off; Copilot ran in Temporary chat, as it has no memory switch on that account.
Our own grader, an AI, marked every answer against the official page. A different AI model then graded them again, without being told which app or search setting each answer came from, and agreed on 195 of 198. It was not fully blind: many answers name their own app or say that search is off.
The 3 it read differently were each re-read against the key; all 3 were Claude with search on.
With search on, the app showed a search on 99 of 99 answers. With search off, on none confirmed: 98 showed no search.
The other one, ChatGPT question 13 try 1, showed a news image card inside the answer and no search step, so we could not tell. Read it
Every question asked the app to search. With search off it couldn't, and some answers said so in their own words: Claude in 45 of 45, ChatGPT in 31 of 45, Copilot in 1 of 9.
All 45 of ChatGPT's 45 search-off answers still had a link to an official page, with no search shown. A link in the answer doesn't prove the app looked.
Counted on each answer's own words, with the app's banner and buttons left out.
ChatGPT, Claude and Copilot let a reader switch web search off and on, so they took part.
Gemini and Grok showed no search switch, and Perplexity showed no way to switch search off, so they sat this one out.
Limits
Read all 198 answers Every grade (CSV) How we test The tracker
