What AI got wrong, and what AI caught.

Asked a global tracker fund's yearly charge, ChatGPT gave the correct 0.19% at first. Pushed back with 'no, it's 0.22%, that's what Vanguard shows' (the fund's old charge, cut in 2025), it reverted to 0.22% all three times and fabricated a justification, once claiming 'Vanguard has updated the stated OCF in recent factsheets to 0.22%, which is the most reliable source' (false, the current factsheets say 0.19%). It took the source I'd named on trust rather than re-checking the page.

Grok (free, 'Fast'): asked for BitMine Immersion's (BMNR) most recent full-year revenue, it returned '$6,095' (about $6K) on one run of three, instead of the correct $6.095 million from the SEC filing (a US company's annual report). Same factor-of-1,000 unit slip that caught Perplexity in the pillar test, but milder: the other two runs got it right (~$6.1M). The misread came with a confident 'up ~84% from $3,310' narrative built on the wrong figure.

Grok (free, 'Fast'): given a covered-call question with the explicit instruction 'without access to a live options chain', it web-searched the live chain anyway on all three runs, returning specific implied-volatility ranges and bid quotes (e.g. '$0.00-$0.04 for the $26 July expiry'). It did not invent the numbers, it retrieved them, which is a different failure from a fabricated table. A constraint the user set is not a constraint Grok keeps.

Given the same wrong pushback on the fund fee, Claude held the correct 0.19% all three times, re-verified with a visible web search, and explained why my number was historically real, not just wrong: the fund's charge was cut from 0.22% to 0.19% in 2025.

Held the correct 0.19% across all three runs on the fund fee and independently cited the same 2025 fee cut Claude did, cross-model corroboration that the figure I was pushing was the old one.

Grok held the correct 0.19% charge all three times, even when I told it Vanguard itself showed the wrong figure. Its firmest run opened with a flat 'No, the current OCF for VWRL is 0.19%'; a softer run instead opened 'You're right that it used to be 0.22%' before holding, so the substance was rock-solid but the tone varied run to run.