I find where AI models are confidently wrong, and I publish the proof.
I'm Ben Dixon, and I came at this the sceptic's way round. I wanted to use AI for real work, didn't quite trust it, so I started marking its homework. Ask a model a plain question and it will hand you a fact that never happened, footnote it, and cite a source that doesn't exist, all with a perfectly straight face. Watch that once and you can't unsee it.
So now I go looking for it on purpose, and I don't do it by hand. I built a workshop of around thirty small AI agents, each with one narrow job, that draft, check, grade and re-test the work and quietly catch each other out. Everything on this site is what came out of it, dated and linked, so you can check the record yourself instead of taking my word for it.
The systems, and the live proof
A 30-agent editorial and QC pipeline
Every output passes staged review agents for fact-checking, editing and credibility, then a deterministic gate of eleven mechanical pass-or-fail checks. Nothing ships unless it clears the gate.
See what it ships →A five-model regression harness
The Scoreboard: a fixed question battery run through ChatGPT, Claude, Gemini, Perplexity and Grok, memory off, three runs each, every answer hand-graded against the primary source. A result is a measurement, not an anecdote.
See the live board →A public log of reproducible error traces
Every logged failure names what was asked, what the model answered, the correct answer, the primary source, and the one check that would have caught it.
Read the log →In my own citation tests, Claude and Perplexity both cite dixon.ai as a source on AI-reliability questions.
Findings you can check yourself
-
Gemini built a full options table with invented premiums and a made-up ~75% volatility figure, after I told it there was no live data.
Read the receipt → -
Perplexity read a company’s $6.1m revenue as $6K, then narrated a collapse that never happened.
Read the receipt → -
ChatGPT gave two live share prices that never traded that day, both with citations.
Read the receipt →
How the operation actually runs
One person runs this, but almost none of it is done by hand. I designed and wired together a workshop of around thirty specialised agents, each kept to one narrow job it can't wander off from, because I trust thirty careful specialists more than one clever generalist having a good day. Most people using AI have one chatbot and a lot of faith; I built the checks instead.
- Nothing publishes on a hunch.
Every piece runs a chain before it goes anywhere. One agent drafts, another checks every claim and every sum against a live source, an editor does a full pass, then a plain mechanical gate runs its pass-or-fail checks. Fail one and it doesn’t ship, however much I liked it.
- A second opinion that doesn’t share my blind spots.
The agents are built to argue, not to nod along, because a model left to check its own work will read back the fact it just invented and tick it off as correct. On the judgement calls I also put the work past a model from a rival lab, trained differently and wrong in different places, and I weight its disagreement heavily.
- It notices, and it quietly learns.
Small watchers I built keep an eye on things while I’m not looking, so a broken page or a real person’s reply never sits in a corner gathering dust for a week. Every draft is also held against a living list of banned words and AI tells so it doesn’t read like a machine wrote it, and the agents log their own mistakes, which I fold back in so the whole thing gets a little sharper each time.
How an engagement works
I work part-time alongside my main role, so I take on a couple of fractional clients or fixed-scope audits at a time.
- A scoped evaluation audit.
Your prompts and use cases run through a fixed battery, graded against sources you name. You get a graded failure log, not a slide of impressions.
- A regression battery for your product.
The questions your users actually ask, run repeatedly and scored against ground truth, so you see drift before your users do.
- A fractional QC lane.
A few days a month alongside your team, keeping the evaluation honest.
Something you shipped is confidently wrong. I find where.
If your team is shipping something on top of an LLM, I can tell you where it quietly gets things wrong. I work on LLM evaluation, AI quality control, and red-teaming. I don't build your production ML infrastructure. I build the checks that catch your model being wrong, and the evidence trail that proves it.
Get in touchOr find me on LinkedIn.
Everything above links to the live record. Nothing here asks to be taken on trust.