Ran 54 tasks each through Claude and Codex. Codex falsely declared Done on 4.1%. Single runs lie, so I measured three rounds.

A June paper (arXiv 2606.09863) found that among failing agent runs that graded themselves, 75.8% still claimed success, and LLM judges catch it at AUROC 0.54 to 0.65. Coin flip. So instead of judging the words, I re-run the reality. nuhuh takes every claim in the final message ("tests pass", "created src/x.ts", "endpoint works") and verifies each one fresh. Full suite in a clean process, real exit codes, actual files, live localhost probes. Zero LLM calls in the verification path.

It ships with a False Done Rate benchmark, deterministic check scripts that know nothing about the tool. Three rounds of 18 tasks per harness, 54 runs each. Codex 4.1%, Haiku 4.5 6.1%, frontier Claude 0.0%.

The part that surprised me. Codex measured 0% on round one and I nearly published that number. Over three rounds it settled at 4.1%. Haiku went the other way, 12.5% down to 6.1%. Whatever benchmark you read this week, ask how many rounds it ran.

The recurring failure is interesting too. It is a config-consistency chore where the model declares victory with a hardcoded port still in place, and the Done message carries no checkable claim at all. Claim verification cannot catch that one, only ground truth can, which is why the bench scripts do not know nuhuh exists.

nuhuh reads Codex rollouts as well as Claude Code session logs, so post-hoc checking works on both. npx nuhuh demo stages a lie and catches it in ten seconds without touching anything. Repo at https://github.com/sjh9714/nuhuh. If you run the bench on other models I would genuinely love a results file.