The prevalent problem of misleading benchmark reporting (re: Astra)
- Published
- 09/03, 21:23
- First discovered
- 09/04, 20:00
- Type
- Dev community · RSS
OpenAI's reported benchmarks for Astra's ARC-AGI-3 are criticized for being misleading. While a screencap shows Astra at 98.6% compared to GPT 5.6 Sol's 7.8% and Claude Opus 5's 30.2%, this was achieved using a specific provider adapter harness. On the standard ARC-AGI-3 harness, Astra scores 62.7% against Opus 5's 30.2% and Sol's 7.8%, a still significant but less exaggerated difference, suggesting a deliberate misrepresentation.
OpenAI's reported benchmarks for Astra's ARC-AGI-3 is one of the most egregious recent examples I have seen of technically true metric reporting being used to deliberately mislead the masses. For context, there is an OpenAI screencap currently at the top of r/singularity 's hot page of Astra achieving 98.6% on ARC-AGI-3 compared to 7.8% for GPT 5.6 Sol and 30.2% for Claude Opus 5. Holy shit, right? ASI achieved, right?
Unfortunately, those figures taken in a vacuum leave out very important context: Astra's agentic harness had significant additional features that GPT 5.6 Sol and Claude Opus 5 did not have access to - specifically reasoning trace retention and custom compaction (source: https://arcprize.org/leaderboard ).
My main takeaway is basically: The most honest way to compare Astra with Opus 5/Sol on this benchmark would have been to either 1) measure their ARC-AGI-3 performances on the same provider adapter harness (where Astra's 98.6% came from), or 2) compare them on the standard ARC-AGI-3 harness. On the standard harness Astra achieves 62.7% vs Opus 5's 30.2% vs Sol's 7.8%. Still a very large gap, but much less misleading than the comparison OpenAI chose to report. (source: https://arcprize.org/leaderboard)
Not an Anthropic fanboy in any sense of the word, btw. I thought Opus 5 was benchmaxxed and pray on Anthropic's downfall every day. But the Astra benchmark glazing made it clear that restraint needs to be had in people's reactions to its benchmarks (if Opus 5 didn't already convince you to not treat benchmarks as gospel) before anyone has even had time to extensively test it in real world use cases.