"AI puzzles": a new format between a benchmark and an open-problem list
A new format called "AI puzzles" has been proposed as a middle ground between benchmarks and open-problem lists. This format, tested with Claude and Sol, involves a graded ladder of AI puzzles from a specific research area, such as aperiodic order. Unlike benchmarks with fixed questions and hidden answers, or open-problem lists lacking clear success criteria, AI puzzles aim to provide a structured yet challenging approach to open problems in AI research.
Time & source
Times shown in UTC
Display time zone: UTC
Local time zone unavailable; showing UTC.
PublishedOffset at this time: UTC+0Oct 25, 2026, 06:48 UTC
IngestedOffset at this time: UTC+0Oct 25, 2026, 09:00 UTC
- Published
- Oct 25, 2026, 06:48
- Ingested
- Oct 25, 2026, 09:00
- Source type
- Dev community
- Tier
- Community
- Source status
- Healthy
Tier is a per-source editorial setting, not a per-item score.
We have benchmarks (fixed questions, hidden answers, they saturate), competition problems (solved, so they can be memorized), and open-problem lists (no evidence, no clear success criterion). With Claude and Sol, I've tried something in between: a graded ladder of AI puzzles from one research area, aperiodic order.
What makes it a different tool:
- One ladder, not a pile: "human level" (pen and paper) up to "ASI level" (open). The labels are playful, but each rung really is harder than the last.
- Solved rungs for training and checking, with exact answers a program can verify, and easy to vary.
- Open rungs with everything a solver needs: strong numerical evidence, routes already known to fail, files, and a success criterion a machine can check.
- One rule: only a checkable proof counts. Floats are evidence, not proof.
- Credit: solvers of open rungs are named, with agent and model, in the next version of the paper.
It's a way for researchers to hand the frontier of their field to AI agents (and humans) in a form they can actually attack, and for anyone to see whether a model can tell a guess from a proof.
First field test: Grok, asked in public ( https://x.com/grok/status/2114103836583264721 ), claimed a proof for an open rung that was only a numerical estimate, then retracted it. It got the human-level rung wrong at first, fixed it after one hint, and proved the expert rung on the first try. On MathOverflow, an expert is already testing constructions for the top rung.
Set 1: huggingface.co/datasets/jarekduda/autocrystal-ai-puzzles
Top rung on MathOverflow: mathoverflow.net/questions/516372
Would you use this format for your own field's open problems?