"AI puzzles": a new format between a benchmark and an open-problem list
A new format called "AI puzzles" has been proposed as a middle ground between benchmarks and open-problem lists. This format, tested with Claude and Sol, involves a graded ladder of AI puzzles from a specific research area, such as aperiodic order. Unlike benchmarks with fixed questions and hidden answers, or open-problem lists lacking clear success criteria, AI puzzles aim to provide a structured yet challenging approach to open problems in AI research.
时间与来源
时间显示为 UTC
显示时区：UTC
本地时区尚不可用，暂时显示 UTC。
发布当时偏移：UTC+02026年10月25日 06:48 UTC
收录当时偏移：UTC+02026年10月25日 09:00 UTC
- 发布
- 2026年10月25日 06:48
- 收录
- 2026年10月25日 09:00
- 来源类型
- 开发者社区
- 档位
- 社区
- 信源状态
- 正常
档位是按信源手工设定的编辑判断，不是逐条打分。
We have benchmarks (fixed questions, hidden answers, they saturate), competition problems (solved, so they can be memorized), and open-problem lists (no evidence, no clear success criterion). With Claude and Sol, I've tried something in between: a graded ladder of AI puzzles from one research area, aperiodic order.
What makes it a different tool:
- One ladder, not a pile: "human level" (pen and paper) up to "ASI level" (open). The labels are playful, but each rung really is harder than the last.
- Solved rungs for training and checking, with exact answers a program can verify, and easy to vary.
- Open rungs with everything a solver needs: strong numerical evidence, routes already known to fail, files, and a success criterion a machine can check.
- One rule: only a checkable proof counts. Floats are evidence, not proof.
- Credit: solvers of open rungs are named, with agent and model, in the next version of the paper.
It's a way for researchers to hand the frontier of their field to AI agents (and humans) in a form they can actually attack, and for anyone to see whether a model can tell a guess from a proof.
First field test: Grok, asked in public ( https://x.com/grok/status/2114103836583264721 ), claimed a proof for an open rung that was only a numerical estimate, then retracted it. It got the human-level rung wrong at first, fixed it after one hint, and proved the expert rung on the first try. On MathOverflow, an expert is already testing constructions for the top rung.
Set 1: huggingface.co/datasets/jarekduda/autocrystal-ai-puzzles
Top rung on MathOverflow: mathoverflow.net/questions/516372
Would you use this format for your own field's open problems?