I built an LLM benchmark harness that lets you browse and compare how models answered each question
一位开发者构建了一个LLM基准测试工具，旨在解决现有基准测试无法直观比较模型如何回答每个问题的问题。目前的基准测试通常只输出原始数据，缺乏便捷的查看工具。该开发者在单个5090上对Qwen3.5-9B、NVIDIA-Nemotron-3.5-Lightning-30B-A3B和Gemma-4-12B-it进行了基准测试，并观察到Qwen的响应时间明显长于其他两个模型。这个项目最初只是一个用于分析单个GSM8K错误答案的脚本，现已发展成为一个更全面的工具。
- 发布
- 2026年9月6日 22:32
- 来源类型
- 开发者社区
- 档位
- 社区
- 信源状态
- 正常
时间以 UTC 显示
更多信息
Hello everyone. I felt current LLM benchmark harnesses hand you headline numbers but offer no tooling to see how models actually answered each question (they dump everything to a JSONL or Parquet file, so you end up writing custom code just to read the answers).
So I built lm-eval-ledger: a benchmark harness that runs the benchmarks, writes everything down, and provides a web app that lets you inspect and compare how each model answered each question.
For the demo I benchmarked three models on a single 5090: Qwen3.5-9B, NVIDIA-Nemotron-3.5-Lightning-30B-A3B (UD-Q4_K_XL GGUF), and Gemma-4-12B-it (QAT w4a16). It looks like Qwen thinks far longer than the other two.
- GPQA Diamond: Qwen 0.717 vs Nemotron 0.657 vs Gemma 0.601 — but 2h26m vs 1h34m vs 1h19m
- LiveCodeBench: Qwen took 22h57m (vs 9h51m / 6h13m), with 0.713 vs 0.837 / 0.820 accuracy
Here are the full results served by lm-eval-ledger on Hugging Face Spaces: https://huggingface.co/spaces/jayminbhan/lm-eval-ledger
- Per question: system prompt, model generation, extracted answer, ground truth, stop reason, generation character count
- Extras: pairwise comparison of two models on the same task/questions, always-wrong / always-right questions across benchmark runs
How it works: everything is written to a single SQLite DB, and a Flask app displays it.
No more custom bash scripts for multi-model x multi-task runs either. lm-eval-ledger is YAML-driven: copy template.yaml, list N models and M tasks, run one command. That’s it.
pip install lm-eval-ledger # add a backend: pip install "lm-eval-ledger[vllm]" lm-eval-ledger init # writes template.yaml, creates results/ and logs/ directories lm-eval-ledger -c bench.yaml # run benchmarks lm-eval-ledger serve # browse at http://localhost:8090
Details and the full task list are in the GitHub repo: https://github.com/jayminbhan/lm-eval-ledger
I verified the vLLM / SGLang / HF / server (llama.cpp) backends on Linux, and HF / server (llama.cpp) on Windows.
This started as a script to look at one wrong GSM8K answer. It got out of hand. Any feedback, ideas, or pull requests are greatly appreciated!