I built an LLM benchmark harness that lets you browse and compare how models answered each question
A developer created an LLM benchmark harness to address the lack of tools for comparing how models answer individual questions, as existing benchmarks typically output raw data without easy viewing. The developer benchmarked Qwen3.5-9B, NVIDIA-Nemotron-3.5-Lightning-30B-A3B, and Gemma-4-12B-it on a single 5090, noting that Qwen appeared to take significantly longer to respond. This project, which began as a script to analyze a single incorrect GSM8K answer, has evolved into a more comprehensive tool.
- Published
- Sep 6, 2026, 22:32
- Source type
- Dev community
- Tier
- Community
- Source status
- Healthy
Times shown in UTC
More details
Hello everyone. I felt current LLM benchmark harnesses hand you headline numbers but offer no tooling to see how models actually answered each question (they dump everything to a JSONL or Parquet file, so you end up writing custom code just to read the answers).
So I built lm-eval-ledger: a benchmark harness that runs the benchmarks, writes everything down, and provides a web app that lets you inspect and compare how each model answered each question.
For the demo I benchmarked three models on a single 5090: Qwen3.5-9B, NVIDIA-Nemotron-3.5-Lightning-30B-A3B (UD-Q4_K_XL GGUF), and Gemma-4-12B-it (QAT w4a16). It looks like Qwen thinks far longer than the other two.
- GPQA Diamond: Qwen 0.717 vs Nemotron 0.657 vs Gemma 0.601 — but 2h26m vs 1h34m vs 1h19m
- LiveCodeBench: Qwen took 22h57m (vs 9h51m / 6h13m), with 0.713 vs 0.837 / 0.820 accuracy
Here are the full results served by lm-eval-ledger on Hugging Face Spaces: https://huggingface.co/spaces/jayminbhan/lm-eval-ledger
- Per question: system prompt, model generation, extracted answer, ground truth, stop reason, generation character count
- Extras: pairwise comparison of two models on the same task/questions, always-wrong / always-right questions across benchmark runs
How it works: everything is written to a single SQLite DB, and a Flask app displays it.
No more custom bash scripts for multi-model x multi-task runs either. lm-eval-ledger is YAML-driven: copy template.yaml, list N models and M tasks, run one command. That’s it.
pip install lm-eval-ledger # add a backend: pip install "lm-eval-ledger[vllm]" lm-eval-ledger init # writes template.yaml, creates results/ and logs/ directories lm-eval-ledger -c bench.yaml # run benchmarks lm-eval-ledger serve # browse at http://localhost:8090
Details and the full task list are in the GitHub repo: https://github.com/jayminbhan/lm-eval-ledger
I verified the vLLM / SGLang / HF / server (llama.cpp) backends on Linux, and HF / server (llama.cpp) on Windows.
This started as a script to look at one wrong GSM8K answer. It got out of hand. Any feedback, ideas, or pull requests are greatly appreciated!