How I combined 11 coding benchmarks without averaging incompatible scores
一位开发者在构建LLMLearner时，创建了一种编码模型比较方法，该方法避免了对不兼容的原始基准分数进行平均。该快照涵盖了98个模型系列代表、11个合格板，以及268个去重后的模型-基准测试结果，成功结合了11个编码基准，而没有平均不兼容的分数。
I’m building LLMLearner and wanted a coding-model comparison that does not average incompatible raw benchmark scores.
The current snapshot covers 98 model-series representatives, 11 qualified boards, and 268 de-duplicated model–benchmark results.
Method:
- Split evidence into repository engineering, agentic coding/tool use, live coding, and function generation.
- Convert each recorded rank to a field-size percentile instead of averaging raw metrics with different scales.
- De-duplicate overlapping tests; for example, HumanEval pass@1/pass@10/pass@100 cannot become three independent votes.
- Weight the overall view 40% repository engineering, 35% agentic coding, 20% live coding, and 5% function generation.
- Renormalize available weights when evidence is missing, while showing a separate coverage label.
- Keep price, context, openness, and release status separate from the capability score.
Known limitations:
- Percentile ranks hide the magnitude of raw-score gaps and depend on the evaluated field.
- Benchmark grouping and weights are editorial choices.
- Agentic results include harness, tool, and scaffolding effects.
- Public evaluations may be contaminated or over-optimized.
- New and open-weight models often have uneven coverage.
The guide and full methodology: https://llmlearner.com/best-llms/coding
Which coding leaderboards should be added or replaced? Should local-deployment evidence such as quantization, VRAM, throughput, and long-context reliability become a separate dimension?
Disclosure: I’m affiliated with LLMLearner. English isn’t my first language, and I used AI to help translate and polish this post.
https://preview.redd.it/aynlionsxfmh1.jpg?width=2038&format=pjpg&auto=webp&s=2a82c038ff5fa6b91a88a79eb80dafd899e81ac5