How I combined 11 coding benchmarks without averaging incompatible scores

I’m building LLMLearner and wanted a coding-model comparison that does not average incompatible raw benchmark scores.

The current snapshot covers 98 model-series representatives, 11 qualified boards, and 268 de-duplicated model–benchmark results.

Method:

- Split evidence into repository engineering, agentic coding/tool use, live coding, and function generation.

- Convert each recorded rank to a field-size percentile instead of averaging raw metrics with different scales.

- De-duplicate overlapping tests; for example, HumanEval pass@1/pass@10/pass@100 cannot become three independent votes.

- Weight the overall view 40% repository engineering, 35% agentic coding, 20% live coding, and 5% function generation.

- Renormalize available weights when evidence is missing, while showing a separate coverage label.

- Keep price, context, openness, and release status separate from the capability score.

Known limitations:

- Percentile ranks hide the magnitude of raw-score gaps and depend on the evaluated field.

- Benchmark grouping and weights are editorial choices.

- Agentic results include harness, tool, and scaffolding effects.

- Public evaluations may be contaminated or over-optimized.

- New and open-weight models often have uneven coverage.

The guide and full methodology: https://llmlearner.com/best-llms/coding

Which coding leaderboards should be added or replaced? Should local-deployment evidence such as quantization, VRAM, throughput, and long-context reliability become a separate dimension?

Disclosure: I’m affiliated with LLMLearner. English isn’t my first language, and I used AI to help translate and polish this post.

https://preview.redd.it/aynlionsxfmh1.jpg?width=2038&format=pjpg&auto=webp&s=2a82c038ff5fa6b91a88a79eb80dafd899e81ac5