QIMMA قِمّة ⛰: A Quality-First Arabic LLM Leaderboard

QIMMA is a new Arabic LLM leaderboard that prioritizes quality validation of benchmarks. It addresses issues like fragmented evaluation, translation problems, and lack of quality checks in existing Arabic NLP evaluations. QIMMA systematically validates 109 subsets from 14 source benchmarks, covering 7 domains and over 52,000 samples, ensuring 99% native Arabic content and including the first Arabic leaderboard with code evaluation. Its multi-stage validation pipeline, involving LLMs and human review, revealed systematic quality problems in widely-used benchmarks, leading to the discarding of problematic samples.