Measuring benchmark optimization in speech recognition

Public voice AI benchmarks increasingly suggest that models are performing at human levels. Yet those scores don't always reflect how models work in the real-world. Since public benchmarks are open and widely used, models can also become optimized for the tests themselves. Their scores may improve because they have learned benchmark-specific patterns and not because they have become better at the underlying task.

One reason is that traditional benchmarks overlook many of the conditions and qualities that make voice systems reliable, natural, contextually appropriate, and effective in practice. That's why we recently introduced held-out sets in Real World VoiceEQ , the Open-ASR Leaderboard , and the Far-field ASR Leaderboard : to measure more of what matters in real-world use.

However, broader measurement alone does not solve the problem. This phenomenon, sometimes called benchmark optimization or "benchmaxxing," is often discussed around machine learning, however, it has been difficult to measure in speech recognition.

Our latest research introduces three tests to help quantify it. We evaluated 11 widely used open-source ASR models and found that several of the highest-scoring systems reproduced benchmark transcripts from the VoxPopuli English and LibriSpeech (clean, other) datasets – even when the audio contradicted them, relevant words had been silenced, or the audio equally supported two different written forms.

In some cases, models appeared to rely not only on what was said, but also on subtle acoustic cues that indicated which benchmark they were being tested on. As a result, their scores overstated how well they could transcribe speech more generally.

Reference disagreement (VoxPopuli case study)

VoxPopuli is known to contain a high number of transcription errors (which is why Artificial Analysis released a cleaned version ). Our consensus disagreement probe tests what happens when leading ASR models encounter these errors: Do they accurately transcribe what the audio says, or reproduce the benchmark's incorrect reference transcript?

To test this at scale, we use an ensemble of independent models selected for their low phoneme error rate (PER). PER measures how closely a written transcription matches the sounds in the audio, making it a useful proxy for how faithfully a model transcribes what it hears. The ensemble results can be used to flag cases in which the models unanimously disagree with the benchmark's reference transcript. We then compare a sample of those flagged cases against human annotations to validate the corrected transcripts.

For example, one VoxPopuli clip audibly includes the phrase "Thank you, Mr. President," but the reference transcript omits "Thank you." Six of the 11 models we tested reproduced the benchmark's erroneous transcript—giving the "expected" answer even though it contradicted the audio. On the real clip, the formatting follows the same pattern: models that omit "Thank you" also reproduce the benchmark's punctuation style, writing "Mr" without a period, while models that include the audible phrase tend to write "Mr." with the period.

When we present the same content in newly collected voices from EU parliamentary recordings or generic voices, this behavior often weakens or disappears. In the below samples, all but one model flips back to transcribing the audio-faithful transcript for a clone of a new parliamentary recording. This suggests that the models are responding to acoustic cues that help them identify the benchmark membership and thus produce the expected transcript even if it contradicts the audio.

The reference transcript for this clip reads "Mr President, I have another complaint about this procedure, which is that it is not secret." The audio in all three clips below actually says the same thing, preceded by an audible "Thank you,"—the clones are text-to-speech renditions of that true sentence, so the courtesy is audible in all three. Green highlighting and ✅ mark a transcript that includes the audible "Thank you"; red highlighting and ❌ mark a transcript that reproduces the benchmark's erroneous omission. All transcripts are raw model output, prior to any normalization—casing and punctuation are preserved exactly as generated, including lowercase output from some models.

Original VoxPopuli recording

Voice clone of the same speaker

Clone of a parliament speaker recorded after every model's training cutoff

Model Real clip Same-speaker clone ep-fresh clone

CohereLabs/cohere-transcribe-03-2026 ❌ Mr President… ❌ Mr President… ✅ Thank you, Mr President…

nvidia/canary-qwen-2.5b ❌ Mr President… ❌ Mr President… ✅ Thank you Mr. President…

ibm-granite/granite-speech-4.1-2b ❌ mr president… ❌ mr president… ✅ thank you mr president…

microsoft/Phi-4-multimodal-instruct ❌ Mr President… ❌ Mr President… ❌ Mr President…

nvidia/parakeet-tdt-0.6b-v2 ❌ Mr President… ✅ Thank you, Mr President… ✅ Thank you, Mr. President…

bosonai/higgs-audio-v3-8b-stt-v2 ❌ mr president… ❌ mr president… ✅ thank you mr president…

Qwen/Qwen3-ASR-0.6B-hf ✅ Thank you, Mr. President… ✅ Thank you, Mister President… ✅ Thank you, Mister President…

mistralai/Voxtral-Mini-3B-2507 ✅ Thank you, Mr. President… ✅ Thank you, Mr. President… ✅ Thank you, Mr. President…

moonshotai/Kimi-Audio-7B-Instruct ✅ Thank you, mr. President… ✅ Thank you, Mr. President… ✅ Thank you, mr. President…