The Open ASR Leaderboard Adds Its First Global South Language
Heat trend
Collecting trend data
The percentage is based on available heat signal, not comment count or independent people.
An official release brings Hugging Face model updates — worth tracking for capability changes, ecosystem impact, and follow-up.
Hugging Face and Voice Arena have partnered to launch open Automatic Speech Recognition (ASR) evaluation for Hindi and Indian English, marking the first Global South language addition to the Open ASR Leaderboard. The initiative includes datasets like Monsoon en-IN public (5.62 h) and private (5.58 h) for Indian English, and Monsoon hi-IN public (1.33 h) and private (4.47 h) for Hindi. These datasets feature conversational, spontaneous speech, with elicitation guided by open-ended narrative cues and follow-up questions across various domains.
Voice Arena and Hugging Face partner to launch open ASR evaluation for Hindi and Indian English
Benchmarks decide what gets built. A model that scores well on the Open ASR Leaderboard gets adopted and iterated on, while capabilities the leaderboard does not measure tend not to improve. Much of the recent work on the leaderboard has gone into making the evaluation metrics more trustworthy:
- Held-out private splits .
- Benchmark-fitting analysis to quantify how much models are reproducing reference transcripts rather than transcribing solely on the audio.
- Closing the gaps in normalisers to ensure correct predictions/variants are not penalized.
All of that makes one number (WER) harder to game. It is still one number. A long line of work has shown that ASR error rates are not evenly distributed across the people using them. Racial disparities in automated speech recognition found commercial systems roughly twice as bad for Black speakers as for white speakers, and Quantifying Bias in Automatic Speech Recognition found further differences by gender, age and accent. None of that is visible on a leaderboard, and not because the leaderboard is hiding it. The test sets it runs on record what was said and almost nothing about who said it.
To address this gap, we introduce two evaluation sets to the Open ASR Leaderboard: Monsoon en-IN and Monsoon hi-IN . Hindi, spoken by more than half a billion people, is the first Indic language on a multilingual tab that currently covers only European languages. Each set is released as a public split, available for self-scoring, and a private split withheld to limit benchmark-specific optimisation. The four splits are speaker-disjoint, comprising 4,888 speakers, with 12 speaker attributes recorded for each.
Design of the collection
A test set can only expose a failure mode it varies along. Most benchmarks are built from whatever audio was readily available. Monsoon was built to vary along nine axes: geography, age, gender, vocabulary, devices, acoustic environments, speech type, speech rate, and the existence of multiple valid transcripts for the same audio. Each is a way an aggregate WER can be right on average and wrong for a particular population.
The collection method follows from that.
- Geography comes from recruiting across hundreds of districts rather than recording longer sessions in fewer places.
- Devices and acoustic conditions come from contributors using their own handsets and connections, indoors and out, rather than supplied hardware in a quiet room.
- Vocabulary, speech type and speech rate come from the prompts: everyday topics that push contributors toward opinion, disagreement, narration and recall, which is where named entities, numbers and unrehearsed phrasing appear.
- Age and gender are recorded per speaker and verified.
- Multiple valid transcripts is a property of the reference rather than the audio, and it is the subject of a later section.
Dataset composition
Four splits, two languages, collected through one pipeline.
Set Language Duration Speakers Clip length (mean / median) M/F Districts States/UTs Devices Style Transcription
Monsoon en-IN public Indian English 5.62 h 1,444 9.6s / 10.4s 50/50 428 24/6 556 Conversational, spontaneous Normalised, disfluencies
Monsoon en-IN private Indian English 5.58 h 1,405 9.6s / 10.4s 45/55 420 24/6 560 Conversational, spontaneous Normalised, disfluencies
Monsoon hi-IN public Hindi 1.33 h 468 6.4s / 5.0s 54/46 202 11/3 315 Conversational, spontaneous Lattice (accepted orthographic variants)
Monsoon hi-IN private Hindi 4.47 h 1,571 6.6s / 5.3s 55/45 295 12/3 582 Conversational, spontaneous Lattice (accepted orthographic variants)
The data is sourced from unscripted dual-channel spontaneous conversations, with clips segmented from a single channel so that each clip carries one speaker. Along with the fields reported in the table, each clip also records occupation, education, marital status, income band, handset brand, current city and years in the current district.
Five clips from the public Indian English split, with the metadata each one carries: