Hugging Face and Voice Arena add Hindi and Indian English ASR tests

Hugging Face and Voice Arena announced new Hindi and Indian English evaluation sets for the Open ASR Leaderboard on August 28. The release adds Monsoon hi-IN and en-IN, with public data for self-scoring and private splits intended to limit benchmark-specific optimization.

Speaker coverage becomes inspectable

The four speaker-disjoint splits cover 4,888 speakers and include 12 speaker attributes. Recordings come from spontaneous conversations across varied locations and devices. Hindi references use a lattice of accepted spellings, allowing more than one valid written form of an utterance.

These details expand what an evaluator can ask beyond the aggregate word error rate. The announcement presents the metadata as a way to inspect variation between groups and recording conditions.

Investigate errors behind a close ranking

Our analysis: start with the populations and environments the application serves. Compare error rates for relevant subsets, while checking sample sizes before drawing conclusions about a small group. Very narrow slices can produce unstable rankings.

Transcript normalization should also be part of the evaluation record. Two systems can appear different because of formatting or spelling conventions rather than a meaningful difference in what they heard. Conversely, an overly broad normalization rule could erase an error that matters to a downstream workflow.

For a voice product, complement word error rate with task-specific checks. Incorrect names, quantities and destinations may have consequences that an average edit count does not capture well.

Keep private evaluation results separate from repeated development on the public split. A model selected through many public-set experiments has already been adapted to that measurement process. The new datasets provide a useful reference, while production recordings and careful subgroup analysis determine how well that reference matches a particular deployment.