Scaling Human and G2P Supervision for Robust Phonetic Transcription

Speech recognition identifies words. Phonetic transcription records the sounds a person makes. Pronunciation feedback, speech therapy, and accent research need those details: two people can say the same word differently.

A phonetic model learns this task from recordings paired with sound transcriptions, called labels. Grapheme-to-phoneme (G2P) software generates labels from written words, giving us a cheaper source of training data.

Machine-generated G2P labels are commonly used to pretrain phonetic models, giving them a starting point before they learn from expert transcriptions. In our Interspeech 2026 study, we find that once more than 20–30 hours of expert-labeled phonetic data are available, G2P supervision stops improving these models. Beyond this threshold, pretraining on broader automatic speech recognition (ASR) word transcripts and then fine-tuning with expert labels yields 2.3 times lower error rates than prior systems.

Background

Expert-annotated phonetic labels are scarce, so it is common to combine self-supervised pretraining with synthetic, machine-generated phonetic labels. Self-supervised pretraining learns patterns directly from recordings without requiring someone to transcribe each one. G2P generates a pronunciation from a word's spelling rather than directly from the audio. As a result, its labels can miss mispronunciations and disfluencies, such as repetitions or interrupted sounds, in non-native speech and speech affected by a medical condition.

Human labels, where an expert listens to the audio and transcribes the exact pronunciation patterns, are the gold standard, but they cannot scale as easily as G2P labels. We investigate the scaling thresholds at which G2P is advantageous for building a state-of-the-art English phonetic transcription model.

Questions

We ask how the quantity and quality of labeled training data trade off in phonetic model performance, particularly for non-standard English speech patterns. We also ask whether a threshold in the amount of expert annotation can help determine when to use G2P, ASR, and human supervision.

Contributions

Our model reduces the error rate by a factor of 2.3 over prior systems, with the largest gains on non-native speech and speech affected by aphasia, a language disorder, reaching 3.4% weighted phone feature error rate (WPFER). We also curate expert phone-labeled data into an English phonetic transcription benchmark.

We identify a supervision-quality threshold: G2P helps only below about 20–30 hours of human annotation. Beyond that point, ASR pretraining is an effective way to introduce speaker diversity without averaging out important disfluency details through approximate G2P labels.

Supervision Quality Threshold

A training curriculum is the order and mix of data used to teach a model.

Each stack in the figures below represents a training curriculum we evaluated. All start with an XLSR self-supervised model and ASR word transcripts. The final layer mixes expert and G2P labels in the proportions shown, and we compare curricula with and without an additional stage of 5,300 hours of G2P supervision.

With 20.4 hours of expert labels and 20.4 hours of G2P labels in the final stage, adding the extra G2P stage lowers WPFER from 4.60% to 4.28%. With 40.8 hours of expert labels, however, extra G2P does not provide a significant gain: WPFER is 3.40% without that stage and 3.45% with it.

Data

Our dataset combines eight speech collections, or corpora, spanning 17 dialects, including eight non-native varieties. After standardization and cleaning, it contains 55 hours of annotated speech: 41 hours for training and 14 hours for testing. The test data include 10 hours from unseen dialects. Keeping test recordings out of training lets us check whether the model can handle new speakers and ways of speaking.

Evaluation Metrics

A phone is one speech sound, and its features describe how it is produced, such as where the tongue or lips are placed.

We use WPFER, weighted phone feature error rate, based on Panphon articulatory features. It enables fairer comparison across transcriptions with different phonetic mappings by giving partial credit to similar sounds. Our confidence intervals use 2,000 bootstrap samples: we repeatedly resample the test results to estimate uncertainty. The intervals and whiskers in the figures show how precise each estimate is. Lower error rates indicate better performance.

ASR Helps on Unseen Accents

On ISLE, which contains unseen Italian and German accents, XLSR trained with expert labels achieves 3.84% WPFER. Adding ASR pretraining lowers that to 3.62%, while adding both ASR and G2P gives 3.69%. All three runs use 40.8 hours of expert labels. The figure's whiskers show 95% confidence intervals.

Limitations & Future Work

Future work should explore more low-resource languages, dialects, and types of speech affected by medical conditions. Our experiments only explore up to about 40 hours of human phonetic labels, so further work should investigate how the pattern evolves with more human data.

Human annotations also contain annotator bias: experts can disagree about the same sound, so labels reflect individual judgments too. Future work should account for annotator agreement and explore approaches such as contrastive learning to reduce this source of training noise.