Multilingual Speech Recognition With Script Normalization for Code-Switching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional multilingual ASR systems face challenges in extending language support to newer languages and handling code-switching due to their dependency on language information, particularly in Indic languages where code-switching is common, leading to inefficiencies in modeling and data sharing.
Innovation Solution
A language-agnostic multilingual end-to-end (E2E) speech recognition model is trained using transliterated text in a target script, without providing any language information, by transliterating native language transcriptions into a common script like Latin, and using sequence-to-sequence neural networks such as RNN-T, to predict speech recognition results across multiple languages.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional multilingual ASR systems use separate language-specific language models for each language, then language recognition accuracy is improved, but device complexity and memory constraints increase
Solution Approach 1:
The patent implements a universal language-agnostic E2E model that can process multiple languages without requiring separate language-specific models. The model uses a shared acoustic model and transliteration-based approach to handle different languages uniformly, eliminating the need for multiple separate LMs and reducing system complexity while maintaining multilingual capability
Solution Approach 2:
The patent extracts and removes language-specific information from the model processing pipeline. By using transliteration to convert all languages into a common script representation and training the model in a language-agnostic manner, the system eliminates the dependency on language-specific models while preserving the ability to recognize and transcribe multiple languages accurately
2Measurement precision
If conventional multilingual ASR systems use multiple separate models for different languages, then language-specific accuracy is improved, but loss of time and computational efficiency worsen
Solution Approach 1:
The patent merges the acoustic model, pronunciation model, and language models into a single end-to-end neural network. This unified E2E model processes all languages through a single architecture rather than switching between multiple separate models, significantly reducing computational overhead and improving processing efficiency while maintaining language-specific accuracy through the transliteration approach
3Measurement precision
If multilingual E2E models are conditioned on language information, then ability to track language switches is improved, but adaptability to new languages and handling of code-switching worsens
Solution Approach 1:
Instead of conditioning the model on language information to track language switches, the patent inverts the approach by making the model language-agnostic. The system uses transliteration to convert all languages into a common representation, allowing the model to handle language switches and code-switching naturally without explicit language conditioning, thereby improving adaptability to new languages and code-switching scenarios
Data Source
AI summary
A method includes obtaining a plurality of training data sets each associated with a respective native language and includes a plurality of respective training data samples. For each respective training data sample of each training data set in the respective native language, the method includes transliterating the corresponding transcription in the respective native script into corresponding transliterated text representing the respective native language of the corresponding audio in a target script and associating the corresponding transliterated text in the target script with the corresponding audio in the respective native language to generate a respective normalized training data sample. The method also includes training, using the normalized training data samples, a multilingual end-to-end speech recognition model to predict speech recognition results in the target script for corresponding speech utterances spoken in any of the different native languages associated with the plurality of training data sets.