Split-Head Multilingual Speech Recognition Without Language ID
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing voice assistant systems face a challenge in maintaining accurate multilingual speech recognition, as improving support for secondary languages often leads to a regression in performance on the primary language.
Innovation Solution
A split head neural network model with a self-attention module is employed, which includes shared acoustic model layers and language-specific projection layers, allowing the model to automatically weigh and combine outputs without requiring a language identifier, thereby maintaining performance on the primary language while enhancing accuracy on secondary languages.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a single acoustic model is trained to recognize multiple languages, then support for secondary languages is improved, but performance on the primary language regresses
Solution Approach 1:
The acoustic model is segmented into multiple specialized models, each trained on a specific language. This allows each model to maintain high accuracy for its target language while the system as a whole supports multiple languages through model selection based on language identification
Solution Approach 2:
A language identification component acts as an intermediary between the audio input and the acoustic models. It first determines which language is being spoken, then routes the input to the appropriate specialized model, thereby preserving primary language accuracy while enabling multilingual support
2Adaptability or versatility
If language-specific projection layers are used without self-attention, then language recognition capability is improved, but model complexity increases and requires language identifier selection
Solution Approach 1:
The self-attention mechanism enables the model to automatically weigh and select relevant language-specific projections without external intervention. The attention weights are computed dynamically based on the input, allowing the model to self-determine which language projection to apply without requiring a separate language identifier or manual selection process
Solution Approach 2:
Multiple language-specific projection layers are merged into a unified structure with shared acoustic model layers. The self-attention mechanism combines outputs from different projection layers by computing attention weights, effectively merging the capabilities of multiple language models into a single cohesive system that handles multiple languages simultaneously
Data Source
AI summary
Systems, methods, and computer-readable storage devices are disclosed for improved recognition of multiple languages in audio data. One method including: receiving a trained split head multilingual neural network model, the trained split head multilingual neural network model including shared acoustic model layers and a plurality of projection layers, each projection layer of the plurality of projection layers corresponding to a language that the trained split head multilingual neural network model recognizes; receiving audio data, the audio data including speech in a plurality of languages in the audio data, the speech in the plurality of languages corresponding the language recognized by a projection layer of the plurality of projection layers of the trained split head multilingual neural network model; and classifying one or more languages of the speech of the audio data using the trained split head multilingual neural network model.