Split-Head Multilingual Speech Recognition Without Primary-Language Regression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current voice assistant systems face a challenge in maintaining accurate multilingual speech recognition, as improving support for secondary languages often leads to a regression in performance on the primary language.
Innovation Solution
A split head neural network model with a self-attention module is used, which includes shared acoustic model layers and language-specific projection layers, allowing the model to automatically weigh and combine outputs without requiring a language identifier, thereby maintaining performance on the primary language while enhancing secondary language recognition.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a single acoustic model is trained to recognize multiple languages, then support for secondary languages is improved, but performance on the primary language regresses
Solution Approach 1:
The acoustic model is segmented into multiple specialized components: a primary language acoustic model trained on primary language data, and multiple secondary language acoustic models trained on respective secondary language data. Each model specializes in its designated language, avoiding the performance degradation that occurs when a single model attempts to handle all languages. The system segments the recognition task by language, with each segment processed by its optimized model.
Solution Approach 2:
A language identification module serves as an intermediary that determines which acoustic model should process the input speech. This mediator analyzes the input and routes it to the appropriate specialized model (primary or secondary language model), ensuring that each speech input is handled by the most suitable model for optimal recognition accuracy.
2Measurement precision
If separate acoustic models are used for each language, then recognition accuracy for each language is improved, but system complexity increases
Solution Approach 1:
The system employs a universal language identification module that handles multiple languages through a single unified component. This intermediary is multi-functional, capable of identifying which of the multiple specialized acoustic models should process the input, thereby managing the complexity of having multiple models through a single versatile routing mechanism rather than requiring separate control logic for each model.
Solution Approach 2:
The system dynamically selects which acoustic model to use based on the language identification result. Rather than using a fixed model for all inputs, the system adaptively routes each input to the appropriate specialized model, allowing the model configuration to change dynamically based on the detected language while maintaining simplicity through automated selection.
3Measurement precision
If language identification is required to select the appropriate model, then recognition accuracy is improved, but processing time increases
Solution Approach 1:
The language identification module performs preliminary analysis of the input speech to determine the language type before routing to the appropriate acoustic model. This preliminary action of language detection is performed quickly using efficient algorithms, enabling the system to prepare and select the correct model in advance, minimizing the time penalty associated with model selection while maintaining high recognition accuracy.
Data Source
AI summary
Systems, methods, and computer-readable storage devices are disclosed for improved recognition of multiple languages in audio data. One method including: receiving a trained split head multilingual neural network model, the trained split head multilingual neural network model including shared acoustic model layers and a plurality of projection layers, each projection layer of the plurality of projection layers corresponding to a language that the trained split head multilingual neural network model recognizes; receiving audio data, the audio data including speech in a plurality of languages in the audio data, the speech in the plurality of languages corresponding the language recognized by a projection layer of the plurality of projection layers of the trained split head multilingual neural network model; and classifying one or more languages of the speech of the audio data using the trained split head multilingual neural network model.