Multilingual On-Device Streaming ASR With Integrated Utterance Endpointing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
On-device end-to-end speech recognition models for multilingual speech recognition are not suitable for streaming applications due to their large model size and complexity, leading to inefficiencies in managing and updating, and they struggle to accurately determine the end of an utterance across different languages.
Innovation Solution
A multilingual ASR system with an encoder that generates higher-order feature representations, a joint network for probability distribution over speech recognition hypotheses, and a prediction network for end-of-utterance token prediction, along with a microphone closer to trigger audio capture termination based on these predictions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a multilingual end-to-end speech recognition model is used to recognize several different languages, then language versatility is improved, but model size increases making it unsuitable for on-device streaming applications
Solution Approach 1:
The multilingual speech recognition model is segmented into a shared base model and multiple language-specific adapter modules. The base model processes acoustic features and generates encoder states, while separate adapter modules handle different languages. This segmentation allows the system to maintain a compact core model while providing multilingual capability through lightweight, interchangeable adapter components that can be loaded as needed.
Solution Approach 2:
The base speech recognition model is designed as a universal component that can process and recognize speech from multiple languages through a shared architecture. Language-specific adaptations are achieved through configurable adapter modules that interface with the universal base model, allowing a single system to serve multiple language recognition functions without requiring separate full-scale models for each language.
2Reliability
If a multilingual end-to-end speech recognition model is used, then multilingual speech recognition quality is improved, but device complexity and management difficulty increase
Solution Approach 1:
The model architecture is segmented into a stable base model and interchangeable adapter modules for different languages. This segmentation simplifies management by allowing individual adapters to be developed, tested, and updated independently without affecting the core model or other language adapters, reducing overall system complexity despite multilingual capability.
Solution Approach 2:
The system manages multilingual capability through parameter changes rather than structural changes. Language-specific parameters and adapter configurations allow the same base model architecture to adapt to different languages, simplifying deployment and management compared to maintaining entirely separate models for each language while preserving speech recognition quality.
3Measurement precision
If endpointing is integrated into the E2E model, then utterance end detection accuracy is improved, but model size increases
Solution Approach 1:
The endpointing detection functionality is merged with the speech recognition model by integrating an endpointing adapter module that works in conjunction with the base model and language adapters. This unified approach allows utterance end detection to leverage the same acoustic feature processing and encoder states as speech recognition, improving accuracy while avoiding the need for a completely separate endpointing model that would increase overall system size.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method (500) includes receiving a sequence of acoustic frames (110) characterizing one or more utterances (106) as input to a multilingual automated speech recognition (ASR) model (200). The method also includes generating a higher order feature representation (204) for a corresponding acoustic frame. The method also includes generating a hidden representation (355) based on a sequence of non-blank symbols output (222) by a final softmax layer (240). The method also includes generating a probability distribution over possible speech recognition hypotheses based on the hidden representation and the higher order feature representation. The method also includes predicting an end of utterance (EOU) token (232) at an end of each utterance. The method also includes classifying each acoustic frame as either speech, initial silence, intermediate silence, or final silence.