Contextual E2E ASR Biasing for Command and Conversational Speech
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing end-to-end automatic speech recognition systems struggle to efficiently handle both conversational and command-and-control speech without degrading recognition performance, particularly when incorporating user-defined commands, and existing solutions either increase complexity or require additional training.
Innovation Solution
An E2E ASR system with integrated additional modules, such as a bias encoder, label encoder, and attention mechanisms, processes sub-word level tokens to handle arbitrary commands without additional training, using prefix penalties to restrict attention to contextually relevant entries.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If two different ASRs are run in parallel to handle conversational speech and C&C speech, then recognition performance for both speech types is maintained, but system complexity increases and additional switching logic is required
Solution Approach 1:
The patent merges the functionality of separate ASR systems into a single E2E ASR model that can handle both conversational speech and C&C speech. The model integrates multiple processing paths (standard processing for conversational speech and biasing processing for C&C speech) within one unified architecture, eliminating the need for parallel systems and switching logic while maintaining recognition performance for both speech types.
Solution Approach 2:
The E2E ASR model is designed with multi-functionality to handle different speech types through a single system. By incorporating configurable biasing mechanisms and adjustable processing paths, the model can adapt to handle both conversational speech and C&C speech without requiring separate specialized systems, thus reducing overall system complexity.
2Measurement precision
If external LMs are used to customize E2E ASR for C&C speech, then recognition accuracy for commands improves, but additional overhead is required and conversational speech recognition may degrade
Solution Approach 1:
The patent integrates the language modeling functionality directly into the E2E ASR model rather than using separate external LMs. The model incorporates biasing mechanisms that can be configured for C&C speech within the same architecture that handles conversational speech, eliminating the need for external LM components and reducing system overhead while maintaining accuracy for both speech types.
Solution Approach 2:
The system dynamically adjusts its processing based on the input type. The biasing configuration can be activated or deactivated depending on whether C&C speech or conversational speech is being processed, allowing the model to optimize its behavior for the current task without requiring separate static systems for each speech type.
3Reliability
If a bias list with many entries is used to handle C&C commands, then recognition of commands improves, but the number of prefixes the ASR must attend to increases, requiring masking
Solution Approach 1:
The patent applies biasing locally to specific processing paths within the E2E model rather than globally affecting all processing. The biasing mechanism can be selectively applied when C&C speech is detected, and the attention mechanism is configured to focus on relevant bias entries based on the current context, reducing the effective number of prefixes that need to be attended to at any given moment without requiring aggressive masking.
Solution Approach 2:
The attention mechanism dynamically adapts to the size of the bias list by adjusting its focus based on contextual relevance. Rather than statically limiting attention to a fixed number of prefixes, the system dynamically weights attention based on the current speech context and the relevance of bias entries, allowing efficient handling of large bias lists without uniform masking.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A contextual end-to-end automatic speech recognition (ASR) system includes: an audio encoder configured to process input audio signal to produce as output encoded audio signal; a bias encoder configured to produce as output at least one bias entry corresponding to a word to bias for recognition by the ASR system; a transcription token probability prediction network configured to produce as output a probability of a selected transcription token, based at least in part on the output of the bias encoder and the output of the audio encoder; a first attention mechanism configured to receive the at least one bias entry and determine whether the at least one bias entry is suitable to be transcribed at a specific moment of an ongoing transcription; and a second attention mechanism configured to produce prefix penalties for restricting the first attention mechanism to only entries fitting a current transcription context.