Asynchronous Speech Recognition Decoders Synchronization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current automatic speech recognition (ASR) systems face challenges in implementing real-time, online/streaming applications due to the asynchronous nature of frame-synchronous decoding (FSD) and label-synchronous decoding (LSD) neural networks, which result in high output delays and inefficiencies, particularly in LSD-based systems that require processing entire speech utterances.
Innovation Solution
The integration of a triggered attention (TA) neural network that synchronizes FSD and LSD modules by providing alignment information from the FSD module to the LSD module, allowing the LSD module to focus attention on relevant parts of the input sequence, reducing processing requirements and output delays, and utilizing a joint scoring module to combine scores and prune prefixes for improved accuracy and efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If LSD-based ASR systems process entire speech utterances to achieve superior recognition accuracy, then word error rate improves, but output delay increases and real-time streaming capability deteriorates
Solution Approach 1:
The patent segments the speech recognition task by introducing a frame-synchronous decoder that processes audio frames independently and generates alignment information. This segmentation allows the label-synchronous decoder to focus only on relevant portions of the utterance identified by the FSD module, reducing the processing burden while maintaining accuracy. The segmentation of decoding operations into two synchronous streams enables real-time processing without requiring the entire utterance to be processed sequentially.
Solution Approach 2:
The frame-synchronous decoder acts as an intermediary between the audio input and the label-synchronous decoder. It generates alignment information that serves as a mediator, guiding the LSD module to focus on relevant parts of the input sequence. This intermediary mechanism reduces the output delay of the LSD module by providing pre-computed alignment data that eliminates the need to process the entire utterance sequentially.
2Measurement precision
If FSD and LSD neural networks are combined to achieve lower word error rates, then recognition accuracy improves, but system complexity increases due to synchronization requirements
Solution Approach 1:
The patent merges the FSD and LSD neural networks into a unified system where both decoders operate in parallel and share common components. The FSD module and LSD module are combined such that they process the same audio input simultaneously but with different decoding strategies. This merging allows the system to leverage the real-time capability of FSD and the accuracy of LSD while managing complexity through shared architecture and coordinated operation.
Solution Approach 2:
The encoder component serves multiple functions: it processes audio frames for both the FSD module and the LSD module, generating representations that are used by both decoding strategies. This multi-functionality reduces overall system complexity by eliminating redundant components. The alignment information generated by the FSD module also serves as input to the LSD module, creating a universal processing pipeline that handles both decoding tasks efficiently.
3Measurement precision
If LSD module processes the entire input sequence to generate accurate transcription, then transcription accuracy improves, but processing speed decreases and real-time streaming becomes impractical
Solution Approach 1:
The frame-synchronous decoder performs preliminary action by generating alignment information in advance for the label-synchronous decoder. This pre-computed alignment data tells the LSD module which parts of the input sequence are most relevant, allowing it to process only those portions rather than the entire sequence. This preliminary action significantly speeds up the LSD module's processing while maintaining high transcription accuracy.
Solution Approach 2:
Instead of requiring the LSD module to process the entire input sequence (excessive action), the patent enables it to process only the relevant portions identified by the FSD module (partial action). The alignment information generated by the FSD module allows the LSD module to focus its computational resources on the most informative parts of the audio signal, achieving both high accuracy and fast processing speed.
Data Source
Figure 1A
Figure 1B
Figure 1C
AI summary
A speech recognition system successively processes each encoder state of encoded acoustic features with a frame- synchronous decoder (FSD) and label- synchronous decoder (LSD) modules. Upon identifying an encoder state carrying information about new transcription output, the system expands a current list of FSD prefixes with FSD module, evaluates the FSD prefixes with LSD module, and prunes the FSD prefixes according to joint FSD and LSD scores. FSD and LSD modules are synchronized by having LSD module to process the portion of the encoder states including new transcription output identified by the FSD module and to produce LSD scores for the FSD prefixes determined by the FSD module.