Sequence-to-Sequence Speech Recognition Latency Threshold
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing automatic speech recognition (ASR) systems face a tradeoff between low latency and low word error rate (WER), with real-time ASR applications often experiencing delays that make them cumbersome and previous attempts to reduce latency increasing WER.
Innovation Solution
A computing system utilizing a sequence-to-sequence speech recognition model with an external alignment model, encoder neural network, and decoder neural network to generate text transcriptions, where each output text token has a latency below a predetermined threshold, allowing for real-time transcription while maintaining low WER.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If real-time ASR is performed to reduce latency, then the delay between input and output is reduced, but the word error rate increases
Solution Approach 1:
The audio input is divided into multiple frames that are processed sequentially through the encoder neural network. The encoder processes frames in order and generates hidden states that are passed to the decoder, allowing the system to produce partial transcriptions at intermediate points while maintaining overall accuracy. This segmentation enables real-time output without sacrificing word error rate by processing audio in manageable chunks rather than requiring complete audio input before transcription.
Solution Approach 2:
The encoder neural network serves as an intermediary that processes audio frames and generates hidden states, which then serve as input to the decoder. This intermediary structure allows the system to maintain temporal relationships in the audio data while enabling incremental transcription output. The encoder-decoder architecture acts as a mediator between the raw audio input and the final transcription output, allowing real-time processing with controlled error rates.
2Reliability
If traditional ASR methods are used to ensure accuracy, then the word error rate is low, but the processing speed and real-time performance decrease
Solution Approach 1:
The encoder neural network performs preliminary processing of audio frames by extracting features and generating hidden states that capture the essential information. This preliminary action prepares the data in advance for the decoder, which then generates the transcription. By performing this preliminary feature extraction and temporal relationship modeling in the encoder, the system enables faster decoding and transcription generation without losing accuracy.
Solution Approach 2:
The encoder-decoder architecture maintains continuous processing of audio frames through time. As each frame is processed by the encoder, the resulting hidden states are immediately available to the decoder for transcription generation. This continuous action allows the system to produce transcriptions incrementally as audio is received, maintaining both high processing speed and low word error rate by never interrupting the processing flow.
Data Source
AI summary
A computing system including one or more processors configured to receive an audio input. The one or more processors may generate a text transcription of the audio input at a sequence-to-sequence speech recognition model, which may assign a respective plurality of external-model text tokens to a plurality of frames included in the audio input. Each external-model text token may have an external-model alignment within the audio input. Based on the audio input, the one or more processors may generate a plurality of hidden states. Based on the plurality of hidden states, the one or more processors may generate a plurality of output text tokens. Each output text token may have a corresponding output alignment within the audio input. For each output text token, a latency between the output alignment and the external-model alignment may be below a predetermined latency threshold. The one or more processors may output the text transcription.