Streaming ASR Partial Hypothesis Re-Ranking for Flicker Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Streaming automatic speech recognition (ASR) systems experience flickering in partial transcriptions due to repeatedly selecting the highest-scored hypothesis in each beam search instance, leading to unstable and distracting displays that increase cognitive load and latency in downstream systems.
Innovation Solution
Implement partial hypothesis re-ranking by adjusting speech recognition scores based on whether the hypothesis shares a prefix with the previous transcription, using a flicker reduction rescorer to select the hypothesis with the highest re-ranked score, while maintaining the beam search scores unchanged.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If streaming ASR systems generate partial transcriptions in real-time, then latency is reduced and responsiveness is improved, but flickering occurs where transcriptions change frequently
Solution Approach 1:
The system performs preliminary action by generating multiple candidate hypotheses during the beam search process and storing them in a lattice structure before final selection. This allows the system to have pre-computed alternatives ready to reduce flickering when transcription changes occur, while maintaining fast streaming operation.
Solution Approach 2:
The system implements feedback by monitoring whether selected hypotheses share a prefix with previous transcriptions and using this information to adjust scoring. When a hypothesis does not share the required prefix, the system penalizes it in the re-ranking process, creating a feedback loop that stabilizes transcriptions while maintaining speed.
2Stability of the object's composition
If the system re-ranks all hypotheses to reduce flickering, then transcription stability improves, but computational complexity increases
Solution Approach 1:
The system applies partial action by re-ranking only the top K hypotheses from the beam search rather than all possible hypotheses. This selective re-ranking approach reduces computational complexity significantly while still providing sufficient flicker reduction, as only the most likely candidates need to be evaluated for prefix matching.
Solution Approach 2:
The system implements local quality by applying different scoring criteria to different hypotheses based on their properties. Hypotheses that share the required prefix receive boosted scores, while those that don't share the prefix are penalized. This localized differentiation allows stable transcriptions without re-ranking the entire hypothesis space.
3Stability of the object's composition
If the system penalizes hypotheses that don't share prefix, then flickering is reduced, but accuracy of final transcription may be affected
Solution Approach 1:
The system segments the transcription process into two distinct phases: beam search generation phase where all hypotheses are explored without penalty, and re-ranking selection phase where prefix matching is enforced. This segmentation allows the system to maximize both accuracy (in generation) and stability (in selection) without compromising either aspect.
Solution Approach 2:
The system applies dynamics by making the scoring function adaptive rather than static. The re-ranking score adjusts based on whether the hypothesis shares the required prefix with previous transcriptions, dynamically penalizing or rewarding hypotheses accordingly. This dynamic scoring maintains accuracy by preserving high-scoring hypotheses while ensuring stability through prefix consistency.
Data Source
AI summary
A method includes processing, using a speech recognizer, a first portion of audio data to generate a first lattice, and generating a first partial transcription for an utterance based on the first lattice. The method includes processing, using the recognizer, a second portion of the data to generate, based on the first lattice, a second lattice representing a plurality of partial speech recognition hypotheses for the utterance and a plurality of corresponding speech recognition scores. For each particular partial speech recognition hypothesis, the method includes generating a corresponding re-ranked score based on the corresponding speech recognition score and whether the particular partial speech recognition hypothesis shares a prefix with the first partial transcription. The method includes generating a second partial transcription for the utterance by selecting the partial speech recognition hypothesis of the second plurality of partial speech recognition hypotheses having the highest corresponding re-ranked score.