Cascaded Audiovisual Speech Recognition for Missing Video Robustness
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional audiovisual automatic speech recognition (AV-ASR) models perform poorly when video is missing, often degrading to worse performance than audio-only ASR models, and are not robust to the absence of video during inference.
Innovation Solution
A cascaded AV-ASR model architecture that includes an audio encoder and an audiovisual encoder stacked together, allowing it to operate in both audiovisual and acoustic-only modes, maintaining performance by routing audio-only processing when video is absent.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a conventional AV-ASR model is designed to process both audio and video inputs, then transcription accuracy is improved when video is available, but performance degrades when video is missing
Solution Approach 1:
The model is segmented into two separate encoder components: an audio encoder that processes acoustic frames and an audiovisual encoder that processes video frames. These encoders operate independently and their outputs are combined, allowing the system to function with or without video input while maintaining transcription accuracy.
2Device complexity
If an AV-ASR model uses a unified architecture for both audio and audiovisual processing, then device complexity is reduced, but adaptability to different input conditions deteriorates
Solution Approach 1:
The audio encoder serves a dual purpose: it processes audio inputs in both audio-only and audiovisual modes. The audiovisual encoder is selectively activated based on video availability. This multi-functionality allows the model to adapt to different input conditions without requiring completely separate architectures.
Solution Approach 2:
The model dynamically adjusts its processing path based on video availability. When video is present, both audio and audiovisual encoders are activated; when video is absent, only the audio encoder is used. This dynamic adaptation maintains performance across varying conditions while managing complexity.
Data Source
AI summary
A method includes receiving a sequence of acoustic frames and generating, by an audio encoder, at each of a plurality of output steps, an acoustic higher-order feature representation for a corresponding acoustic frame in the sequence of acoustic frames. For each acoustic frame in the sequence of acoustic frames paired with a corresponding video frame, the method includes generating, by an audiovisual encoder, an audiovisual higher-order feature representation for the corresponding acoustic higher-order feature frame and the corresponding video frame; and generating, by a joint network, at an output step, a probability distribution over possible speech recognition hypotheses based on the audiovisual higher-order feature representation. The method, for each corresponding acoustic frame in the sequence of acoustic frames not paired with a corresponding video frame, includes generating, by the joint network, at an output step, a probability distribution over possible speech recognition hypotheses based on the acoustic higher-order feature representation.