CTC-Based Visual Speech Recognition for Audio-Free Word Boundaries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional visual speech recognition techniques struggle with accurately detecting word boundaries in lip movements, often relying on audio information which can be unreliable in noisy environments or when enunciation is poor.
Innovation Solution
An electronic apparatus utilizing a Deep Neural Network (DNN) trained with a combination of Connectionist Temporal Classification (CTC) loss and cross-entropy loss functions to analyze lip movements, enabling the detection of word boundaries directly from video frames without relying on audio, and dividing the video into clips corresponding to spoken words.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional lip reading techniques are used, then the system can process visual speech input, but the detection of word boundaries is inaccurate and relies on audio information
Solution Approach 1:
The patent segments the visual speech recognition task into two distinct components: character recognition and word boundary detection. By introducing a special boundary token in the output vocabulary, the model can identify word boundaries independently from character recognition, eliminating the need for audio-based boundary detection and improving accuracy in visual-only conditions.
Solution Approach 2:
The patent extends the output space of the CTC model by adding a boundary token dimension. This dimensional extension allows the model to simultaneously output characters and word boundary markers, transforming the detection task from a sequential two-step process into a unified multi-dimensional classification problem that operates purely on visual input.
2Measurement precision
If audio information is used to extract word boundaries, then boundary detection may be achieved, but the system fails in noisy environments or when enunciation is poor
Solution Approach 1:
The patent extracts and removes the dependency on audio information for word boundary detection. By designing a visual-only CTC model that outputs boundary tokens directly from video frames, the system eliminates the harmful factor of audio noise and poor enunciation, achieving robust boundary detection that works reliably in challenging acoustic environments.
Solution Approach 2:
The patent substitutes the audio-based boundary detection mechanism with a visual-based mechanism. Instead of using acoustic signals to identify word boundaries, the model processes visual features from lip movements and facial expressions, replacing the acoustic detection pathway with a visual one that is immune to noise and enunciation quality issues.
3Adaptability or versatility
If a Deep Neural Network with CTC loss is used, then the model can process variable-length sequences, but training convergence and accuracy need improvement
Solution Approach 1:
The patent employs a composite loss function that combines CTC loss with character-level cross-entropy loss. This composite approach leverages the strengths of both loss functions: CTC provides temporal alignment and variable-length sequence handling, while cross-entropy ensures accurate character-level prediction. The combination accelerates convergence and improves overall accuracy compared to using CTC loss alone.
Solution Approach 2:
The patent implements a dynamic training strategy where the model alternates between CTC-based temporal alignment learning and cross-entropy-based character recognition learning. This dynamic training approach allows the model to adaptively balance between understanding sequence timing and mastering precise character prediction, leading to better overall performance on variable-length speech sequences.
Data Source
AI summary
An electronic apparatus and method for visual speech recognition based on connectionist temporal classification (CTC) loss is disclosed. The electronic apparatus receives a video that includes human speakers and generates a prediction corresponding to lip movements of the human speakers. The prediction is generated based on application of a Deep Neural Network (DNN) on the video and the DNN is trained using a CTC loss function. The electronic apparatus detects, based on the prediction, word boundaries in a sequence of characters that correspond to the lip movements and divides the video into a sequence of video clips based on the detection. Each video clip corresponds to a word spoken by the human speakers. The electronic apparatus generates a sequence of word predictions by processing the sequence of video clips and generates a sentence, or a phrase based on the generated sequence of word predictions.