Visual Speech Recognition With CTC Word Boundary Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for visual speech recognition fail to accurately detect word boundaries and rely on audio information in lip reading techniques do not perform well in existing environments.
Innovation Solution
The technical solution is to use a combination of a sentence or phrase using a neural network to detect word boundaries in a sequence of word boundaries in a sequence of a sentence or phrase using a combination of a neural network to detect word boundaries in a sequence of characters that correspond to lip movements of the video clips in a sequence of video clips, based on connectionist temporal classification loss.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional lip reading techniques are used, then audio information can be utilized for speech recognition, but word boundary detection accuracy deteriorates
Solution Approach 1:
The patent segments the speech recognition task into two independent components: phoneme recognition from visual lip movements and word boundary detection from temporal patterns in the phoneme sequence. This segmentation allows accurate word boundary detection without relying on audio information, as the model identifies boundaries through transitions in phoneme probability distributions over time
Solution Approach 2:
The patent replaces the acoustic signal processing mechanism with a visual mechanism. Instead of using audio waveforms and spectral analysis to detect word boundaries, the system uses a CNN-LSTM architecture that processes visual features from lip movements and temporal patterns in phoneme sequences to identify word boundaries
2Adaptability or versatility
If visual speech recognition is implemented, then audio independence is achieved, but detection precision of lip movements deteriorates
Solution Approach 1:
The patent merges multiple visual feature extraction approaches by combining CNN-based spatial feature extraction from lip regions with temporal feature extraction from phoneme sequences. This combination of spatial and temporal processing enhances lip movement detection accuracy while maintaining audio independence
Solution Approach 2:
The patent performs preliminary phoneme recognition from visual lip movements before conducting word boundary detection. The LSTM network processes the temporal sequence of phoneme probabilities to identify word boundaries, building upon the previously extracted visual phoneme features to achieve accurate detection without audio input
3Reliability
If conventional speech-to-text models are used, then audio processing is simplified, but performance in noisy environments deteriorates
Solution Approach 1:
The patent replaces the audio processing mechanism with visual processing. Instead of analyzing acoustic signals that are degraded by noise, the system processes visual information from lip movements and facial expressions, which are inherently more robust to acoustic noise environments
Solution Approach 2:
The patent applies local quality enhancement by focusing the visual analysis on specific regions of the face (lips, mouth, jaw) that provide the most informative cues for speech recognition. The CNN architecture selectively processes local visual features from these key regions rather than the entire face, improving reliability in noisy environments
Data Source
Figure 1
Figure 2
Figure 3
AI summary
An electronic apparatus and method for visual speech recognition based on connectionist temporal classification (CTC) loss is disclosed. The electronic apparatus receives a video that includes human speakers and generates a prediction corresponding to lip movements of the human speakers. The prediction is generated based on application of a Deep Neural Network (DNN) on the video and the DNN is trained using a CTC loss function. The electronic apparatus detects, based on the prediction, word boundaries in a sequence of characters that correspond to the lip movements and divides the video into a sequence of video clips based on the detection. Each video clip corresponds to a word spoken by the human speakers. The electronic apparatus generates a sequence of word predictions by processing the sequence of video clips and generates a sentence, or a phrase based on the generated sequence of word predictions.