Visual Speech Recognition via Phoneme Prediction and FST Decoding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing visual speech recognition systems suffer from low accuracy and require retraining for different applications, and they struggle with uncertainty in mapping lip movements to phonemes and words, especially for individuals with speech impairments.
Innovation Solution
A system utilizing a volumetric convolutional neural network with bi-directional long short-term memory layers and group normalization, decoupling phoneme prediction from word decoding, and employing a decoder with a finite state transducer and n-gram language model for improved accuracy and adaptability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If existing visual speech recognition systems are used, then the system can process video frames to predict words, but the accuracy is low and word error rates are high (92.9%)
Solution Approach 1:
The system segments the speech recognition task into distinct components: phoneme prediction from video frames, phoneme sequence generation, and word decoding using a finite state transducer. This segmentation allows each component to be optimized independently, with the phoneme prediction network focusing on accurate temporal sequence generation and the FST handling the mapping to words, thereby improving overall recognition accuracy and reducing word error rates
Solution Approach 2:
The system introduces phonemes as an intermediary between visual lip movement input and final word output. Instead of directly mapping video frames to words, the model first predicts a sequence of phonemes that represent the spoken language, then uses a finite state transducer to convert phonemes to words. This intermediary layer captures the temporal dynamics of speech more effectively and improves recognition accuracy
2Adaptability or versatility
If existing systems are used, then they can perform visual speech recognition, but they require retraining for different applications
Solution Approach 1:
The system achieves universality by decoupling the phoneme prediction model from the application-specific word decoding layer. The core phoneme prediction network trained on general speech data can be universally applied to different languages and applications by simply changing the finite state transducer and language model components, eliminating the need for complete retraining and enabling rapid adaptation to new applications
Solution Approach 2:
By segmenting the system into a general-purpose phoneme prediction component and application-specific decoding components, the system allows the time-consuming training to be performed once on the phoneme predictor, while application adaptation only requires configuring the FST and language model, dramatically reducing adaptation time
3Measurement precision
If existing systems map lip movements directly to words, then the process is straightforward, but uncertainty in mapping reduces accuracy
Solution Approach 1:
The system introduces phonemes as an intermediary representation between visual lip movements and final word output. The phoneme sequence captures the temporal dynamics and phonetic structure of speech, providing a more accurate intermediate representation that reduces mapping uncertainty. The finite state transducer then handles the deterministic mapping from phonemes to words, separating the uncertain visual-phoneme mapping from the phoneme-word conversion
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for performing visual speech recognition. In one aspect, a method comprises receiving a video comprising a plurality of video frames, wherein each video frame depicts a pair of lips; processing the video using a visual speech recognition neural network to generate, for each output position in an output sequence, a respective output score for each token in a vocabulary of possible tokens, wherein the visual speech recognition neural network comprises one or more volumetric convolutional neural network layers and one or more time-aggregation neural network layers; wherein the vocabulary of possible tokens comprises a plurality of phonemes; and determining a sequence of words expressed by the pair of lips depicted in the video using the output scores.