Lip Detection for Voice Query Endpointing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Automated speech recognition systems often struggle to accurately endpoint audio data in noisy environments, leading to misrecognition of voice queries due to background noise, resulting in false positives and missed detections.
Innovation Solution
The system uses synchronized video data, specifically detecting lip movement, to differentiate between speech and non-speech inputs, determining the start and end points of voice queries, and truncating audio segments accordingly for improved transcription accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If audio data is processed using traditional speech recognition techniques, then the system can recognize voice queries, but the accuracy deteriorates in noisy environments due to background noise causing false positives and missed detections
Solution Approach 1:
The patent transitions from unidimensional audio processing to bidimensional audio-video synchronization. By incorporating video data capturing lip movements alongside audio data, the system creates a new dimension of information. The lip movement detection provides visual verification of speech portions, enabling the system to distinguish actual speech from background noise more reliably in noisy environments.
Solution Approach 2:
The patent introduces lip movement detection as an intermediary verification mechanism between audio input and speech recognition output. The video processing component acts as a mediator that validates audio segments by checking for corresponding lip movements, thereby filtering out false positives caused by background noise before they reach the speech recognition system.
2Reliability
If the system processes all audio data as potential speech, then no speech portions are missed, but false positives increase due to background noise being misrecognized as speech
Solution Approach 1:
The patent implements a feedback mechanism where video data of lip movements is used to verify and validate audio segments. The system processes both audio and video data, uses lip movement detection to identify genuine speech portions, and feeds this verification information back to confirm or reject audio segments as valid speech. This feedback loop significantly reduces false positives while maintaining high detection accuracy.
Solution Approach 2:
The patent merges audio data processing with video data processing into a unified speech verification system. By combining the temporal information from audio with the visual confirmation from lip movements, the system creates a more robust speech detection mechanism that reduces false positives while maintaining comprehensive speech coverage.
3Measurement precision
If the system uses only audio data for speech recognition, then the device complexity is low, but the measurement precision of speech portions deteriorates in noisy conditions
Solution Approach 1:
The patent adds a visual dimension to speech detection by incorporating video processing. The system captures video data, extracts lip movement features, and synchronizes this temporal information with audio data. This additional dimension provides precise endpoint detection for speech portions by identifying when lip movements begin and end, significantly improving measurement precision over audio-only methods.
Data Source
AI summary
Systems and methods are described for improving endpoint detection of a voice query submitted by a user. In some implementations, a synchronized video data and audio data is received. A sequence of frames of the video data that includes images corresponding to lip movement on a face is determined. The audio data is endpointed based on first audio data that corresponds to a first frame of the sequence of frames and second audio data that corresponds to a last frame of the sequence of frames. A transcription of the endpointed audio data is generated by an automated speech recognizer. The generated transcription is then provided for output.