Depth-Camera Lip Fusion for In-Vehicle Speech Recognition
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In-vehicle speech recognition systems face challenges in maintaining accuracy due to environmental noise variations, which can lead to navigation errors and increased driving risk.
Innovation Solution
A method that combines facial depth image analysis and voice feature extraction using a depth camera and audio processing to fuse mouth shape and voice features, enhancing speech recognition accuracy by utilizing a convolutional neural network and audio-video feature fusion.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If voice interaction is used in in-vehicle environment, then convenience and safety are improved, but speech recognition accuracy deteriorates due to environmental noise
Solution Approach 1:
The patent combines audio signal processing with visual lip movement detection to create an audio-video fusion recognition system. The mouth shape feature from depth images is integrated with voice features through feature fusion modules, allowing the system to leverage both auditory and visual cues for improved recognition accuracy in noisy in-vehicle environments
Solution Approach 2:
The patent introduces depth camera technology as an intermediary to capture mouth shape information that is independent of environmental light conditions. This intermediary visual channel provides reliable lip movement data even when audio quality degrades due to noise, serving as a complementary recognition pathway
2Device complexity
If audio-only recognition is used, then device complexity is reduced, but recognition accuracy deteriorates in noisy environments
Solution Approach 1:
The system merges audio processing pipelines with visual processing pipelines by integrating depth camera data capture and processing. Multiple feature extraction modules (audio features, mouth shape features) are combined through fusion mechanisms to create a more robust recognition system that outperforms audio-only approaches
3Measurement precision
If frequency band separation is applied, then audio recognition accuracy is improved, but processing time increases
Solution Approach 1:
The audio signal is segmented into multiple frequency bands using filter banks, allowing parallel processing of different frequency components. This segmentation enables the system to focus computational resources on relevant frequency ranges and improves recognition accuracy by analyzing spectral characteristics across multiple bands simultaneously
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
Improves speech recognition accuracy by leveraging mouth shape features from facial depth images, unaffected by environmental light, and separates frequency bands to enhance audio recognition, reducing errors and improving user safety.
Implementation Method 1
The face of the user emits an infrared light pulse and obtains an infrared light pulse reflected by the face of the user. A time difference is calculated based on a change between phases of the emitted and reflected infrared light pulses. Depth information of the face of the user is calculated with reference to a propagation speed of the infrared light pulse and the time difference.
Data Source
AI summary
Embodiments of this application provide a speech recognition method. The speech recognition method includes: obtaining a facial depth image and a to-be-recognized voice of a user, where the facial depth image is an image collected by using a depth camera; recognizing a mouth shape feature from the facial depth image, and recognizing a voice feature from a to-be-recognized audio; and fusing the voice feature and the mouth shape feature into an audio-video feature, and recognizing, based on the audio-video feature, a voice uttered by the user.