Audio-Visual Speech Enhancement for Far-Field Noise and Reverberation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing far-field sound pickup technologies in intelligent devices face challenges due to noise, interference, and reverberation, leading to reduced speech recognition efficiency, especially in large, open rooms with high acoustic reflection coefficients.
Innovation Solution
A signal processing method that utilizes a microphone array and a camera to determine the target sound source direction, enhances voice quality using a voice quality enhancement model based on user lip videos, and integrates audio and video information to suppress noise and interference.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If beamforming technology and echo cancellation algorithm are used to suppress ambient noise and echo, then audio signal clarity is improved, but speech recognition rate is greatly reduced due to severe reverberation and noise interference from cooking, television programs, and family chatting
Solution Approach 1:
The patent transitions from purely audio-based sound source localization to a video-assisted approach by introducing camera capture of lip movements. This adds a visual dimension to the problem-solving process, enabling the system to distinguish between actual speech and background noise by analyzing lip synchronization across multiple sound sources.
Solution Approach 2:
The patent introduces video information as an intermediary element that mediates between the audio signal and the speech recognition system. By capturing and analyzing lip movements visually, the system can verify which audio signals correspond to actual speech, thereby improving speech recognition reliability in noisy environments.
2Ease of operation
If microphone array is used to pick up audio signal from far field, then speech interaction from distance is enabled, but sound definition is greatly reduced due to noise and reverberation in large open rooms
Solution Approach 1:
The patent merges audio signal processing with video analysis by combining data from both microphone arrays and cameras. This multi-modal approach allows the system to maintain far-field speech interaction capability while compensating for sound definition loss through visual verification of lip movements.
Solution Approach 2:
The patent adds a visual dimension to far-field speech interaction by incorporating camera-based lip analysis. This enables the system to distinguish genuine speech from reverberation and noise even at distances of 5 meters or more, thereby maintaining both ease of operation and sound definition.
3Productivity
If voice quality enhancement model integrating audio and video information is used, then speech recognition efficiency is greatly improved, but device complexity increases due to integration of camera and multiple processing algorithms
Solution Approach 1:
The patent makes the electronic device multi-functional by integrating both audio pickup and video capture capabilities into a single system. The camera and microphone array work together as a unified speech recognition system, allowing the device to perform both audio and visual processing functions simultaneously.
Solution Approach 2:
The system performs self-verification by using its own camera to capture lip movements and cross-check them against detected audio sources. This self-service mechanism automatically validates speech authenticity without requiring external verification, thereby improving speech recognition efficiency while managing complexity through autonomous operation.
Data Source
AI summary
Example signal processing methods and example electronic devices are disclosed. One example method is applied to an electronic device, where the electronic device includes a microphone array and a camera. The example method includes performing sound source localization on a first audio signal obtained by using the microphone array, to obtain sound source direction information. A first video obtained by using the camera is processed to obtain user direction information. A target sound source direction is determined based on the sound source direction information and the user direction information. A user lip video is obtained in the target sound source direction by using the camera. A second audio signal is obtained by using the microphone array. A third audio signal is obtained based on the second audio signal and the user lip video by using a voice quality enhancement model.