Face-Guided Multi-Channel Beamforming for Distractor Audio Separation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Adaptive beamformers struggle to distinguish between target and distractor audio sources in far-field applications, leading to ineffective suppression of background noise when multiple speakers are present.
Innovation Solution
A speech enhancement system that integrates face detection from a camera with multi-channel beamforming, using image-based face detection to verify and adjust beam directions, ensuring the beam is aligned with the target audio source.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If adaptive beamforming is used to enhance speech quality, then signal-to-noise ratio is improved, but the ability to distinguish between target and distractor audio sources deteriorates in far-field applications
Solution Approach 1:
The patent combines audio signal processing with visual face detection to create a multi-modal system. The beamformer processes audio signals while the face detection system simultaneously identifies target and distractor speakers, allowing the system to distinguish between them by cross-referencing visual and acoustic information.
Solution Approach 2:
The face detection system acts as an intermediary that provides visual context to disambiguate audio sources. By detecting faces and their orientations, the system can identify which speaker is the target and which are distractors, then use this information to guide the beamforming process.
2Measurement precision
If adaptive beamforming suppresses background noise, then speech quality is improved, but distortion of target audio sources increases when multiple speakers are present
Solution Approach 1:
The system dynamically adjusts beamforming parameters based on real-time face detection results. When multiple speakers are detected, the system can selectively steer the beam toward the target speaker's direction while suppressing other speakers, adapting the beamforming weights based on visual information.
Solution Approach 2:
The patent applies different processing qualities to different spatial regions. By identifying the target speaker's location through face detection, the system concentrates processing resources and beamforming energy specifically toward the target direction while applying stronger suppression to distractor directions.
3Reliability
If face detection is integrated with beamforming, then separation of target and distractor audio sources is improved, but device complexity increases
Solution Approach 1:
The system uses a camera for multiple purposes: it captures images for face detection, determines speaker orientations, and provides spatial information for beamforming. This multi-functional use of the camera reduces the need for additional dedicated sensors.
Solution Approach 2:
The system creates a visual copy or representation of the audio scene through face detection and orientation estimation. This visual model serves as a reference that guides the audio processing, allowing the system to separate speakers based on correspondence between visual and acoustic spatial information.
Data Source
AI summary
This disclosure provides methods, devices, and systems for speech enhancement. The present implementations more specifically relate to utilizing multiple modalities to suppress audio originating from a distractor audio source without distorting audio originating from a target audio source. In some aspects, a speech enhancement system may receive a multi-channel audio signal via a microphone array and may further receive an image associated with a respective frame of the audio signal. The speech enhancement system detects one or more target faces in the image and determines whether the audio frame originates from a target audio source. For example, the speech enhancement system may compare a respective direction of each target face with a direction-of-arrival (DOA) of the audio frame. The speech enhancement system may selectively steer a beam associated with a multi-channel beamformer toward the DOA of the audio frame based on whether the audio frame originates from a target face.