Gaze-Guided Speech Enhancement for Cocktail Party Noise
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Individuals, including those with hearing loss, struggle to discern a specific speaker's voice in noisy environments due to the 'cocktail party effect', where existing technologies fail to accurately identify and enhance one voice amidst multiple speakers and background noise.
Innovation Solution
Utilizing a user's gaze direction and image analysis to determine the active speaker, combined with beamforming algorithms and machine learning models, to enhance and isolate the speaker's audio, and optionally display enhanced audio and visual cues on XR glasses.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a microphone array with narrow pickup angle is used to focus on a specific audio source, then audio directionality is improved, but device size and complexity increase
Solution Approach 1:
The patent replaces the mechanical solution of using physically narrow pickup angle microphones (which require large, cumbersome arrays) with an optical-based approach. Image sensors capture visual information about speaker locations and orientations, which then guides audio processing. This substitution of mechanical acoustic filtering with optical sensing and computational processing achieves precise audio directionality without the physical complexity of large microphone arrays.
Solution Approach 2:
The patent introduces image sensors and visual processing as an intermediary between the user and the audio sources. Instead of directly using microphones to spatially filter audio, the system first captures images to identify which speaker the user is looking at, then uses this visual information to guide audio enhancement. This intermediary visual processing step simplifies the audio processing requirements while maintaining precise target selection.
2Measurement precision
If image analysis is used to identify the active speaker, then speaker identification accuracy is improved, but processing power requirements increase
Solution Approach 1:
The patent performs preliminary action by using image sensors to capture and analyze visual information about speakers before processing the audio. By identifying the target speaker through image analysis (such as detecting gaze direction, facial orientation, or lip movement), the system pre-sorts through potential audio sources visually, which simplifies subsequent audio processing. This preliminary visual identification reduces the computational burden on audio processing compared to analyzing all audio sources simultaneously.
3Use of energy by moving object
If traditional hearing aids amplify all surrounding sounds, then overall volume is improved, but ability to distinguish specific speakers deteriorates
Solution Approach 1:
The patent applies local quality by enhancing audio selectively based on spatial and visual information. Instead of uniformly amplifying all sounds, the system identifies the specific speaker the user is looking at through image analysis and applies enhancement primarily to that speaker's audio channel. This localized audio enhancement preserves speaker distinction while providing amplification, allowing users to hear the target speaker clearly without being overwhelmed by all surrounding sounds equally.
Data Source
AI summary
A method and device for enhancing speech audio signals of an individual in a noisy environment based on a user's gaze and a captured image of the user's environment. A direction of a user's gaze is determined using image sensors configured to capture an orientation of a user's eyes and an image of the user environment is captures. Spatial audio is captured and analyzed along with the direction of gaze and image of the user environment to enhance audio of an active speaker.