Speaker-Conditioned Voice Filtering for Overlapping Speech Isolation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing automated assistants struggle to accurately isolate and process voice data from a single speaker amidst background noise and overlapping voices, leading to inaccuracies in voice-to-text conversion and interactive dialog responses.
Innovation Solution
A trained voice filter model generates a predicted mask using speaker embeddings to isolate the voice of a single speaker by processing audio spectrograms, employing convolutional and recurrent neural networks to refine the audio data and remove background noise.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional speech recognition processes audio data directly, then processing speed is maintained, but voice isolation accuracy deteriorates due to background noise and overlapping voices
Solution Approach 1:
The patent segments the audio processing task into distinct stages: first generating a spectrogram from the audio signal, then creating a predicted mask to isolate the target speaker's voice, and finally converting the masked spectrogram back to audio. This segmentation allows each stage to specialize in a specific function, improving voice isolation accuracy while managing processing complexity through modular architecture.
Solution Approach 2:
The patent introduces intermediate representations (spectrogram and predicted mask) as mediators between the raw audio data and the final isolated voice output. The spectrogram transforms audio into a visual frequency-time representation, and the predicted mask acts as a filter that selectively enhances target speaker components. These intermediaries enable more accurate voice isolation by operating in a transformed domain where speaker characteristics are more distinguishable.
2Measurement precision
If speaker embeddings are used to condition the voice filter model, then voice isolation precision is improved, but computational requirements increase
Solution Approach 1:
The patent performs preliminary extraction of speaker embeddings from audio data before using them to condition the voice filter model. By pre-computing these embeddings that capture speaker-specific characteristics, the system prepares targeted information in advance that guides the masking process. This preliminary action improves speaker identification accuracy while allowing the main processing to focus on voice isolation rather than speaker analysis.
Data Source
AI summary
Techniques are disclosed that enable processing of audio data to generate one or more refined versions of audio data, where each of the refined versions of audio data isolate one or more utterances of a single respective human speaker. Various implementations generate a refined version of audio data that isolates utterance(s) of a single human speaker by processing a spectrogram representation of the audio data (generated by processing the audio data with a frequency transformation) using a mask generated by processing the spectrogram of the audio data and a speaker embedding for the single human speaker using a trained voice filter model. Output generated over the trained voice filter model is processed using an inverse of the frequency transformation to generate the refined audio data.