Speaker-Conditioned Spectrogram Masking for Single-Voice Separation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing automated assistants struggle to accurately isolate and process voice data from a single speaker amidst background noise and multiple speakers, leading to inaccuracies in voice-to-text conversion and interactive dialog systems.
Innovation Solution
A trained voice filter model generates a predicted mask using speaker embeddings to isolate the voice of a single speaker by processing audio spectrograms, employing convolutional and recurrent neural networks to refine the audio data and remove background noise.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional speech recognition processes audio data directly, then processing speed is maintained, but voice isolation accuracy deteriorates due to background noise and multiple speakers
Solution Approach 1:
The system segments the audio processing task into distinct stages: first generating speaker embeddings from audio data, then creating spectrograms, applying predicted masks to isolate specific speakers, and finally converting to text. This segmentation allows each component to specialize in one aspect of voice isolation, improving accuracy while managing complexity through modular architecture
Solution Approach 2:
The system introduces intermediate representations (speaker embeddings, spectrograms, and predicted masks) between the raw audio data and final text output. These intermediaries enable sophisticated voice isolation by transforming audio into frequency domain representations that can be selectively masked, thereby improving voice isolation accuracy without requiring direct manipulation of raw audio
2Measurement precision
If speaker embeddings are generated and processed through trained models, then voice-to-text accuracy is improved, but computational time increases
Solution Approach 1:
The system performs preliminary actions by pre-training speaker embedding models and voice filter models offline before deployment. During actual voice-to-text conversion, these pre-trained models rapidly process audio data through cached representations and learned patterns, reducing real-time computational time while maintaining high accuracy
Solution Approach 2:
The system creates and processes copies of audio data in different representations (speaker embeddings, spectrograms) rather than directly manipulating raw audio. These copied representations enable parallel processing and optimization, improving voice-to-text accuracy while managing computational time through efficient manipulation of derived data structures
Data Source
AI summary
Techniques are disclosed that enable processing of audio data to generate one or more refined versions of audio data, where each of the refined versions of audio data isolate one or more utterances of a single respective human speaker. Various implementations generate a refined version of audio data that isolates utterance(s) of a single human speaker by processing a spectrogram representation of the audio data (generated by processing the audio data with a frequency transformation) using a mask generated by processing the spectrogram of the audio data and a speaker embedding for the single human speaker using a trained voice filter model. Output generated over the trained voice filter model is processed using an inverse of the frequency transformation to generate the refined audio data.