Multi-Talker Audio Separation Using Talker Embeddings
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing speech transcription systems struggle in noisy conditions or multi-talker scenarios, leading to degraded performance and compromised talker labeling when multiple talkers are active concurrently.
Innovation Solution
Derive talker embedding vectors for each speaker, use personalized noise suppression models to generate single-talker audio streams, and merge transcriptions using a consistent time representation scheme to create a multi-talker output transcription.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If speech transcription systems are trained for single talker or clear turn-taking scenarios, then transcription performance is good in those conditions, but performance degrades in multi-talker concurrent scenarios
Solution Approach 1:
The patent segments the audio stream into multiple single-talker streams by separating talker embeddings and processing each talker's audio independently through personalized noise suppression models. This segmentation allows the transcription system to handle multi-talker scenarios by processing each talker's speech separately, then combining the results, thereby maintaining high transcription performance while adapting to complex multi-talker environments.
2Device complexity
If transcription systems process multi-talker audio directly, then system complexity remains low, but diarization accuracy and talker labeling are compromised
Solution Approach 1:
The system segments the mixed audio signal into separate talker streams using talker embedding vectors and personalized noise suppression models. Each segment corresponds to a single talker's audio, enabling accurate diarization and talker labeling while maintaining manageable processing complexity through modular architecture.
Solution Approach 2:
The patent introduces talker embedding vectors and personalized noise suppression models as intermediary components between the audio input and transcription output. These intermediaries process and separate the audio signals before transcription, improving diarization accuracy without requiring the entire system to become significantly more complex.
Data Source
AI summary
A plurality of talker embedding vectors may be derived that correspond to a plurality of talkers in an input audio stream. Each talker embedding vector may represent respective voice characteristics of a respective talker. The talker embedding vectors may be generated based on, for example, a pre-enrollment process or a cluster-based embedding vector derivation process. A plurality of instances of a personalized noise suppression model may be executed on the input audio stream. Each instance of the personalized noise suppression model may employ a respective talker embedding vector. A plurality of single-talker audio streams may be generated by the plurality of instances of the personalized noise suppression model. A plurality of single-talker transcriptions may be generated based on the plurality of single-talker audio streams. The plurality of single-talker transcriptions may be merged into a multi-talker output transcription.