Longform Word-Level Speaker Diarization via Dynamic Audio Cohorts
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing speaker diarization systems struggle to accurately represent speech variations of speakers throughout a conversation, leading to inaccurate speech recognition results due to the use of fixed-length segments and speaker-discriminative embeddings that may not capture the dynamic changes in a speaker's voice over time, and are also restricted by regulations that prohibit computing speaker-discriminative embeddings.
Innovation Solution
A training process that utilizes a dynamic audio cohort matrix, which is updated based on diarization results, to train a joint speech recognition and speaker diarization model, allowing for cross-attention and dynamic representation of speaker speech variations, enabling word-level end-to-end diarization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If fixed-length segments and speaker-discriminative embeddings are used for speaker diarization, then the system structure is simple, but the accuracy of speech recognition deteriorates because they cannot capture dynamic changes in speaker voice over time
Solution Approach 1:
The patent applies the Dynamics principle by replacing fixed-length segments with dynamic audio cohorts that are continuously updated during processing. The audio cohort for each speaker evolves over time, incorporating new speech snippets and removing older ones, allowing the system to adapt to changing speaker characteristics throughout the conversation. This dynamic structure enables accurate capture of voice variations without requiring complex static model architectures.
Solution Approach 2:
The patent implements Parameter changes by transforming the representation of speaker characteristics from fixed embeddings to variable audio cohorts. The cohort size, composition, and temporal characteristics are dynamically adjusted based on the conversation flow and speaker behavior. This allows the system to adapt parameters like cohort length and update frequency to optimize accuracy for different speaking patterns and durations.
2Measurement precision
If dynamic audio cohort with continuous updates is implemented, then speaker diarization accuracy improves, but computational complexity increases
Solution Approach 1:
The patent applies Segmentation by dividing the audio cohort into discrete, manageable slots rather than maintaining a continuous, unstructured collection. Each cohort has a defined capacity with specific slots for storing speech snippets, enabling efficient memory management and reducing computational overhead. This segmented structure allows the system to process and update cohorts in discrete steps rather than continuously, lowering computational complexity while maintaining accuracy.
Solution Approach 2:
The patent implements Periodic action through the use of sliding windows and periodic updates of the audio cohort. Instead of continuously updating the entire cohort, the system processes audio in periodic intervals and updates cohorts at structured checkpoints. This periodic approach reduces computational load by batching operations while still capturing dynamic speaker characteristics effectively throughout the conversation.
3Measurement precision
If word-level end-to-end diarization is performed, then the granularity of speaker attribution is improved, but the processing time increases
Solution Approach 1:
The patent applies Preliminary action by pre-computing and storing audio cohorts during the processing of earlier conversation segments. These pre-computed cohorts are then reused as reference data when processing subsequent segments, eliminating the need to re-analyze entire conversation histories. This preliminary preparation enables fast word-level speaker attribution without requiring time-consuming full-reanalysis at each step.
Solution Approach 2:
The patent implements Continuity of useful action by maintaining active audio cohorts throughout the conversation that continuously accumulate and update speaker characteristics. Rather than performing discrete, repeated analysis from scratch, the system continuously refines cohort representations as new audio arrives, enabling seamless word-level diarization. This continuous approach eliminates redundant processing while maintaining high granularity speaker attribution accuracy.
Data Source
AI summary
A method includes obtaining a series of segmented labeled training samples. Each respective segmented labeled training sample includes one or more spoken terms spoken during a conversation by multiple speakers. Each respective spoken term is characterized by a corresponding sequence of acoustic frames and is paired with a corresponding transcription and a corresponding speaker label. For each respective segmented labeled training sample, the method includes obtaining a corresponding dynamic audio cohort associated with an immediately prior segmented labeled training sample, generating diarization results that include a corresponding speech recognition result having one or more predicted terms, and training a joint speech recognition and speaker diarization model based on a loss derived from the generated diarization results, the corresponding transcriptions, and the corresponding speaker labels.