Speaker Diarization LLM Post-Processing for Overlapping Speech
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing speaker diarization models often produce diarization results with errors, particularly in environments with multiple speakers and overlapping speech, leading to inaccuracies in identifying and segmenting speakers.
Innovation Solution
A computer-implemented method using a joint speech recognition and speaker diarization model, followed by post-processing with a large language model (LLM) to refine diarization results by replacing identity-agnostic speaker tokens with identity-specific tokens and correcting misalignments using semantic interpretation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a joint speech recognition and speaker diarization model is used, then speaker diarization can be performed, but errors in identifying and segmenting speakers occur particularly in environments with multiple speakers and overlapping speech
Solution Approach 1:
The patent introduces an intermediary post-processing module that receives the initial diarization results and refines them by re-evaluating speaker boundaries and identities. This intermediary step corrects errors in speaker segmentation and identification without requiring changes to the core diarization model, thereby improving reliability and precision simultaneously.
Solution Approach 2:
The system implements a feedback mechanism where the post-processing module analyzes the output of the diarization model and generates corrected results that are fed back as improved diarization output. The feedback loop identifies and corrects errors in speaker boundaries and identities, particularly in challenging environments with overlapping speech, thereby resolving the contradiction between reliability and measurement precision.
2Reliability
If post-processing with large language model is applied, then accuracy and reliability of speaker identification is enhanced, but computational complexity and processing time increase
Solution Approach 1:
The post-processing module applies large language model analysis selectively rather than to all diarization results uniformly. It focuses computational resources on segments where errors are most likely to occur, such as overlapping speech regions or ambiguous speaker transitions, thereby enhancing accuracy while limiting the increase in device complexity to only where necessary.
Solution Approach 2:
The system performs preliminary error identification before applying the computationally intensive large language model. By first detecting potential error regions in the diarization output, the system prepares targeted inputs for the LLM, reducing unnecessary computational complexity while maintaining enhanced accuracy in critical segments.
Data Source
AI summary
A method includes receiving audio data including a plurality of spoken terms spoken by one or more speakers during a conversation. The method includes generating diarization results based on the plurality of spoken terms spoken by the one or more speakers during the conversation. The diarization results include a speech recognition result including a series of predicted terms and a series of identity-agnostic speaker tokens. The method also includes processing the diarization results conditioned on a diarization prompt to predict, as output from an LLM, updated diarization results. The updated diarization results include the speech recognition result including the series of predicted terms and a series of identity-specific speaker tokens.