Longform Speaker Diarization With Chunk-Wise LLM Self-Enrollment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing speaker diarization systems struggle with accurately identifying speaker turns in conversations due to speaker-discriminative embeddings that fail to represent speech variations throughout a conversation and are restricted by regulations, leading to inaccurate diarization results.
Innovation Solution
A joint speech recognition and speaker diarization model using a tokenizer and a trained large language model (LLM) generates audio encoding chunks and diarization results, associating predicted terms with speaker tokens, and conditions on a prior audio cohort to improve diarization accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If speaker-discriminative embeddings are used to represent speaker identity, then speaker identification capability is improved, but the system fails to represent speech variations throughout a conversation and is restricted by regulations
Solution Approach 1:
The patent segments the conversation into chunks and processes each chunk independently through the LLM. This allows the system to handle speech variations locally within each chunk while maintaining speaker identification accuracy through the sequential processing of segmented audio data, resolving the contradiction between fixed embeddings and speech variations.
Solution Approach 2:
The system dynamically processes audio chunks sequentially, allowing the speaker representation to adapt and evolve throughout the conversation. The LLM maintains a dynamic understanding of speaker identities across chunks, enabling the system to represent speech variations while preserving speaker discrimination capability.
2Productivity
If existing speaker diarization models are used, then processing speed is maintained, but diarization results include errors and accuracy is insufficient
Solution Approach 1:
The patent introduces a tokenizer as an intermediary component that converts audio data into encoding chunks before processing by the LLM. This intermediary layer enables the system to leverage the LLM's high accuracy while maintaining efficient processing speeds, resolving the contradiction between speed and accuracy in diarization results.
3Measurement precision
If a trained LLM with frozen parameters is used, then diarization accuracy is enhanced, but model adaptability to new speakers may be limited
Solution Approach 1:
The system performs preliminary processing by segmenting audio into chunks and preparing encoding representations before LLM processing. This preliminary action allows the frozen-parameter LLM to achieve high accuracy on familiar speakers while the chunk-wise processing framework naturally accommodates new speakers through sequential adaptation, resolving the contradiction between frozen parameters and adaptability.
Data Source
AI summary
A method includes receiving a prompt including a textual diarization request and corresponding audio data characterizing a conversation between multiple speakers. The method also includes generating a sequence of audio encoding chunks based on the corresponding data. For each respective audio encoding chunk, the method includes using a trained large language model (LLM) generating corresponding diarization results based on the respective audio encoding chunk and the textual diarization request and generating a new audio cohort for the respective audio encoding chunk based on the corresponding diarization results. The corresponding diarization results include a speech recognition result that has one or more predicted terms. Each respective predicted term is associated with a corresponding speaker token representing a predicted identity of a respective speaker that spoke the respective predicted term. The trained LLM is conditioned on a prior audio cohort generated by the trained LLM for a prior audio encoding chunk.