Online Speaker Diarization Using Turn Tokens and Spectral Clustering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing speaker diarization systems face challenges in accurately segmenting speech from multiple speakers due to computational inefficiencies and the need for manual annotation of speaker turns, leading to inaccurate speech recognition results.
Innovation Solution
An online speaker diarization system that segments audio based on detected speaker turns using a speech recognition model, extracts speaker-discriminative embeddings, and performs spectral clustering with pairwise constraints to reduce computational cost and latency, allowing for on-device execution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual annotation of speaker turns is used, then speaker diarization accuracy can be improved, but productivity deteriorates due to high time cost and labor requirements
Solution Approach 1:
The system uses automatically detected speaker turns from the speech recognition model to segment the audio signal, eliminating the need for manual annotation. The speaker turn tokens generated by the ASR model directly indicate speaker change locations, allowing the system to self-segment the audio without human intervention while maintaining accuracy.
Solution Approach 2:
The patent introduces speaker turn tokens as an intermediary element that bridges speech recognition and speaker diarization. These tokens are generated by the ASR model and serve as automatic markers for speaker transitions, replacing manual annotation while enabling accurate segmentation for downstream clustering tasks.
2Measurement precision
If traditional offline spectral clustering is used, then clustering accuracy can be improved, but latency increases making real-time processing infeasible
Solution Approach 1:
The patent segments the audio signal into speaker-specific segments using automatically detected speaker turns, then performs spectral clustering on these segmented embeddings. This segmentation approach enables online processing by treating each speaker segment independently, reducing the computational burden compared to traditional offline clustering that processes entire audio streams at once.
Solution Approach 2:
The system performs preliminary segmentation of the audio signal into speaker-specific segments using speaker turn tokens before applying spectral clustering. This preliminary action organizes the data in advance, allowing the clustering algorithm to work on pre-structured embeddings rather than raw audio, thereby reducing latency while maintaining accuracy.
3Quantity of substance
If speaker segments with many speaker turns are used, then more speech content can be processed, but computational cost increases due to excessive embedding extraction and clustering operations
Solution Approach 1:
The patent applies partial action by selectively performing spectral clustering only on speaker segments that contain actual speaker turns, rather than processing every segment uniformly. Speaker segments without speaker turns are excluded from clustering operations, reducing unnecessary computational cost while still processing the relevant speech content.
Solution Approach 2:
The system applies different processing strategies to different speaker segments based on their local characteristics. Segments with speaker turns undergo full spectral clustering, while segments without speaker turns are handled more efficiently. This local quality approach optimizes computational resource allocation to match the actual information content of each segment.
Data Source
AI summary
A method includes receiving an input audio signal that corresponds to utterances spoken by multiple speakers. The method also includes processing the input audio to generate a transcription of the utterances and a sequence of speaker turn tokens each indicating a location of a respective speaker turn. The method also includes segmenting the input audio signal into a plurality of speaker segments based on the sequence of speaker tokens. The method also includes extracting a speaker-discriminative embedding from each speaker segment and performing spectral clustering on the speaker-discriminative embeddings to cluster the plurality of speaker segments into k classes. The method also includes assigning a respective speaker label to each speaker segment clustered into the respective class that is different than the respective speaker label assigned to the speaker segments clustered into each other class of the k classes.