Online Speaker Diarization Using Turn-Based Spectral Clustering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing speaker diarization systems face challenges in accurately segmenting speech from multiple speakers with high computational cost and latency due to the use of fixed-length segments and offline clustering algorithms, which are inefficient and require manual annotation of speaker turns.
Innovation Solution
An online speaker diarization system that segments audio based on detected speaker turns using a streaming speech recognition model, extracts speaker-discriminative embeddings, and performs spectral clustering with pairwise constraints to reduce computational cost and latency, allowing on-device deployment.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If fixed-length segments and offline clustering algorithms are used for speaker diarization, then speaker segmentation can be performed, but computational cost and latency increase significantly
Solution Approach 1:
The audio stream is segmented into speaker turns using a streaming speech recognition model that detects speaker turn tokens at word boundaries, rather than using fixed-length segments. This allows segmentation to occur naturally at speaker transition points, reducing the need for extensive clustering computation on arbitrary fixed segments while maintaining accurate speaker boundary detection.
Solution Approach 2:
The system transitions from static offline clustering algorithms to dynamic online spectral clustering that processes speaker-discriminative embeddings in real-time as they become available. The clustering is performed incrementally with a rolling window approach, allowing the system to adapt to new speakers dynamically without reprocessing entire audio streams, thus reducing latency while maintaining segmentation accuracy.
2Measurement precision
If offline clustering algorithms are used, then speaker clustering can be performed, but computational cost increases
Solution Approach 1:
Instead of performing clustering on the entire audio stream at once (excessive action), the system performs spectral clustering on a rolling window of recent speaker-discriminative embeddings (partial action). This incremental approach processes only the necessary subset of embeddings at each step, reducing overall computational cost while maintaining clustering accuracy through iterative refinement.
Solution Approach 2:
The system extracts speaker-discriminative embeddings using a speaker encoder model before performing spectral clustering. This preliminary extraction of discriminative features prepares the data in advance, allowing the subsequent clustering operation to work with compact, informative representations rather than raw audio features, thereby reducing computational cost while preserving clustering accuracy.
3Measurement precision
If manual annotation of speaker turns is performed, then speaker turn locations can be identified, but processing time and cost increase
Solution Approach 1:
The system uses a streaming speech recognition model to automatically detect speaker turns by identifying speaker turn tokens in the transcription stream. This self-service approach eliminates the need for manual annotation while maintaining high accuracy in speaker turn detection, as the model learns to identify speaker transitions from training data and applies this knowledge autonomously to new audio streams.
Solution Approach 2:
The system replaces the mechanical process of manual speaker turn annotation with an automated computational approach using a streaming speech recognition model. The model processes audio in real-time and generates speaker turn tokens based on acoustic and linguistic features, substituting human labor with an efficient automated system that maintains high detection accuracy while dramatically increasing processing throughput.
Data Source
AI summary
A method includes receiving an input audio signal that corresponds to utterances spoken by multiple speakers. The method also includes processing the input audio to generate a transcription of the utterances and a sequence of speaker turn tokens each indicating a location of a respective speaker turn. The method also includes segmenting the input audio signal into a plurality of speaker segments based on the sequence of speaker tokens. The method also includes extracting a speaker-discriminative embedding from each speaker segment and performing spectral clustering on the speaker-discriminative embeddings to cluster the plurality of speaker segments into k classes. The method also includes assigning a respective speaker label to each speaker segment clustered into the respective class that is different than the respective speaker label assigned to the speaker segments clustered into each other class of the k classes.