Real-Time Active Speaker Detection With Composite Embeddings
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing active speaker detection algorithms face a tradeoff between accuracy and computational speed, requiring significant processing time for improved accuracy due to the use of long sequences of audiovisual data.
Innovation Solution
A system and method utilizing a detection model with an audiovisual encoder and classifier to generate composite embeddings, determining active speaker detection scores in real-time by aggregating these scores to adjust the display focus on the speaking person, incorporating greater temporal context without a prohibitive increase in computation time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If long sequences of audiovisual data are used for detection, then accuracy is improved, but processing time increases significantly
Solution Approach 1:
The patent divides the long audiovisual sequence into multiple shorter segments or clips. Each segment is processed independently by the detection model to generate individual detection results. These results are then aggregated (e.g., through voting or averaging) to produce the final detection outcome. This segmentation approach maintains detection accuracy by preserving temporal context within each segment while significantly reducing the computational burden compared to processing the entire long sequence at once.
2Measurement precision
If more computational resources are allocated for processing, then detection accuracy improves, but system complexity increases
Solution Approach 1:
By segmenting the processing task into smaller independent units (shorter audiovisual clips), the system can use simpler detection models or lighter computational resources for each segment. The overall accuracy is maintained through the aggregation of multiple segment results. This approach avoids the need for a single complex model processing entire long sequences, thereby reducing system complexity while preserving detection performance.
Data Source
AI summary
An active speaker detection (ASD) system includes a visual sensor that captures a visual scene including a first person. The ASD system further includes a computer system including an audiovisual encoder and a classifier. The computer system is configured to obtain a first set of frames and a second set of frames from the visual sensor and to produce a first embedding and a second embedding from the first set of frames and the second set of frames, respectively, using the audiovisual encoder. The computer is further configured to generate one or more composite embeddings from the first embedding and the second embedding and determine, using the classifier, an ASD score for each of the one or more composite embeddings. The computer is further configured to aggregate the one or more ASD scores forming a detection result and determine whether the first person is speaking based on the detection result.