Active Speaker Detection With Temporal Segment Embeddings
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing active speaker detection algorithms face a tradeoff between accuracy and computational speed, particularly when processing long sequences of audiovisual data, which can take over three minutes to process a one-and-a-half-minute video clip, making real-time detection challenging.
Innovation Solution
A system and method using a detection model with an audiovisual encoder and classifier to generate composite embeddings from frames, determining an active speaker detection score, and adjusting the display to focus on the speaker, while maintaining real-time processing by using machine-learned models like neural networks and RNNs to incorporate temporal context without significantly increasing computation time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If machine-learned models are used to detect active speakers using audiovisual data, then detection accuracy is improved, but computational processing time increases significantly
Solution Approach 1:
The patent segments the audiovisual data into multiple clips divided into temporal segments, processing each segment independently through the machine-learned detection model. This segmentation allows the system to maintain high detection accuracy while reducing overall computational time by avoiding processing the entire long video sequence at once.
Solution Approach 2:
The system performs preliminary actions by pre-processing the audiovisual data into clips and segments before feeding them to the detection model. This preliminary organization of data enables more efficient processing and reduces the computational burden on the machine-learned model while maintaining accurate active speaker detection.
2Measurement precision
If long sequences of audiovisual data are processed to improve detection accuracy, then measurement precision improves, but processing time increases proportionally
Solution Approach 1:
The patent divides long audiovisual sequences into smaller clips and temporal segments, processing each segment independently. This segmentation maintains detection accuracy by preserving temporal context while significantly improving processing speed by avoiding the need to analyze entire long videos sequentially.
Solution Approach 2:
The system processes only the necessary portions of the audiovisual data (clips and segments) rather than the entire long sequence. This partial processing approach maintains sufficient accuracy for active speaker detection while dramatically reducing processing time and computational requirements.
Data Source
Figure 1~2
Figure 3A~3B
Figure 4
AI summary
An active speaker detection (ASD) system includes a visual sensor that captures a visual scene including a first person. The ASD system further includes a computer system including an audiovisual encoder and a classifier. The computer system is configured to obtain a first set of frames and a second set of frames from the visual sensor and to produce a first embedding and a second embedding from the first set of frames and the second set of frames, respectively, using the audiovisual encoder. The computer is further configured to generate one or more composite embeddings from the first embedding and the second embedding and determine, using the classifier, an ASD score for each of the one or more composite embeddings. The computer is further configured to aggregate the one or more ASD scores forming a detection result and determine whether the first person is speaking based on the detection result.