Speaker Recognition Across Scene Switches Using Face Depth Cues
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing speaker recognition methods in multi-person conversation videos suffer from poor accuracy due to frequent scene switching and speaker movement, leading to poor segmentation and continuity, especially when considering the entire video as a whole.
Innovation Solution
The method divides the video into segments using scene detection, extracts facial and audio features, and employs cross-scene face tracking through face depth and audio features to improve recognition accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If the entire video is considered as a whole for speaker recognition, then the processing is simple, but the recognition accuracy deteriorates due to scene switching and speaker movement
Solution Approach 1:
The video is divided into multiple video segments based on scene detection results. Each segment contains continuous frames belonging to the same scene, allowing independent speaker recognition while maintaining temporal coherence. This segmentation resolves the contradiction by simplifying processing within each segment while improving overall recognition accuracy through consistent feature extraction across segments.
2Measurement precision
If scene switching is detected and video is divided into segments, then recognition accuracy improves, but processing complexity increases
Solution Approach 1:
Scene detection is performed in advance to identify scene switching points before speaker recognition. The video is pre-divided into segments based on these detection results, allowing the subsequent speaker recognition to proceed systematically through each segment. This preliminary action reduces the complexity of handling scene transitions during recognition while maintaining high accuracy.
3Measurement precision
If cross-scene face tracking is performed using face depth features, then speaker identification accuracy improves across scenes, but computational complexity increases
Solution Approach 1:
Face depth features are extracted and processed locally for each video frame or small group of frames within segments. The cross-scene tracking is performed by comparing these local features between adjacent segments, focusing computational resources only where needed at scene transition points rather than processing the entire video uniformly. This local quality approach improves cross-scene accuracy while controlling computational complexity.
Data Source
Figure 1~2
Figure 3
Figure 4
AI summary
The present application provides a speaker recognition method and apparatus, an electronic device, a medium, and a program product, where the speaker recognition method includes: performing scene detection on a video to be recognized, and dividing the video to be recognized into a plurality of video segments based on a result of the scene detection; for each video segment in the plurality of video segments, separating the video segment to obtain audio data and video frames in the video segment; extracting facial features of the video frames and extracting an audio feature of the audio data; for a plurality of video frames with scene switching in the plurality of video segments, extracting face depth features of the plurality of video frames, and calculating a distance between the face depth features of adjacent video frames in the plurality of video frames, so as to obtain a cross-scene distance feature; and recognizing a speaker from faces included in the video segments based on the cross-scene distance feature, the facial features and the audio feature. Through the present disclosure, the speaker recognition is achieved scene by scene, and at the same time, the face association is achieved by combining cross-scene features so as to overcome the problem of poor accuracy of speaker recognition caused by scene switching.