Face-Aware Speaker Diarization for Video Editing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional video editing tools are tedious and challenging for users due to their reliance on time-based interactions, and they lack effective speaker diarization techniques that can associate voices with faces and accurately distinguish speech from music, leading to over-segmentation and inaccuracies in transcribing singing.
Innovation Solution
The implementation of a face-aware speaker diarization technique that combines audio-only and audio-visual diarization using the Hungarian algorithm, along with music-aware speaker diarization to remove transcribed singing, and transcript segmentation to facilitate text-based video editing by identifying candidate boundaries for video segments based on detected sentences and words, allowing users to select and edit video segments through a transcript interface.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If audio-only speaker diarization is used to identify speakers, then speaker identification can be performed, but over-segmentation occurs and accuracy deteriorates
Solution Approach 1:
The patent combines audio-only diarization results with audio-visual diarization results by merging the two sets of speaker segments. This integration allows the system to leverage both audio-based speaker detection and visual face-based speaker detection, thereby reducing over-segmentation and improving overall speaker identification accuracy while maintaining automated operation.
2Ease of operation
If conventional video editing tools are used, then video frames can be selected and edited, but the process becomes tedious and operation difficulty increases
Solution Approach 1:
The patent replaces the conventional mechanical/time-based video editing interaction (manually selecting and manipulating video frames on a timeline) with a text-based interaction system. Users can select and edit video segments by interacting with transcript text, which automatically maps to corresponding video segments. This substitution simplifies the editing process, making it more intuitive and easier to operate while reducing the complexity of the editing workflow.
3Productivity
If speech transcription is performed without music awareness, then all audio can be transcribed, but singing and music are incorrectly transcribed as speech
Solution Approach 1:
The patent extracts and separates music segments from the audio signal before performing speech transcription. By identifying and removing music portions (including singing) from the transcription process, the system ensures that only speech segments are transcribed. This extraction approach maintains high transcription productivity for speech while eliminating the accuracy problem of incorrectly transcribing singing and music as speech.
Data Source
AI summary
Embodiments of the present invention provide systems, methods, and computer storage media for face-aware speaker diarization. In an example embodiment, an audio-only speaker diarization technique is applied to generate an audio-only speaker diarization of a video, an audio-visual speaker diarization technique is applied to generate a face-aware speaker diarization of the video, and the audio-only speaker diarization is refined using the face-aware speaker diarization to generate a hybrid speaker diarization that links detected faces to detected voices. In some embodiments, to accommodate videos with small faces that appear pixelated, a cropped image of any given face is extracted from each frame of the video, and the size of the cropped image is used to select a corresponding active speaker detection model to predict an active speaker score for the face in the cropped image.