Audio Video Synchronization Detection Using Mouth Feature Extraction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for audio and video synchronization detection in video editing, encoding, and playback often result in false detections, requiring manual verification and consuming significant time and manpower.
Innovation Solution
An audio and video synchronization detection method that extracts image and audio data from a target-length video segment, performs face detection and tracking to obtain face image lists, extracts mouth features to characterize lip movements, and determines synchronization based on audio data and mouth features.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If reference-based methods or non-reference methods are used for audio and video synchronization detection, then the detection process can be performed, but the detection accuracy is low and false detections occur frequently
Solution Approach 1:
The patent segments the detection process into multiple specialized modules: face detection module, mouth region extraction module, audio feature extraction module, and synchronization detection module. Each module focuses on a specific aspect of the synchronization problem, improving overall detection accuracy by breaking down the complex task into manageable, optimized components.
Solution Approach 2:
The patent introduces mouth feature vectors as an intermediary representation that bridges visual mouth movements and audio speech content. By extracting and comparing mouth feature vectors from video frames with audio feature vectors from audio data, the system creates a reliable intermediate representation that enables accurate synchronization detection without direct comparison of raw audio and video streams.
2Reliability
If manual secondary verification is performed to confirm false detections, then detection reliability can be improved, but time consumption and manpower requirements increase significantly
Solution Approach 1:
The system performs self-verification through multiple detection mechanisms working together. The face detection module verifies the presence of valid face regions, the mouth feature extraction ensures proper mouth region identification, and the audio-visual feature comparison provides cross-modal verification. This self-service approach eliminates the need for external manual verification while maintaining high reliability.
Solution Approach 2:
The patent implements feedback loops where detection results are continuously refined. The system uses face detection results to guide mouth region extraction, uses extracted mouth features to inform audio-visual comparison, and adjusts detection parameters based on synchronization results. This feedback mechanism ensures high reliability without requiring manual intervention.
3Measurement precision
If face detection and tracking are performed on video data, then mouth features can be accurately extracted, but the processing complexity and computational requirements increase
Solution Approach 1:
The patent performs face detection and mouth region extraction as preliminary actions before the main synchronization detection task. By pre-identifying and isolating mouth regions in video frames and extracting mouth feature vectors in advance, the system simplifies the subsequent audio-visual comparison process. This preliminary processing organizes the data structure and reduces the complexity of the main detection algorithm.
Data Source
AI summary
The disclosure provides an audio and video synchronization detection method, an audio and video synchronization detection device, an electronic equipment and a terminal, and relates to a field of image processing, in particular to the technical fields of computer vision, artificial intelligence and the like. The method includes: extracting image data and audio data of a video segment of a target length; obtaining a plurality of face image lists by performing face detection and tracking based on the extracted image data; extracting mouth features corresponding to each face image list based on a traversal result of the face image list, in which the mouth features are used for representing changes in lip shape; and determining a synchronization result of the video segment based on the audio data and the mouth features.