Lip-Sync Estimation Training With Ranking for Partial Synchronization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional contrastive learning approaches for lip-sync estimation models fail to accurately assess varying degrees of partial synchronization, particularly in dubbed content, and lack understanding and utilization of partial-sync examples, leading to ineffective identification of misalignments between visual and audio content.
Innovation Solution
Implement a Ranking Supervised Multi-Similarity (RSMS) loss function and a multi-stage training approach that incorporates partially-synchronized examples, using a hard example miner to enforce hierarchical supervision and learn a continuous spectrum of synchronization quality through fine-tuning with real-world dubbed audio examples.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional contrastive learning approaches are used for training lip-sync estimation models, then binary classification between synchronized and unsynchronized content is achieved, but accurate assessment of varying degrees of partial synchronization is lost
Solution Approach 1:
The patent transforms the binary classification parameter into a continuous ranking parameter. Instead of classifying only synchronized vs. unsynchronized, the model now ranks audio samples along a continuous spectrum of synchronization quality, enabling accurate assessment of partial synchronization degrees through parameter transformation
Solution Approach 2:
The patent adds a new dimension to the classification space by introducing ranking-based supervision that operates alongside traditional contrastive learning. This creates a multi-dimensional training framework where models learn both binary discrimination and continuous ranking, enhancing adaptability to partial synchronization while maintaining binary classification capabilities
2Ease of manufacture
If binary classification training is used, then simple training objective is achieved, but understanding and utilization of partial-sync examples is lost
Solution Approach 1:
The patent merges contrastive learning objectives with ranking-based supervision objectives into a unified training framework. By combining these two approaches, the system maintains training simplicity while incorporating partial-sync examples, as both objectives can be optimized jointly through a combined loss function that leverages information from all training examples including partial-synchronization cases
3Productivity
If traditional contrastive learning is applied, then computational efficiency is maintained, but differentiation among partially synchronized content is lost
Solution Approach 1:
The patent segments the training objective into two distinct components: contrastive learning for binary classification and ranking-based supervision for continuous differentiation. This segmentation allows each component to maintain its computational efficiency while contributing specialized capabilities, with the contrastive part handling efficient binary discrimination and the ranking part adding fine-grained differentiation for partial synchronization
Data Source
AI summary
One embodiment sets forth a technique for training lip-sync estimation models. According to some embodiments, the method can be implemented by a computing device, and includes the steps of obtaining video training data comprising a plurality of training videos and corresponding audio training data; selecting an anchor video from the plurality of training videos; identifying, with respect to the anchor video and based on a similarity evaluation generated by a machine learning (ML) model, a plurality of audio samples; generating a training loss from the plurality of audio samples; applying backpropagation from the training loss to update parameters of the ML model until a convergence criteria is satisfied; and generating a trained lip-sync estimation model based on the updated parameters.