Audio-Driven Lip Geometry Generation for Dubbed Video Sync
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Dubbed audio-visual media often fails to synchronize audio with lip geometries of characters, leading to mismatched lip movements.
Innovation Solution
A computing system utilizing an autoencoder machine learning system to predict and generate lip geometries based on audio waveforms, trained with reference facial identities to enhance synchronization accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional dubbing methods are used, then audio can be translated to different languages, but lip geometries do not match the spoken words
Solution Approach 1:
The patent introduces an autoencoder model as an intermediary component that receives audio waveforms and generates corresponding lip geometry expressions. This mediator bridges the gap between audio content and visual lip movements, enabling accurate lip-sync in dubbed videos without requiring manual adjustment or complex generative adversarial networks
Solution Approach 2:
The system segments the lip-sync synchronization task into distinct functional components: audio waveform processing by the encoder, latent space mapping, and expression generation by the decoder. This segmentation allows each component to be optimized independently while working together to achieve accurate lip synchronization across different languages
2Manufacturing precision
If autoencoder machine learning system is used to predict expressions from audio waveforms, then lip geometries can be accurately generated, but system complexity increases
Solution Approach 1:
The autoencoder model learns to copy the essential relationship between audio waveforms and lip expressions by training on paired data. The encoder-codes the audio waveform into a latent representation, and the decoder-decodes this representation back into the corresponding lip geometry expression, creating an accurate copy of the original synchronization pattern that can be applied to dubbed content
Solution Approach 2:
The system transforms the lip-sync problem into a parameter optimization problem by adjusting the autoencoder's internal parameters (weights and biases) during training. By changing these parameters to minimize the difference between predicted and actual lip geometries, the model achieves high synchronization accuracy without increasing operational complexity during inference
Data Source
AI summary
Techniques described herein relate to a computing system that includes at least one processor and a storage device that stores instructions executable by the at least one processor to generate an encoding vector by providing an audio waveform to an encoder module. The instructions executable by the at least one processor may further cause the at least one processor to determine, based on the encoding vector, an embedding vector of a plurality of embedding vectors of a codebook. The instructions executable by the at least one processor may further cause the at least one processor to provide the embedding vector to a decoder module. The instructions executable by the at least one processor may further cause the at least one processor to output, based on the embedding vector, an expression with the decoder module, wherein the expression includes a series of one or more lip geometries.