Mouth Shape Synthesis Using Neural Network Embeddings
Solution Overview
Problem
Existing artificial intelligence algorithms for generating synthesized videos struggle to accurately synthesize mouth shapes through audio input, often resulting in desynchronized or unrealistically rendered mouth shapes, particularly in transition sections between audio and non-audio segments.
Innovation Solution
A mouth shape synthesis device using an artificial neural network that includes an original video encoder, audio encoder, synthesized video decoder, discriminator, and pre-learned embedding modules to improve synchronization and realism by minimizing reconstruction, sync, and emotion losses during the learning process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If existing generalized AI models are used for mouth shape synthesis, then the model can process various video inputs, but the audio-video synchronization precision deteriorates and mouth shape accuracy decreases
Solution Approach 1:
The patent divides the video processing into segments with and without audio, applying different synthesis strategies to each segment type. This segmentation allows the model to optimize for audio synchronization in audio-present segments while maintaining versatility across different video types.
Solution Approach 2:
The patent applies different loss weights and synthesis parameters to different segments of the video based on whether audio is present. By adjusting local quality parameters (loss weights, synthesis strength) according to segment characteristics, the model achieves high precision mouth shape synthesis in audio-driven segments while maintaining overall model versatility.
2Adaptability or versatility
If existing algorithms synthesize mouth shapes without audio input, then the model can process silent video sections, but the mouth shape realism deteriorates in transition sections
Solution Approach 1:
The patent performs preliminary actions by pre-processing video segments to identify audio presence and preparing appropriate synthesis parameters in advance. This allows the model to maintain reliability and realism in transition sections by having pre-configured synthesis strategies ready for both audio-present and audio-absent contexts.
Solution Approach 2:
The patent dynamically adjusts synthesis parameters and loss weights based on the detected audio presence in each segment. This dynamic adaptation enables the model to maintain mouth shape realism across transition sections by switching between audio-driven and visual-driven synthesis modes as needed.
3Measurement precision
If the model focuses on audio synchronization, then the sync precision improves, but the visual realism of the synthesized mouth shape deteriorates
Solution Approach 1:
The patent changes synthesis parameters and loss function weights based on segment characteristics. In audio-present segments, sync precision is prioritized with higher audio loss weights, while in audio-absent segments, visual realism is prioritized with higher reconstruction loss weights. This parameter adaptation resolves the contradiction by optimizing for the appropriate metric in each context.
Solution Approach 2:
The patent uses a composite loss function that combines multiple loss components (audio sync loss, reconstruction loss, identity loss) with dynamically adjusted weights. This composite approach allows simultaneous optimization of both sync precision and visual realism by balancing multiple objectives through weighted combination, resolving the trade-off between the two requirements.
Data Source
AI summary
Provided is a mouth shape synthesis device and a method using an artificial neural network. To this end, an original video encoder that encodes original video data which is a target of a mouth shape synthesis as a video including a face of a synthesis target; an audio encoder that encodes audio data that is a basis for the mouth shape synthesis and outputs an audio embedding vector; and a synthesized video decoder that uses the original video embedding vector and the audio embedding vector as input data, and outputs synthesized video data in which a mouth shape corresponding to the audio data is synthesized on the synthesis target face may be provided.