Lip-Sync Video Generation Using Facial Feature Migration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing virtual digital human technologies face challenges in generating high-quality, lifelike virtual human videos due to high computing consumption and limited computing resources, requiring large amounts of training data and resulting in poor generalization and low image quality.
Innovation Solution
A method that utilizes a single target person image, audio, and a driving video to migrate and enhance dynamic facial features, generating a target person lip synchronization video with high image quality and synchronization, decoupling person image, audio, and video from the model, and using pre-trained models for key-point detection and enhancement.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a large amount of video data of the real human is recorded to train a virtual human generation model, then the virtual human has desirable human-like effect, but the computing consumption increases greatly
Solution Approach 1:
The patent extracts only the essential facial feature information from the driving video through key-point detection and feature migration, rather than using the entire video dataset for training. This allows the system to capture the necessary human-like characteristics while significantly reducing the amount of data processing and computing resources required.
Solution Approach 2:
The patent creates a virtual digital human by copying and migrating dynamic facial features from the driving video to the target person image. This feature migration approach allows the system to generate realistic virtual humans without requiring extensive training data, as the essential characteristics are directly transferred from the source video.
2Manufacturing precision
If a large amount of video data is recorded to train the model, then the virtual human generation quality improves, but the device computing resources are insufficient
Solution Approach 1:
The patent segments the complex task of generating high-quality virtual humans into distinct modules: key-point detection, feature migration, lip synchronization, and image enhancement. Each module processes specific aspects independently, reducing the computational burden on any single component while maintaining overall image quality.
Solution Approach 2:
The patent introduces an intermediary feature migration process that bridges the driving video and target person image. This intermediary step extracts and transfers essential facial features, enabling high-quality generation without requiring the device to process entire video datasets, thus reducing computing resource requirements.
3Manufacturing precision
If the virtual human generation model is made complex to achieve high-definition image quality, then the image quality improves, but the computing consumption increases
Solution Approach 1:
The patent changes the approach from using complex models with many parameters to using a simpler feature migration framework. By transforming the problem into one of feature extraction and transfer rather than complex generation, the system achieves high image quality with reduced computing consumption.
Solution Approach 2:
The patent replaces complex computational generation processes with a feature-based approach. Instead of using complex neural networks to generate pixels directly from scratch, the system migrates pre-extracted facial features, substituting a more efficient computational mechanism that achieves similar or better quality with lower resource consumption.
4Adaptability or versatility
If a single target person image is used to generate the video, then the process is simplified and generalization capability is improved, but the initial feature migration accuracy may be reduced
Solution Approach 1:
The patent incorporates feedback mechanisms in the feature migration process, where the system iteratively refines the transferred features based on the target person image characteristics. This feedback loop ensures high accuracy in feature migration while maintaining the ability to work with a single input image, thereby preserving generalization capability.
Solution Approach 2:
The patent performs preliminary key-point detection and feature extraction from the driving video before the actual generation process. This preliminary action prepares and pre-processes the essential features, ensuring that when these features are migrated to the target person image, the accuracy is maintained despite using only a single image for generation.
Data Source
AI summary
A method includes obtaining a target person image of a target person, target audio that is set for the target person, and a preset driving video, where a face of a person in the driving video is dynamic; migrating a dynamic facial feature of the person in the driving video to the target person image to obtain a target person dynamic video, where a face of the target person in the target person dynamic video is dynamic; generating a lip synchronization video based on the target audio and the target person dynamic video, where a dynamic facial feature in the lip synchronization video is synchronous with the target audio; and enhancing the dynamic facial feature in the lip synchronization video and image quality of the lip synchronization video based on the target person image to obtain a target person lip synchronization video.