Audio-Driven Virtual Persona Video via Landmark Sequence Generation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing face reconstruction technologies, such as 3DMM-based methods, struggle with generating detailed high-precision textures and teeth, while deep learning-based approaches like GANs face instability and high computational resource demands, limiting the generation of realistic and diverse audio-driven virtual persona videos.
Innovation Solution
A data processing method involving facial landmark extraction and audio feature extraction, followed by inputting these features into a landmark generation network model to generate a facial landmark image sequence, which is then used to produce a video using a video generation model, optimizing computational resources and improving video generation quality.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If 3DMM-based methods are used for face reconstruction, then the process is relatively stable, but the generation of detailed high-precision textures and teeth is poor
Solution Approach 1:
The patent segments the face reconstruction process into multiple stages: first extracting facial landmarks and expressions from reference images, then generating intermediate facial feature maps, and finally synthesizing the complete face with high-precision textures and teeth. This segmentation allows each stage to optimize for specific quality aspects while maintaining overall stability.
Solution Approach 2:
The patent performs preliminary extraction of facial landmarks and expression features from reference images before generating the final face reconstruction. This preliminary action captures essential facial structure and expression information, which then guides the subsequent generation process to achieve both stability and high precision in textures and teeth.
2Manufacturing precision
If deep learning-based GANs are used for face reconstruction, then high-quality video generation is achieved, but computational resource demands are high and stability is poor
Solution Approach 1:
The patent divides the video generation process into multiple stages with different computational complexities. The first stage extracts facial landmarks and expressions using lighter computational models, while the second stage generates high-quality video frames using more resource-intensive models only when necessary. This segmentation reduces overall computational resource consumption while maintaining video generation quality.
Solution Approach 2:
The patent applies partial deep learning-based GAN generation only to specific regions or frames where high quality is most important, rather than applying it uniformly across the entire video generation process. This partial action approach reduces computational resource demands while maintaining acceptable video generation quality for critical aspects.
3Adaptability or versatility
If deep learning-based GANs are used for face reconstruction, then realistic and diverse video generation is achieved, but the process is unstable and requires high computational resources
Solution Approach 1:
The patent performs preliminary extraction of facial landmarks, facial structure, and expression features from reference images before generating the final diverse and realistic video. This preliminary action establishes a stable foundation that guides subsequent generation, ensuring consistency in identity and expression while allowing for diverse and realistic variations in the generated video content.
Data Source
AI summary
A method includes: obtaining audio data and a first target image including the face of a target object; performing a facial landmark extraction on the first target image to obtain a first facial landmark image; performing, based on the audio data, an audio feature extraction to obtain an audio feature; inputting the first facial landmark image and the audio feature into a predefined landmark generation network model to obtain a facial landmark image sequence corresponding to the audio data; obtaining, based on the facial landmark image sequence and the first target image, a video corresponding to the audio data generated based on the first target image.