Audio-Guided Motion Priors for Realistic Digital Human Generation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods struggle to generate continuous and realistic digital human images that accurately match audio data in terms of facial expressions and limb movements, leading to inefficiencies in model training and image generation quality.
Innovation Solution
A training method for an image generation model that incorporates motion prior information by estimating keypoints and optical flow based on audio and reference images, using networks to predict and refine motion parameters, thereby reducing learning difficulty and improving training efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If existing methods are used to generate digital human images, then the basic image generation function is achieved, but the alignment between facial expressions and limb movements with audio data is insufficient
Solution Approach 1:
The patent performs preliminary motion estimation and optical flow calculation before the main image generation process. By pre-computing motion parameters from audio data and reference images, the system prepares motion priors that guide subsequent image generation, improving alignment precision without significantly increasing overall training time
Solution Approach 2:
The patent introduces optical flow as an intermediary representation that bridges audio data and image generation. The optical flow field serves as a mediator that encodes motion information from audio, enabling the image generation model to produce coherent facial expressions and limb movements that align with the audio input
2Manufacturing precision
If motion estimation and optical flow calculation are performed, then the alignment of facial expressions and limb movements with audio data is improved, but the model training complexity increases
Solution Approach 1:
The patent segments the training process into distinct modules: audio encoding, motion estimation, optical flow calculation, and image generation. Each module processes specific information independently and passes results to the next stage, making the complex training process more manageable and efficient. The segmentation allows for optimized processing of motion parameters and optical flow fields separately from the main image generation task
3Manufacturing precision
If dense optical flow is computed for both predicted keypoints and non-key pixel points, then the realism of digital human images is improved, but the computational cost increases
Solution Approach 1:
The patent applies different levels of computational detail to different regions. Dense optical flow is computed for non-key pixel points to ensure realism in areas requiring fine detail, while keypoint motion is handled through parameter estimation. This localized approach to computational intensity ensures high image realism where needed while managing overall computational cost
Data Source
AI summary
Provided are a training method for an image generation model, an image generation method, apparatus, and a device. The training method includes extracting reference keypoints of a character from a sample reference image; based on a model to be trained, performing motion estimation using sample audio data and the reference keypoints to obtain predicted keypoints that match the sample audio data; performing parameter estimation using the reference keypoints and the predicted keypoints to obtain motion parameters of the predicted keypoints, and performing prior motion estimation using the motion parameters of the predicted keypoints to obtain optical flow of non-key pixel points; performing image prediction using the sample reference image and dense optical flow to obtain predicted image data that matches the sample audio data; performing model training using the predicted image data and annotated image data to obtain the image generation model.