Digital Human Voiceover Video Generation With Resolution Scaling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing speech-to-lip conversion models, such as the wav2lip model, produce low-resolution blurred images, resulting in poor visual quality of digital human voiceover videos, and super-resolution models are inefficient for real-time applications like live streaming.
Innovation Solution
A video generation model that includes down-sampling and up-sampling processes to enhance image resolution without additional image processing models, addressing the inefficiencies of existing methods by improving resolution and generation efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If the wav2lip model is used for speech-to-lip conversion, then the digital human voiceover video can be generated, but the output image resolution is low and visual quality is poor
Solution Approach 1:
The patent segments the image processing into two stages: first down-sampling the input image to a lower resolution for efficient model processing, then up-sampling the generated output image to high resolution for quality output. This segmentation allows the model to work efficiently at low resolution while producing high resolution output.
Solution Approach 2:
The patent introduces a resolution dimension transformation by applying down-sampling followed by up-sampling operations. This dimensional change in resolution allows the system to achieve both high generation efficiency (at low resolution) and high output quality (at high resolution) without compromising either aspect.
2Manufacturing precision
If super-resolution models are used to enhance image resolution, then the visual quality improves, but the generation efficiency decreases for real-time applications
Solution Approach 1:
The patent applies preliminary down-sampling action before the main generation process to reduce computational complexity, enabling faster generation. The high-resolution output is then achieved through up-sampling after generation, avoiding the need for time-consuming super-resolution processing during the main generation phase.
Solution Approach 2:
The patent uses a copying approach where the low-resolution generated image is up-sampled to create the high-resolution output. This copying method preserves the generated content while enhancing resolution, avoiding the need for complex super-resolution modeling that would increase processing time.
3Manufacturing precision
If additional image processing models are added to improve resolution, then the visual quality enhances, but the system complexity increases
Solution Approach 1:
The patent merges the down-sampling and up-sampling operations into a single integrated video generation model architecture. This consolidation allows the model to handle resolution transformation internally without requiring separate image processing models, thereby reducing overall system complexity while achieving high-resolution output.
Solution Approach 2:
The patent makes the video generation model universal by embedding resolution transformation capabilities (down-sampling and up-sampling) within its architecture. This multi-functional design allows the single model to perform both generation and resolution enhancement, eliminating the need for additional specialized image processing models.
Data Source
Figure 1
Figure 2~3
Figure 4~5
AI summary
The present disclosure relates to a video generation method and apparatus, a readable medium, and an electronic device, so as to improve the resolution of a digital human voiceover video and enhance the visual effect of the digital human voiceover video. The video generation method includes: obtaining a talking video of a target object and a target text for video generation; and generating, by using the talking video, the target text, and a video generation model, a target video of a digital human corresponding to the target object talking according to the target text, where the video generation model is configured to generate the target video by: extracting an initial image sequence from the talking video, and down-sampling images in the initial image sequence to obtain a target image sequence, where each of the images in the initial image sequence includes a face of the target object; generating a video frame sequence corresponding to a video of the digital human corresponding to the target object talking according to the target text, according to the target image sequence and an audio sequence corresponding to the target text; and up-sampling video frames in the video frame sequence to obtain the target video.