Image-to-Video Generation with Local Redrawing for Object Integrity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current image-to-video generation models suffer from object deformation and distortion, and require users to preset motion trajectories, making them complex and inaccurate.
Innovation Solution
An image-to-video generation method that determines an interframe transform matrix sequence from a pretrained model, generates a masked image sequence, and uses a second video generation model with local redrawing to create a target video without preset motion parameters, ensuring clear and diverse motion trajectories.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a pretrained image-to-video model is used to generate videos, then video generation capability is achieved, but object deformation and distortion occur
Solution Approach 1:
The patent segments the video generation process into two distinct stages: a coarse generation stage using a pretrained model to create the overall video, and a fine-tuning stage using a local redrawing model to specifically refine the target object. This segmentation allows each stage to focus on its strength - the pretrained model provides video generation capability while the local redrawing model ensures object shape accuracy by selectively processing only the target object region.
Solution Approach 2:
The patent applies local quality by using a local redrawing function that specifically targets and processes only the region containing the target object. Instead of reprocessing the entire video frame, the system identifies the object's location and applies the fine-tuning model only to that local region, thereby maintaining high object shape accuracy while preserving the overall video content generated by the pretrained model.
2Ease of operation
If a user presets motion trajectory parameters, then motion control is achieved, but the process becomes complicated
Solution Approach 1:
The patent implements self-service by enabling the system to automatically extract motion trajectory information directly from the input image itself, rather than requiring external parameter input. The target object image sequence embedded in the input image serves as the motion guide, allowing the system to self-determine how the object should move throughout the video generation process, thereby eliminating the need for complex preset parameters while maintaining intuitive motion control.
Data Source
AI summary
Provided is an image-to-video generation method. A source image including a target object is inputted into a first video generation model to obtain a material video. An interframe transform matrix sequence is determined according to the material video. An object masked image corresponding to the target object is obtained from the source image. The interframe transform matrix sequence is applied to the object masked image to obtain a masked image sequence including a plurality of masked images. The interframe transform matrix sequence is applied to the source image to obtain a target object image sequence including a plurality of target object images. Target input data is determined according to the source image, the masked image sequence and the target object image sequence. The target input data is inputted into a second video generation model supporting local redrawing to obtain a target video.