Reference-Image Media Generation With Guided Subject Motion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current machine learning-based methods for generating videos or animations from a single image or descriptive text fail to accurately meet user intentions, leading to inadequate user experience.
Innovation Solution
A method involving obtaining a reference image and a noise image, along with control information including target subject information, processing category, and movement guidance, to perform denoising processing using a target model to generate media where the target subject moves according to user-defined movement guidance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If video or animation is generated based only on one frame of existing image or one frame plus descriptive text, then the generation process is simple, but the accuracy of meeting user intentions is insufficient
Solution Approach 1:
The patent segments the control information into multiple distinct components: subject information (identifying the target subject), processing category information (specifying the type of movement), and movement guidance information (defining the movement trajectory and parameters). This segmentation allows the system to process each aspect separately, improving accuracy while maintaining manageable complexity through modular information handling.
Solution Approach 2:
The patent transitions from traditional 2D image generation to 4D media generation by adding temporal and spatial movement dimensions. The movement guidance information introduces trajectory coordinates and movement parameters that define how subjects move through space and time, enabling precise control over subject motion while generating dynamic media from static reference images.
2Ease of operation
If control information includes detailed movement guidance, then user experience improves, but information processing complexity increases
Solution Approach 1:
The patent creates a universal control information structure that can handle multiple types of processing categories (different movement types) through a unified framework. The same basic structure (subject information + processing category + movement guidance) accommodates various movement scenarios, simplifying the interface for users while providing comprehensive control capabilities.
Solution Approach 2:
The patent utilizes parameter changes to encode movement guidance information, including trajectory coordinates, movement speed, direction, and other quantifiable attributes. By transforming qualitative user intentions into quantitative parameters, the system enables precise control over subject movement while maintaining ease of operation through parameter-based adjustment rather than complex procedural programming.
Data Source
AI summary
The present disclosure provides a media generation method, an apparatus, a device, and a medium. A specific implementation of the method includes: obtaining a reference image and a noise image; obtaining control information for guiding media generation, the control information including information of a target subject in the reference image, information of a processing category to which the target subject belongs, and movement guidance information of the target subject; and performing, using a target model and based on the reference image and the control information, denoising processing on the noise image to obtain a target media, such that the target subject in the target media moves according to the movement guidance information.