Stepwise Subject-Action Image Generation With Frequency Guidance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for generating images with specific subjects performing actions struggle with precise control over the action pose and often require extensive training datasets or additional annotations like keypoints or pose data, limiting their generalization and accuracy.
Innovation Solution
A stepwise inference process using a denoising diffusion model guided by amplitude and phase information from source and reference images, allowing generation of synthetic images with a subject performing an action based on a single pair of input images without extensive training, by first guiding the generation towards the action and then the subject.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If conventional methods use extensive training datasets and additional annotations (keypoints, pose data) to generate images with specific subjects performing actions, then the accuracy of action pose control is improved, but the device complexity and data requirements increase significantly
Solution Approach 1:
The patent segments the image generation process into two distinct phases: first generating the action pose structure through a diffusion model, then refining and controlling the pose using pose data. This segmentation allows each phase to focus on specific aspects (action generation vs. pose control) without requiring all data types simultaneously, thereby reducing overall system complexity while maintaining accuracy
Solution Approach 2:
The patent performs preliminary generation of action poses using a diffusion model before applying pose control refinements. By pre-generating plausible action structures and then iteratively refining them with pose data, the system achieves accurate pose control without needing extensive pre-annotated training datasets, thus reducing data complexity requirements
2Manufacturing precision
If conventional methods require additional annotations like keypoints or pose data to control action pose, then the action accuracy is improved, but the ease of operation and accessibility are worsened due to increased data preparation requirements
Solution Approach 1:
The patent enables the system to generate its own pose data through the diffusion model's preliminary action generation, reducing dependence on externally annotated pose data. The model self-generates plausible action structures that can then be refined with minimal additional pose information, making the system more self-sufficient and easier to operate without extensive manual annotation efforts
3Adaptability or versatility
If conventional methods use extensive training datasets to achieve generalization, then the adaptability to different subjects and actions is improved, but the loss of time for training and the productivity are worsened
Solution Approach 1:
The patent performs preliminary generation of action poses using a pre-trained diffusion model before fine-tuning with specific pose data. This preliminary action leverages the model's pre-existing knowledge for quick adaptation to new subjects and actions, reducing the time required for extensive retraining while maintaining generalization capabilities through the two-stage generation-refinement process
Data Source
AI summary
A method, apparatus, non-transitory computer readable medium, and system for generating images includes obtaining an image generation prompt and a reference prompt. The image generation prompt includes a first element and a second element and the reference prompt includes the second element. Embodiments then generate, using an image generation model, an intermediate image based on the reference prompt. Subsequently, embodiments generate, using the image generation model, a synthetic image including the first element and the second element based on the intermediate image and the image generation prompt.