Drag-Based Image Editing with Optical Flow and Diffusion Models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional drag-based image editing methods require significant computational resources and memory, necessitate additional user input like text or masks, and are not optimized for real-time high-quality editing.
Innovation Solution
An image editing apparatus and method utilizing a two-stage AI model architecture, comprising a GAN-based optical flow generation module (FlowGen) and a diffusion model-based image generation module (FlowDiffusion), trained on video data to generate optical flow and edit images in real-time without additional user input, optimizing computation and memory usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If conventional drag-based image editing methods are used to achieve natural motion reflection, then editing quality is improved, but computational resources and memory requirements increase significantly
Solution Approach 1:
The patent segments the image editing process into two distinct AI models: FlowGen (generative adversarial network) for optical flow generation and FlowDiffusion (diffusion model) for image editing. This segmentation allows each model to specialize in specific tasks, improving overall efficiency while maintaining high editing quality with reduced computational overhead compared to using a single comprehensive model.
Solution Approach 2:
The patent performs preliminary action by pre-training the FlowGen and FlowDiffusion models on video data before actual image editing. This pre-training enables the models to learn natural motion patterns in advance, so that during actual editing operations, the system can quickly generate optical flow and edit images without requiring extensive real-time computation, thus reducing memory and computational resource requirements.
2Manufacturing precision
If conventional drag-based image editing methods are used to achieve natural motion reflection, then editing quality is improved, but processing time increases
Solution Approach 1:
By dividing the editing process into two specialized models (FlowGen for motion analysis and FlowDiffusion for image synthesis), the patent enables parallel processing and optimized computation paths. This segmentation reduces the time required for each individual task while maintaining overall editing quality, achieving real-time or near-real-time performance.
Solution Approach 2:
The pre-training of models on video data performs preliminary learning of natural motions, so that during actual editing, the models can quickly infer and apply appropriate transformations without extensive real-time computation. This preliminary action significantly reduces processing time while preserving the quality of motion reflection.
3Manufacturing precision
If additional user input (text or masks) is required for image editing, then editing precision is improved, but ease of operation deteriorates
Solution Approach 1:
The patent implements self-service by enabling the drag-based editing system to automatically determine editing parameters and motion patterns without requiring additional user input such as text prompts or mask drawings. The FlowGen model extracts motion information directly from the drag gesture, and FlowDiffusion uses this information to perform editing autonomously, maintaining high precision while significantly improving ease of operation.
4Quantity of substance
If optimization-based schemes are used for image editing, then resource efficiency is improved, but editing speed deteriorates
Solution Approach 1:
The patent replaces traditional optimization-based mechanical systems with AI-based probabilistic models. Instead of iteratively optimizing parameters through computation-intensive optimization algorithms, the pre-trained FlowGen and FlowDiffusion models directly generate and synthesize images based on learned patterns, achieving both resource efficiency and high editing speed without the trade-off present in optimization-based approaches.
Data Source
Figure 1~2
Figure 3~5
Figure 6~8
AI summary
Proposed are an image editing apparatus and method. According to an aspect of the present invention, there is provided an image editing apparatus including: an input/output interface configured to obtain a drag input instruction and an image; and a controller configured to obtain an optical flow based on the drag input instruction and the image by using a first artificial intelligence model that is trained to receive a drag input instruction and an image as input and output an optical flow, to input the optical flow and the image to a second artificial intelligence model that is different from the first artificial intelligence model, thereby obtaining an edited image as an output of the second artificial intelligence model, and to provide the edited image.