Instruction-Guided Image Editing with LLM Feature Fusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing image editing models based on stable diffusion (SD) struggle with weak text interpretation and reasoning capabilities, leading to difficulties in comprehending complex editing instructions and generating images that do not conform to the intended content.
Innovation Solution
An image editing method that integrates a large language model (LLM) to enhance text comprehension and reasoning, fusing image and instruction features to accurately determine objects and perform editing operations, using an end-to-end integrated structure with modules for instruction interpretation and image editing, and incorporating dimension transformation to improve computational efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If an image editing model based on stable diffusion is used, then image editing functionality is provided, but text interpretation capability and reasoning capability are weak, making it difficult to comprehend complex editing instructions
Solution Approach 1:
The patent combines the stable diffusion model with a large language model into an integrated image editing system. The text encoder from the large language model is used to enhance text interpretation capability, while the stable diffusion model handles image generation. This merging allows the system to comprehend complex editing instructions accurately while maintaining image editing functionality.
Solution Approach 2:
The patent introduces a text encoder as an intermediary component between the editing instruction input and the image generation process. This text encoder, derived from the large language model, processes and interprets the textual editing instructions, transforming them into a format that the stable diffusion model can utilize for accurate image editing.
2Measurement precision
If feature fusion is performed between image features and instruction text features, then accuracy of determining objects and editing operations is improved, but computational complexity increases
Solution Approach 1:
The patent segments the feature fusion process into distinct components: image feature extraction from the input image, text feature extraction from the editing instruction, and subsequent fusion of these features. This segmentation allows for systematic processing and reduces the overall complexity by breaking down the fusion task into manageable stages.
Solution Approach 2:
The patent transforms the editing instruction from pure text into a multi-dimensional feature space by extracting text features and fusing them with image features. This dimensionality change enables the system to process and compare textual instructions with visual data in a unified feature space, improving object identification accuracy while managing computational complexity through structured transformation.
Data Source
AI summary
An image editing method includes obtaining an input image and an editing instruction for the input image; extracting, from the input image, a first image feature comprising a plurality of feature items; extracting, from the editing instruction, an instruction text feature comprising a plurality of feature items; fusing a target image feature with the instruction text feature, to obtain a fused feature, the fused feature being configured to represent a description of executing the editing instruction on the input image, and the target image feature comprising the first image feature; determining, according to the fused feature, a first object in the input image and an editing operation on the first object; performing the editing operation on the first object, to generate an edited image; and merging the edited image with the input image, to obtain a target image.