Multimodal Image Editing Masks for Multi-Instruction Prompts
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing image editing systems using generative AI models struggle with handling multi-instruction prompts and non-applicable editing instructions, leading to over-editing and inaccurate results due to imprecise attention masks and external preprocessing dependencies.
Innovation Solution
The use of multimodal large language models (MLLMs) to generate precise editing masks and negative tokens, enabling the generative AI models to distinguish applicable and non-applicable instructions, thereby improving image editing accuracy and efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If external preprocessing is used to generate attention masks, then image editing can be performed, but device complexity increases and preprocessing time is lost
Solution Approach 1:
The patent extracts the attention mask generation capability from external preprocessing tools and integrates it directly into the diffusion model's transformer architecture. The transformer decoder generates attention masks internally during the diffusion process, eliminating the need for separate external preprocessing steps while reducing overall system complexity.
Solution Approach 2:
The patent merges the attention mask generation function with the diffusion model's transformer architecture. The transformer decoder simultaneously performs both the diffusion process and attention mask generation, combining what were previously separate external preprocessing and editing operations into a single integrated system.
2Productivity
If imprecise attention masks are used, then image editing can be performed, but manufacturing precision deteriorates leading to over-editing
Solution Approach 1:
The patent implements feedback mechanisms where the transformer decoder continuously refines attention masks based on the diffusion process outputs. The cross-attention mechanism provides feedback loops that adjust attention weights in real-time, ensuring precise editing regions are identified and preventing over-editing while maintaining efficient processing speeds.
Solution Approach 2:
The patent employs dynamic attention masks that evolve during the diffusion process rather than using static pre-computed masks. The attention mechanism adapts dynamically to the image content and editing requirements, allowing precise control of editing regions while maintaining computational efficiency throughout the generation process.
3Adaptability or versatility
If multi-instruction prompts are processed without distinction, then versatility improves, but reliability deteriorates due to non-applicable instructions
Solution Approach 1:
The patent segments the prompt processing into distinct instruction types using token classification. The transformer decoder identifies and separates applicable instructions (生成 token) from non-applicable instructions (忽略 token), allowing the system to handle multi-instruction prompts while filtering out irrelevant ones to maintain reliability.
Solution Approach 2:
The patent changes the parameter state of instructions through token embedding and classification. By transforming prompt instructions into different token types (applicable vs. non-applicable) with distinct embeddings, the system can process diverse multi-instruction prompts while reliably distinguishing which instructions should be executed and which should be ignored.
Data Source
AI summary
Provided are systems, methods, and apparatuses for systems and methods of image editing based on multimodal large language models. In one or more examples, the systems, devices, and methods include generating image tokens from an input image and word tokens from an editing prompt; generating a mask token based on an artificial intelligence model processing the image tokens and the word tokens; and generating an editing mask based on a mask decoder processing the mask token, word embeddings of the editing prompt, and visual embeddings of the input image. In one or more examples, the systems, devices, and methods include generating a correlation map that correlates the editing mask to a set of one or more words of the editing prompt; and generating an output image based on the correlation map, the output image comprising an edited version of the input image according to the editing prompt.