Image Editing Model Selection via LLM Prompt Rewriting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing generative AI models for image editing face challenges in interpreting ambiguous user prompts, misinterpreting context, and being computationally expensive, and struggle to handle a wide range of user intentions for modifying images, such as preserving structure and shape or replacing objects.
Innovation Solution
A method involving a large language model (LLM) to rewrite prompts and select from different machine-learning models based on user intent, including structure-preserving, shape-preserving, and non-structure/non-shape preserving models, to generate output images that meet clear and concise instructions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a single generative AI model is used for image editing, then the system is simple to operate, but the model cannot accurately handle different user intentions (structure preservation, shape preservation, object replacement)
Solution Approach 1:
The patent segments the image editing task into three distinct model types (structure-preserving, shape-preserving, and non-structure/non-shape preserving models), each specialized for specific user intentions. The system divides the broad image editing functionality into specialized sub-tasks, allowing each model to excel at its specific function rather than one model trying to do everything.
Solution Approach 2:
The patent introduces an LLM-based prompt rewriting system as an intermediary between the user and the generative models. This mediator translates ambiguous user prompts into precise instructions and determines which specialized model should be used, enabling the system to handle diverse user intentions without requiring users to understand the complex model distinctions.
2Measurement precision
If multiple specialized machine-learning models are used, then the accuracy of interpreting user prompts improves, but the computational cost and processing time increase
Solution Approach 1:
The patent applies preliminary action by using the LLM to rewrite and clarify prompts before they are processed by any generative model. This pre-processing step ensures that the prompt is optimized and the appropriate model is selected in advance, preventing the need for multiple iterative attempts and reducing overall processing time despite having multiple specialized models.
Solution Approach 2:
The patent implements dynamics by making the model selection process adaptive and context-dependent. The system dynamically chooses which specialized model to use based on the rewritten prompt content and user intentions, rather than using a fixed model for all tasks. This dynamic selection ensures computational efficiency by deploying only the necessary model for each specific editing task.
3Manufacturing precision
If existing generative AI models are used directly, then the system is computationally efficient, but the models produce artifacts and fail to preserve structure and shape
Solution Approach 1:
The patent applies local quality by using different specialized models for different regions and types of editing tasks. The structure-preserving model is used when structural integrity is critical, the shape-preserving model when maintaining object shape is important, and the non-structure/non-shape preserving model for complete replacements. This localized application of specialized models ensures high precision where needed without wasting computational resources on tasks that don't require such precision.
Data Source
AI summary
A computer-implemented method includes receiving an initial image and an original prompt from a user, wherein the original prompt includes a request to modify the initial image. The method further includes selecting, based on the original prompt, a machine-learning model from a set of machine-learning models. The method further includes providing the original prompt and the initial image as input to a large language model (LLM). The method further includes receiving, from the LLM and based on the original prompt and the initial image, a rewritten prompt. The method further includes selecting, based on the rewritten prompt, a machine-learning model from a set of machine-learning models. The method further includes generating, by the selected machine-learning model, an output image that satisfies the rewritten prompt.