Composed Image Retrieval With Trainable Prompts for Zero-Shot Adaptation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing zero-shot composed image retrieval systems face limitations in adaptability and expandability due to the use of predefined connection prompts, which reduce expressiveness and performance.
Innovation Solution
A method and system that trains a zero-shot composed image retrieval system by dynamically generating and training base and condition prompts using text-only inputs, employing multimodal encoders for image and text compatibility, and utilizing a prompt learning technique to enhance adaptability and accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If predefined connection prompts are used to construct inputs for the retrieval system, then the training process is simplified, but the expressiveness and adaptability of the model are reduced
Solution Approach 1:
The patent applies dynamics by making the prompts trainable and adaptive rather than static and predefined. The base prompt and condition prompt are learned during training to dynamically adapt to different input distributions and retrieval tasks, allowing the model to optimize prompt representations based on actual data patterns while maintaining training feasibility through structured learning objectives.
Solution Approach 2:
The patent changes the parameters of the prompts from fixed predefined values to trainable parameters that can be optimized during training. This allows the system to learn optimal prompt representations that balance expressiveness and adaptability, with the prompt embeddings being adjusted based on training data to achieve better retrieval performance across diverse scenarios.
2Ease of operation
If predefined connection prompts are used to construct inputs for the retrieval system, then implementation is easier, but the performance and responsiveness to various image and text expressions are reduced
Solution Approach 1:
The patent implements dynamic prompts that are trained to adapt to different retrieval scenarios, improving performance and responsiveness while maintaining ease of operation through automated training procedures. The system learns optimal prompt representations that enhance its ability to handle various image and text expressions without requiring complex manual tuning.
Solution Approach 2:
The patent enables the system to self-optimize its prompts through automated training processes. The model learns to generate effective prompts by itself during training, reducing the need for manual intervention and expert tuning while achieving improved performance across diverse retrieval tasks through self-directed optimization.
Data Source
AI summary
Provided are a zero-shot composed image retrieval method and system. The zero-shot composed image retrieval method which is performed by the zero-shot composed image retrieval system includes acquiring, by a zero-shot composed image retrieval system, an image embedding by inputting an input image into a visual encoder, generating, by the zero-shot composed image retrieval system, an image-projected token by inputting the image embedding into a projection module, generating, by the zero-shot composed image retrieval system, a composed string based on a pre-trained base prompt, the image-projected token, a pre-trained condition prompt, and input text, generating, by the zero-shot composed image retrieval system, a composed embedding by inputting the composed string into a text encoder, and extracting, by the zero-shot composed image retrieval system, one candidate image from among a plurality of candidate images that are retrieval targets using the composed embedding.