Text-Image Compatibility Ranking for Ambiguous Visual Retrieval
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing image search systems struggle with limited expressivity and ambiguity when using single-modality queries, as textual descriptions are restrictive and visual queries are ambiguous, making it difficult to accurately retrieve desired images.
Innovation Solution
A neural network architecture that leverages a multi-modal query comprising a reference image and a text modifier to rank candidate images by computing pairwise interactions and compatibility scores, using interaction and compatibility modules to generate vector representations and measure the relationship between the query elements and candidate images.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a textual query is used for image search, then the description is accurate but only partial, but it is difficult to provide an exhaustive textual description
Solution Approach 1:
The patent combines textual queries with visual queries (image uploads) into a multi-modal search system. The text encoder processes textual descriptions while the image encoder processes uploaded reference images, and their embeddings are merged to create a comprehensive query representation that captures both explicit textual requirements and implicit visual characteristics.
Solution Approach 2:
The patent adds a visual dimension to the traditional text-based search by accepting image uploads as queries. This transforms the search from a single-modality (text) to multi-modality (text + image), allowing users to search based on visual similarity while maintaining the ability to provide textual modifiers for refinement.
2Loss of information
If a visual query is used for image search, then the query is much richer but a lot more ambiguous, and it may be difficult to find a query image that corresponds to what a user is looking for
Solution Approach 1:
The patent introduces text modifiers as an intermediary element that bridges the gap between the visual query and the target images. Users can provide textual descriptions that modify or refine the visual query, such as specifying attributes like color, size, or style, which helps disambiguate the visual search and improve precision.
Solution Approach 2:
The search system is designed to be dynamic and adaptive, allowing users to iteratively refine their queries. The system can provide feedback on search results, enabling users to adjust their text modifiers or upload different reference images until they achieve the desired search accuracy.
3Measurement precision
If multi-modal queries with text modifiers are used, then search accuracy is improved, but the system complexity increases
Solution Approach 1:
The patent segments the search system into distinct modular components: a text encoder for processing textual queries, an image encoder for processing visual queries, and a compatibility scorer for evaluating query-image pairs. Each module operates independently with its own embedding space, allowing for simplified training and maintenance while achieving high search accuracy through their coordinated interaction.
Data Source
AI summary
An interaction module includes: a first text-image interaction module configured to generate a vector representation of a first text-image pair based on an encoded representation of a reference image and an encoded representation of a text modifier, the reference image and the text modifier received from a computing device. A second text-image interaction module is configured to generate a vector representation of a second text-image pair based on the encoded representation of the text modifier and an encoded representation of a candidate target image. A compatibility module is configured to compute, based on the vector representation of the first text-image pair and the vector representation of the second text-image pair, a compatibility score for a triplet including the reference image, the text modifier, and the candidate target image. A ranking module is configured to rank a set of candidate target images including the candidate target image by compatibility scores.