Unified Vector Search for Image-Text Retrieval
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current mechanisms for searching large volumes of digital content using images as queries are inefficient due to the need for multiple classifiers to convert images into textual content, leading to errors and classification difficulties, especially in cases with high cardinality classification problems.
Innovation Solution
Encoding images and textual content into multidimensional vectors using image and text models, allowing input images to directly search for close vectors in both image and text graphs without relying on classifiers, by tuning the models to map corresponding text and images to proximate points in a shared high-dimensional space.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If classifier-based mechanisms are used to convert images into textual content for searching, then text-based search functionality can be achieved, but classification errors propagate through the system and reduce search accuracy
Solution Approach 1:
The patent extracts and eliminates the classifier component from the search system. Instead of using classifiers to convert images to text, the system directly encodes images into vectors that can be searched alongside text vectors in a unified space, removing the source of classification errors entirely
Solution Approach 2:
The patent introduces vector encoding as an intermediary mechanism between images and text search. Both images and text are transformed into vectors through separate encoders, creating a common representation space that allows direct comparison without requiring image-to-text classification
2Adaptability or versatility
If multiple classifiers are used to handle diverse image classifications, then broader image recognition capability is achieved, but system complexity and training difficulty increase significantly
Solution Approach 1:
The patent merges the previously separate image classification and text search systems into a unified vector-based search system. Both images and text are represented as vectors in the same space, allowing a single search mechanism to handle both modalities without requiring multiple specialized classifiers
Solution Approach 2:
The patent creates a universal vector encoding system that can handle both images and text through the same underlying mechanism. The image encoder and text encoder both produce vectors that can be searched using the same proximity-based search algorithm, providing multi-functionality without additional complexity
3Productivity
If traditional text-based search mechanisms are used, then search efficiency is maintained, but the ability to search using input images is limited or non-existent
Solution Approach 1:
The patent creates a vector copy or representation of images that exists in the same space as text vectors. By encoding images into vectors, the system can use the same efficient text search mechanisms to search both text and image data, maintaining search efficiency while expanding input flexibility
Data Source
AI summary
Images are encoded into multidimensional vectors in a high-dimensional space utilizing an image model and textual content utilizing a text model. At least one of the image model and/or the text model are tuned such that the point within the multidimensional space pointed to by a vector encoded from an image is proximate to, as determined within the context of that multidimensional space, a point pointed to by a vector encoded from correlated textual content. Received images and textual content are encoded into image vectors and text vectors, respectively, and stored in an image graph and text graph, respectively. An input image can then be encoded as an input image vector and utilized to find close vectors in both the image graph and the text graph, thereby enabling an input image to be utilized to search textual content without using classifiers to deduce textual content therefrom.