Relational Image Querying Using Neural Language Models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing information retrieval systems fail to effectively identify and prioritize images that satisfy specific relationships between objects in search queries, leading to irrelevant results and increased user effort in finding relevant images.
Innovation Solution
A computer-operated system that maps input images to saliency maps to identify objects and their relationships, using a neural language model to generate query vectors for relational querying, allowing users to specify relationships in natural language and prioritize search results based on matching relationships and salient regions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If keyword searching techniques are used for image retrieval, then the system can process generic searches, but it fails to recognize and prioritize images satisfying specified relationships between objects
Solution Approach 1:
The patent segments the image into multiple regions and generates separate vector representations for each region. This segmentation allows the system to independently analyze and compare object relationships in different parts of the image, thereby improving relationship recognition accuracy while maintaining search efficiency through parallel processing of region vectors.
Solution Approach 2:
The patent transforms the image data into a vector space representation, adding a dimensional transformation layer. By converting visual features into vectors and performing operations in this transformed space, the system can effectively measure and compare relationships between objects, improving recognition accuracy without sacrificing search speed.
2Reliability
If the system returns all matching images without prioritization, then comprehensive results are provided, but users must manually filter through large numbers of irrelevant images
Solution Approach 1:
The patent implements a feedback mechanism where the system calculates relationship satisfaction scores for each image based on vector comparisons. These scores provide feedback that enables automatic prioritization and ranking of results, allowing the system to present comprehensive results while simultaneously highlighting the most relevant images, thereby reducing user search time without sacrificing result completeness.
3Measurement precision
If the system performs detailed relationship analysis for each image, then relationship satisfaction is accurately determined, but processing time and computational resources increase
Solution Approach 1:
The patent performs preliminary action by pre-processing images into region vectors and storing them in an indexed structure. This preliminary vectorization allows the system to quickly retrieve and compare image features without performing complex pixel-level analysis during the actual search operation, thereby maintaining high relationship satisfaction accuracy while reducing processing complexity and computational burden during query execution.
Data Source
AI summary
Various aspects of the subject technology relate to systems, methods, and machine-readable media for relational image querying. A system may receive a search query for content from a client device, where the query specifies one or more objects and one or more spatial relationships between the one or more objects. The system may generate a query vector for the query using a computer-operated neural language model. The system may compare the query vector to an indexed vector for each of the one or more spatial relationships between the one or more objects of an image. The system may determine a listing of relational images from a collection of images based on the comparison. The system may determine a ranking for each image in the listing of relational images, and provide search results responsive to the search query to the client device, which may include a prioritized listing of the relational images based on the determined ranking.