Composition-Aware Image Querying Using Saliency Maps
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional image retrieval systems face challenges in identifying and returning images that match a user's desired composition, as they often require users to sift through numerous irrelevant images, making it difficult to find images with specific objects in desired locations within the image.
Innovation Solution
A computer-operated system that generates a saliency map to identify objects and their locations within an image, allowing users to annotate search queries with region-specific queries, and uses neural language models to prioritize search results based on composition-specific matches, enabling users to find images with specific objects in precise locations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional image retrieval systems return images based on general search queries, then the system can process simple queries, but users must sift through numerous irrelevant images to find desired images with specific objects in specific locations
Solution Approach 1:
The patent segments the image search problem by dividing the image into multiple regions and associating different search queries with different regions. The saliency map divides the image into salient and non-salient regions, allowing users to specify region-specific search criteria. This segmentation enables precise control over which objects should appear in which locations, dramatically improving retrieval accuracy and reducing time to find desired images.
Solution Approach 2:
The patent applies local quality by allowing different search queries to be applied to different regions of the image rather than treating the entire image uniformly. Each region can have its own query requirements, enabling users to specify that certain objects should be in certain locations. This regional query approach directly addresses the problem of finding images with specific compositional requirements.
2Loss of information
If the system indexes and processes entire images for search, then comprehensive image content is captured, but data storage and processing loads increase significantly
Solution Approach 1:
The patent extracts only the essential visual features and semantic content from images rather than processing entire images. The saliency detection algorithm identifies and extracts only the salient regions that contain important objects, ignoring non-salient background areas. This extraction approach maintains comprehensive image content coverage while significantly reducing data storage and processing requirements by focusing only on relevant portions.
Solution Approach 2:
The system performs preliminary saliency detection and region identification before the actual image search and matching processes. By pre-processing images to identify salient regions and generate region-level feature representations, the system prepares condensed image descriptors in advance. This preliminary action reduces the computational load during query processing while maintaining comprehensive content coverage.
Data Source
AI summary
Various aspects of the subject technology relate to systems, methods, and machine-readable media for composition aware image querying. A system may receive user input identifying a search query for content from a client device, where the user input indicates one or more queries assigned to one or more regions of a search input page. The system may generate a query vector for each query using a computer-operated neural language model. The system may compare the query vector to an indexed vector for each region of an image. The system may determine a listing of composition-specific images from a collection of images based on the comparison. The system may determine a ranking for each image in the listing of composition-specific images, and provide search results responsive to the search query to the client device. The search results may include a prioritized listing of the composition-specific images based on the determined ranking.