Guided Visual Search Sketch Refinement for Relevant Results
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current visual search systems face challenges when users lack a suitable visual example, leading to ambiguous or incomplete sketched queries, resulting in low-quality search results and user frustration, as they often produce large sets of irrelevant images.
Innovation Solution
A guided visual search system that iteratively presents users with clusters of search results and sketch suggestions, using neural networks to encode and cluster images based on structural and semantic similarities, allowing users to interactively refine their queries through incremental modifications.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If users sketch visual queries manually, then the system can accept queries without requiring suitable visual examples, but the sketches are often ambiguous or incomplete leading to low quality search results
Solution Approach 1:
The system presents clusters of search results to users and receives feedback through their selections. This feedback loop allows the system to iteratively refine and disambiguate incomplete sketches by understanding user intent from their choices among result clusters.
Solution Approach 2:
The system performs preliminary clustering of search results based on the initial sketch before presenting them to the user. This preliminary action organizes ambiguous results into manageable groups, making it easier for users to provide meaningful feedback even when their sketch is incomplete.
2Quantity of substance
If the system presents many search results to users, then comprehensive coverage is achieved, but users experience frustration due to large sets of irrelevant images
Solution Approach 1:
The system segments the large set of search results into multiple clusters based on semantic similarity. Instead of presenting all results in a single undifferentiated list, users see organized groups that represent different interpretations of their sketch, making it easier to identify relevant content.
Solution Approach 2:
The system performs clustering beyond what a single query might strictly require, creating multiple potential result groups. This excessive action ensures that even if the sketch is ambiguous, at least one cluster will contain relevant results, improving user experience without losing comprehensive coverage.
3Shape
If visual search systems cluster results based on structural similarity, then matching images are grouped together, but results of different semantic categories are included (e.g., mushrooms, umbrellas, street signs)
Solution Approach 1:
The system adds a semantic dimension to the clustering process by analyzing both structural similarity and semantic meaning of images. This multi-dimensional approach ensures that results are grouped not only by visual appearance but also by conceptual category, preventing semantically unrelated items from being clustered together.
Data Source
AI summary
Embodiments of the present invention provide systems, methods, and computer storage media for guided visual search. A visual search query can be represented as a sketch sequence that includes ordering information of the constituent strokes in the sketch. The visual search query can be encoded into a structural search encoding in a common search space by a structural neural network. Indexed visual search results can be identified in the common search space and clustered in an auxiliary semantic space. Sketch suggestions can be identified from a plurality of indexed sketches in the common search space. A sketch suggestion can be identified for each semantic cluster of visual search results and presented with the cluster to guide a user towards relevant content through an iterative search process. Selecting a sketch suggestion as a target sketch can automatically transform the visual search query to the target sketch via adversarial images.