User Drawing Based Image Search Using Convolutional Neural Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current image search engines using text-based and upload-based methods often fail to accurately identify the most relevant visual content, as they may interpret user intent differently or require burdensome image uploads.
Innovation Solution
A computer-implemented method and system that uses a convolutional neural network to analyze user drawings and identify semantic concepts, comparing feature vectors of images to those of user drawings to generate relevant search results, thereby improving the accuracy and efficiency of image retrieval.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If text-based image search is used, then the search process is simple and fast, but the search accuracy does not reflect user's desired content
Solution Approach 1:
The patent introduces an intermediary system that translates user drawings into search queries. Instead of directly using text keywords or uploaded images, the system uses a drawing analysis module to interpret user sketches and convert them into effective search queries, thereby improving search accuracy while keeping the user interface simple
Solution Approach 2:
The patent replaces the traditional mechanical approach of text parsing or image uploading with a computational drawing analysis system. The system uses algorithms to analyze drawing strokes, identify shapes and patterns, and map them to relevant search terms, substituting manual text input or image selection with automated drawing interpretation
2Measurement precision
If upload-based image search is used, then the search can find visually similar images, but it requires burdensome image uploads
Solution Approach 1:
The patent replaces the need for permanent image uploads with temporary, lightweight drawing inputs. Users create simple drawings that are processed immediately and discarded, rather than uploading substantial image files that occupy storage and require file management
Solution Approach 2:
Instead of requiring users to upload complete images and then search for similarities, the patent inverts the approach by having users draw simplified representations and having the system generate the full search queries and results, reversing the traditional flow of image-based search
3Reliability
If standard text-based or upload-based search is used, then the search process is straightforward, but it may not accurately reflect user's desired content
Solution Approach 1:
The patent changes the input parameter from text keywords or complete images to drawing strokes and shapes. This parameter change allows the system to capture user intent more naturally through visual sketching, while the underlying system processes these drawings by extracting features and mapping them to relevant content
Data Source
AI summary
Various aspects of the subject technology relate to systems, methods, and machine-readable media for user drawing based image search. These aspects include an image retrieval system using a convolutional neural network trained to identify how users draw semantic concepts and using an image search engine to search against images having a similar concept. The aspects include mapping between concepts of the user drawing space and concepts of the image space such that images associated with the same concept are identified. For each input user drawing, the drawing is first processed through a concept classifier to identify a corresponding concept, and then through a feature extractor to form a corresponding feature vector. The results from the concept classifier and the feature extractor may be combined to search against a collection of images having a similar concept to determine a listing of images ranked by visual and semantic similarity to the input.