Contextualized Visual Search Using Deep Learning for Unseen Query Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current visual search systems in e-commerce are ineffective in identifying product images that contain a query image, especially when the query image is small or unseen, and struggle to provide real-time search results for large numbers of users, due to reliance on similarity of scale-invariant features and lack of extensive metadata in training datasets.
Innovation Solution
The implementation of a contextualized visual search system using machine learning techniques that scans and isolates target images containing a query image, employing a smart caching system and a Deep Learning model trained on object detection datasets to detect query images regardless of size or transformation, and personalizes search results based on user behavior.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If state-of-the-art visual search systems use similarity of scale-invariant features, then they can quickly process images, but they fail to reliably identify product images with 'contains' relationship when query image is small or unseen
Solution Approach 1:
The system changes the parameter of feature extraction by using Deep Learning models (Inception-v3, ResNet-50) that can learn hierarchical representations of images. These models process query images and target images through multiple layers, transforming pixel data into abstract features that capture semantic meaning, enabling reliable detection of small or unseen objects while maintaining speed
Solution Approach 2:
The patent replaces traditional mechanical image processing methods with neural networks. Instead of using hand-crafted features like SIFT or HOG that rely on scale-invariant geometric properties, the system uses deep neural networks that automatically learn optimal features from data, achieving both reliability and adaptability to unseen images
2Productivity
If visual search systems process large numbers of images for real-time search, then user engagement improves, but system complexity and processing time increase
Solution Approach 1:
The system segments the image processing task into distinct stages: (1) extracting features from query and target images using pre-trained neural networks, (2) computing similarity scores between extracted features, and (3) ranking and returning results. This segmentation allows each component to be optimized independently and improves overall processing efficiency
Solution Approach 2:
The system performs preliminary actions by pre-processing images to extract features and compute similarity scores before final ranking. The neural networks are pre-trained on large datasets, and image features are pre-computed and stored, enabling fast retrieval during actual search operations without reprocessing entire images
3Measurement precision
If the query image occupies only a small area of the product image, then the search becomes more challenging, but current systems cannot reliably identify such relationships
Solution Approach 1:
The system applies local quality by focusing the neural network attention on specific regions of interest within images. The Inception-v3 and ResNet-50 models use multi-scale convolutional filters that can detect patterns at different locations and sizes, enabling precise identification of small query images within larger product images while maintaining overall context understanding
Data Source
AI summary
Embodiments of a contextualized visual search (CVS) system are disclosed capable of isolating target images of items that contain instances of a previously-unseen query image from a large database of target images. In embodiments, the system is used to implement an interactive query interface of an e-commerce portal, which allows the user to specify the query image (e.g. a logo) to be searched. The system converts the query image into a feature vector using a first machine learning model, and compares the feature vector to feature vectors of target images using a second machine learning model to find matching target images that contain an instance of the query image. The system then returns a query result indicating a list of items associated with matched target images. In embodiments, the query results may be ranked based on a set of personalized factors associated with the user.