Sketch and Keyword Fusion for Image Retrieval
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing content-based image retrieval methods face challenges in accurately retrieving images due to subjective text descriptions and the difficulty in expressing semantic information using objective features like color and texture, often requiring the original image for retrieval and lacking precision in matching semantic features.
Innovation Solution
A method and apparatus that acquire feature matrices from a sketch of a target item and a keyword set, combined with feature extraction matrices from an image set, to determine matching degrees and select relevant images based on these matrices, allowing for retrieval without the original image and improving semantic matching accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If text-based image retrieval is used with text descriptions, then the retrieval process can be performed without original images, but the retrieval accuracy deteriorates due to subjective text descriptions
Solution Approach 1:
The patent combines sketch-based retrieval and keyword-based retrieval into a unified framework. The final retrieval score is computed by fusing the sketch matching degree and keyword matching degree, thereby merging two different retrieval approaches to overcome their individual limitations and achieve both operational convenience and retrieval accuracy.
2Reliability
If content-based image retrieval uses only original images with color and texture features, then rich visual features are available, but the method requires the original image and cannot retrieve items from sketches
Solution Approach 1:
The patent creates a universal retrieval system that can accept multiple types of input (original images, sketches, and keywords) and process them through a unified framework. The system maintains the ability to use rich color and texture features when original images are provided, while also gaining the flexibility to retrieve items from sketches and keyword descriptions.
Solution Approach 2:
The patent introduces an intermediary mapping process that converts sketches into feature matrices comparable with image features. This intermediary representation allows sketches to be bridged to the content-based retrieval space, enabling retrieval without requiring the original image while maintaining access to rich visual features.
3Ease of manufacture
If objective features like color and texture are extracted from images, then feature extraction is straightforward, but semantic information of the image cannot be effectively expressed
Solution Approach 1:
The patent combines objective visual features (color, texture) with subjective semantic features (keywords) in a unified retrieval framework. By fusing the matching degrees from both sketch-based visual comparison and keyword-based semantic comparison, the system recovers semantic information that would be lost using only objective feature extraction.
Data Source
AI summary
A method for retrieving an image is provided. The method comprises: acquiring a first matrix obtained by performing feature extraction on a sketch of a target item; acquiring a second matrix composed of word vectors of keywords in a keyword set corresponding to the target item; acquiring a third matrix set obtained by performing feature extractions on images in an image set respectively; determining, for a third matrix in the third matrix set, a comprehensive matching degree between an item presented in an image corresponding to the third matrix and the target item, based on a first matching degree between the first matrix and the third matrix and a second matching degree between the second matrix and the third matrix; and selecting a preset number of images from the image set based on the determined comprehensive matching degrees, and sending the selected images.