Zero-Shot Sketch Image Retrieval via Unified Binary Codes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current sketch-based image retrieval (SBIR) techniques are inadequate for identifying images from unknown categories, as they require prior training on specific categories and fail to effectively retrieve images outside the learned categories, leading to limitations in real-world applications.
Innovation Solution
The implementation of a neural network architecture comprising three interconnected networks - a sketch encoding network, an image encoding network, and a multi-modal training network - that generates unified binary codes for sketches and images, allowing zero-shot image recognition and retrieval without prior training on all categories, leveraging semantic knowledge to generalize across unseen categories.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If pre-training is performed on specific categories, then retrieval accuracy for known categories is improved, but the system fails to retrieve images from unknown categories
Solution Approach 1:
The patent applies universality by training the neural network on a limited set of known categories during pre-training, but enabling the system to perform retrieval functions for unknown categories as well. The encoder learns universal visual features that generalize across categories, allowing the same model to handle both seen and unseen categories without requiring separate training for each category.
Solution Approach 2:
The patent changes the parameter representation by using learned visual features and binary codes instead of category-specific parameters. The encoder transforms images and sketches into a unified feature space where category information is encoded in the visual features themselves rather than in explicit category labels, enabling generalization to unknown categories.
2Productivity
If traditional SBIR techniques are used with pre-training on specific categories, then retrieval speed is improved for known categories, but computational resources are wasted on categories that may not be queried
Solution Approach 1:
The patent applies preliminary action by performing pre-training on a limited set of categories to establish universal visual feature representations, rather than training on all possible categories. This preliminary training enables fast retrieval for known categories while the universal features allow the system to handle unknown categories without additional training overhead, optimizing the trade-off between retrieval speed and computational resource usage.
3Adaptability or versatility
If the system is trained on all possible categories, then complete coverage is achieved, but the complexity of training and storage requirements increase significantly
Solution Approach 1:
The patent achieves complete category coverage through universality by learning visual features that are category-agnostic. The encoder captures fundamental visual patterns (edges, textures, shapes) that are universal across all categories, allowing the system to retrieve images from any category without requiring explicit training on that category, thereby avoiding the complexity of training on all possible categories while maintaining complete adaptability.
Solution Approach 2:
The patent uses copying by creating binary code representations of visual features that can be stored and retrieved efficiently. Instead of storing and processing full images or complex feature vectors for all categories, the system copies essential visual information into compact binary codes, reducing storage requirements and computational complexity while preserving the ability to perform accurate retrieval.
Data Source
AI summary
This disclosure relates to improved sketch-based image retrieval (SBIR) techniques. The SBIR techniques utilize an architecture comprising three interconnected neural networks to enable zero-shot image recognition and retrieval based on free-hand sketches. Zero-shot learning may be implemented to retrieve one or more images corresponding to the sketches without prior training on all categories of the sketches. The neural network architecture may do so, at least in part, by training encoder hashing functions to mitigate heterogeneity of sketches and images, and by applying semantic knowledge that is learned during a limited training phase to unknown categories.