CNN Image Retrieval Training with Listwise Ranking Loss
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current image retrieval methods face challenges in efficiently training convolutional neural networks (CNNs) for ranking images based on similarity, particularly with high-resolution images, due to memory constraints and the need for costly pre-training and hard-negative mining.
Innovation Solution
A listwise ranking loss function is used to directly optimize the mean average precision (mAP) metric, enabling training on existing GPUs with arbitrary batch sizes and image resolutions, without pre-training or hard-negative mining, through a multi-stage optimization scheme that computes global feature descriptors and gradients efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional image retrieval methods are used with pre-training and hard-negative mining, then ranking accuracy can be improved, but training time and computational cost increase significantly
Solution Approach 1:
The patent extracts and removes the unnecessary pre-training and hard-negative mining steps from the traditional training pipeline. By directly optimizing the mean average precision metric through listwise ranking loss, the method eliminates these intermediate steps while maintaining or improving ranking accuracy, thereby reducing training time and computational cost.
Solution Approach 2:
The patent performs preliminary optimization by directly formulating the training objective to maximize mean average precision from the start. Instead of requiring sequential pre-training followed by fine-tuning with hard-negative mining, the method establishes the final optimization goal upfront through listwise ranking loss, enabling direct training that achieves comparable or better accuracy faster.
2Measurement precision
If high-resolution images are used for training, then retrieval quality improves, but memory requirements exceed available GPU memory
Solution Approach 1:
The patent segments the training process into multiple stages: first computing global feature descriptors for all training images and storing them, then processing query-images and database-images in smaller batches. This segmentation allows high-resolution images to be processed by extracting their global descriptors once, then reusing these descriptors in subsequent training iterations without loading entire high-resolution images into memory repeatedly.
Solution Approach 2:
The patent creates and stores copies of global feature descriptors extracted from high-resolution training images. These descriptor copies are saved to disk or memory and reused during the actual training process, allowing the model to work with compressed representations rather than the full high-resolution images, thereby reducing memory requirements while preserving retrieval quality.
3Productivity
If arbitrary batch sizes are used for training, then training efficiency can be optimized, but training becomes infeasible on existing GPUs due to memory constraints
Solution Approach 1:
The patent segments the training data processing into two phases: global descriptor extraction for all images, then batched processing of query-image and database-image pairs. This allows flexible batch sizes to be used in the second phase without requiring all high-resolution images to be in memory simultaneously, enabling training on existing GPUs with arbitrary batch sizes while maintaining training efficiency.
4Productivity
If listwise ranking loss function is used to directly optimize mAP, then training computational efficiency improves, but the complexity of the loss function and optimization algorithm increases
Solution Approach 1:
The patent introduces global feature descriptors as an intermediary representation between high-resolution images and the ranking optimization process. By first extracting these descriptors and using them as the basis for listwise ranking loss computation, the method simplifies the optimization algorithm while maintaining the ability to directly optimize mAP, as the descriptors serve as a compact intermediate representation that reduces computational complexity.
Data Source
AI summary
A method of performing image retrieval includes: obtaining a query image; generating a global feature descriptor of the query image by inputting the query image into a convolutional neural network (CNN) and obtaining the global feature descriptor as an output of the CNN, where parameters of the CNN are learned during training of the CNN on a batch of training images using a listwise ranking loss function and optimizing a quantized mean average precision ranking evaluation metric; determining similarities between the query image and other images based on distances between the global feature descriptor of the query image and global feature descriptors of the other images, respectively; ranking the other images based on the similarities, respectively; and selecting a set of the other images based on the similarities between the query image and the other images.