Multi-task Image Search Model with Balanced Embeddings
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current search engines struggle to effectively process user queries involving images, particularly in identifying similar images from diverse datasets, including user-uploaded and professional images, due to biases in training data and sparse attributes, leading to suboptimal retrieval results.
Innovation Solution
A multi-task classification model is trained using separate classification heads and datasets for various tasks, including low-resolution and high-resolution image classification, with a data sampler that balances training data across tasks and utilizes triplet loss for deep metric learning, to generate embeddings that capture visual attributes like color, material, and shape, improving the retrieval of similar images.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a single classification model is used for image search, then the model structure is simple, but the retrieval accuracy for diverse image types deteriorates
Solution Approach 1:
The patent divides the classification model into multiple independent classification heads, each specialized for a specific task (e.g., user-uploaded image classification, professional image classification, attribute-based classification). This segmentation allows each head to optimize for its specific domain while sharing the same backbone network, thereby improving retrieval accuracy for diverse image types without completely redesigning the entire model architecture.
Solution Approach 2:
The patent implements a multi-functional classification model where a single backbone network supports multiple classification heads that can handle different image types and classification tasks simultaneously. This universal structure enables the model to process both user-uploaded and professional images, as well as perform attribute-based retrieval, all within one integrated system, thus improving overall retrieval accuracy without proportionally increasing complexity.
2Measurement precision
If training data is not balanced across classification tasks, then the training process is simple, but the model performance on rare image types deteriorates
Solution Approach 1:
The patent applies different training strategies to different classification heads based on their specific requirements. For tasks with rare image types, the model uses reweighted sampling and task-specific loss functions that emphasize rare categories during training. This localized quality adjustment ensures that each classification head receives appropriately balanced training data for its specific domain, improving performance on rare image types without requiring complete redesign of the training pipeline.
Solution Approach 2:
The patent modifies training parameters including sampling weights and loss function parameters for different classification tasks. By adjusting these parameters dynamically based on task requirements and data distribution, the model can handle imbalanced datasets effectively, ensuring that rare image types receive adequate training attention without complicating the overall training process.
3Measurement precision
If separate loss functions are used for each classification task, then each task optimizes independently, but the training time increases
Solution Approach 1:
The patent combines multiple classification tasks into a single unified training framework where different classification heads share the same backbone network and are trained simultaneously using combined loss functions. This merging approach allows the model to learn shared features across tasks while maintaining task-specific optimization through separate loss weighting, thereby achieving task-specific performance improvements without proportionally increasing training time.
Solution Approach 2:
The patent applies partial optimization strategies where not all classification heads are trained with equal intensity. Instead, the model uses reweighted sampling and selective loss function application to focus training on critical tasks and rare image types, while maintaining reasonable performance on common tasks. This partial action approach achieves effective task-specific optimization without the full computational cost of uniformly optimizing all tasks.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for searching with images One of the methods includes receiving a query requesting to search a dataset of images using a first input image; inputting the first image data into a multi-task classification model trained to identify one or more images from the dataset of images; obtaining, as output from a layer of the neural network preceding the plurality of classification heads used for training and in response to the first image data, a first embedding for the first image data; identifying, using the first embedding and from among a plurality of embeddings corresponding to images in the dataset of images, a set of images that are similar to the first input image; and providing, in response to the received query and for display on a client device, the set of images.