ANNS Hard-Negative Sampling for Scalable Extreme Classifier Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current extreme classification methods face challenges in achieving state-of-the-art accuracy while keeping per-epoch training costs low, especially when dealing with extremely large label sets, as they either require excessive computing resources or prolonged training times due to inefficient negative sampling strategies.
Innovation Solution
The ASTRA algorithm employs a mixed negative sampling strategy that combines importance sampling and uniform sampling, using an Approximate Nearest Neighbor Search (ANNS) index on classifier vectors to identify hard negatives, and updates these indices periodically to maintain efficiency and accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If joint training of classifiers and encoder is used, then accuracy is improved, but per-epoch training time scales as O(L) which is too slow for large label sets
Solution Approach 1:
The training process is segmented into two distinct stages: a first stage for pre-training the encoder and a second stage for training the classifiers. This segmentation allows each stage to be optimized independently, achieving both accuracy and efficiency. The encoder is pre-trained on a subset of data, then the classifiers are trained in a second stage using the pre-trained encoder, avoiding the need to re-train everything from scratch and reducing per-epoch time complexity.
Solution Approach 2:
The encoder is trained in advance during a first stage before the classifiers are trained in the second stage. This preliminary action of pre-training the encoder allows the more computationally intensive encoder training to be completed once, and then the classifiers can be trained more efficiently in subsequent stages using the pre-trained encoder, reducing overall training time.
2Loss of time
If staged training approach is used, then per-epoch costs are reduced to O(log L), but the clustering procedure becomes very expensive as N can be larger than L
Solution Approach 1:
The expensive clustering procedure is extracted and removed from the training process. Instead of performing clustering on all query embeddings to identify hard negatives, the patent uses a simpler approach where negative labels are sampled uniformly at random from the complement of positive labels, eliminating the need for costly clustering operations while maintaining training efficiency.
Solution Approach 2:
The patent replaces the expensive and complex clustering procedure with a simple and efficient uniform random sampling method for selecting negative labels. This substitution uses a cheap and straightforward sampling mechanism that is computationally inexpensive and can be performed efficiently during training, avoiding the need for complex clustering algorithms.
3Productivity
If uniform random negative sampling is used, then training is simple and fast, but accuracy is insufficient compared to hard negative sampling methods
Solution Approach 1:
The patent merges two sampling strategies: uniform random sampling for most negative labels and importance sampling for a subset of hard negative labels. This combination allows the training process to benefit from both the simplicity and speed of uniform sampling and the accuracy improvement of importance sampling, achieving a balance between training efficiency and classification accuracy.
Solution Approach 2:
Different sampling strategies are applied to different portions of the negative labels. Uniform random sampling is used for the majority of negative labels to maintain simplicity and speed, while importance sampling is applied to a specific subset of hard negative labels to improve accuracy for difficult cases. This local differentiation allows each sampling method to be applied where it is most effective.
Data Source
AI summary
An extreme classification method includes receiving training data-points and classifier vectors associated with the training data-points. A plurality of training epochs are performed wherein each training epoch includes generating query embeddings for each data-point, sampling a predetermined number of negative labels from a set of negative labels for each of the training data-points; and training an encoder and the classifier vectors using the sampled negative labels. Positive labels and the sampled negative labels are then used to compute a loss. Encoder parameters and the classifier vectors are then updated based on the computed loss. For a first portion of epochs, the sampled negative labels include only uniformly random negative labels. For a second portion of the epochs, the sampled negative labels include uniformly random negative labels and hard negative labels. The hard negative labels are identified using an Approximate Nearest Neighbor Search (ANNS) index (308) built on the classifier vectors.