Label Classifier Training With Dynamic Margin for XMC Accuracy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning models, particularly Extreme Multi-label Classification (XMC) models, face diminished accuracy during inference due to the use of a static margin in loss functions, which fails to account for varying degrees of similarity and dissimilarity between positive and negative embeddings, leading to reduced differentiation between matching and non-matching objects.
Innovation Solution
Implementing a dynamic margin in the training process that adjusts based on the relative hardness of embeddings, using a Label Prototype Network (LPN) to aggregate text-based embeddings and learnable free vectors, thereby optimizing model training and inference accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a static margin is used in the loss function, then the model training is simplified and computationally efficient, but the model accuracy during inference is diminished due to reduced differentiation between positive and negative embeddings
Solution Approach 1:
The patent applies the dynamics principle by transforming the static margin into a dynamic margin that adapts during training. The margin is no longer a fixed value but changes based on the training progress and the specific embedding pairs being processed, allowing the model to achieve better differentiation between positive and negative embeddings while maintaining training feasibility
Solution Approach 2:
The patent implements parameter changes by modifying the margin parameter from a constant to a variable that changes during training. The dynamic margin adjusts its value based on the relative hardness of embedding pairs, enabling the loss function to better capture the nuanced differences between positive and negative samples without requiring excessive computational resources
2Measurement precision
If a dynamic margin is implemented to improve differentiation between positive and negative labels, then the model training accuracy is enhanced, but the computational complexity increases
Solution Approach 1:
The patent manages computational complexity through parameter changes by formulating the dynamic margin as a function of easily computable features. The margin adjustment is based on the relative hardness of embedding pairs, which can be determined from the current training state without requiring additional heavy computations or complex architectural modifications
Solution Approach 2:
The patent applies partial action by implementing dynamic margin adjustment only where necessary - specifically for embedding pairs that benefit from differentiated treatment. The system selectively applies the dynamic margin based on the relative hardness of specific embedding pairs, rather than uniformly increasing complexity across all training operations
Data Source
AI summary
Systems, apparatus, articles of manufacture, and methods are disclosed to train models with a dynamic margin. An example apparatus includes interface circuitry, machine-readable instructions, and at least one processor circuit to be programmed by the machine-readable instructions to determine a positive label similarity value and a negative label similarity value, the positive and negative similarity values based on a query embedding, determine a negative-to-positive difference value and a positive-to-negative difference value associated with the positive label similarity value and the negative label similarity value, determine a positive-to-negative difference value associated with the positive label similarity value and the negative label similarity value, and cause training of a label classifier with a loss function having a dynamic margin when the positive-to-negative difference value satisfies a threshold.