Neural Network Training with Logit Adjustment for Noisy Labels
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning approaches fail to effectively address both label noise and class imbalance in training data, leading to degraded performance in real-world datasets with significant noise and imbalance issues.
Innovation Solution
A computer-implemented method for training an artificial neural network that involves pre-training to generate invariant representations under data augmentations and fine-tuning using a logit adjustment loss function, which adjusts weights of the projection head while freezing the encoder weights, and employs curriculum learning with a separation parameter and per-sample confidence parameter.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If noise-aware losses or label correction methods are used to address label noise, then label noise robustness is improved, but performance degrades in the presence of imbalanced training data
Solution Approach 1:
The training process is segmented into two distinct stages: pre-training on augmented views to learn robust representations, and fine-tuning with logit adjustment for classification. This segmentation allows each stage to optimize for its specific goal without interference from the other, resolving the contradiction between noise robustness and accuracy.
Solution Approach 2:
The method changes parameters dynamically during training by adjusting logits based on estimated class distribution and using temperature scaling. These parameter changes allow the model to adapt to both noisy labels and class imbalance, improving both robustness and accuracy simultaneously.
2Quantity of substance
If sampling methods or loss function modifications are used to address class imbalance, then class balance is improved, but performance degrades when labels are noisy
Solution Approach 1:
The method performs preliminary action by pre-training the model on augmented views of the data before fine-tuning on the actual labeled data. This pre-training stage allows the model to learn robust representations that are invariant to data augmentations, preparing it to handle noisy labels more effectively during the subsequent fine-tuning stage.
Solution Approach 2:
The method introduces an intermediary mechanism (logit adjustment based on class distribution estimation) that mediates between the balanced and imbalanced classes during fine-tuning. This intermediary allows the model to account for class imbalance without directly manipulating the data distribution, thereby maintaining label noise robustness.
3Measurement precision
If data augmentation is applied to improve representation learning, then representation quality is improved, but training complexity increases
Solution Approach 1:
The method applies partial action by using a predefined set of data augmentations during pre-training rather than continuously increasing augmentation complexity. This allows the model to learn robust representations without the training complexity escalating indefinitely, as the augmentation set remains fixed and manageable.
Data Source
AI summary
A computer-implemented method for training an artificial neural network with training data including samples and corresponding labels for performing a task includes: pre-training the artificial neural network to generate matrix representations that are invariant to a predetermined set of data augmentations applied to a sample, where the artificial neural network includes an encoder module and a projection module configured to generate the matrix representations based on ones of the samples, respectively; and after the pre-training, fine-tune training the artificial neural network using a loss function, wherein fine-tuning the artificial neural network includes adjusting, based on the labels, one or more weights of the projection module while maintaining constant weights of the encoder module, and where the loss function is based on a logit adjustment loss that is based on logits that are adjusted based on a class distribution of the training data.