Instance-Adaptive Noise-Robust Loss for Noisy Label Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Natural language processing models trained with automated annotation datasets face challenges due to noisy labels generated by machine learning algorithms, as existing noise-robust losses are not effective in handling varying noise levels across different data points, leading to overfitting and reduced accuracy.
Innovation Solution
Implementing an instance-specific noise-robust loss function that uses a beta value calculated for each training instance, combining it with an entropy term to regularize the loss, and jointly training the classifier model, input encoder, and label quality predictor model to enhance noise-robustness and accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If Cross Entropy loss is used as the optimization objective, then training speed is improved, but the model overfits to label noise and accuracy deteriorates
Solution Approach 1:
The patent transforms the fixed Cross Entropy loss into a flexible noise-robust loss function by introducing instance-specific beta parameters. This parameter change allows the loss function to adapt to varying noise levels across different training instances, resolving the contradiction between training efficiency and accuracy by maintaining fast convergence while preventing overfitting to noisy labels through dynamic parameter adjustment.
Solution Approach 2:
The patent makes the loss function dynamic by calculating instance-specific beta values that adapt during training. The beta parameter dynamically adjusts the weight of the loss for each training instance based on its predicted noise level, allowing the model to automatically focus more on clean instances and less on noisy ones, thereby improving accuracy without sacrificing training speed.
2Reliability
If a dataset-wide noise-robust parameter is used, then noise resistance is improved, but the ability to capture varying noise levels across different data points deteriorates
Solution Approach 1:
The patent segments the single dataset-wide noise-robust parameter into multiple instance-specific beta parameters. Each training instance receives its own beta value based on its individual noise characteristics, allowing the model to capture varying noise levels across different data points while maintaining overall noise resistance through the collective effect of all instance-specific parameters.
Solution Approach 2:
The patent applies local quality by making the noise-robust parameter local to each training instance rather than global to the entire dataset. Each instance's beta parameter is locally optimized based on its specific noise characteristics, enabling the model to adapt to local noise variations while maintaining global noise resistance through the aggregated effect of all local adaptations.
3Reliability
If instance-specific beta values are calculated for each training instance, then noise-robustness is improved, but computational complexity increases
Solution Approach 1:
The patent implements self-service by having the model automatically calculate instance-specific beta values during training without requiring external intervention or complex preprocessing. The beta parameters are derived internally from the model's own predictions and the training data characteristics, enabling the system to improve noise-robustness while keeping computational complexity manageable through self-directed parameter optimization.
Solution Approach 2:
The patent incorporates feedback mechanisms where the instance-specific beta values are calculated based on the model's predictions and fed back into the loss function during training. This feedback loop allows the model to continuously adjust its learning process based on the estimated noise levels, improving noise-robustness while maintaining computational efficiency through iterative refinement rather than complex pre-computation.
Data Source
AI summary
There is included a method and apparatus comprising computer code for a joint training method using neural networks with noise-robust losses comprising encoding input tokens from a noisy dataset into input vectors using an input encoder; predicting a label based on the input vectors using a classifier model; calculating a beta value based on the input vectors and the label using a label quality predictor model, wherein the beta value is instance-specific for each training instance; and j oint training more than one model using a first modified loss function based on the beta value and an entropy value.