Iterative Label Noise Correction in Deep Learning Models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep learning models face difficulties in processing label noise, particularly in distinguishing between different types of noise and correcting label errors, which can lead to overfitting and performance degradation due to the reduction in the number of refined data points as label noise increases.
Innovation Solution
A method involving iterative training of two neural network models, where one model detects clean data and the other corrects labels, using a process that alternates between identifying clean data and refining labels, thereby improving the accuracy of label correction and noise detection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If label noise is removed by filtering data, then training data quality is improved, but the number of available training samples decreases leading to overfitting
Solution Approach 1:
The patent divides the training data into clean data and noisy data subsets using a noise detection model. The noise detection model identifies and segments noisy samples, allowing the training process to use only clean samples for supervised learning while preserving the overall dataset structure. This segmentation resolves the contradiction by enabling quality filtering without permanently discarding data samples.
Solution Approach 2:
The patent performs preliminary noise detection and correction before the main training process. A noise detection model is trained first to identify noisy labels, then these noisy samples are corrected using the detected noise patterns. This preliminary action cleans the data beforehand, allowing all samples to be used in subsequent training without introducing overfitting from late-stage filtering.
2Quantity of substance
If deep learning models are trained with label noise, then more data can be utilized, but model performance degrades due to inability to properly process noise
Solution Approach 1:
The patent introduces a noise detection model as an intermediary between the raw data with noisy labels and the main training process. This intermediary model detects and corrects noise in labels before the main model trains on the data. By inserting this intermediary processing step, the system can utilize all available training samples while protecting the main model from the harmful effects of label noise.
3Measurement precision
If typical noise estimation schemes are used based on cross-entropy loss, then label noise can be estimated, but the ability to distinguish between different types of heterogeneous noise is reduced
Solution Approach 1:
The patent applies different noise detection and correction strategies to different subsets of data based on local characteristics. The noise detection model is trained to identify specific noise patterns in different regions of the data space, allowing it to distinguish between instance-dependent noise and feature-dependent noise. This local quality approach enables the system to adapt to different noise types in different parts of the dataset, resolving the contradiction between estimation accuracy and noise type discrimination.
Data Source
AI summary
A processor-implemented method with label noise processing includes: iteratively training a first model for correcting a label of a data set, the label comprising noise, and a second model for detecting the noise of the label; and processing the data set comprising the noise using either one or both of the trained first model and the trained second model, wherein the iterative training comprises: identifying clean data in the data set using the second model; training the first model using the clean data; correcting the label of the data set using the trained first model; and training the second model based on the data set comprising the corrected label.