Contrastive Learning With Label Correction for Noisy Image Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural networks are susceptible to overfitting when trained on noisy data, particularly with inaccurate labels and out-of-distribution inputs, leading to poor performance in real-world scenarios.
Innovation Solution
A contrastive learning mechanism that projects training images into a low-dimensional subspace, using consistency and prototypical losses to regularize the geometric structure, combined with iterative label correction and reconstruction, to learn robust representations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If DNNs are trained on noisy data with inaccurate labels, then the amount of training data increases and manual labeling costs decrease, but the model performance deteriorates due to overfitting to noise
Solution Approach 1:
The patent introduces an intermediary module that identifies and corrects noisy labels during training. This mediator processes the relationship between training data and model learning, filtering out harmful noise while preserving useful information, thus enabling the use of large-scale noisy data without performance degradation
Solution Approach 2:
The system implements a feedback mechanism where the model's predictions are used to identify and correct noisy labels in the training data. The corrected labels are then fed back into the training process, creating a self-improving loop that enhances model performance while utilizing noisy data sources
2Reliability
If manual data cleaning and labeling are performed to ensure data quality, then model performance improves, but labor intensity and costs increase significantly
Solution Approach 1:
The system enables self-service data cleaning where the model automatically identifies and corrects noisy labels without human intervention. The training process itself generates the cleaning mechanism, eliminating the need for separate manual labeling efforts while maintaining high data quality
Solution Approach 2:
Through feedback loops, the model uses its own predictions to identify labeling errors and automatically correct them. This self-correcting mechanism replaces manual data cleaning processes, achieving high data quality without the associated labor costs and time requirements
3Quantity of substance
If DNNs are trained on real-world noisy images from the Internet, then data availability increases, but label accuracy decreases due to lack of human review
Solution Approach 1:
An intermediary correction mechanism is introduced that processes the relationship between automated web-scraped labels and actual image content. This mediator identifies discrepancies and corrects inaccurate labels, enabling the system to utilize large volumes of internet-sourced data while maintaining label accuracy through automated verification
Data Source
AI summary
Embodiments described herein provide systems and methods for noise-robust contrastive learning. In view of the need for a noise-robust learning system, embodiments described herein provides a contrastive learning mechanism that combats noise by learning robust representations of the noisy data samples. Specifically, the training images are projected into a low-dimensional subspace, and the geometric structure of the subspace is regularized with: (1) a consistency contrastive loss that enforces images with perturbations to have similar embeddings; and (2) a prototypical contrastive loss augmented with a predetermined learning principle, which encourages the embedding for a linearly-interpolated input to have the same linear relationship with respect to the class prototypes. The low-dimensional embeddings are also trained to reconstruct the high-dimensional features, which preserves the learned information and regularizes the classifier.