Neural Network Training via Robust Temporal Ensembling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning models, such as deep neural networks, face performance issues due to mislabeled data during training, which can inhibit their effectiveness in image classification tasks.
Innovation Solution
The approach involves augmenting training data by generating pseudo-labels using a neural network that averages weights over multiple training rounds, and combining task loss and consistency loss to update the model weights, allowing the network to infer labels consistently across different augmentations of the same data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If training data with mislabeled data is used to train neural networks, then the training process can proceed without data filtering, but the neural network performance and accuracy deteriorate
Solution Approach 1:
The patent implements a feedback mechanism where the trained model generates pseudo-labels for unlabeled data, which are then used to retrain the model. This iterative feedback loop allows the model to progressively improve its performance by learning from its own predictions, effectively resolving the contradiction between using noisy training data and achieving high accuracy.
Solution Approach 2:
The model performs self-labeling by generating its own training labels through pseudo-labeling. Instead of relying entirely on external labeled data, the system enables the model to create its own training signals from unlabeled data, allowing it to continue training even when initial labeled data contains errors or is insufficient.
2Adaptability or versatility
If more training data is used to improve model robustness, then the model can handle unforeseen data shifts better, but the computational resources and training time increase
Solution Approach 1:
The patent applies preliminary data augmentation transformations to the training data before model training. By pre-processing the data with various augmentations (rotations, flips, color adjustments, etc.), the model is exposed to diverse variations upfront, which improves its robustness to unseen data shifts without requiring extensive additional training time.
3Adaptability or versatility
If data augmentation is applied to increase training data diversity, then the model's adaptability improves, but the computational complexity and processing time increase
Solution Approach 1:
The patent applies data augmentation selectively and partially rather than exhaustively. Instead of applying all possible augmentations to all data points, the system uses a subset of augmentation techniques on relevant portions of the data, achieving sufficient diversity for robust training while controlling computational complexity.
Data Source
AI summary
Apparatuses, systems, and techniques to use one or more neural networks to generate data labels. In at least one embodiment, one or more neural networks is trained based, at least in part on, one or more labels, pseudo-labels, training data, and modified training data.