Contrastive Learning with Positive Pseudo Labels for Low-Label Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network training methods are inefficient in utilizing unlabeled data and require significant computational resources and labeled data, leading to high carbon emissions and electricity consumption.
Innovation Solution
A semi-supervised learning technique using positive pseudo labels (SEMPPL) that combines labeled and unlabeled data to generate informative representations, allowing neural networks to be pre-trained with a small amount of labeled data and adapted efficiently for specific tasks, reducing computational requirements and data needs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If neural networks are trained using traditional supervised learning methods with labeled data, then training accuracy can be achieved, but the computational resources and labeled data requirements become excessively large
Solution Approach 1:
The patent applies preliminary action by pre-training a target neural network on a large corpus of unlabeled data to create a target embedding queue. This pre-training phase prepares the network to efficiently handle labeled data during the actual training task, reducing the computational burden during supervised learning. The target network is initialized with weights derived from unsupervised pre-training, allowing the fine-tuning phase to require significantly fewer labeled examples and computational resources.
2Reliability
If neural networks are trained using traditional supervised learning methods with labeled data, then training accuracy can be achieved, but the amount of labeled data required becomes excessively large
Solution Approach 1:
The patent introduces an intermediary mechanism in the form of a target embedding queue that stores embeddings of unlabeled data. During training, this queue serves as an intermediary resource that provides contextual information and representation learning without requiring labels. The queue acts as a mediator between the unlabeled pre-training data and the labeled training data, allowing the network to leverage both types of data effectively and reducing the dependency on large quantities of labeled examples.
3Reliability
If neural networks are trained using traditional supervised learning methods, then model performance can be achieved, but carbon emissions and electricity consumption increase significantly
Solution Approach 1:
The patent applies parameter changes by modifying the training objective and data utilization parameters. Instead of training solely on labeled data with high computational intensity, the system changes the approach to use a combination of unlabeled pre-training and efficient fine-tuning with labeled data. This parameter change in the training methodology reduces the total computational footprint and associated carbon emissions while maintaining model performance through the hybrid approach of semi-supervised learning with target network initialization.
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training a neural network to perform a machine learning task on one or more received inputs by using a hybrid training dataset with a semi-supervised learning technique. The hybrid training dataset includes multiple unlabeled training inputs and multiple labeled training inputs and, in some cases, more unlabeled training inputs than labeled training inputs.