Bootstrapped Latent Representation Learning Without Negative Pairs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural networks for image processing require labeled data for training, which can be costly and time-consuming, and methods using contrastive loss functions face challenges with negative pairs and high computational requirements.
Innovation Solution
A self-supervised learning process using a pair of neural networks with a moving average mechanism, where one network (target) is not updated during training, allowing the other (online) to learn representations from unlabeled data through transformed views of the same data, without the need for negative pairs or large batch sizes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If labeled data is used for training neural networks, then training accuracy is improved, but data acquisition cost and time increase
Solution Approach 1:
The system performs self-supervised learning by automatically generating supervision signals from the data itself through transformations. The neural network learns to predict transformed versions of its own inputs, eliminating the need for external labeled data while maintaining training effectiveness.
2Measurement precision
If contrastive loss functions are used for self-supervised learning, then representation learning is achieved, but computational requirements and batch size requirements increase
Solution Approach 1:
The patent extracts and removes the computationally expensive components from traditional contrastive learning approaches. By eliminating the need for negative pairs and large batch sizes, the method retains the beneficial representation learning while reducing computational overhead to manageable levels.
3Speed
If neural networks are updated during training, then learning speed is improved, but training stability decreases
Solution Approach 1:
The method performs preliminary actions by pre-processing inputs through transformations before they reach the neural network. This allows the network to learn from varied representations in advance, improving both learning speed and stability by presenting diverse yet controlled inputs during training.
Solution Approach 2:
Transformations act as intermediaries between the raw input data and the neural network processing. These intermediary transformations provide stable, controlled variations that facilitate learning without introducing the instability that would result from direct network updates alone.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A computer-implemented method of training a neural network. The method comprises processing a first transformed view of a training data item, e.g. an image, with a target neural network to generate a target output, processing a second transformed view of the training data item, e.g. image, with an online neural network to generate a prediction of the target output, updating parameters of the online neural network to minimize an error between the prediction of the target output and the target output, and updating parameters of the target neural network based on the parameters of the online neural network. The method can effectively train an encoder neural network without using labelled training data items, and without using a contrastive loss, i.e. without needing "negative examples" which comprise transformed views of different data items.