Co-training Framework for Semi-supervised Learning with Contrastive Graph Regularization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing semi-supervised learning methods for neural models face limitations such as reliance on pseudo-label quality, confirmation bias, and task-agnostic representations, which can lead to suboptimal performance in classification tasks.
Innovation Solution
A co-training framework that jointly learns two representations of training data: class probabilities and low-dimensional embeddings. This framework uses a CNN with parallel classification and projection heads, generating pseudo-labels and embeddings that interact to improve each other, and employs contrastive graph regularization to enhance representation learning.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If pseudo-labeling methods are used to train unlabeled data, then the model can learn from unlabeled data, but the prediction mistakes accumulate due to confirmation bias
Solution Approach 1:
The patent introduces a memory bank as an intermediary component that stores high-confidence predictions from previous iterations. This memory bank acts as a mediator between the current model predictions and the training process, allowing the model to learn from reliable historical predictions while filtering out low-confidence uncertain predictions that would otherwise propagate errors.
Solution Approach 2:
The patent implements iterative training with feedback loops where model predictions are stored in a memory bank and reused in subsequent training iterations. High-confidence predictions provide positive feedback to improve the model, while low-confidence predictions are filtered out to prevent error accumulation, creating a selective feedback mechanism that improves reliability.
2Adaptability or versatility
If self-supervised learning methods are used, then the model can learn representations without labels, but the learned representations are suboptimal for specific classification tasks
Solution Approach 1:
The patent merges self-supervised learning (contrastive learning on unlabeled data) with supervised learning (classification on labeled data) into a unified training framework. The model simultaneously learns task-agnostic representations from unlabeled data and task-specific representations from labeled data, combining the advantages of both approaches to achieve optimal representation quality for specific classification tasks.
Solution Approach 2:
The patent segments the training process into distinct components: contrastive learning on unlabeled data for general representation learning, and supervised classification on labeled data for task-specific optimization. This segmentation allows each component to focus on its strength while working together to achieve overall performance improvement.
3Measurement precision
If a large amount of manually annotated training data is used, then the model learns accurately, but the training cost increases significantly
Solution Approach 1:
The patent enables the model to serve itself by generating pseudo-labels for unlabeled data through self-training. The model uses its own predictions (with confidence filtering via memory bank) to create training labels, reducing the need for expensive manual annotation while maintaining learning accuracy through iterative self-improvement.
Solution Approach 2:
The patent performs preliminary contrastive learning on unlabeled data before supervised fine-tuning. This preliminary action pre-trains the model with task-agnostic representations, reducing the amount of labeled data needed for subsequent supervised training and thereby reducing overall training time and cost.
Data Source
AI summary
Embodiments described herein provide an approach (referred to as “Co-training” mechanism throughout this disclosure) that jointly learns two representations of the training data, their class probabilities and low-dimensional embeddings. Specifically, two representations of each image sample are generated: a class probability produced by the classification head and a low-dimensional embedding produced by the projection head. The classification head is trained using memory-smoothed pseudo-labels, where pseudo-labels are smoothed by aggregating information from nearby samples in the embedding space. The projection head is trained using contrastive learning on a pseudo-label graph, where samples with similar pseudo-labels are encouraged to have similar embeddings.