Contrastive Model Training With Graph-Based Unlabeled Sample Partitioning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing contrastive learning frameworks are limited by the assumption that sample pairs are known, making them unsuitable for real-world scenarios requiring both labeled and unlabeled samples, and there is a need for more efficient training methods that leverage unlabeled samples to reduce the reliance on labeled data.
Innovation Solution
The Self-Evolving Contrastive Training (SECT) method uses unlabeled samples to iteratively update a pre-training model by creating undirected graphs, dividing them into sub-graphs based on features, and training using both labeled and unlabeled samples, minimizing entropy and incorporating a priori knowledge to improve model accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If existing contrastive learning frameworks are used, then model training can be performed with known sample pairs, but the method is limited and cannot effectively utilize unlabeled samples in real-world scenarios
Solution Approach 1:
The patent segments the training process into two distinct phases: a pre-training phase using only unlabeled samples to build initial feature representations, and a contrastive learning phase that incorporates labeled samples. This segmentation allows the model to first learn from abundant unlabeled data without requiring labeled pairs, then refine its capabilities with labeled data, thereby reducing overall reliance on labeled data while adapting to real-world scenarios where unlabeled data is prevalent.
Solution Approach 2:
The patent applies preliminary action by performing pre-training on unlabeled samples before conducting contrastive learning with labeled samples. This preliminary phase initializes the model with meaningful feature representations extracted from unlabeled data, creating a solid foundation that reduces the amount of labeled data needed in subsequent training stages, thus addressing the limitation of existing frameworks that require known sample pairs from the outset.
2Measurement precision
If more labeled samples are used for training, then model accuracy can be improved, but the cost and time for data labeling increases
Solution Approach 1:
The patent implements self-service through self-supervised learning mechanisms where the model learns to generate its own training signals from unlabeled data without human intervention. The pre-training phase automatically creates learning tasks from unlabeled samples, and the contrastive learning phase uses the model's own predictions to guide further training, eliminating the need for expensive and time-consuming manual labeling while maintaining model accuracy.
Solution Approach 2:
The patent changes the training parameters by introducing a two-stage training approach with different objective functions: unsupervised pre-training objectives that leverage unlabeled data structure, followed by supervised contrastive learning objectives that refine accuracy. This parameter change allows the model to achieve high accuracy by strategically using a small portion of labeled data after extensively training on unlabeled data, thereby reducing labeling time and costs.
3Productivity
If unlabeled samples are leveraged to reduce labeled data reliance, then training efficiency improves, but the complexity of handling mixed labeled and unlabeled data increases
Solution Approach 1:
The patent applies dynamics by making the training process adaptive and flexible through its two-stage architecture. The system dynamically transitions from unsupervised pre-training to supervised contrastive learning, allowing it to efficiently process mixed labeled and unlabeled data. This dynamic approach simplifies the overall process by providing clear stage transitions and objective function switches, managing the complexity of handling mixed data types while maintaining high training efficiency.
Data Source
AI summary
The present disclosure relates to a method, a device, and a program product for training a model. The method includes: receiving at least one unlabeled sample and at least one labeled sample for training a pre-training model, the pre-training model being used to extract features of the samples; creating an undirected graph associated with the pre-training model using the at least one unlabeled sample and a set of training samples associated with the pre-training model; dividing the undirected graph to form a plurality of sub-graphs based on corresponding features of the unlabeled sample and the set of training samples, the plurality of sub-graphs corresponding to a plurality of classifications of the samples, respectively; and training, based on the plurality of sub-graphs and the at least one labeled sample, the pre-training model to generate a training model. A corresponding device and a corresponding computer program product are provided.