Contrastive Learning Gradient Update via Segmented Data Batches
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Contrastive learning models face resource depletion issues in computing devices due to the need to load all training data batches simultaneously for determining update gradients, as the gradients generated by different batches are interdependent.
Innovation Solution
The method involves dividing the training data into groups, determining a global gradient factor in a preprocessing stage that is not used for backpropagation, and a local gradient factor in each training stage that is used for backpropagation, allowing only the current group's data to be loaded, reducing the computational workload.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If all batches of training data are loaded simultaneously to determine update gradients for contrastive learning model, then the gradient calculation is accurate, but the computational resources are rapidly depleted
Solution Approach 1:
The patent segments the training data into multiple batches and divides the gradient calculation into two distinct types: first-type gradients computed from all batches collectively, and second-type gradients computed from individual batches. This segmentation allows accurate gradient calculation to be distributed across multiple processing stages, reducing the peak computational resource requirements while maintaining overall gradient accuracy.
Solution Approach 2:
The patent performs preliminary computation of the first-type gradient factors by loading all training data batches initially. These pre-computed first-type gradient factors are then reused in subsequent training stages where only the current batch needs to be loaded for second-type gradient calculation. This preliminary action eliminates the need to reload all batches in every training stage, significantly reducing computational resource depletion.
2Productivity
If gradient accumulation method is used to integrate multiple batches of training data, then the model training performance is improved, but the interdependence of gradients from different batches requires loading all batches simultaneously
Solution Approach 1:
The patent segments the gradient computation into two independent components: first-type gradient factors that capture inter-batch relationships and are computed once from all batches, and second-type gradient factors that capture intra-batch relationships and are computed per batch. This segmentation decouples the data loading requirements, allowing second-type gradients to be computed independently for each batch without requiring other batches to be loaded, thus reducing device complexity.
Solution Approach 2:
The patent performs preliminary computation of first-type gradient factors that encode the interdependence relationships between batches. These pre-computed factors are stored and reused in subsequent training stages, eliminating the need to reload and reprocess all batches to maintain gradient interdependence. This preliminary action preserves model training performance while simplifying the data loading process.
Data Source
AI summary
There are provided method, apparatus, device, and medium for determining update gradient for contrastive learning model. In the method, a gradient factor of a first type for the contrastive learning model is determined based on a first group of training data and a second group of training data for training the contrastive learning model. The gradient factor of the first type is not used for backpropagation during a training process. In a first stage of the training process, a gradient factor of a second type associated with the first group of training data is determined based on the contrastive learning model. The gradient factor of the second type is used for backpropagation during the training process. Gradient is obtained for updating the contrastive learning model based on the gradient factor of the first type and the gradient factor of the second type associated with the first group of training data.