Multi-GPU Contrastive Loss Grouping for Lower Video Memory Use
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for calculating contrastive loss through multiple graphics processing units (GPUs) face challenges due to high video memory consumption, limiting the increase in training batch size and hindering efficiency improvements.
Innovation Solution
Grouping graphics processing units to calculate group contrastive loss separately, reducing video memory consumption by storing similarity matrices within each group, and determining overall contrastive loss based on these group losses.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a large quantity of graphics processing units and training batch of samples are used, then model training efficiency is improved, but video memory consumption of each graphics processing unit increases
Solution Approach 1:
The patent divides N graphics processing units into M processing unit groups, where each group independently calculates contrastive loss for its subset of samples. This segmentation allows each GPU to process only a portion of the total similarity matrix, reducing individual video memory consumption while maintaining overall training efficiency through parallel group processing.
2Quantity of substance
If video memory consumption of each graphics processing unit is reduced, then a quantity of each training batch of samples can be increased, but calculation complexity distribution becomes more challenging
Solution Approach 1:
The patent segments both the graphics processing units and the training samples into M groups, with each group handling a specific subset. This dual segmentation simplifies the memory management problem by creating independent calculation units, while the overall system maintains scalability through the modular group structure.
Solution Approach 2:
The patent combines the results from M processing unit groups to compute the overall contrastive loss. Each group's partial contrastive loss calculations are merged through a weighted average or summation operation, achieving the final result without requiring any single GPU to handle the complete dataset in memory.
Data Source
AI summary
Embodiments of this specification provide a method and apparatus for calculating contrastive loss through multiple graphics processing units. The method includes: processing a feature of a target batch of samples through N graphics processing units divided into M processing unit groups, where each processing unit group includes one or more graphics processing units; separately determining, by each processing unit group, a partial feature similarity between features processed by a graphics processing unit, and storing the partial feature similarity into a corresponding video memory of the graphics processing unit included in the processing unit group; separately determining, according to the partial feature similarity stored in the corresponding video memory of the graphics processing unit included in each processing unit group, group contrastive loss corresponding to each processing unit group; and determining overall contrastive loss according to the group contrastive loss corresponding to each processing unit group.