Accelerator Pipeline Training With Deferred Micro-Batch Updates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning model training methods are computationally expensive, resource-intensive, and introduce discrepancies when using accelerator pipelines, particularly for large and complex recommendation models.
Innovation Solution
Implementing an accelerator pipeline with deferred updates to model parameters, where frequently accessed embeddings are stored in GPU memory and non-frequently accessed embeddings are stored in main memory, allowing for efficient resource utilization and minimizing discrepancies by deferring updates until after training with all micro-batches is complete.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If an accelerator pipeline is used for training operations, then resource efficiency is improved and training time is reduced, but discrepancies are introduced into the training results compared to training without an accelerator pipeline
Solution Approach 1:
The system performs preliminary classification of micro-batches into popular and non-popular categories before training. Frequently accessed embeddings are pre-loaded into GPU memory, while less frequently accessed embeddings remain in CPU memory. This preliminary organization enables the accelerator pipeline to process different micro-batch types through optimized paths, maintaining both speed improvements and training result accuracy.
Solution Approach 2:
The training data is segmented into different classes of micro-batches (popular and non-popular) based on access frequency. This segmentation allows the system to apply different processing strategies: popular micro-batches are processed immediately with pre-loaded embeddings in GPU memory, while non-popular micro-batches are processed after loading required embeddings from CPU memory. This segmentation resolves the contradiction by enabling selective optimization without compromising overall training accuracy.
2Use of energy by moving object
If frequently accessed embeddings are stored in GPU memory and non-frequently accessed embeddings are stored in main memory, then resource efficiency is improved, but memory management complexity increases
Solution Approach 1:
Different memory locations (GPU memory vs. CPU memory) are assigned different qualities based on access frequency requirements. Frequently accessed embeddings are placed in the faster, more energy-efficient GPU memory, while less frequently accessed embeddings are stored in the larger, cheaper CPU memory. This local quality differentiation optimizes energy efficiency while maintaining a manageable memory hierarchy through clear access patterns.
Solution Approach 2:
The system dynamically changes the memory location parameter for embeddings based on their access frequency. By monitoring and reclassifying embeddings as popular or non-popular, the system can migrate embeddings between GPU and CPU memory, optimizing energy efficiency adaptively. This parameter change approach manages complexity through automated classification rather than static, complex memory allocation.
3Productivity
If micro-batches are processed in an accelerator pipeline with deferred updates, then resource utilization is improved, but the complexity of coordinating processing and updates increases
Solution Approach 1:
The system implements periodic update cycles within the accelerator pipeline, processing multiple micro-batches and then performing model parameter updates at defined intervals. This periodic approach allows the system to maintain high resource utilization during processing phases while concentrating coordination complexity into manageable update phases. The pipeline alternates between processing mode and update mode, improving overall resource utilization without requiring continuous complex coordination.
Solution Approach 2:
The accelerator pipeline maintains continuous useful action by overlapping data preparation, processing, and update operations. While one micro-batch is being processed, the system prepares the next micro-batch and pre-loads required embeddings into GPU memory. This continuity maximizes resource utilization by ensuring the accelerator is rarely idle, while the coordinated updates are scheduled to occur at optimal points in the pipeline without interrupting the flow of useful computational work.
Data Source
AI summary
Innovations in training a machine learning model using an accelerator pipeline with deferred updates to model parameters are described. An accelerator identifies one or more first-class micro-batches (“MBs”) and a second-class MB of a working set. A first-class MB contains, as inputs, frequently accessed embeddings stored in graphics processing unit (“GPU”) memory. The accelerator schedules the first-class MB(s) for training using one or more GPUs. During the training, the accelerator obtains the second-class MB, which contains, as inputs, non-frequently accessed embeddings stored in main memory. At least some updates to the model parameters from the training with the first-class MB(s) are deferred until after training with the second-class MB. The accelerator schedules the second-class MB for training. Finally, after the training with the second-class MB, the accelerator updates the non-frequently accessed values for the second-class MB. At this point, the GPU(s) also update model parameters, applying deferred updates.