Batch Importance Selection for Heterogeneous ML Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning in heterogeneous processing systems faces challenges due to memory bandwidth bottlenecks, particularly in large-scale training of models where the volume of data exceeds the memory capacity of accelerators like GPUs, leading to inefficient processing and performance degradation, especially in sparse datasets.
Innovation Solution
The method involves partitioning the training data matrix into batches, calculating batch importance values based on vector contributions to sub-optimality, and selectively providing these batches to the accelerator unit for optimization, reducing memory copy operations and alleviating bandwidth bottlenecks by processing larger memory blocks as single units.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If individual data vectors are transferred to accelerator memory for each iteration, then the accelerator can process relevant vectors, but memory bandwidth bottleneck between host CPU and GPU memory severely degrades performance
Solution Approach 1:
Multiple individual data vectors are merged into a single batch matrix that is transferred to accelerator memory as one unit. This combining approach reduces the frequency of memory copy operations between host and accelerator, thereby alleviating the memory bandwidth bottleneck while maintaining processing efficiency.
Solution Approach 2:
Data vectors are pre-organized into batches in host memory before being transferred to the accelerator. This preliminary grouping allows the system to perform multiple iterations using the same batch data without repeated transfers, reducing memory bandwidth consumption and improving overall performance.
2Quantity of substance
If the entire training data matrix is stored in accelerator memory, then all data is available for processing, but the memory capacity of accelerators is typically very limited and cannot hold large datasets
Solution Approach 1:
The training data matrix is segmented into multiple batches that can be stored in host memory. Only the necessary batch size required for each iteration is transferred to accelerator memory, allowing the system to work with large datasets that exceed accelerator memory capacity by processing data in manageable segments.
Solution Approach 2:
The system transitions from requiring all data to be simultaneously available in accelerator memory to a batch-based approach where data is available in sequential batches. This dimensional change in data access pattern allows processing of large datasets by utilizing host memory capacity while maintaining efficient accelerator utilization.
3Loss of time
If batches are processed as single units, then fewer memory copy operations are needed, but the host must calculate and store batch importance values for selection
Solution Approach 1:
The system introduces batch importance values as a new parameter to guide batch selection. By calculating and utilizing these importance parameters, the system can intelligently select which batches to process in each iteration, optimizing the trade-off between reducing memory copy operations and maintaining processing efficiency.
Data Source
AI summary
Computer-implemented methods are provided for implementing training of a machine learning model in a heterogeneous processing system that includes a host computer operatively interconnected to an accelerator unit. The training operation involves an iterative optimization process for optimizing a model vector defining the model. Such a method includes, in the host computer, storing a matrix of training data and partitioning the matrix into a plurality of batches of data vectors. For each of successive iterations of the optimization process, a selected subset of the batches is provided to the accelerator unit. In the accelerator unit, each iteration of the optimization process is performed to update the model vector in dependence on vectors in the selected subset for that iteration. In the host computer, batch importance values are calculated for respective batches. The batch importance value is dependent on contributions of vectors in that batch to sub-optimality of the model vector.