Memory-Based Data Selection for ML Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for training machine learning models in mixed environments with CPUs and GPUs often result in idle times for either the CPU or GPU due to limited memory resources, leading to inefficient use of hardware resources.
Innovation Solution
A method that loads a complete training data set into the CPU memory, determines importance values for training data vectors, and dynamically transfers the vectors with the highest importance values to the GPU memory for processing, allowing both CPUs and GPUs to work in parallel and optimizing resource utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the complete training data set is loaded into GPU memory, then the GPU can process all training data in parallel, but the GPU memory capacity is insufficient to hold the complete training data set
Solution Approach 1:
The training data set is divided into multiple partitions, with each partition being transferred to the GPU sequentially for processing. This allows the GPU to handle large data sets that exceed its memory capacity by processing data in manageable segments.
Solution Approach 2:
The CPU pre-processes and prepares data partitions before transferring them to the GPU. Importance values are calculated and stored in advance, allowing the GPU to immediately begin processing upon receiving data partitions without waiting for data preparation.
2Productivity
If the training data set is partitioned between CPU and GPU processing, then both processors can work in parallel, but idle times occur when one processor waits for the other
Solution Approach 1:
The CPU and GPU are kept continuously busy through overlapping operations. While the GPU processes one data partition, the CPU simultaneously prepares the next partition by calculating importance values and organizing data, eliminating idle waiting time for both processors.
Solution Approach 2:
The system monitors the processing status and dynamically adjusts data transfer timing. When the GPU finishes processing a partition, it signals the CPU to transfer the next prepared partition, ensuring continuous operation and minimizing idle time through responsive coordination.
3Reliability
If all training data vectors are transferred to GPU memory, then the GPU can utilize all available data for training, but the transfer time and memory bandwidth consumption increase significantly
Solution Approach 1:
Instead of transferring all training data vectors to the GPU, the system extracts and transfers only the most important data vectors based on calculated importance values. This selective approach maintains training effectiveness while significantly reducing transfer time and memory bandwidth consumption.
Solution Approach 2:
Different data vectors are treated differently based on their importance values. High-importance vectors are prioritized for transfer and processing, while lower-importance vectors may be processed later or with reduced priority, optimizing the balance between training quality and transfer efficiency.
Data Source
AI summary
A method for a machine learning model training is provided which operates in a mixed CPU/GPU environment. The amount of general processing unit memory is larger than the amount of special processing unit memory. The method includes loading a complete training data set into the memory of the general processing unit, determining importance values relating to training data vectors in the provided training data set of the training data vectors, dynamically transferring training data vectors of the training data set from the general processing unit memory to a special processing unit memory using as decision criteria the importance value of the training data vector, wherein the importance value used is taken from an earlier training round of the machine learning model, and executing a training algorithm on the special processing unit with the training data vectors having the highest available importance values of one of the earlier training rounds.