Deep Learning Training Pipeline Across CPU-GPU Resources
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current deep learning model training methods face inefficiencies in utilizing computing resources, particularly GPUs, due to insufficient device utilization in synchronous computing modes and suboptimal concurrency in data parallelism.
Innovation Solution
A method and apparatus for training deep learning models that segment model description information into sections based on configuration variables, allocating operations with frequent memory interaction to CPUs and computationally intensive operations to GPUs, enabling asynchronous and concurrent execution across heterogeneous devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If synchronous computing mode is used in model parallelism pipeline, then parameter synchronization is ensured, but device utilization rate decreases due to blocking and waiting
Solution Approach 1:
The model is segmented into multiple sections that can be executed independently on different devices. Each device processes a specific section and passes intermediate results to the next device, eliminating the need for all devices to block and wait for parameter synchronization while maintaining computational correctness.
Solution Approach 2:
Intermediate results from forward propagation are prepared and passed in advance to subsequent devices before backward propagation begins. This preliminary preparation of context information allows devices to proceed with their computations without blocking, improving overall device utilization while ensuring proper parameter synchronization when needed.
2Productivity
If data parallelism is used, then concurrency is improved, but resource allocation optimization becomes insufficient
Solution Approach 1:
Different computing resources (CPU, GPU, NPU) are allocated to different sections of the model based on their specific computational characteristics. Memory-intensive sections are assigned to devices with faster memory access, while computationally intensive sections are assigned to devices with higher computational power, optimizing resource utilization while maintaining concurrency through data parallelism.
3Quantity of substance
If model parallelism pipeline is used, then large batch processing is enabled, but calculation context transfer between devices reduces efficiency
Solution Approach 1:
Context information buffers or queues are introduced as intermediaries between devices in the pipeline. These intermediaries store intermediate results and gradient information, allowing devices to process data batches independently without frequent direct communication and context transfer, thereby maintaining large batch processing capability while improving calculation efficiency.
Data Source
AI summary
A method for training a deep learning model may include: acquiring model description information and configuration information of a deep learning model; segmenting the model description information into at least two sections based on segmentation point variable in the configuration information, and loading the model description information to a corresponding resource to run; inputting a batch of training samples into a resource corresponding to a first section of model description information, then starting training and using obtained context information as an input of a resource corresponding to a subsequent section of model description information; and so on until an operation result of a resource corresponding to a final section of model description information is obtained; if a training completion condition is met, outputting a trained deep learning model; and otherwise, keeping on acquiring a subsequent batch of training samples and performing the above training steps until the condition is met.