Distributed AI Model Training via Parameter Server Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep learning models require substantial computing power and memory, especially for large AI models, which can be constrained by the memory capacity of GPUs, leading to inefficiencies in training and inference processes, such as model parallelism and data parallelism, which are limited by memory capacity and synchronization requirements.
Innovation Solution
A system comprising a parameter server and memory-constrained target devices, where the AI model is dissected into smaller portions, executed efficiently on the target device, and synchronized using multi-level parallel reduction of parameters, allowing for dynamic execution and mixed-precision training to optimize computation and memory usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data parallelism is used to improve throughput in distributed training, then computing efficiency is improved, but memory capacity requirements increase and synchronization overhead increases
Solution Approach 1:
The AI model is divided into multiple subportions that are distributed across different target devices. Each device processes a specific subportion, allowing the system to handle large models without requiring each individual device to have substantial memory capacity. The parameters are segmented and managed in chunks rather than requiring full model storage on each device.
2Productivity
If data parallelism is used to improve throughput in distributed training, then computing efficiency is improved, but synchronization overhead increases
Solution Approach 1:
The system performs preliminary actions by pre-processing and preparing parameter portions before they are needed for synchronization. The parameter server proactively manages and prepares parameter chunks, reducing the need for frequent synchronization operations during training. This preliminary preparation minimizes the time lost to synchronization overhead.
3Quantity of substance
If the AI model is divided into subportions for execution on memory-constrained devices, then memory efficiency is improved, but device complexity increases
Solution Approach 1:
A parameter server acts as an intermediary between multiple target devices, managing the distribution and coordination of parameter portions. This central coordinator simplifies the complexity by providing a single point of control for parameter management, rather than requiring complex peer-to-peer coordination between all devices. The parameter server handles the orchestration of microbatches and parameter synchronization.
4Power
If multiple GPUs are run in data-parallel manner to improve throughput, then computing power is improved, but memory capacity requirements increase
Solution Approach 1:
The system transitions from requiring vertical scaling (increasing memory capacity of individual GPUs) to horizontal scaling (adding more GPUs with standard memory capacity). By distributing the model across multiple devices in a multi-dimensional architecture with a parameter server coordinating multiple target devices, the system achieves high computing power using devices with conventional memory sizes rather than requiring each device to have exponentially increasing memory capacity.
Data Source
AI summary
Methods, systems, apparatuses, and computer program products are described herein that enable execution of a large AI model on a memory-constrained target device that is communicatively connected to a parameter server, which stores a master copy of the AI model. The AI model may be dissected into smaller portions (e.g., layers or sub-layers), and each portion may be executed as efficiently as possible on the target device. After execution of one portion of the AI model is finished, another portion of the AI model may be downloaded and executed at the target device. To improve efficiency, the input samples may be divided into microbatches, and a plurality of microbatches executing in sequential order may form a minibatch. The size of the group of microbatches or minibatch can be adjusted to reduce the communication overhead. Multi-level parallel parameters reduction may be performed at the parameter server and the target device.