Serverless Gradient Chunking for Large DL Model Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Serverless platforms face challenges in training large deep learning models due to limited compute power, short instance lifetimes, and the inability to perform peer-to-peer communication, which restricts their use in distributed training, especially when gradients or models exceed the maximum size allowed by cloud storage services.
Innovation Solution
A method and system for training large deep learning models using serverless architecture that involves chunking gradients into segments based on the maximum size allowed by cloud storage services, utilizing multiple aggregators and threads to optimize communication, and employing data types like float16 and int8 to reduce overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If serverless instances are used for distributed training, then scalability and resource management are improved, but peer-to-peer communication capability deteriorates
Solution Approach 1:
The patent introduces cloud storage services as an intermediary communication channel between serverless instances. Instead of direct peer-to-peer communication, workers write gradients to and read from cloud storage, enabling distributed training without requiring communication capabilities in the serverless instances themselves.
2Loss of energy
If cloud storage services are used for gradient exchange, then communication overhead is reduced, but data item size is constrained
Solution Approach 1:
The patent segments large gradient data into smaller chunks that fit within cloud storage data item limits. By dividing the gradient exchange process into multiple smaller operations, the system overcomes the data size constraint while maintaining efficient communication through the cloud storage intermediary.
Data Source
AI summary
State of the art methods require size of DL model, or its gradients be less than maximum data item size of storage used as a communication channel for model training with serverless platform. Embodiments of the present disclosure provide method and system for training large DL models via serverless architecture using communication channel when the gradients are larger than maximum size of one data item allowed by the channel. Gradients that are generated by each worker during current training instance, are chunked into segments and stored in the communication channel. Corresponding segments of each worker are aggregated by aggregators and stored back. Each of the aggregated corresponding segments are read by each worker to generate an aggregated model to be used during successive training instance. Optimization techniques are used for reading-from and writing-to the channel resulting in significant improvement in performance and cost of training.