Deep Learning Accelerator Co-Optimization for Model Partitioning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing methods for optimizing hardware architecture and distribution strategies for deep learning models are inefficient and typically consider these aspects separately, leading to suboptimal performance in terms of memory usage and throughput during training.
Innovation Solution
A system that integrates algorithmic solutions to simultaneously optimize accelerator architecture and model partitioning through a novel integer linear program (ILP) and dynamic programming (DP) approach, reducing the complexity of the optimization space by employing area-based heuristics and time-indexed variable avoidance, to determine the optimal hardware configuration and distribution strategy for deep learning training.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If distributed training strategies and hardware architecture are optimized separately, then each aspect can be simplified and implemented independently, but the overall training efficiency and throughput are suboptimal due to lack of co-optimization
Solution Approach 1:
The patent merges the optimization of distributed training strategies with hardware architecture selection into a unified co-optimization framework. The system simultaneously evaluates multiple distribution schemes (data parallelism, pipeline parallelism, tensor model parallelism) across various hardware configurations, allowing both aspects to be optimized together rather than separately, thereby achieving superior training throughput while managing complexity through automated search algorithms
2Productivity
If larger hardware accelerators are used to increase training throughput, then more computational power is available, but memory requirements and area constraints are violated
Solution Approach 1:
The patent changes key parameters of the hardware architecture (such as memory size, compute unit count, interconnect bandwidth) to find optimal configurations that maximize throughput within area constraints. The automated search framework evaluates multiple parameter combinations and selects configurations that achieve high throughput without violating memory or area limitations, effectively tuning the hardware parameters to the specific deep learning workload
3Quantity of substance
If more memory is allocated to accommodate larger models and batch sizes, then training capacity increases, but the area constraint for the accelerator is exceeded
Solution Approach 1:
The patent adjusts memory-related parameters (memory size, memory bandwidth, memory hierarchy) within the hardware architecture to achieve optimal balance between memory capacity and area constraints. The co-optimization framework evaluates different memory configuration scenarios and selects parameters that provide sufficient capacity for large models and batch sizes while staying within the physical area limitations of the accelerator
Data Source
AI summary
A training optimization system implements algorithmic solutions to solve the conjoined problem of accelerator architecture search and model partitioning for distributed training. The system makes the multi-dimensional optimization space of architecture search and device placement tractable by reducing the number of accelerator architectures explored through area-based heuristics and employing a novel integer linear program (ILP), the size of which is dependent only on the number of operators. The ILP scheduling optimization also explores the partitioning of operators across cores, known as intra-operator parallelism. Despite the vast space, the ILP described herein requires significantly less time to perform the optimizations across all explored accelerator configurations. Based on the optimal backward and forward pass latencies, the system leverages a novel dynamic programming (DP) approach to determine the device placement and model partitioning scheme.