Neural Network Parallelization Plan Generation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks (DNNs) require significant computational resources and memory, making it challenging to train large models on a single GPU due to limited memory, necessitating the use of parallel GPUs to distribute model weights, but existing solutions are error-prone and lack flexibility in designing parallelization plans.
Innovation Solution
A computing system generates a parallelization plan by transforming a data flow graph into fine-grained tasks and scheduling them across multiple devices, using primitives like operator transformation, assignment, and data dependency materialization, with virtual tensors to track dependencies and prevent deadlocks, allowing for flexible and efficient parallelization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If parallel GPUs are used to distribute model weights, then memory limitation is overcome, but system complexity and error-proneness increase
Solution Approach 1:
The patent introduces an intermediary parallelization plan generator that sits between the neural network model and the execution system. This generator automatically transforms the model into a parallelization plan, mediating the complexity of distributed memory management and operator scheduling, thereby reducing user burden while enabling large model training across multiple GPUs
Solution Approach 2:
The patent segments the neural network model into multiple operators and tensors that can be distributed across different GPUs. By dividing the model into manageable segments (operators and tensors), the system can manage complexity through structured decomposition while achieving distributed memory utilization
2Ease of operation
If manual parallelization design is performed, then control over parallelization plan is precise, but time consumption and error rate increase
Solution Approach 1:
The parallelization plan generator performs self-service by automatically analyzing the neural network model and generating appropriate parallelization plans without requiring manual intervention. The system autonomously handles operator transformation, tensor virtualization, and scheduling, eliminating time-consuming manual design while maintaining precision through automated validation
Solution Approach 2:
The patent applies preliminary action by pre-defining transformation rules and scheduling strategies in the parallelization plan generator. These pre-established rules enable rapid automatic generation of parallelization plans, avoiding the need for time-consuming manual design while ensuring correctness through built-in validation mechanisms
3Productivity
If existing parallelization solutions are used, then model training is enabled on multiple GPUs, but flexibility and adaptability are limited
Solution Approach 1:
The patent introduces dynamics by enabling the parallelization plan generator to adaptively create different parallelization strategies based on the specific neural network model characteristics. The system dynamically adjusts operator transformations and scheduling plans according to model architecture, data types, and hardware configurations, providing flexibility while maintaining training capability
Solution Approach 2:
The patent utilizes parameter changes by allowing the parallelization plan generator to modify key parameters such as operator partitioning strategies, tensor virtualization levels, and scheduling priorities. These parameter adjustments enable the system to adapt to different model types and hardware configurations, enhancing versatility while preserving productivity
Data Source
AI summary
Embodiments of the present disclosure include techniques for designing and generating a parallelization plan for a neural network so that workloads in the neural network may be split amongst multiple devices. Operators and tensors in the neural network are transformed into a set of functionally equivalent operators and tensors. These functionally equivalent operators and tensors are then scheduled to separate devices for execution.