Distributed Tensor Layout Conversion for Hybrid Model Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deep learning model training methods face bottlenecks due to memory limitations and increased communication overheads as datasets and models grow larger, restricting the implementation of hybrid parallelism between data and model parallelism.
Innovation Solution
A data processing method that involves determining redistribution operators to convert inconsistent tensor layouts between operators in a distributed cluster, allowing for hybrid parallelism by inserting reshape and communication operators into the sliced computational graph.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data parallelism is used for training deep learning models, then training capacity is improved, but memory limitations and communication overheads increase as datasets and models grow larger
Solution Approach 1:
The patent applies segmentation by dividing the model parallelism training process into multiple stages with different tensor layouts. Each stage processes specific operators with optimized layouts, breaking down the complex training task into manageable segments that can be executed efficiently across distributed devices, thereby reducing communication overhead while maintaining training capacity.
Solution Approach 2:
The patent introduces tensor layout transformation as an additional dimension of optimization. By converting between different tensor layouts (e.g., from layout A suitable for data parallelism to layout B suitable for model parallelism), the system can switch parallelism strategies without changing the underlying hardware architecture, thus managing communication overhead effectively.
2Productivity
If hybrid parallelism of data parallelism and model parallelism is implemented, then training efficiency is improved, but tensor layout constraints restrict conversion between parallelism manners
Solution Approach 1:
The patent employs parameter changes by systematically transforming tensor layouts between different operators. Each tensor layout transformation involves changing parameters such as dimension permutations, sharding strategies, and memory mapping configurations. This allows seamless conversion between data parallelism and model parallelism modes, enhancing both training efficiency and adaptability.
3Productivity
If tensor layouts are optimized for specific operators, then computation efficiency is improved, but inconsistent tensor layouts between consecutive operators require redistribution
Solution Approach 1:
The patent applies preliminary action by pre-planning tensor layout transformations through computational graph analysis. Before executing operators, the system identifies where layout conversions are needed and prepares redistribution operations in advance. This proactive approach minimizes runtime overhead and ensures smooth transitions between operators with different optimal layouts.
Data Source
Figure 1
Figure 2
Figure 3~4
AI summary
Embodiments of this application disclose a data processing method, and relate to the field of artificial intelligence. The method is applied to distributed parallel model training, for example, distributed training of a text translation model, a speech recognition model, a facial recognition model, a three-dimensional reconstruction model, and a virtual reality model. The method can support implementation of hybrid parallelism in a distributed cluster. The method includes: inserting, based on tensor layouts of tensors of each operator in a deep neural network model, a redistribution operator between operators that have an input-output dependency relationship, to implement conversion between different tensor layouts; inserting the redistribution operator into a sliced computational graph; and determining an updated sliced computational graph to implement parallel model training of the deep neural network.