Computational Graph Execution via Task Bodies and Memory Allocation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deep learning frameworks are not flexible or effective for training large-scale deep neural networks on distributed devices, as they require complex parallelism and complicate the use and implementation of distributed deep learning.
Innovation Solution
A method for executing a computational graph in a neural network model that creates task execution bodies and allocates idle memory blocks to enable pipeline parallelism, allowing the computational graph to participate in deep learning training tasks across different batches of data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If distributed devices are used for training large-scale deep neural networks, then the processing capability and throughput are improved, but the system complexity and implementation difficulty increase
Solution Approach 1:
The computational graph is segmented into multiple operator kernels that can be executed in parallel across distributed devices. Each operator kernel is treated as an independent executable unit, allowing the training workload to be divided and distributed across multiple devices, thereby improving throughput while managing complexity through modular organization
Solution Approach 2:
The deep learning framework is designed with universal interfaces that can operate across single devices and distributed devices. The same computational graph execution mechanism works whether deployed on one device or multiple devices, providing flexibility and reducing implementation complexity for distributed training scenarios
2Productivity
If distributed training interfaces are developed to enhance parallelism, then the training efficiency is improved, but the ease of operation and use deteriorates
Solution Approach 1:
The framework provides unified interfaces that automatically adapt to both single-device and distributed-device training scenarios. Users interact with the same simple interface regardless of the deployment scale, while the underlying system handles the complexity of parallel execution transparently, maintaining ease of operation while achieving high training efficiency
Solution Approach 2:
The computational graph execution system automatically manages the complexity of distributed parallelism without requiring users to manually configure complex parallel training interfaces. The system self-organizes the execution of operator kernels across devices based on the computational graph structure, eliminating the need for complicated user-facing distributed training interfaces
Data Source
AI summary
The present disclosure discloses a method for execution of a computational graph in a neural network model and an apparatus thereof, including: creating task execution bodies on a native machine according to a physical computational graph compiled and generated by a deep learning framework, and designing a solution for allocating a plurality of idle memory blocks to each task execution body, so that the entire computational graph participates in deep learning training tasks of different batches of data in a pipelining and parallelizing manner.