Subgraph Compilation Using Reconstructed Nodes for Faster AI Scheduling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network computation graphs have low execution concurrency due to stream-based execution methods, leading to long scheduling times and inefficient node execution.
Innovation Solution
The computation graph is segmented into subgraphs, which are then reconstructed and compiled using reconstructed nodes as scheduling units, allowing for concurrent execution of multiple nodes and threads, and incorporating cache management and dependency relationships to optimize execution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If stream-based execution method is used, then nodes can be executed in sequence with some concurrency between streams, but execution concurrency is low and scheduling time is long
Solution Approach 1:
The computation graph is segmented into multiple subgraphs, and each subgraph is further segmented into multiple reconstructed nodes. This multi-level segmentation enables finer-grained parallel execution units, increasing execution concurrency while reducing scheduling overhead through localized optimization within each reconstructed node.
Solution Approach 2:
The patent introduces dynamic scheduling mechanisms where reconstructed nodes can be dynamically created, merged, or split based on runtime conditions. The scheduling system dynamically adjusts the granularity and distribution of execution units across streams, optimizing both concurrency and scheduling efficiency adaptively.
2Productivity
If computation graph is segmented into subgraphs with stream-based execution, then some parallel execution is possible, but overall execution concurrency remains low
Solution Approach 1:
By segmenting the computation graph into subgraphs and further into reconstructed nodes, the patent creates manageable execution units that can be independently scheduled. This segmentation transforms the complex global scheduling problem into multiple smaller, more tractable local scheduling problems, reducing overall scheduling complexity while enabling parallel execution.
Solution Approach 2:
The patent introduces an additional dimension of organization by creating reconstructed nodes that span multiple original nodes across different subgraphs. This dimensional transformation allows execution units to operate at a higher level of abstraction, enabling more flexible parallelism and reducing the complexity of managing fine-grained dependencies.
3Ease of operation
If node-level scheduling is used, then fine-grained control is achieved, but scheduling time becomes very long
Solution Approach 1:
The patent merges multiple original nodes into reconstructed nodes that can be scheduled as unified execution units. This merging reduces the total number of scheduling decisions required, significantly decreasing scheduling time while maintaining fine-grained control through the structured organization of reconstructed nodes and their internal node compositions.
Solution Approach 2:
The scheduling system dynamically adjusts the granularity of execution units by creating reconstructed nodes with variable compositions. This dynamic approach allows the system to optimize between fine-grained control and scheduling efficiency based on the specific computation graph structure and runtime conditions, achieving both goals simultaneously.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Embodiments of this application provide a subgraph compilation method, a subgraph execution method, and a related device in the field of artificial intelligence technologies. The method includes: obtaining a first subgraph, where the first subgraph is any one of a plurality of subgraphs obtained by segmenting a computation graph, and the first subgraph includes a plurality of first nodes and directed edges between the plurality of first nodes; reconstructing the first subgraph to obtain a first reconstructed subgraph, where the first reconstructed subgraph includes at least one first reconstructed node and directed edges between the at least one first reconstructed node; and compiling a first target reconstructed node, to obtain compiled data of the first target reconstructed node, where the first target reconstructed node is any one of the at least one first reconstructed node, the first target reconstructed node includes M first nodes in the plurality of first nodes and directed edges between the M first nodes, and M is a positive integer. Therefore, embodiments of this application can be used to reduce scheduling time.