Stream-Based Accelerator Processing for Computational Graphs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems processing computational graphs for neural networks on a single device, such as a GPU, face inefficiencies due to the large number and size of operations, leading to significant processing time.
Innovation Solution
The method involves assigning subgraphs of a computational graph to multiple streams on a hardware accelerator device, like a GPU, where operations that do not depend on each other are executed in different streams, and operations that change the internal state are executed sequentially, allowing for concurrent execution of logically independent operations and efficient memory management.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If operations of a computational graph are processed by a single device, then device complexity is reduced, but processing time increases significantly
Solution Approach 1:
The computational graph is divided into multiple subgraphs, which are then assigned to different streams on the same device or to multiple devices. This segmentation allows parallel processing of independent operations while maintaining manageable device complexity. The system identifies operations that can be executed independently and assigns them to different streams, thereby reducing overall processing time without significantly increasing device complexity.
2Productivity
If operations are assigned to multiple streams on a hardware accelerator, then processing throughput increases, but device complexity and memory management complexity increase
Solution Approach 1:
The system dynamically assigns operations to streams based on dependencies and resource availability. The assignment is not static but adapts to the computational graph structure and runtime conditions. This dynamic approach allows the system to maximize throughput by efficiently utilizing multiple streams while managing complexity through adaptive scheduling rather than fixed complex configurations.
Solution Approach 2:
The system introduces an intermediary layer (the stream assignment mechanism) that manages the complexity of coordinating multiple streams. This intermediary handles dependency tracking, resource allocation, and synchronization, thereby enabling high throughput processing while shielding the underlying hardware complexity from the computational graph representation.
3Speed
If operations that do not depend on each other are assigned to different streams, then parallel execution speed increases, but ensuring correct execution order becomes more complex
Solution Approach 1:
The system implements feedback mechanisms to track the execution status of operations across different streams. By monitoring completion events and dependency satisfaction, the system ensures correct execution order while maintaining parallel speedup. The feedback loop allows the scheduler to dynamically adjust stream assignment and synchronization points based on actual execution progress rather than static analysis alone.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for receiving, by a computational graph system, a request to process a computational graph; obtaining data representing a subgraph of the computational graph, the computational graph comprising a plurality of nodes and directed edges, wherein each node represents a respective operation, wherein each directed edge connects a respective first node to a respective second node, the subgraph assigned to a first device by a placer in the computational graph system; determining that the first device comprises a hardware accelerator having a plurality of streams; in response to determining, generating instructions that when executed by the first device cause the first device to: assign the operation represented by each node in the subgraph to a respective stream; and perform the operations represented by the nodes in the subgraph in accordance with the assignment.