Neural Network Execution Streams for Parallel GPU Scheduling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing application programming interfaces (APIs) for executing neural networks are inefficient and result in complex code, leading to suboptimal performance.
Innovation Solution
An application programming interface that allows for defining a neural network as a graph with nodes and edges, where operations are assigned to execution streams using a compiler-optimized approach, leveraging techniques like loop fusion, parallelization, and synchronization to improve performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional APIs are used to execute neural networks, then code can be written in conventional manner, but execution efficiency is poor and performance is suboptimal
Solution Approach 1:
The neural network execution is segmented into multiple independent streams, where each stream contains a sequence of operations that can be executed in parallel. This segmentation allows the compiler to optimize each stream independently while maintaining overall execution efficiency, resolving the contradiction between productivity and device complexity.
Solution Approach 2:
The API introduces dynamic stream creation and management capabilities, allowing the execution plan to be adapted at runtime. Streams can be created, destroyed, and synchronized dynamically based on the specific neural network operations, enabling optimal execution efficiency without requiring complex static code structures.
2Loss of time
If operations are executed sequentially in traditional APIs, then code is simpler to write, but memory copies and kernel executions cannot be overlapped
Solution Approach 1:
The compiler performs preliminary analysis of the neural network operations and automatically generates optimized execution streams before runtime execution. This preliminary stream generation includes determining operation dependencies, allocating streams, and preparing synchronization points, which enables time overlap of memory copies and kernel executions without requiring complex runtime management.
Solution Approach 2:
The execution stream acts as an intermediary structure between the neural network operations and the GPU hardware. It mediates the execution by organizing operations into parallel streams, managing memory copies asynchronously, and coordinating kernel launches, thereby reducing execution time while abstracting away the hardware complexity.
3Speed
If multiple streams are executed concurrently to improve performance, then execution speed increases, but synchronization and coordination become more complex
Solution Approach 1:
The execution stream system incorporates feedback mechanisms where the compiler analyzes operation dependencies and automatically adjusts stream assignments and synchronization points. This feedback loop ensures that streams are synchronized correctly at dependency boundaries while maintaining concurrent execution, achieving high speed without manual synchronization complexity.
Data Source
AI summary
Apparatuses, systems, and techniques to perform a neural network. In at least one embodiment, an application programming interface schedules two or more graph nodes to be performed by two or more parallel processing pipelines based, at least in part, on an order of layers in the neural network.