Neural Network Graph Partitioning for Hardware Resource Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional neural network graph partitioning techniques lead to inefficient use of hardware resources due to sequential traversal of layers and frequent switching between hardware accelerators, resulting in increased model inference time and overhead.
Innovation Solution
A method for determining an optimal order of processing nodes in a neural network by identifying independent nodes and assigning them to appropriate hardware resources based on type and frequency, using techniques like depth-first search and caching to reduce redundant searches and merge sub-graphs, thereby improving hardware resource utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If conventional sequential traversal of neural network layers is used, then the processing order is simple to implement, but hardware resource utilization becomes inefficient due to frequent switching between hardware accelerators
Solution Approach 1:
The patent segments the neural network graph into independent nodes and groups them by hardware resource type. This segmentation allows the system to identify which nodes can be processed by which hardware accelerator, reducing unnecessary switching between different accelerators while maintaining implementation feasibility through systematic graph analysis.
Solution Approach 2:
The patent performs preliminary analysis of the neural network graph to determine the optimal execution order and hardware resource allocation before actual processing begins. By pre-computing the execution plan that groups nodes by hardware type and identifies independent nodes, the system avoids frequent runtime switching between accelerators, thereby improving hardware resource utilization without complicating the actual execution phase.
2Adaptability or versatility
If frequent switching between hardware accelerators is performed, then different node types can be processed by specialized hardware, but model inference time increases due to switching overhead
Solution Approach 1:
The patent merges consecutive nodes that can be processed by the same hardware accelerator into contiguous execution blocks. By combining multiple nodes that share the same hardware resource requirement into a single execution sequence, the system minimizes the number of switches between different hardware accelerators, thereby reducing switching overhead and inference time while still utilizing specialized hardware for different node types.
Solution Approach 2:
The patent ensures continuous execution of nodes on the same hardware accelerator by identifying and grouping independent nodes that share hardware resource requirements. This continuity eliminates idle switching periods between accelerators, maintaining useful computational action throughout the inference process and reducing overall inference time while preserving hardware specialization benefits.
3Productivity
If more subnets are created in the partitioning process, then finer-grained hardware resource allocation is achieved, but the number of hardware switches and overhead increases
Solution Approach 1:
The patent dynamically determines the optimal number and size of subnets based on the neural network graph structure and hardware resource availability. Rather than using a fixed fine-grained partitioning scheme, the system adapts the subnet granularity to match the actual dependencies and hardware requirements of the neural network nodes, achieving efficient resource allocation without creating excessive subnets that would increase complexity and switching overhead.
Data Source
AI summary
A method for processing a neural network includes receiving a graph corresponding to an artificial neural network including multiple nodes connected by edges. The method determines a set of independent nodes of multiple nodes to be executed in a neural network. The method also determines a next node in the set of independent nodes to add to an ordered set of the multiple nodes corresponding to an order of execution via a hardware resource for processing the neural network. The next node is determined based on a common hardware resource with a first preceding node in the ordered set or a frequency of nodes in the set of independent nodes to be executed via a same hardware resource. The ordered set of the plurality of nodes is generated based on the next node. The method may be repeated until each of the nodes of the graph are included in the ordered set of the nodes.