Neural Network Subgraph Partitioning for Memory Allocation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current hardware technologies, such as CPUs and GPUs, face challenges in storing and executing neural networks due to the large number of weights required, leading to slow performance and extensive dynamic memory allocation.
Innovation Solution
Implementing static memory allocation by partitioning neural networks into subgraphs and assigning them to multiple processing units, allowing weights to fit within dedicated memory, thereby avoiding dynamic memory allocation and optimizing access times.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If dynamic memory allocation is used to store neural network weights, then the system can handle large numbers of weights, but performance decreases and computational overhead increases
Solution Approach 1:
The patent segments the neural network computation into multiple stages and partitions weights into different memory regions based on their access patterns during each stage. This segmentation allows static allocation of memory for each weight group, eliminating dynamic memory allocation overhead while maintaining the ability to handle large numbers of weights across multiple computation stages.
2Loss of time
If all neural network weights are loaded into dedicated memory, then access times are optimized, but memory capacity requirements exceed available memory
Solution Approach 1:
The patent performs preliminary analysis of the neural network computation graph to identify which weights are needed at each computation stage. Based on this preliminary action, weights are statically allocated to specific memory regions in advance, allowing fast access during execution without requiring all weights to be loaded simultaneously into memory.
Solution Approach 2:
The patent implements a multi-stage computation approach where the system dynamically transitions between different computation stages, each with its own statically allocated weight sets. This allows the system to optimize for fast access within each stage while managing overall memory capacity constraints across the entire neural network execution.
3Productivity
If static memory allocation is implemented by partitioning neural networks, then dynamic memory allocation is avoided and access times are optimized, but system complexity increases
Solution Approach 1:
The patent segments the neural network into multiple computation stages with clearly defined weight requirements for each stage. This segmentation simplifies the static allocation process by breaking down the complex problem of allocating all weights simultaneously into simpler, stage-by-stage allocation decisions, reducing overall system complexity while maintaining execution speed benefits.
Data Source
AI summary
Techniques for time-based memory allocation for a neural network inference are disclosed. A description of a neural network comprising a plurality of operations to be executed across a set of accelerators is received. A plurality of interconnect times at a plurality of partition points within the neural network are calculated. Each of the plurality of interconnect times corresponds to a duration of time for transferring an output feature map from one of the set of accelerators to another of the set of accelerators to be used as an input feature map. A partitioning scheme that divides the plurality of operations into a set of subgraphs is determined based on the plurality of interconnect times. Each of the set of subgraphs is assigned to a different accelerator of the set of accelerators in accordance with the partitioning scheme.