Multi-Core Neural Network Execution With On-Chip Weight Partitioning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The performance and efficiency of AI accelerators in executing convolutional neural networks (CNNs) are limited by memory bottlenecks due to the time it takes to fetch data from off-chip memory, which is often longer than the time required for computation.
Innovation Solution
The method involves partitioning layer weights and input data into subsets that are stored in on-chip memories across multiple cores, allowing efficient execution of neural network computations without writing intermediate results to off-chip memory, thereby reducing off-chip memory bandwidth usage and enhancing computational efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is fetched from off-chip memory in layer-by-layer order, then the neural network computation can be performed, but the memory bandwidth becomes the bottleneck and reduces computational efficiency
Solution Approach 1:
The patent segments the neural network computation into multiple micro-batches, allowing parallel processing of different data batches through the same layer. This segmentation enables better utilization of on-chip memory and reduces the frequency of off-chip memory accesses, thereby improving computational efficiency and reducing the time lost to data fetching.
Solution Approach 2:
The patent performs preliminary actions by pre-loading weights and activations into on-chip memory before computation begins. By preparing data in advance and organizing it in a format suitable for parallel processing, the system minimizes the need for frequent off-chip memory accesses during the actual computation phase, thus reducing data fetching time and improving overall productivity.
2Quantity of substance
If layer weights are stored in off-chip memory, then memory capacity is sufficient, but the time to fetch weights to cores is longer than computation time
Solution Approach 1:
The patent implements a nested memory hierarchy where off-chip memory stores the complete weight matrices, while on-chip memory stores segmented portions of these weights. The system nests multiple levels of memory organization, allowing weights to be progressively loaded from off-chip to on-chip memory in an organized manner, thereby maintaining sufficient storage capacity while improving access speed through closer proximity to computing units.
Solution Approach 2:
The patent changes the dimensionality of weight storage by organizing weights into multiple segments along different dimensions (e.g., splitting weight matrices into row-wise or column-wise segments). This dimensional transformation allows parallel loading of weight segments into on-chip memory, effectively increasing data access speed while maintaining the total quantity of stored weights.
3Reliability
If intermediate results are written back to off-chip memory after each layer, then data can be persisted, but the memory bandwidth is consumed and performance is reduced
Solution Approach 1:
The patent merges multiple micro-batch computations together, allowing intermediate results from different batches to be accumulated and processed jointly. By combining multiple small computation tasks into larger batch operations, the system maintains data persistence reliability while reducing the frequency of write-back operations to off-chip memory, thereby preserving memory bandwidth for computation and improving overall inference performance.
Data Source
Figure 1
Figure 2
Figure 3A
AI summary
This disclosure relates to artificial intelligence. A method applied to an apparatus for running a neural network is disclosed. The method comprises partitioning weights of a first layer and a second layer into multiple subsets of first weights and multiple subsets of second weights, respectively, which are distributed among multiple cores of the apparatus and are stored only in on-chip memory. Each core performs computation of the first layer using a first input partition and the respective subset of first weights, to obtain a first output partition of the first layer. The first output partition of the first layer is stored in the on-chip memory. Each core performs computation of the second layer using a respective first output partition of the first layer and the corresponding subset of second weights, to obtain a first output partition of the second layer. In this way, neural network computation efficiency can be enhanced.