Neural Network Tiling for On-Chip Buffer Memory Access Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional neural network processing methods frequently access external memory, leading to limited calculation performance and high power consumption due to bandwidth constraints and system power consumption.
Innovation Solution
A neural network tiling method that tiles a neural network graph to obtain depth subgraphs, allowing for the generation of depth subnetworks that do not require external memory access, thereby reducing memory access frequency and power consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a single vertex is used as a basic unit to perform calculation layer-by-layer, then the neural network can process data sequentially, but the calculation performance is limited and system power consumption is high due to frequent external memory access
Solution Approach 1:
The neural network graph is segmented into multiple depth subgraphs, where each subgraph represents a portion of the network that can be processed independently. This segmentation allows intermediate results to be stored in on-chip buffers rather than requiring frequent external memory access, thereby improving calculation performance and reducing power consumption.
2Quantity of substance
If calculation result data of a single vertex is stored in external memory, then data storage capacity is sufficient, but the frequency of external memory access increases, limiting calculation performance
Solution Approach 1:
The patent implements a hierarchical memory structure where on-chip buffers are nested within the processing unit, and external memory serves as a larger nested storage layer. Intermediate calculation results are stored in the nested on-chip buffers, reducing the need to access the outer external memory layer, thus improving calculation performance while maintaining sufficient storage capacity.
3Use of energy by moving object
If the neural network is tiled to generate depth subnetworks, then external memory access frequency decreases, but the device complexity increases due to additional buffer management
Solution Approach 1:
Multiple depth subgraphs are merged and processed in parallel using shared on-chip buffers. This merging approach reduces the total number of buffer management operations required compared to processing each subgraph separately, thereby reducing device complexity while maintaining the power consumption benefits of reduced external memory access.
Data Source
Figure 1
Figure 2
Figure 3A
AI summary
Embodiments of this application disclose a neural network tiling method, a prediction method, and a related apparatus. The neural network tiling method includes: obtaining a neural network graph, where the neural network graph is used to represent a neural network; tiling the neural network graph to obtain a depth subgraph, where a plurality of vertices included in the depth subgraph exchange data with each other by reading and writing an on-chip buffer, the depth subgraph is used to successively process at least two groups of data obtained by tiling first input data, to obtain first output data, and the first input data is input data of the depth subgraph. In the method, a neural network tiling apparatus tiles a neural network graph to obtain one or more depth subgraphs, so as to generate one or more depth subnetworks based on the one or more depth subgraphs. When these depth subnetworks are used to perform a processing task of the neural network, a quantity of times of accessing an external memory can be greatly decreased.