Neural Network Accelerator Buffer Tile Sizing for Off-Chip Transfer Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network accelerators face challenges in reducing data transmission between their buffers and external memory, which hinders computational efficiency due to high off-chip memory transfer times.
Innovation Solution
The proposed electronic apparatus and method optimize data transmission by determining optimal combinations of fused and non-fused layers in a neural network model, calculating data transmission times, and adjusting tile sizes to minimize data movement between the buffer and memory.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is stored in external memory for neural network computation, then storage capacity is sufficient, but data transmission time between buffer and memory increases
Solution Approach 1:
The patent segments the neural network computation into multiple tiles, where each tile can be processed independently. This allows the system to load smaller data chunks from external memory into the buffer, process them, and write results back, reducing the total data transmission time while maintaining sufficient storage capacity through multiple sequential operations
Solution Approach 2:
The patent performs preliminary actions by pre-processing and tiling the neural network data before computation. The data is divided into tiles in advance, and the computation is scheduled to process these tiles in an optimized sequence, reducing the need for repeated data transmission between buffer and external memory during the computation process
2Loss of time
If layer fusion is applied to reduce data transmission, then off-chip memory transfer decreases, but buffer capacity requirements increase
Solution Approach 1:
The patent segments the fused layer computation into multiple tiles that can be processed in batches. Instead of loading all data for fused layers into the buffer at once, the system processes smaller tile segments sequentially, reducing the required buffer capacity while maintaining the benefits of layer fusion by keeping intermediate results in the buffer for subsequent layer computations
Solution Approach 2:
The patent dynamically adjusts the computation schedule to optimize buffer usage. By carefully ordering the processing of different tiles from fused and non-fused layers, the system ensures that data remains in the buffer only as long as necessary, dynamically allocating buffer space to minimize peak memory requirements while still achieving reduced off-chip transfers through fusion
Data Source
AI summary
An electronic apparatus includes a memory configured to store data corresponding to a neural network model, a neural network accelerator including a buffer configured to temporarily store the data corresponding to the neural network model, and a core configured to perform a computation on the neural network model based on the data stored in the buffer, and a processor configured to determine a plurality of combinations including fused layers and non-fused layers based on a method of selecting and fusing adjacent layers of the neural network model, based on a capacity of the buffer, determine a size of a tile capable of being processed in one computation in the core to acquire feature values output by the fused layers and the non-fused layers, and based on a first memory usage and computation time for storing the feature values in the buffer, determine whether to store the feature values in the memory.