Neural Network Accelerator Nested Tiling and Bit-Width Extension
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Channel loop tiling in neural network accelerators increases data movement and memory usage, leading to bottlenecks in data throughput and energy efficiency, while also reducing the accuracy of calculations due to errors.
Innovation Solution
A neural network accelerator that includes an operator for calculating operation results based on tiled input feature maps and filter data, a quantizer for extending bit width, a compressor for generating partial sums through compression, and a decompressor for restoring operation results, minimizing external memory usage and communication overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If loop tiling is performed to implement large CNN on hardware platform, then the CNN can be accommodated in on-chip memory, but repeated data movement between on-chip memory and off-chip memory causes bottleneck phenomenon reducing data throughput and energy efficiency
Solution Approach 1:
The patent implements nested tiling by dividing the CNN computation into multiple hierarchical levels: first tiling the entire CNN into macro-tiles, then further dividing each macro-tile into micro-tiles. This nested structure allows intermediate results to be stored in on-chip memory at the macro-tile level, reducing the frequency of data movement between on-chip and off-chip memory while maintaining the ability to process large CNN models.
2Quantity of substance
If loop tiling is performed to implement large CNN on hardware platform, then the CNN can be accommodated in on-chip memory, but repeated data movement increases energy consumption
Solution Approach 1:
The nested tiling structure enables intermediate computation results to be retained in on-chip memory across multiple micro-tile iterations within the same macro-tile. This reduces the number of times data must be transferred to and from off-chip memory, thereby significantly reducing energy consumption associated with data movement while still enabling processing of large CNN models that exceed on-chip memory capacity.
3Device complexity
If channel loop is tiled to process large CNN, then the computation can be divided into manageable blocks, but errors during tiling cause decrease in calculation accuracy
Solution Approach 1:
The patent performs preliminary bit-width extension before the tiling and compression operations. By extending the bit-width of data representations prior to segmentation and compression, the system preserves numerical precision throughout the tiling process. This preliminary action ensures that even when data is divided into multiple tiles and subjected to compression, the calculation accuracy is maintained and errors are minimized.
Data Source
AI summary
A neural network accelerator includes an operator that calculates a first operation result based on a first tiled input feature map and first tiled filter data, a quantizer that generates a quantization result by quantizing the first operation result based on a second bit width extended compared with a first bit width of the first tiled input feature map, a compressor that generates a partial sum by compressing the quantization result, and a decompressor that generates a second operation result by decompressing the partial sum, the operator calculates a third operation result based on a second tiled input feature map, second tiled filter data, and the second operation result, and an output feature map is generated based on the third operation result.