CNN Accelerator On-Chip Buffering for Memory Bandwidth Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional methods for processing convolutional neural network (CNN) layers face memory-bandwidth limitations due to the large amount of intermediate data that needs to be transferred off-chip to and from external memory, which hampers performance gains even with increased computational resources.
Innovation Solution
The system processes CNN layers by loading a three-dimensional tile structure of input data, reusing intermediate convolved data values when they overlap, computing non-overlapping values using weight data values, and buffering these in an on-chip buffer to minimize off-chip data transfers, allowing for iterative processing across fused convolutional layers without writing intermediate data to external memory.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional layer-by-layer CNN processing is used, then computational accuracy is maintained, but memory bandwidth requirements increase due to frequent off-chip data transfers
Solution Approach 1:
The patent combines multiple CNN layers into a single processing pipeline, allowing intermediate data to be reused across layers without being written to and read from external memory. This merging of layer processing eliminates redundant memory transfers while maintaining computational accuracy through proper dataflow management and on-chip buffer utilization.
Solution Approach 2:
The patent performs preliminary loading of input data into on-chip buffers before processing begins, and pre-positions intermediate results in appropriate buffer locations during processing. This preliminary action ensures that data is readily available for reuse in subsequent layers without requiring off-chip memory accesses, thereby reducing memory bandwidth requirements.
2Productivity
If more computational resources are allocated to CNN processing, then processing speed increases, but memory bandwidth limitations become the bottleneck
Solution Approach 1:
By merging multiple CNN layers into a unified processing architecture with shared on-chip buffers, the system enables computational resources to work continuously on different layers simultaneously. This eliminates the memory bandwidth bottleneck that would otherwise limit the effectiveness of increased computational resources, as intermediate results remain accessible on-chip for reuse.
Solution Approach 2:
The patent establishes continuous dataflow through the CNN layers by keeping intermediate results in on-chip buffers where they can be immediately reused. This continuity ensures that computational units remain actively productive without idle waiting for memory transfers, thereby maximizing processing speed while minimizing memory bandwidth consumption.
3Quantity of substance
If intermediate data is stored in on-chip buffers, then memory bandwidth requirements are reduced, but on-chip buffer size requirements increase
Solution Approach 1:
The patent implements a hierarchical buffer architecture where different regions of on-chip memory are allocated to different layers and data types based on their specific requirements. This local quality approach ensures that buffer space is efficiently utilized by storing only the necessary intermediate results for each layer in appropriate buffer regions, minimizing total on-chip buffer size while still reducing memory bandwidth requirements.
Solution Approach 2:
The patent employs a sliding window approach where intermediate data in on-chip buffers is discarded once it has been reused by subsequent layers, and buffer space is recovered for reuse. This dynamic management of buffer space ensures that the on-chip buffer size remains manageable while still capturing all necessary intermediate results for computation reuse across layers.
Data Source
AI summary
System, method, and accelerator to process a convolutional neural network. In accordance therewith, a tile structure having input data values is loaded for a convolution layer. Each tile of the tile structure corresponds to a respective feature map in a set of input feature maps. The tile structure of each iteration represents a different subset of data values in the input feature maps. Intermediate data values associated with a subset of the data values of the input feature maps in the current intermediate tile structure are reused, when the intermediate data values of a previous tile structure overlap values to be computed in the current tile structure. Intermediate non-overlapping data values that are associated with the subset of the data values in the current tile structure are computed using associated filters having weight data values. Available reused intermediate data values and computed intermediate data values are buffered as intermediate data.