2D CNN Kernel Tiling With Overlap Rows to Cut IFM I/O
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The implementation of Convolution Neural Networks (CNNs) in real-time applications faces challenges due to high computational requirements and significant data transfer bottlenecks, particularly in the transfer of input feature maps (IFMs) to and from larger memory units, which necessitates segmentation into multiple tiles and multiple fetches of IFM rows for kernel calculations, leading to an input/output (I/O) bottleneck.
Innovation Solution
A method is introduced where overlap rows of the current IFM tile are fetched once and maintained in a buffer until the completion of calculations for the next IFM tile, reducing the need for repeated fetching and enhancing buffer utilization by using a cyclic buffer mechanism.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If the input feature map is segmented into multiple tiles to fit the buffer, then the buffer can be utilized, but the I/O bandwidth traffic increases due to multiple fetches
Solution Approach 1:
The patent applies preliminary action by fetching and storing overlap rows of the input feature map into the buffer before they are actually needed for computation. By pre-loading these overlapping rows that will be required for both current and next tile calculations, the system avoids repeated I/O operations during the computation phase, thereby reducing overall I/O bandwidth traffic while maintaining effective buffer utilization
Solution Approach 2:
The patent segments the input feature map into multiple tiles for processing, but introduces an overlap region between adjacent tiles. This segmentation strategy allows the buffer to be efficiently utilized for each tile while the overlap rows serve as a shared resource between tiles, reducing the total number of fetch operations required compared to non-overlapping segmentation
2Productivity
If rows are fetched multiple times for different OFM tiles, then calculations for multiple tiles can be completed, but the I/O bottleneck worsens
Solution Approach 1:
The patent implements discarding and recovering by allowing the buffer to be overwritten with new tile data after the overlap rows have been utilized for computation. The overlap rows are effectively 'recovered' and reused for the next tile's calculations, while non-overlap rows are discarded and replaced. This cyclic reuse pattern minimizes repeated fetching from external memory, reducing I/O bottleneck and processing time
Solution Approach 2:
The patent ensures continuity of useful action by maintaining the overlap rows in the buffer continuously across tile boundaries. Instead of completing one tile calculation and then fetching data for the next tile, the overlap rows remain available in the buffer, allowing computation to proceed continuously without I/O interruptions, thereby reducing the I/O bottleneck
Data Source
AI summary
A method, integrated circuit, and a computer readable medium that stores instructions for reducing IO traffic from a global or remote memory unit to a buffer of a neural network unit, by using overlap rows of an input feature map tile.