3D CNN Tiling for Lower External Memory Bandwidth
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Convolutional neural networks require significant processing resources and external memory bandwidth, leading to performance bottlenecks that hinder efficient execution.
Innovation Solution
The system partitions input data into three-dimensional blocks and processes them in a specific pattern to minimize external memory bandwidth utilization, combining convolution operations within internal memory before writing back to external memory.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If convolutional neural networks are executed using conventional methods, then processing can be performed, but external memory bandwidth consumption is excessive causing performance bottlenecks
Solution Approach 1:
The input data is divided into multiple tiles that are processed independently. Each tile is loaded from external memory, processed through the convolutional neural network layers, and results are accumulated. This segmentation reduces the amount of data that needs to be transferred between external and internal memory, thereby reducing memory bandwidth consumption while maintaining processing throughput.
Solution Approach 2:
The patent implements a nested memory hierarchy where tiles are loaded from external memory into internal memory, processed through multiple convolutional layers with intermediate results stored in buffers, and final results written back to external memory. This nested structure allows intermediate results to be retained in faster internal memory without requiring repeated external memory accesses.
2Productivity
If more processing resources are allocated to convolutional neural networks, then performance improves, but power consumption increases
Solution Approach 1:
The patent extracts and optimizes the memory access patterns from the convolutional neural network execution. By separating the computation-intensive operations from memory-intensive operations, the system can allocate processing resources more efficiently. The tiling approach allows compute units to process multiple tiles using the same loaded data, reducing redundant memory accesses and associated power consumption.
Solution Approach 2:
The patent performs preliminary tiling and data organization before execution. Input data is pre-processed into tile formats that optimize memory access patterns during inference. This preliminary organization reduces the need for complex memory management during execution, lowering overall power consumption while maintaining high processing throughput.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Systems, apparatuses, and methods for implementing memory bandwidth reduction techniques for low power convolutional neural network inference applications are disclosed. A system includes at least a processing unit and an external memory coupled to the processing unit. The system detects a request to perform a convolution operation on input data from a plurality of channels. Responsive to detecting the request, the system partitions the input data from the plurality of channels into 3D blocks so as to minimize the external memory bandwidth utilization for the convolution operation being performed. Next, the system loads a selected 3D block from external memory into internal memory and then generates convolution output data for the selected 3D block for one or more features. Then, for each feature, the system adds convolution output data together across channels prior to writing the convolution output data to the external memory.