Zero Skipping Convolution Engine for CNN Memory Bandwidth
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Matrix multiplication-based architectures for Convolutional Neural Networks (CNNs) require high memory bandwidth due to the duplication of 2D image data, increasing memory requirements and computation costs.
Innovation Solution
A novel convolution engine that combines zero coefficient skipping with a multi-kernel accumulation approach, where only one kernel coefficient is applied to a large portion of the input image at a time, reducing computation and memory bandwidth by skipping zero coefficient multiplications and using a low-cost kernel decompressor, and performing convolutions using accumulation instead of summation to enable efficient reuse of input data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If matrix multiplication-based architecture is used for CNN, then computation can be performed, but memory bandwidth requirement increases due to duplication of 2D image data
Solution Approach 1:
The patent segments the convolution operation into independent coefficient-based processing units. Each coefficient is applied to the entire 2D image independently, allowing the image data to be segmented into tiles that can be processed in parallel without duplication. This segmentation eliminates the need to duplicate image data for each kernel coefficient while maintaining computation performance.
Solution Approach 2:
The patent transforms the traditional matrix multiplication approach (operating on flattened 1D vectors) into a 2D tile-based convolution approach. By maintaining the spatial structure of the image data in 2D tiles and applying coefficients across the entire tile, the system reduces memory bandwidth requirements while preserving computational efficiency through the added dimensional organization.
2Loss of energy
If zero coefficient skipping is implemented, then computation cost is reduced, but device complexity increases due to need for coefficient checking and selective processing
Solution Approach 1:
The patent extracts only the non-zero coefficients from the kernel and processes them separately from the zero coefficients. By taking out the non-zero coefficients and applying them directly to the image tiles without requiring full matrix multiplication logic, the system reduces computation cost while avoiding the complexity of conditional checking for every coefficient.
Solution Approach 2:
The patent implements a self-service mechanism where the convolution engine automatically identifies and processes only the necessary non-zero coefficients without external control logic. The accumulation buffer automatically accumulates results from non-zero coefficient applications, eliminating the need for complex control structures to manage zero-skipping logic.
3Reliability
If traditional convolution with full kernel application is used, then complete convolution is achieved, but memory bandwidth is wasted due to processing zero coefficients
Solution Approach 1:
The patent implements skipping of zero coefficient applications by directly accumulating the results of non-zero coefficient multiplications into the accumulation buffer. This rushing through of zero coefficients maintains convolution accuracy while eliminating wasted memory bandwidth on unnecessary data transfers and computations.
Solution Approach 2:
The patent performs preliminary identification of non-zero coefficients before the main convolution process. By pre-processing the kernel to identify which coefficients are non-zero and organizing them for efficient access, the system ensures that only necessary computations are performed, maintaining accuracy while optimizing memory bandwidth utilization.
Data Source
AI summary
A convolution engine, such as a convolution neural network, operates efficiently with respect to sparse kernels by implementing zero skipping. An input tile is loaded and accumulated sums are calculated for the input tile for non-zero coefficients by shifting the tile according to a row and column index of the coefficient in the kernel. Each coefficient is applied individually to tile and the result written to an accumulation buffer before moving to the next non-zero coefficient. A 3D or 4D convolution may be implemented in this manner with separate regions of the accumulation buffer storing accumulated sums for different indexes along one dimension. Images are completely processed and results for each image are stored in the accumulation buffer before moving to the next image.