Hardware-Accelerated Neural Network Convolution via FFT Patching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deep learning accelerator technologies perform convolutions in a suboptimal and inefficient manner, particularly in neural networks, leading to high computational costs and resource usage due to the repetitive and intensive nature of convolutional layers in image processing tasks.
Innovation Solution
The implementation of overlap-add and overlap-save algorithms in conjunction with Fast Fourier Transform (FFT) methods to efficiently compute convolutions, allowing for the division of input images into partially overlapping patches, transformation into the frequency domain, multiplication by filters, and reassembly into the spatial domain, thereby reducing the number of operations required.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional convolution methods are used in deep learning accelerators, then the implementation is simple and straightforward, but the computational load is extremely high and processing time is long
Solution Approach 1:
The patent divides the input image into multiple smaller patches that overlap with each other. Each patch is then processed independently through the convolution operation. This segmentation approach reduces the computational complexity by breaking down the large convolution operation into smaller, more manageable operations on smaller patches, thereby increasing processing speed while maintaining the overall convolution functionality.
Solution Approach 2:
The patent transforms the convolution operation from the spatial domain to the frequency domain using Fast Fourier Transform (FFT). By converting the image and filter into frequency representations, the convolution operation becomes a simple element-wise multiplication rather than complex spatial sliding operations. This dimensional transformation significantly reduces computational complexity and improves processing efficiency.
2Adaptability or versatility
If the filter is repeatedly tiled across the input image in a sliding window fashion, then translation variance is achieved, but the computational intensity increases significantly
Solution Approach 1:
The input image is segmented into overlapping patches, and the filter is applied to each patch through frequency-domain multiplication. This segmentation approach maintains translation variance by processing different regions of the image while reducing the total computational energy consumption by avoiding the repetitive tiling and sliding operations required in conventional methods.
Solution Approach 2:
The patent changes the domain in which the convolution operation is performed, transforming from spatial domain sliding window operations to frequency domain multiplication. This parameter change (domain transformation) significantly reduces the computational energy consumption while preserving the translation variance property, as the frequency-domain approach inherently handles the sliding window effect through the convolution theorem.
3Productivity
If Fast Fourier Transform methods are applied to compute convolutions, then the number of operations is reduced, but the implementation complexity increases
Solution Approach 1:
The patent segments the image into smaller patches and applies FFT to each patch independently. This segmentation allows the use of efficient FFT algorithms on smaller data sizes, reducing the overall computational complexity and making the hardware implementation more feasible. The overlapping patches ensure that the segmentation does not lose any information while maintaining computational efficiency.
Solution Approach 2:
The patent transforms the convolution operation to the frequency domain using FFT, changing the computational approach from spatial sliding window operations to frequency domain multiplication. This dimensional change reduces the number of operations required and improves computation efficiency. The frequency-domain approach, while algorithmically more complex, is highly amenable to hardware acceleration and can be implemented efficiently with dedicated FFT hardware units.
Data Source
AI summary
A system for providing efficient-hardware accelerated neural network convolution is disclosed. The system receives an input image for an artificial intelligence task for a deep learning accelerator. The system divides the image in to equally-sized partially overlapping image patches and applies a Fast Fourier Transform to the image patches. A size of an image filter is padded to match a size of the image patches and a Fast Fourier Transform to applied to the image filter. For each pixel in each patch, a matrix-vector product is computed between channels of each image patch and a matrix from a corresponding pixel location in the image filter. An inverse Fast Fourier Transform is applied to the matrix-vector product to convert each image to the spatial domain. A convolved version of the image is reconstructed by summing overlapping edges of the patches or by discarding overlapping regions of the patches.