On-the-fly Fourier Transform for CNN Memory Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing FFT-based approach for convolution in CNNs generates a large number of frequency domain coefficients proportional to the image dimensions, exceeding the memory capacity of most computers and making it impractical for hardware implementation.
Innovation Solution
The method performs on-the-fly Fourier transforms (OTF-FTs) on kernel coefficients and multiplies them with pruned Fourier transforms of input features, followed by inverse Fourier transforms to reduce storage requirements and achieve convolution in the frequency domain with spatial domain storage, similar to direct convolution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If FFT-based convolution is used, then computational complexity is reduced, but memory usage increases significantly
Solution Approach 1:
The patent pre-computes and stores the Fourier transform of the kernel coefficients in on-chip memory before processing images. This preliminary action allows the frequency domain convolution to proceed with reduced runtime computation and memory access requirements, resolving the contradiction between computational complexity reduction and memory usage.
Solution Approach 2:
The patent dynamically adjusts the processing approach by using pre-stored kernel frequency domain representations that can be efficiently accessed and combined with image frequency domain data. This dynamic utilization of pre-computed data structures enables fast convolution operations without requiring large amounts of runtime memory.
2Productivity
If FFT-based convolution is used, then processing speed is improved, but on-chip memory capacity is exceeded
Solution Approach 1:
The kernel Fourier transform coefficients are pre-computed and stored in on-chip memory before image processing begins. This preliminary preparation enables fast frequency domain convolution during runtime without requiring large on-chip memory capacity, as the pre-stored coefficients are compact and efficiently accessible.
Solution Approach 2:
The patent extracts the computationally intensive kernel Fourier transform operation from the runtime processing path and performs it beforehand. This extraction moves the heavy computation to a setup phase, allowing the main processing loop to operate with reduced memory requirements and improved speed.
3Device complexity
If frequency domain coefficients are stored, then convolution computation is simplified, but storage requirements become impractical
Solution Approach 1:
The patent pre-computes the Fourier transform of kernel coefficients and stores them in compact on-chip memory structures. This preliminary computation simplifies the convolution operation to efficient point-wise multiplication in the frequency domain, while the pre-stored coefficients minimize runtime storage requirements to practical levels.
Solution Approach 2:
The patent changes the representation parameters of kernel data from spatial domain to frequency domain, and from full-precision floating point to optimized fixed-point or reduced-precision formats suitable for on-chip storage. This parameter transformation maintains computational accuracy while dramatically reducing storage requirements.
Data Source
AI summary
A method for analyzing images to generate a plurality of output features includes receiving input features of the image and performing Fourier transforms on each input feature. Kernels having coefficients of a plurality of trained features are received and on-the-fly Fourier transforms (OTF-FTs) are performed on the coefficients in the kernels. The output of each Fourier transform and each OTF-FT are multiplied together to generate a plurality of products and each of the products are added to produce one sum for each output feature. Two-dimensional inverse Fourier transforms are performed on each sum.