Convolutional Neural Network Memory Layout Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for performing convolution operations in convolutional neural networks (CNNs) face inefficiencies due to high computational requirements, memory access issues, and poor performance on CPUs, particularly when dealing with large models and sparse convolution layers, which can lead to slow operations and increased cache misses.
Innovation Solution
The method involves storing input arrays continuously along the input channel depth and convolution arrays continuously along the filter channel depth, allowing for an aggregate matrix multiply operation that reduces the number of operations and memory accesses, while leveraging sparse convolutions without the need for shuffle and reshuffle operations, thereby improving computation efficiency and reducing cache misses.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional convolution operations are performed in CNNs, then the model can process data, but the computational requirements are high and operation speed is slow
Solution Approach 1:
The patent changes the parameter of memory access pattern from non-continuous to continuous along the channel depth dimension. By reorganizing how convolution arrays are stored and accessed in memory, the system achieves better cache utilization and reduces memory access latency, thereby improving operation speed without increasing computational power requirements
Solution Approach 2:
The patent segments the convolution operation into independent channel groups that can be processed separately. By dividing the computation into smaller independent segments that can be executed in parallel and stored contiguously in memory, the system improves both speed and reduces the peak computational burden
2Productivity
If convolution arrays are stored in traditional memory layouts, then memory access is simple, but cache misses increase and computation efficiency decreases
Solution Approach 1:
The patent introduces a new dimension for memory organization by storing convolution arrays continuously along the channel depth dimension rather than following traditional spatial or batch dimensions. This dimensional reorganization ensures that during channel-wise convolution operations, the required data elements are accessed sequentially in memory, maximizing cache hit rates and eliminating cache misses
Solution Approach 2:
The patent performs preliminary organization of convolution arrays into channel-continuous layouts before the actual convolution operation begins. This pre-processing of memory layout ensures that when the computation starts, all necessary data is already positioned for optimal cache access, preventing cache misses during the critical computation phase
3Productivity
If sparse convolutions are implemented, then computation can be optimized, but additional shuffle and reshuffle operations are required which increase complexity
Solution Approach 1:
The patent merges the storage organization of convolution arrays with the computation structure of sparse convolutions. By storing arrays continuously along channel depth, the system naturally supports sparse operation patterns without requiring separate shuffle/reshuffle steps, as the channel-continuous layout aligns perfectly with how sparse convolutions access data
Solution Approach 2:
The patent extracts and eliminates the unnecessary shuffle and reshuffle operations from the sparse convolution pipeline. By recognizing that channel-continuous storage inherently supports sparse computation patterns, the system removes these intermediate transformation steps, reducing operational complexity while maintaining computation efficiency
Data Source
AI summary
A system and method for convolutional layer in convolutional neural networks is provided. The convolution is performed via a transformation that includes relocating input, relocating convolution filters and performing an aggregate matrix multiply.