Convolution Layer Processor Pad Supervisor Module
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current neural network accelerators face challenges in efficiently processing convolution operations due to high computational complexity and memory requirements, particularly with the need for additional memory to store padding pixels and sequential traversal of input feature maps.
Innovation Solution
A convolution layer processor with a memory access module and multiple convolution modules that perform parallel operations, utilizing a pad supervisor module to manage padding and reduce memory access for padding pixels, allowing for efficient convolution operations with arbitrary padding, stride, and dilation without storing physical padding in memory.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If additional memory is allocated to store padding pixels for convolution operations, then the convolution operation can be performed correctly with proper padding, but the memory requirements and device complexity increase
Solution Approach 1:
The patent introduces a pad supervisor module as an intermediary component that manages padding operations. This module receives pixel data from the memory access module, determines whether padding is needed based on position information, and generates appropriate padding pixels on-the-fly without requiring physical storage of padding data in memory. The pad supervisor acts as a mediator between the memory system and convolution modules, resolving the contradiction by providing correct padding functionality while avoiding additional memory allocation.
Solution Approach 2:
The pad supervisor module performs preliminary actions by pre-calculating and generating padding pixels before they are needed for convolution operations. By determining padding requirements in advance based on position information and generating padding pixels proactively, the system ensures correct convolution operations without needing to allocate additional memory space for storing padding data, thus resolving the memory requirement issue.
2Ease of operation
If padding pixels are stored physically in memory, then memory access for convolution is simplified, but the memory bandwidth and memory capacity requirements increase
Solution Approach 1:
The pad supervisor module implements self-service by autonomously generating padding pixels based on position information received from the memory access module. Instead of requiring the memory system to provide padding data, the pad supervisor serves itself by creating padding pixels on-demand using simple logic operations. This eliminates the need for additional memory bandwidth to transfer padding data, while still providing the ease of operation through automated padding management.
Solution Approach 2:
The pad supervisor module creates copies of existing pixel values to generate padding pixels, rather than retrieving padding data from memory. By copying and reusing valid pixel data (such as replicating border pixel values for external padding), the system simplifies the padding operation while avoiding additional memory bandwidth consumption, as the copying operation is performed in-place by the pad supervisor.
3Device complexity
If sequential traversal is used to access input feature map pixels, then memory access is straightforward, but the processing speed and productivity decrease
Solution Approach 1:
The patent segments the pixel access operation by dividing the input feature map traversal into multiple independent parallel streams. Each convolution module receives a specific segment of pixels through dedicated data paths from the memory access module. This segmentation allows simultaneous access to different regions of the input feature map, maintaining straightforward memory access patterns for each segment while achieving parallel processing that significantly improves convolution processing speed.
Solution Approach 2:
The system transitions from sequential one-dimensional traversal to parallel multi-dimensional access by organizing pixel flow into multiple simultaneous streams corresponding to different convolution modules. The memory access module operates in another dimension by providing parallel data paths that feed multiple convolution modules concurrently, thereby maintaining access simplicity while dramatically improving productivity through parallelism.
4Productivity
If multiple convolution modules operate in parallel, then processing speed improves, but the complexity of managing padding and data distribution increases
Solution Approach 1:
The pad supervisor module implements a universal solution that serves all convolution modules through a single centralized interface. Rather than implementing separate padding logic in each convolution module, the pad supervisor provides multi-functional padding management that adapts to the needs of different convolution modules based on their position and configuration. This universal approach improves processing speed through parallelism while avoiding the complexity multiplication that would result from distributed padding management.
Solution Approach 2:
The pad supervisor acts as an intermediary layer between the memory access module and multiple convolution modules, centralizing the complexity of padding management. It receives position information from the memory access module, determines padding requirements for each convolution module's data stream, and generates appropriate padding pixels for each module. This intermediary approach enables parallel operation of multiple convolution modules while containing the complexity of padding management in a single dedicated component.
Data Source
AI summary
A convolution layer processor for a neural network accelerator includes a memory access module to access elements of an input feature map having a first array of pixels and a plurality of convolution modules. Each convolution module receives an element of the input feature map and performs a convolution operation on the received element of the input feature map with a convolution kernel having a second array of pixels to provide a corresponding element of an output feature map. The memory access module includes a DMA requester to request elements of the input feature map, a data buffer to provide the requested elements to each of the plurality of convolution modules, and a pad supervisor module to provide to the data buffer, for each element requested by the DMA requester, padding pixels of the input feature map when the requested element extends beyond a boundary of the input feature map.