3D Feature Map Segmentation With Line Buffers for FPGA CNNs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Convolutional neural networks (CNNs) face challenges in efficiently processing three-dimensional input feature maps due to memory limitations in Field Programmable Gate Arrays (FPGAs), leading to latency and bandwidth imbalances, especially when the entire 3-D IFM does not fit in the available local storage, making it difficult to perform convolution operations efficiently.
Innovation Solution
A circuit arrangement that divides a three-dimensional input feature map into multiple sub-volumes based on the available local storage dimensions, using a request generator circuit and line buffers to manage data access, allowing for scalable and adaptable processing across various CNN layers, regardless of the size of the IFM.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the entire 3-D IFM is stored in local memory for processing, then convolution operations can be performed efficiently, but this is not feasible when the IFM size exceeds available local storage capacity
Solution Approach 1:
The patent divides the large 3-D input feature map into multiple smaller sub-volumes that can be stored in local memory simultaneously. The request generator circuit segments the IFM based on available local storage capacity, allowing processing to proceed in manageable chunks while maintaining efficient convolution operations on each segment.
2Productivity
If static expansion of the IFM volume is performed in FPGA accelerators, then matrix multiplication can be executed, but latency and bandwidth limitations prevent high efficiency operation
Solution Approach 1:
The request generator circuit preliminarily organizes and segments the input feature map into appropriate sub-volumes before processing begins. By pre-dividing the IFM into manageable sections that fit local storage, the system avoids latency associated with dynamic memory management during processing, enabling continuous efficient operation.
3Adaptability or versatility
If the IFM is divided into sub-volumes to fit local storage, then processing can proceed, but the system must manage complex data access patterns across multiple memory regions
Solution Approach 1:
The request generator circuit automatically determines the optimal segmentation of the input feature map based on available local storage capacity. The system self-adjusts the sub-volume dimensions and quantities without external intervention, simplifying the interface while managing the complexity of data access patterns internally through automated memory address generation and sub-volume assignment.
Data Source
AI summary
Disclosed circuits and methods include N line buffers. Each line buffer is configured for storage of M data elements of a three-dimensional (3-D) input feature map (IFM). A request generator circuit is coupled to the N line buffers and to a memory configured for storage of the 3-D IFM. The request generator circuit is divides the 3-D IFM into a plurality of IFM sub-volumes based on values of N, M, and dimensions of the 3-D IFM. The request generator circuit reads from the memory, data elements at addresses of an unprocessed one of the IFM sub-volumes and stores the data elements of the unprocessed one of the IFM sub-volumes in the N line buffers. In response to a completion signal, the request generator circuit repeats the reading of an unprocessed one of the IFM sub-volumes and storing the data elements in the N line buffers.