Convolution Engine With Unidirectional Dataflows For Neural Network Energy Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current hardware architectures for neural network applications, particularly convolutional deep neural networks, face challenges in energy efficiency due to frequent memory access and large buffer sizes, leading to high energy overhead and latency, especially in large-scale deployments.
Innovation Solution
A method and hardware system that accumulates partial results of convolutional neural network layers on an array of processing elements, utilizing unidirectional dataflows to reduce memory access and implement parallel computation of different convolution kernels, thereby enhancing throughput and energy efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If large quantities of input data and weights are frequently accessed from external memory or on-chip buffers, then convolution computations can be performed, but energy consumption increases significantly due to large energy overhead from frequent memory access
Solution Approach 1:
The patent applies preliminary action by pre-loading input data and weight coefficients into on-chip buffers before convolution computations begin. The system divides the input map into blocks and pre-loads them along with corresponding weight coefficients, so that during the computation phase, processing elements can perform multiple convolution operations using the pre-loaded data without frequent external memory accesses. This reduces energy consumption by minimizing memory access frequency during the actual computation phase.
Solution Approach 2:
The patent implements a nested memory hierarchy where small on-chip buffers (SRAM) are nested within each processing element, and these are nested within larger on-chip buffer regions, which are in turn nested within the external memory system. This nested structure allows data to be cached at multiple levels, with the most frequently accessed data residing in the smallest, fastest, and most energy-efficient on-chip buffers, reducing the need for energy-consuming external memory accesses.
2Loss of energy
If on-chip SRAM buffer sizes are increased to reduce memory access frequency, then energy efficiency improves, but chip area occupied by buffers increases significantly
Solution Approach 1:
The patent segments the input map into multiple smaller blocks that can be independently processed. Each processing element or small group of processing elements handles one block at a time, loading only the necessary portion of input data and weight coefficients into on-chip buffers. This segmentation allows the system to use smaller buffer sizes while maintaining energy efficiency, as each buffer only needs to hold data for one block rather than the entire input map, thus reducing the total chip area required for buffers.
Solution Approach 2:
The patent introduces a temporal dimension to the processing by using a pipeline architecture where different blocks of data are processed at different time stages. While one processing element is computing with data from block 1, another is loading block 2, and a third is preparing block 3. This temporal multiplexing allows the system to maintain high throughput with smaller buffer sizes, as buffers are reused across different time stages rather than needing to hold all data simultaneously, thereby reducing chip area.
3Device complexity
If multiple passes are used to carry out full multi-dimensional convolution, then computation can be completed with limited local storage, but latency increases due to frequent retrieval of filter coefficients from global data buffer
Solution Approach 1:
The patent merges the storage of input data blocks and weight coefficients into a unified on-chip buffer structure. Instead of separately storing input maps and filter coefficients in different memory locations, the system combines them in integrated buffer regions that are optimally positioned for the convolution computations. This merging allows processing elements to access both input data and weights efficiently during each computation pass, reducing the need for multiple passes and minimizing latency while maintaining manageable local storage sizes.
4Loss of energy
If row-stationary dataflow is used for energy-efficient mapping, then local data reuse is improved, but each processing element still requires relatively large local storage including I/O FIFO buffers
Solution Approach 1:
The patent makes the on-chip buffers universal by designing them to serve multiple functions: storing input data blocks, storing weight coefficients, holding intermediate computation results, and providing data to multiple processing elements. This multi-functionality allows the same buffer memory to be shared across different processing elements and computation stages, eliminating the need for each processing element to have dedicated large local storage including I/O FIFO buffers. The universal buffers are accessed by multiple processing elements in a coordinated manner, maintaining energy efficiency through local data reuse while reducing per-processing-element storage requirements.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method and hardware system for mapping an input map of a convolutional neural network layer to an output map are disclosed. An array of locally interconnected processing elements arranged on a regular two-dimensional grid is provided. The processing elements are interconnected to support unidirectional dataflows through the array along at least three different spatial directions. Each processing element is adapted to combine values of dataflows along different spatial directions into a new value for at least one of the supported dataflows. For each data entry in the output map, a plurality of products from pairs of weights of a selected convolution kernel and selected data entries in the input map is provided and arranged into a plurality of associated partial sums. The plurality of products is stored in the processing elements of at least one parallelogram set of the array. Products associated with a same partial sum are first accumulated on the array by starting and updating first and second unidirectional dataflows. Partial sums are then accumulated on the array into at least one data entry in the output map by starting an updating third and fourth unidirectional dataflows, wherein the third and fourth flow direction are selected from the third spatial direction and the other one of the first or second spatial direction, not selected for the first and second flow direction.