Systolic Array Dot-Product Processing for Energy-Efficient Machine Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current deep learning technologies, particularly in computer vision and speech processing, face challenges in achieving power efficiency for convolutional neural networks (CNNs) and long short-term memory (LSTM) models, especially in mobile devices, due to the high energy and area costs associated with convolution (CONV) and GEMM/GEMV operations.
Innovation Solution
The proposed apparatus includes a global memory and a systolic array with processing elements (PEs) that perform dot-product operations on input feature maps and kernel vectors in parallel, optimizing energy efficiency and reducing storage area by exploiting massive parallelism and maximizing data reuse.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Use of energy by moving object
If conventional memory and processing units are used for CNN and LSTM operations, then computational functionality is achieved, but power consumption and area occupancy are excessively high
Solution Approach 1:
The system segments the computational workload into distinct phases: data fetching from global memory, buffering in local memory, and processing in the systolic array. This segmentation allows each component to be optimized for its specific function, with the systolic array dedicated to compute operations and separate buffer memories for data staging, thereby reducing overall power consumption while maintaining computational efficiency
Solution Approach 2:
Buffer memories are introduced as intermediary components between the global memory and the systolic array. These buffers pre-load and stage data locally, reducing the frequency and energy cost of accesses to the higher-power global memory. The buffers act as mediators that enable the compute unit to operate continuously with minimal high-power memory access
2Reliability
If data is frequently accessed from global memory during processing, then data availability is ensured, but power consumption increases due to repeated high-bandwidth access
Solution Approach 1:
The buffer memories perform preliminary actions by pre-fetching and staging data from global memory before the systolic array needs it for processing. This advance data preparation ensures that when the compute unit requires data, it is immediately available in the low-power buffer memory, maintaining data availability while avoiding repeated high-energy global memory access
3Productivity
If a systolic array with multiple processing elements is implemented, then parallel processing capability is enhanced, but area occupancy increases
Solution Approach 1:
Multiple processing elements in the systolic array are merged into a tightly-coupled grid structure where adjacent PEs share common data pathways and control logic. This merging allows the system to achieve high parallel throughput while minimizing the area overhead that would otherwise be required for separate processing units with duplicate infrastructure
4Productivity
If buffer memories are added to stage data for the systolic array, then data reuse is maximized, but device complexity increases
Solution Approach 1:
The buffer memories are designed with local quality optimized for the specific requirements of the systolic array - they provide the exact data dimensions and access patterns needed by the PE grid. Rather than implementing a complex general-purpose memory hierarchy, the buffers are tailored locally to match the compute unit's data consumption patterns, achieving high data reuse with minimal complexity
Data Source
AI summary
An apparatus includes a global memory and a systolic array. The global memory is configured to store and provide an input feature map (IFM) vector stream from an IFM tensor and a kernel vector stream from a kernel tensor. The systolic array is configured to receive the IFM vector stream and the kernel vector stream from the global memory. The systolic array is on-chip together with the global memory. The systolic array includes a plurality of processing elements (PEs) each having a plurality of vector units, each of the plurality of vector units being configured to perform a dot-product operation on at least one IFM vector of the IFM vector stream and at least one kernel vector of the kernel vector stream per unit clock cycle to generate a plurality of output feature maps (OFMs).