Weight-Stationary Matrix Multiply Accelerator With Shared L2 Cache
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional processors struggle to efficiently sustain the data bandwidth required for running large machine learning models, particularly in processing the number of matrix multiplications needed for neural networks, falling short of the computational demands.
Innovation Solution
A weight-stationary systolic array accelerator is tightly coupled with a shared L2 cache, enabling high computational throughput for matrix multiplications by maintaining locality of data within the model, allowing efficient processing of machine learning workloads.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional processors are used for matrix multiplications, then general-purpose computing is maintained, but computational throughput and data bandwidth efficiency deteriorate
Solution Approach 1:
The system is divided into distinct segments: a general-purpose processor core for control and a specialized accelerator for matrix multiplications. The accelerator is further segmented into multiple systolic array tiles that can process different matrix operations simultaneously. This segmentation allows each component to be optimized for its specific function, achieving high computational throughput while maintaining overall system manageability.
Solution Approach 2:
A tightly coupled L2 cache is introduced as an intermediary between the processor core and the accelerator. This cache serves as a buffer that stores weight matrices and activation matrices, reducing the computational burden on the processor core while ensuring rapid data access for the accelerator. The cache mediates data transfer, allowing the accelerator to operate at high speeds without overwhelming the processor core.
2Productivity
If data is frequently transferred between processor and memory, then computational flexibility is maintained, but data bandwidth efficiency deteriorates
Solution Approach 1:
Weight matrices are pre-loaded into the tightly coupled L2 cache before inference operations begin. This preliminary action ensures that the accelerator has immediate access to the weight data during matrix multiplications, eliminating the need for frequent data transfers between main memory and the processor during computation. The activation matrices are also managed in the cache to minimize transfer delays.
Solution Approach 2:
The L2 cache is tightly coupled with both the processor core and the accelerator, merging these components into a unified data sharing structure. This merged architecture allows both the processor and accelerator to access the same cached data simultaneously, reducing redundant data transfers and improving overall data bandwidth efficiency. The weight matrices reside in this shared cache, serving both components efficiently.
3Power
If more compute units are added to increase processing capacity, then computational power improves, but energy consumption increases
Solution Approach 1:
The systolic array tiles in the accelerator are designed to perform matrix multiplications using local data from the tightly coupled L2 cache, minimizing the need for additional data movement operations. Each tile processes data locally, reducing the energy associated with data transfer. The architecture allows the compute units to serve themselves by maintaining weight matrices in the cache, reducing the overall energy footprint while maintaining high computational power.
Data Source
AI summary
An accelerator is accessed. The accelerator includes a weight-stationary systolic array of one or more multiply-accumulate units. The accelerator is coupled to a memory hierarchy and a processor core. The processor core sends a work request to the accelerator. The work request is based on execution of a machine learning model and an activation matrix. In response to the work request, the accelerator loads a weight matrix and the activation matrix. The loading uses the memory hierarchy. The accelerator multiplies the weight matrix by the activation matrix. The multiplication results in an answer matrix. The accelerator stores the answer matrix in the memory hierarchy. The processor core obtains the answer matrix that was stored. The machine learning model is trained. The training produces the weight matrix, which is transposed and saved to the memory hierarchy.