Neural Network Weight Prefetching in Systolic Arrays
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems for computing neural network inferences, particularly convolutional neural networks, are inefficient due to the need for numerous matrix multiplications in large dimensional spaces, which limits parallelization and increases computational intensity.
Innovation Solution
A special-purpose hardware circuit with a systolic array and associated units for efficiently processing neural network layers, including weight fetcher and sequencer units, allows for parallel processing of weight and activation inputs, enabling efficient computation of neural network inferences without the need for external memory coupling.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If software-based convolution calculations are used with brute force matrix multiplications, then computational accuracy is maintained, but processing speed and productivity deteriorate due to numerous sequential operations and limited parallelization
Solution Approach 1:
The patent segments the convolution calculation into multiple independent processing elements (PEs) arranged in a grid, where each PE handles a specific portion of the computation. This segmentation enables parallel processing of different regions of the input data simultaneously, dramatically improving processing speed while distributing computational intensity across multiple units rather than concentrating it in a single processor.
Solution Approach 2:
The patent introduces spatial dimensions by arranging processing elements in a two-dimensional grid structure, transforming the traditionally sequential one-dimensional processing approach. This dimensional change allows data and weights to flow through the grid in multiple directions, enabling simultaneous computations across multiple PEs and significantly increasing parallelization capability.
2Productivity
If external memory coupling is used to store weight inputs, then device complexity is reduced, but processing speed deteriorates due to memory access bottlenecks
Solution Approach 1:
The patent implements nested memory structures where weight buffers are embedded within processing elements, and intermediate results are stored in local memory associated with each PE. This nested arrangement allows frequently accessed weights to be stored close to the computation units that need them, reducing memory access latency and eliminating bottlenecks associated with external memory coupling.
Solution Approach 2:
The patent pre-loads weight values into buffers within the processing elements before the actual convolution computation begins. This preliminary action ensures that all necessary weights are already available in fast memory locations when computation starts, eliminating the need for repeated memory accesses during the computation phase and thereby improving processing speed.
3Productivity
If weight inputs are shifted synchronously across processing elements, then processing efficiency is improved, but device complexity increases due to coordination requirements
Solution Approach 1:
The patent implements feedback mechanisms where control signals are propagated through the processing element grid to coordinate weight shifting operations. Each PE receives control signals from neighboring PEs or from a central controller, allowing synchronized weight movement across the entire grid. This feedback-based coordination ensures that all PEs advance their computation in unison, maintaining computational efficiency while managing the complexity of synchronization through distributed control.
Data Source
AI summary
A circuit for performing neural network computations for a neural network, the circuit comprising: a systolic array comprising a plurality of cells; a weight fetcher unit configured to, for each of the plurality of neural network layers: send, for the neural network layer, a plurality of weight inputs to cells along a first dimension of the systolic array; and a plurality of weight sequencer units, each weight sequencer unit coupled to a distinct cell along the first dimension of the systolic array, the plurality of weight sequencer units configured to, for each of the plurality of neural network layers: shift, for the neural network layer, the plurality of weight inputs to cells along the second dimension of the systolic array over a plurality of clock cycles and where each cell is configured to compute a product of an activation input and a respective weight input using multiplication circuitry.