Systolic Matrix Multiply Unit With Parallel Weight Loading
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network computation hardware faces high latency due to slow weight loading into matrix multiply units, which hinders efficient processing.
Innovation Solution
Implementing a matrix multiply unit with a systolic array architecture that includes two chains of weight shift registers per column and separate registers for vertical and horizontal weight shift chains, allowing for parallel loading of weights from both directions, thereby increasing the weight loading rate.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If a traditional systolic array with uniform structure is used, then the device complexity is low, but the weight loading rate is slow causing high latency
Solution Approach 1:
The systolic array is segmented into multiple columns, each containing independent weight shift registers. This segmentation allows parallel weight loading paths while maintaining modular complexity. Each column can load weights independently, achieving higher aggregate loading rates without proportionally increasing overall system complexity.
Solution Approach 2:
The patent introduces a temporal dimension to weight loading by implementing multi-cycle weight shift registers that can hold and progressively shift weights into matrix registers over multiple clock cycles. This transforms the loading process from a single-dimensional operation to a multi-dimensional process involving time, enabling higher effective loading rates without immediate increases in hardware complexity.
2Loss of time
If weights are loaded at a higher rate into matrix multiply units, then the latency is reduced, but the device complexity increases
Solution Approach 1:
Weights are pre-loaded into weight shift registers before being transferred to matrix registers for computation. This preliminary action allows weights to be staged and ready for rapid sequential loading during computation, reducing the critical path latency without requiring complex real-time loading mechanisms during the actual matrix multiplication operation.
Solution Approach 2:
Weight shift registers serve as intermediary structures between the external weight source and the matrix registers. These intermediaries buffer and manage the weight data flow, enabling higher loading rates by decoupling the weight source from the computation units and allowing controlled, parallel loading without directly increasing matrix multiply unit complexity.
3Adaptability or versatility
If separate transposed and non-transposed weight shift registers are implemented, then the adaptability is improved, but the device complexity increases
Solution Approach 1:
Each column's weight shift register structure is designed to handle both transposed and non-transposed weight configurations through the same hardware path. The shift registers can operate in different modes depending on the computation requirements, providing universal functionality that reduces the need for completely separate dedicated structures for each weight type, thereby limiting the increase in device complexity.
Data Source
AI summary
Methods, systems, and apparatus for a matrix multiply unit implemented as a systolic array of cells are disclosed. The matrix multiply unit may include cells arranged in columns of the systolic array. Two chains of weight shift registers per column of the systolic array are in the matrix multiply unit. Each weight shift register is connected to only one chain and each cell is connected to only one weight shift register. A weight matrix register per cell is configured to store a weight input received from a weight shift register. A multiply unit is coupled to the weight matrix register and configured to multiply the weight input of the weight matrix register with a vector data input in order to obtain a multiplication result.