Systolic Matrix Multiply Unit with Dual Weight Loading Paths
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network computation systems face high latency due to slow weight loading rates into matrix multiply units, which hampers efficient processing.
Innovation Solution
Implementing a matrix multiply unit with a systolic array architecture that includes two chains of weight shift registers per column, separate registers for vertical and horizontal weight shift chains, and a multiplexer to select weight inputs, allowing for increased weight loading rates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If a conventional systolic array with a single weight shift register chain is used, then the device complexity is low, but the weight loading rate is slow resulting in high latency
Solution Approach 1:
The weight shift register chain is segmented into multiple parallel chains (first and second weight shift register chains), each capable of independently loading weights into different sets of weight registers. This segmentation enables concurrent weight loading operations, increasing the overall weight loading rate without proportionally increasing device complexity
Solution Approach 2:
The patent introduces a temporal dimension to weight loading by enabling simultaneous loading operations across multiple chains and cycles. Weights can be loaded in parallel across different chains and staggered across clock cycles, effectively utilizing time as an additional dimension to increase throughput without linearly increasing hardware resources
2Loss of time
If the weight loading rate is increased using multiple shift register chains, then the latency is reduced, but the device complexity increases
Solution Approach 1:
The multiple weight shift register chains are designed to be universal and interchangeable, with each chain capable of loading weights into different weight registers depending on the operational mode. The select logic enables dynamic configuration, allowing the same hardware structure to serve multiple functions and reducing overall system complexity
Solution Approach 2:
The patent implements dynamic control through select logic that can configure the weight loading path based on operational requirements. The system can dynamically switch between different loading modes (e.g., loading into first weight registers vs. second weight registers, or transposed vs. non-transposed modes), optimizing performance without requiring dedicated hardware for each mode
3Adaptability or versatility
If separate transposed and non-transposed weight shift registers are implemented, then the adaptability is improved, but the device complexity increases
Solution Approach 1:
The weight shift register chains are designed to handle both transposed and non-transposed weight configurations through a single universal path. The select logic dynamically routes weights to appropriate registers based on whether transposed or non-transposed loading is required, eliminating the need for separate dedicated chains for each mode and reducing overall complexity
Data Source
AI summary
Methods, systems, and apparatus for a matrix multiply unit implemented as a systolic array of cells are disclosed. Each cell of the matrix multiply includes: a weight matrix register configured to receive a weight input from either a transposed or a non-transposed weight shift register; a transposed weight shift register configured to receive a weight input from a horizontal direction to be stored in the weight matrix register; a non-transposed weight shift register configured to receive a weight input from a vertical direction to be stored in the weight matrix register; and a multiply unit that is coupled to the weight matrix register and configured to multiply the weight input of the weight matrix register with a vector data input in order to obtain a multiplication result.