Dynamic Serialization Buffer for Matrix Multiplication Throughput
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing matrix multiplication units are constrained to process a limited number of vectors per cycle, leading to decreased throughput and slower processing for accelerating matrix multiplication operations.
Innovation Solution
The introduction of a dynamic serialization buffer and a dynamic deserialization buffer between the vector processing unit and the matrix multiplication unit allows for streaming any integer of vectors per cycle, enabling arbitrary throughput for matrix multiplication operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the matrix multiplication unit processes M vectors per cycle with each vector containing N elements, then the throughput is constrained to be a multiple-of-two ratio, but this constraint decreases the throughput of the matrix multiplication unit
Solution Approach 1:
The patent divides the data processing into two separate units: a vector processing unit that handles data in fixed-size vectors and a matrix multiplication unit that processes data in flexible multiples. The vector processing unit segments the input data into vectors of E elements, while the matrix multiplication unit can process any integer M vectors per cycle, with M not necessarily being a factor of E/N. This segmentation allows each unit to operate independently with its own optimal parameters.
Solution Approach 2:
The patent introduces an intermediary buffer between the vector processing unit and the matrix multiplication unit. This buffer acts as a mediator that decouples the fixed output rate of the vector processing unit from the flexible input rate requirements of the matrix multiplication unit. The buffer absorbs timing differences and allows the matrix multiplication unit to process any integer number of vectors per cycle without being constrained by the vector register dimensions.
2Reliability
If the number of cycles is configured to be an integer value to maintain atomicity of the single register value, then the value of M is constrained, but such constraint decreases the throughput
Solution Approach 1:
The patent separates the processing into two independent stages: the vector processing unit maintains atomicity by processing complete vector registers, while the matrix multiplication unit operates independently with its own cycle timing. This segmentation allows the matrix multiplication unit to process any integer M vectors per cycle without compromising the atomicity guarantee of the vector processing unit, as each stage operates with its own integrity boundaries.
Solution Approach 2:
The patent enables continuous processing by allowing the matrix multiplication unit to operate at its optimal throughput rate independently of the vector processing unit's fixed cycle timing. The intermediary buffer ensures that the matrix multiplication unit can continuously process any integer number of vectors per cycle without waiting for alignment with the vector processing unit's cycle boundaries, maintaining continuous useful action while preserving atomicity at the vector register level.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Aspects of the disclosure are directed to an architecture including a dynamic serialization buffer and/or dynamic deserialization buffer coupled between a vector processing unit and a matrix multiplication unit. The dynamic serialization buffer and/or dynamic deserialization buffer allow for streaming any integer of vectors per cycle when performing acceleration of matrix multiplication operations. The matrix multiplication unit receives vectors equivalent to an amount of data from the vector processing unit at an arbitrary rate of vectors per cycle. The matrix multiplication unit processes the vectors to generate resulting vectors that are output at the arbitrary rate.