Tensor Calculation Unit Cascading for RAM-Bandwidth-Limited Matrix Operations
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing GPU architectures face challenges in efficiently performing large-scale parallel matrix calculations due to limitations in RAM bandwidth and latency issues when cascading multiply-and-accumulate operators, which hinder the acceleration of matrix operations.
Innovation Solution
A tensor calculation unit is introduced with cascaded multiply-and-accumulate operators that allow for efficient data flow by directly sending intermediate results between operators, doubling calculation capability without increasing RAM bandwidth requirements, and an interleaved operation method is used to manage latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If multiply-and-accumulate operators are cascaded to double calculation capability, then productivity is improved, but latency increases due to sequential dependency
Solution Approach 1:
The patent segments the calculation process by dividing matrix operations into multiple independent tensor calculation units, each handling specific portions of the computation. This allows parallel execution of segmented tasks while maintaining the benefits of cascaded operators within each unit, thereby reducing overall latency without sacrificing productivity.
Solution Approach 2:
The patent implements preliminary action by pre-loading data into local storage and preparing computation pipelines before actual calculation begins. This allows the cascaded multiply-and-accumulate operators to operate on pre-prepared data, reducing waiting time and latency while maintaining doubled calculation capability.
2Productivity
If more data is processed through RAM to support parallel calculations, then productivity is improved, but RAM bandwidth requirements increase
Solution Approach 1:
The patent transitions from a single-dimension RAM-based data flow to a multi-dimensional architecture by introducing local storage layers and hierarchical memory structures. This allows data to be cached and reused locally, reducing the quantity of data that must traverse RAM bandwidth while maintaining high parallel calculation capability through efficient data locality.
3Manufacturing precision
If cascaded operators are used to enhance matrix operations, then manufacturing precision is improved, but device complexity increases
Solution Approach 1:
The patent uses copying by replicating the cascaded multiply-and-accumulate operator structure across multiple tensor calculation units. Rather than creating a single complex cascaded structure, identical copies are distributed in parallel, maintaining manufacturing precision through consistent operation while reducing individual unit complexity and enabling easier fabrication and maintenance.
Data Source
AI summary
A tensor calculation unit, a use method of the tensor calculation unit, a data processing apparatus and an operation method of the data processing apparatus are provided. The tensor calculation unit includes the first multiply-and-accumulate operator and the second multiply-and-accumulate operator that are cascaded, wherein the first multiply-and-accumulate operator includes the first input port, the second input port, the third input port and the first output port, the first multiply-and-accumulate operator is configured to compute D0=A0×B0+C, and a calculation result DO is output at the first output port; and the second multiply-and-accumulate operator includes the fourth input port, the fifth input port, the sixth input port and the second output port, the sixth input port is coupled to the first output port to receive the calculation result DO, the second multiply-and-accumulate operator is configurated to compute D1=A1×B1+D0, and a calculation result D1 is output at the second output port.