Cascaded Tensor Multiply-Accumulate Units for Bandwidth-Limited AI Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing GPU architectures face challenges in efficiently performing large-scale parallel matrix calculations due to limitations in the number of multiply-and-accumulate operations and bandwidth constraints, leading to latency issues and reduced computational efficiency.
Innovation Solution
A tensor calculation unit is introduced, featuring cascaded multiply-and-accumulate operators that allow for increased parallel processing by doubling calculation capability without increasing RAM bandwidth, achieved through a cascading method that combines multiple operators to perform operations in different clock cycles.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If general-purpose processors are used for AI inference, then versatility is maintained, but processing speed and power efficiency deteriorate
Solution Approach 1:
The processor is segmented into a general-purpose CPU and specialized AI inference units (tensor calculation units), allowing each component to handle specific tasks optimally while maintaining overall system versatility
Solution Approach 2:
A memory management unit acts as an intermediary between the CPU and tensor calculation units, coordinating data flow and enabling both general-purpose and specialized operations to function efficiently together
2Adaptability or versatility
If general-purpose processors are used for AI inference, then adaptability is maintained, but power consumption increases
Solution Approach 1:
The processing system is divided into energy-efficient specialized tensor calculation units for AI workloads and a CPU for other tasks, reducing overall power consumption while maintaining adaptability across different computational needs
Solution Approach 2:
The tensor calculation units utilize low-precision arithmetic (e.g., 8-bit integers) instead of standard floating-point operations, significantly reducing power consumption while maintaining adequate precision for neural network inference
3Device complexity
If conventional memory systems are used with tensor calculation units, then system complexity is reduced, but data transfer bottlenecks increase
Solution Approach 1:
The memory system uses asymmetric memory hierarchies with different memory types (e.g., SRAM for frequently accessed weights, DRAM for less frequently accessed data) optimized for their specific access patterns, improving data transfer efficiency without excessive complexity
Solution Approach 2:
Data is pre-loaded and cached in faster memory layers before being needed by the tensor calculation units, reducing wait times and improving overall data transfer efficiency
Data Source
Figure 1
Figure 2
Figure 3~4
AI summary
A tensor calculation unit and a use method, and a data processing apparatus and an operation method. The tensor calculation unit comprises a first multiply-add operator and a second multiply-add operator, which are cascaded, wherein the first multiply-add operator comprises a first input port, a second input port, a third input port and a first output port, the first input port, the second input port and the third input port are used for respectively receiving parameters A0, B0 and C, and the first multiply-add operator is configured to perform the calculation of D0 = A0 × B0 +C, and output a calculation result D0 at the first output port; and the second multiply-add operator comprises a fourth input port, a fifth input port, a sixth input port and a second output port, the fourth input port and the fifth input port are used for respectively receiving parameters A1 and B1, the sixth input port is coupled to the first output port so as to receive the calculation result D0, and the second multiply-add operator is configured to perform the calculation of D1 = A1 × B1 + D0, and output a calculation result D1 at the second output port. The tensor calculation unit can be infinitely extended and stacked.