Large Systolic Array Modes for AI Matrix Throughput
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional processors face inefficiencies in handling the complex algorithms and data-intensive workflows of artificial intelligence tasks, failing to meet the demands of AI processing efficiently.
Innovation Solution
An AI-accelerating processor system with a systolic array of computation tiles arranged in a grid for performing matrix multiplication, operating in multiple modes to optimize computations for various dataset sizes, and utilizing collective operations like broadcast, reduce, and gather to enhance performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional processors are used for AI tasks, then general-purpose computation capability is maintained, but computational efficiency and processing speed for AI algorithms deteriorate
Solution Approach 1:
The processor is segmented into distinct functional units: a control processor for managing operations and a systolic array for parallel computational tasks. This segmentation allows the systolic array to specialize in AI computations while the control processor handles orchestration, thereby improving computational efficiency without requiring the entire processor to be redesigned for AI-specific workloads.
Solution Approach 2:
The systolic array is designed with dynamic reconfigurability, allowing it to adapt its operation modes based on the specific AI algorithm being executed. This dynamic capability enables the hardware to optimize its performance for different computational patterns while maintaining a relatively simple base architecture, resolving the contradiction between efficiency and complexity.
2Productivity
If systolic array size is increased to handle large matrix multiplications, then computational throughput is improved, but data movement time and latency worsen
Solution Approach 1:
The patent transitions from conventional two-dimensional systolic arrays to a three-dimensional architecture by stacking multiple computation layers vertically. This dimensional expansion allows data to flow through multiple layers in parallel, significantly increasing computational throughput for large matrix operations while the interconnected structure reduces data movement bottlenecks by providing multiple pathways for data flow.
Solution Approach 2:
Multiple computation layers are merged into a single integrated systolic array structure with shared control and data pathways. This merging enables simultaneous processing across layers, improving throughput while the unified architecture reduces redundant data movement operations that would otherwise increase latency.
3Adaptability or versatility
If systolic array operates in multi-stage accumulation mode for smaller datasets, then resource utilization is improved, but computation time increases compared to single-stage mode
Solution Approach 1:
The systolic array incorporates dynamic operation modes that can be switched based on dataset characteristics. For small datasets, the array operates in multi-stage accumulation mode to maximize resource utilization by processing multiple small matrices through the same hardware pipeline. For large datasets, it switches to single-stage mode to minimize computation time. This dynamic adaptability resolves the contradiction between versatility and computation time.
Data Source
AI summary
An AI-accelerating processor system that includes memory configured to store weights of a machine learning model. The processor system also includes a systolic array that has 1000×1000 or more computation tiles arranged in a grid for performing matrix multiplication involving the weights. In some embodiments, the systolic array may perform first multiplications of data in a first set of columns of the computation tiles. The systolic array may accumulate first multiplication results in the first set of columns. The systolic array may transmit accumulated results from the first set of columns to a second set of columns of the computation tiles. The systolic array may perform second multiplications of the accumulated results in a second set of columns. The systolic array may accumulate second multiplication results in the second set of columns.