Chiplet Matrix Multiplication via Striped Systolic Data Routing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for multiplying matrices in large language models face challenges in achieving low latency, particularly when dealing with large matrices that require efficient processing across multiple chiplets.
Innovation Solution
The method involves decomposing matrices into horizontal and vertical stripes, which are then distributed and processed across a plurality of chiplets using systolic arrays, with striped data being passed between chiplets along closed paths to facilitate efficient matrix multiplication.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If matrices are multiplied using a single chiplet, then the processing is simpler, but the latency increases and throughput decreases
Solution Approach 1:
The patent divides large matrices into smaller stripe segments that are distributed across multiple chiplets. Matrix A is divided into horizontal stripes and Matrix B into vertical stripes, with each chiplet processing specific stripe combinations. This segmentation enables parallel processing across multiple chiplets, reducing overall computation latency while managing complexity through structured data distribution.
2Productivity
If matrices are decomposed into stripes and distributed across multiple chiplets, then the processing throughput increases, but the data communication overhead increases
Solution Approach 1:
The patent implements continuous data flow through systolic arrays where input stripes are continuously fed and processed without interruption. The pipelined architecture ensures that while one stripe pair is being multiplied, the next stripe pairs are being prepared and loaded, maintaining continuous useful action across all chiplets. This continuity maximizes throughput while minimizing idle communication cycles.
3Reliability
If vertical stripes are passed between all pairs of chiplets, then complete matrix multiplication is achieved, but the communication paths become complex
Solution Approach 1:
The patent organizes chiplets in a two-dimensional array layout where communication follows structured rows and columns. Vertical stripes are passed along defined paths through the chiplet array, with each chiplet receiving stripes from specific sources and sending to specific destinations. This spatial organization transforms the communication complexity into a manageable two-dimensional routing problem, ensuring all necessary stripe combinations are processed for complete matrix multiplication.
Data Source
AI summary
A plurality of chiplets may be used to multiply two matrices A and B. Matrix A may be decomposed into horizontal stripes and matrix B may be decomposed into vertical stripes. Each of the horizontal stripes may be multiplied by each of the vertical stripes to form the output matrix C. Specifically, horizontal stripes may be stored in a stationary, distributed manner across the chiplets, while the vertical stripes (or sub-vertical stripes) may be passed between respective pairs of the chiplets until each of the vertical stripes (or sub-vertical stripes) of matrix B has been received and processed by each of the chiplets. The vertical stripes may be passed along one or more paths that interconnect the chiplets. Similar techniques can be applied to an arrangement in which the vertical stripes are stationary and the horizontal stripes (or sub-horizontal stripes) are passed between respective pairs of the chiplets.