Matrix Multiplication Memory Tiling for Multithreaded Cache Bottlenecks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Matrix multiplication in multithreaded processing systems is bottlenecked by data handling limitations, particularly in systems with limited cache storage and bandwidth, which reduces the parallelization efficiency of large matrices.
Innovation Solution
The method involves storing portions of input matrices in dedicated caches and local memories, using workgroups to perform concurrent multiplications, and optimizing matrix subunit dimensions based on cache and memory capacities to minimize data retrieval from external memory.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If more threads are used to parallelize matrix multiplication, then processing speed should increase, but data handling bottlenecks worsen due to limited cache storage and bandwidth
Solution Approach 1:
The patent divides large matrices into smaller matrix subunits that can be stored in dedicated caches and local memories. Each processing unit handles specific subunits, breaking down the large-scale matrix multiplication into manageable segments that fit within available memory resources, thereby reducing data handling bottlenecks while maintaining parallel processing efficiency
Solution Approach 2:
The patent introduces a hierarchical memory architecture with multiple levels (dedicated caches and local memories) between global memory and processing units. This adds a dimensional layer to the memory hierarchy, allowing data to be cached at intermediate levels closer to the processing units, thereby reducing access time and bandwidth constraints without limiting parallelization
2Productivity
If cache storage capacity is increased to store more matrix data, then data reuse efficiency improves, but system complexity and cost increase
Solution Approach 1:
The patent assigns different cache sizes and characteristics to different processing units based on their specific needs. Each processing unit has dedicated cache and local memory configured for its particular matrix subunit processing requirements, allowing optimization of data reuse efficiency without uniformly increasing complexity across the entire system
Solution Approach 2:
The patent pre-loads matrix subunits into dedicated caches and local memories before processing begins. By preparing data in advance at appropriate memory levels, the system maximizes data reuse efficiency during computation without requiring excessively large caches, as data is staged in advance in manageable portions
3Productivity
If matrix dimensions are increased to process larger matrices, then computational capability improves, but data handling bottlenecks worsen due to limited cache and memory bandwidth
Solution Approach 1:
The patent segments large matrices into smaller subunits that can be processed in parallel by multiple processing units. Each subunit fits within the cache and memory capacity of individual processing units, allowing the system to handle large overall matrix dimensions by dividing and conquering the data volume across multiple parallel processing streams
Solution Approach 2:
The patent uses the additional memory hierarchy levels (dedicated caches and local memories) to store intermediate matrix subunits, effectively increasing the usable data volume beyond what a single processing unit could handle alone. This hierarchical storage enables processing of larger matrices by distributing data across multiple memory levels and processing units
Data Source
Figure 1A~1B
Figure 2
Figure 3A~3B
AI summary
The present disclosure relates to a method of performing fast matrix multiplication in a multithreaded processing system. The multithreaded processing system comprises one or more processing units, each processing unit coupled with i) a dedicated cache and ii) a dedicated local memory, and wherein each processing unit is configured to operate a plurality of threads grouped into a plurality of workgroups, the method comprising obtaining address data identifying a first matrix input and a second matrix input; for a first processing unit of the multithreaded processor: storing, in a cache dedicated to the first processing unit, at least a portion of the first matrix input as one or more first matrix subunits; storing, in a local memory dedicated to the first processing unit, at least a portion of the second matrix input as a plurality of second matrix subunits; and generating a plurality of output matrix subunits. The generating comprising: launching, at the first processing unit, a plurality of workgroups; for each launched workgroup of the launched plurality of workgroups: assigning a subset of one or more second matrix subunits to the workgroup; multiplying, by threads of the workgroup, each of the one or more first matrix subunits with a corresponding second matrix subunit of the subset of second matrix subunits to obtain an output matrix subunit. During the generation of the plurality of output matrix subunits, the one or more first matrix subunits are concurrently accessed by each launched workgroup from the cache dedicated to the first processing unit.