Matrix Multiplication Layout for Cache-Limited Parallel Threads
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Matrix multiplication in multi-threaded processing systems is bottlenecked by data handling limitations, particularly in systems with limited cache storage and finite bandwidth, which reduces the parallelization efficiency of large matrices.
Innovation Solution
A method involving data transformation to redistribute adjacent linear arrays into a single linear array, utilizing multi-value reading and processing registers with mismatched capacities, and organizing threads into workgroups to perform concurrent multiplication operations, with data stored in dedicated caches and local memories to optimize data reuse and minimize external memory access.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If more threads are used to parallelize matrix multiplication, then processing speed should increase, but data handling bottlenecks worsen due to limited cache storage and finite bandwidth
Solution Approach 1:
The patent divides the second input matrix into multiple submatrices and distributes them across different memory locations and processing units. This segmentation allows parallel processing threads to access smaller, manageable data portions simultaneously, reducing cache conflicts and memory bandwidth bottlenecks while maintaining high processing speed.
Solution Approach 2:
The patent transforms the traditional row-major or column-major matrix storage layout into a swizzled layout where adjacent linear arrays are redistributed. This dimensional reorganization enables better spatial locality and allows processing threads to access required data elements more efficiently, reducing memory access latency and improving parallel processing throughput.
2Reliability
If cache storage is increased to hold more matrix data, then data reuse improves, but memory bandwidth requirements increase
Solution Approach 1:
The patent performs preliminary data transformation and reorganization of the second input matrix into a swizzled layout before processing. This preliminary action ensures that data is pre-positioned in optimal locations for parallel access, maximizing cache utilization and data reuse efficiency without requiring excessive memory bandwidth during the actual computation phase.
3Ease of manufacture
If straightforward SIMD parallel processing is used, then implementation simplicity is maintained, but data handling bottlenecks reduce parallelization efficiency
Solution Approach 1:
The patent introduces an intermediate data transformation step that converts the second input matrix into a swizzled layout. This intermediary transformation acts as a mediator between the straightforward SIMD processing approach and the need for efficient parallel data access, enabling simple parallel processing to achieve high efficiency by presenting data in an optimized format without complicating the core computation logic.
Data Source
Figure 1A~1B
Figure 2
Figure 3A~3B
AI summary
The present disclosure relates to a method of performing matrix multiplication on a processing unit. The processing unit comprises processing threads, each thread being associated with a multi-value reading register and a multi-value processing register, wherein a capacity of the multi-value processing register is smaller than a capacity of the multi-value reading register. The method comprising: obtaining address data identifying a first matrix input and a second matrix input; transforming at least a portion of the second matrix input to obtain a transformed second matrix input, comprising redistributing a plurality of adjacent linear arrays of the second matrix input to form a single linear array in the transformed second matrix input; obtaining a first matrix subunit from the first matrix input and a transformed second matrix subunit from the transformed second matrix input; and obtaining, using a multi-value reading register, a linear array, R, from the single linear array of the transformed second matrix subunit. The method further comprises, at a thread of the processing unit: obtaining a plurality of values from the first matrix subunit; and for each value in the linear array R, multiplying the value with a corresponding value from the plurality of values of the first matrix subunit and accumulating the results of the multiplying in a linear array P, stored in a multi-value processing register associated with the thread, wherein the accumulated results form at least a portion of an output matrix.