Matrix Subunit Caching for Fast Multithreaded Multiplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Matrix multiplication in multithreaded processing systems is bottlenecked by data handling and memory resource limitations, particularly for large matrices, which reduces the efficiency of parallel processing.
Innovation Solution
A method and system for matrix multiplication in a multithreaded processing system, where each processing unit is coupled with a dedicated cache and local memory, storing matrix inputs as subunits and launching workgroups to concurrently access and multiply subunits, optimizing data reuse and reducing memory access bottlenecks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If more threads are used for parallel processing of matrix multiplication, then processing speed should increase, but memory access bottlenecks worsen due to limited cache and memory bandwidth
Solution Approach 1:
The patent divides large matrices into smaller submatrices and distributes them across multiple processing units. Each processing unit handles a specific portion of the computation with its own local cache, reducing the burden on shared memory and allowing parallel processing without overwhelming the memory subsystem.
Solution Approach 2:
The patent introduces a hierarchical memory architecture with multiple cache levels (L1, L2, L3) and separate instruction/data caches for each processing unit. This multi-dimensional memory organization allows simultaneous access from multiple threads without creating bottlenecks, as data can be cached at different levels and accessed in parallel.
2Productivity
If cache size is increased to store more matrix data, then data reuse efficiency improves, but device complexity and cost increase
Solution Approach 1:
Instead of one large cache, the patent divides cache memory into multiple smaller caches distributed across different processing units. Each processing unit has its own L1 cache, with L2 and L3 caches serving as intermediate and shared storage. This segmentation allows efficient use of smaller cache sizes while maintaining high data reuse through local caching.
Solution Approach 2:
The patent implements separate instruction caches and data caches for each processing unit, with different cache sizes and hierarchies optimized for specific access patterns. This local optimization allows each processing unit to have cache characteristics tailored to its computational needs without requiring all units to have large, complex caches.
3Productivity
If more processing units are added to the system, then parallel processing capability increases, but memory bandwidth requirements and bottlenecks worsen
Solution Approach 1:
The patent divides the memory system into separate caches associated with each processing unit, allowing parallel access without contending for the same memory bandwidth. Each processing unit can access its local cache independently, and only when data is not in local cache does it need to access higher-level caches or external memory, reducing overall bandwidth requirements.
Solution Approach 2:
The patent introduces intermediate cache levels (L2, L3) that act as mediators between individual processing unit caches and external memory. These intermediate caches buffer data transfers and allow multiple processing units to access data without directly competing for external memory bandwidth, reducing the overall bandwidth burden.
Data Source
AI summary
Fast matrix multiplication in a multithreaded processing system having one or more processing units, each processing unit operates a plurality of threads grouped into a plurality of workgroups. At least a portion of a first matrix input is stored in a cache dedicated to a first processing unit as a first matrix subunit. At least a portion of a second matrix input is stored in a local memory dedicated to the first processing unit as a second matrix subunit. A plurality of output matrix subunits is generated by launching a plurality of workgroups. A subset of one or more second matrix subunits is assigned to a workgroup. Each of the first matrix subunits is multiplied with a corresponding second matrix subunit to obtain an output matrix subunit. During the generation of the plurality of output matrix subunits, the first matrix subunits are concurrently accessed by each launched workgroup from a cache dedicated to the first processing unit.