Compiler Vector-Matrix Combine for Parallel Processing Unit Threads
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current approaches for evaluating matrix-vector multiply operations in parallel processing units, such as GPUs, are inefficient due to assumptions of whole-thread concurrency and divergence, which limits the utilization of specialized circuits like Tensor Cores and leads to suboptimal performance in graphics rendering tasks.
Innovation Solution
A compiler is used to generate executable code that combines vectors into input matrices for parallel processing units, enabling matrix-matrix multiply operations using specialized circuits in a Single Instruction-Multiple Thread (SIMT) manner, allowing for efficient execution of vector-matrix multiply operations across multiple threads.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If whole-thread concurrency and divergence assumptions are used for parallel processing, then thread-level parallelism can be maintained, but utilization of specialized circuits like Tensor Cores is limited and performance is suboptimal
Solution Approach 1:
The patent segments threads into warps of 32 threads that are further divided into 4 warp blocks of 8 threads each. This segmentation enables fine-grained parallelism where each warp block can independently execute matrix-matrix multiply operations on Tensor Cores, resolving the contradiction between maintaining thread-level parallelism and utilizing specialized circuits efficiently.
Solution Approach 2:
The patent introduces a hierarchical dimension to thread organization by creating warp blocks as an intermediate level between individual threads and full warps. This dimensional change allows the system to organize computations in a way that maximizes Tensor Core utilization while preserving the ability to handle thread divergence through the warp block structure.
2Productivity
If vector-matrix multiply operations are executed thread-by-thread independently, then programming simplicity is maintained, but execution efficiency and throughput are reduced
Solution Approach 1:
The patent merges multiple vector-matrix multiply operations into a single matrix-matrix multiply operation by combining vectors from multiple threads into matrices. This merging enables batched execution on Tensor Cores, significantly improving execution efficiency and throughput while the compiler handles the complexity of combination automatically.
Solution Approach 2:
The patent introduces a compiler as an intermediary that automatically combines vector-matrix multiply operations into matrix-matrix multiply operations. This intermediary layer handles the complexity of operation combination and memory management, allowing simple thread-level programming while achieving high execution efficiency through optimized batched operations.
3Productivity
If threads execute operations sequentially, then memory access patterns are simple, but computational throughput is limited
Solution Approach 1:
The patent performs preliminary actions by having the compiler pre-organize threads into warp blocks and pre-combine their vector-matrix multiply operations into matrix-matrix multiply operations before execution. This preliminary organization enables efficient parallel execution on Tensor Cores while the runtime simply needs to execute the pre-prepared operations, reducing the complexity of parallel coordination.
Data Source
AI summary
Disclosed are systems and techniques for efficient vector-matrix multiply operations across parallel processing unit threads. The techniques include receiving first data of a first thread, the first data comprising a first input vector and a first matrix. The techniques further include receiving second data of a second thread, the second data comprising a second input vector and a second matrix. The techniques further include combining the first input vector and the second input vector into an input matrix and generating a result matrix at least by multiplying the input matrix by the first matrix using a matrix-multiply circuit. The techniques further include separating the result matrix into a first result value and a second result value, the first result value corresponding to the first thread and the second result value corresponding to the second thread.