GPU Matrix Multiplication Pipelining Across Compute Unit Subsets
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional GPU scheduling methods for matrix multiplication operations require a high number of memory fetch cycles relative to compute cycles, leading to inefficient processor performance.
Innovation Solution
The GPU scheduler divides CUs into subsets and clusters, assigning recurrent matrix multiplication operations to these subsets and clusters, allowing for pipelined execution and concurrent processing, reducing data fetches and enhancing compute efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional GPU scheduling methods are used for matrix multiplication operations, then operations are scheduled at compute units, but the number of memory fetch cycles is high relative to compute cycles, leading to inefficient processor performance
Solution Approach 1:
The patent divides compute units into multiple subsets and further divides matrix multiplication operations into multiple stages. Each subset processes a portion of the computation, allowing overlapping of memory fetches and compute operations across different subsets. This segmentation enables the system to hide memory latency by keeping multiple subsets busy with different stages of computation simultaneously.
Solution Approach 2:
The patent implements a pipelined approach where data is pre-loaded into intermediate storage buffers before being processed by compute units. Memory fetches are initiated in advance and overlapped with compute operations. The scheduler prepares and queues operations ahead of time, ensuring that compute units have data ready when needed, thereby reducing idle time and improving the compute-to-fetch cycle ratio.
2Quantity of substance
If more memory fetch cycles are performed, then more data is available for processing, but the ratio of compute cycles to memory fetch cycles decreases, negatively impacting processor performance
Solution Approach 1:
The patent introduces intermediate storage buffers as mediators between memory and compute units. These buffers hold data that has been fetched from memory, allowing compute units to process data without waiting for additional memory fetches. The buffers act as a staging area that decouples memory access from compute operations, enabling compute units to maintain high utilization while data is gradually supplied from memory through the buffer.
Solution Approach 2:
The patent ensures continuous useful action by maintaining a pipeline of compute operations across multiple subsets. While one subset is fetching data from memory, another subset is processing data from buffers, and a third subset is writing results. This continuous overlapping of fetch, compute, and write operations ensures that compute units are rarely idle, maximizing the compute-to-fetch cycle ratio while maintaining adequate data availability.
Data Source
AI summary
A graphics processing unit (GPU) schedules recurrent matrix multiplication operations at different subsets of CUs of the GPU. The GPU includes a scheduler that receives sets of recurrent matrix multiplication operations, such as multiplication operations associated with a recurrent neural network (RNN). The multiple operations associated with, for example, an RNN layer are fused into a single kernel, which is scheduled by the scheduler such that one work group is assigned per compute unit, thus assigning different ones of the recurrent matrix multiplication operations to different subsets of the CUs of the GPU. In addition, via software synchronization of the different workgroups, the GPU pipelines the assigned matrix multiplication operations so that each subset of CUs provides corresponding multiplication results to a different subset, and so that each subset of CUs executes at least a portion of the multiplication operations concurrently.