Fused Kernels for GPU Machine Learning Workloads
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning (ML) workloads on Graphics Processing Units (GPUs) face inefficiencies due to inflexible customization and high processing costs, particularly in handling memory-bound computations like matrix-vector multiplication, which are memory-bound and suffer from uncoalesced memory accesses and synchronization overhead.
Innovation Solution
The implementation of a pipelined approach using fused kernels that exploit temporal locality and hierarchical aggregation across the GPU memory hierarchy, optimizing thread occupancy and minimizing atomic writes, along with specialized processing techniques for both sparse and dense matrices to reduce data loads and synchronization overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If multiple operator-level kernels are launched for ML computations, then flexibility and customization are improved, but processing efficiency and productivity deteriorate due to synchronization overhead and uncoalesced memory accesses
Solution Approach 1:
The patent combines multiple operator-level kernels into a single fused kernel that performs multiple ML operations (matrix-vector multiplication, activation functions, normalization) in one GPU kernel invocation. This eliminates the synchronization overhead between separate kernels and ensures coalesced memory accesses throughout the entire computation pipeline, thereby improving processing efficiency while maintaining flexibility through configurable operation sequences.
2Productivity
If hand-crafted libraries of GPU kernels are created for specific ML algorithms, then processing efficiency is improved, but adaptability and customization deteriorate
Solution Approach 1:
The patent creates a universal fused kernel that can perform multiple different ML operations including matrix-vector multiplication, various activation functions (ReLU, sigmoid, tanh), batch normalization, and layer normalization. The kernel accepts configurable parameters to select different operations and their combinations, providing both high processing efficiency through unified execution and broad adaptability through parameterized functionality.
3Adaptability or versatility
If primitive operations are stitched together for ML processes, then adaptability is improved, but processing efficiency deteriorates due to memory access patterns and synchronization overhead
Solution Approach 1:
The patent merges multiple primitive operations (matrix-vector multiplication, activation functions, normalization) into a single fused kernel execution. This unified approach ensures that all memory accesses within the kernel use coalesced access patterns, eliminating the uncoalesced memory accesses that occur when primitive operations are executed as separate kernels. The fused kernel maintains adaptability by allowing configuration of different operation sequences and parameters.
Data Source
AI summary
A method for optimization of machine learning (ML) workloads on a graphics processor unit (GPU). The method includes identifying a computation having a generic pattern commonly observed in ML processes. Hierarchical aggregation spanning a memory hierarchy of the GPU for processing is performed for the identified computation including maintaining partial output vector results in shared memory of the GPU. Hierarchical aggregation for vectors is performed including performing intra-block aggregation for multiple thread blocks of a partial output vector results on GPU global memory.