Fused GPU Neural-Network Layers to Cut Global Memory Bottlenecks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training and inference processes for artificial neural networks on GPUs are computationally expensive due to the sequential layer-based approach, which limits GPU optimizations and results in significant performance bottlenecks from global memory transactions.
Innovation Solution
A fused neural network layer architecture that combines multiple neural network operations into a single GPU kernel, utilizing shared memory to reduce global memory transactions by performing operations on tiles within the shared memory, thereby eliminating the need for separate GPU kernel invocations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If multiple neural network layers are executed as separate GPU kernels with intermediate results stored in global memory, then each layer can be implemented independently, but global memory transactions become a significant performance bottleneck
Solution Approach 1:
The patent combines multiple neural network layers into a single fused GPU kernel, eliminating the need to store intermediate results in global memory between layers. This merging of previously separate layer implementations removes the performance bottleneck caused by repeated global memory transactions while maintaining the functional sequence of operations across multiple layers.
2Reliability
If intermediate tensor results are stored in GPU global memory between layers, then data can be persisted for subsequent layer access, but global memory is the slowest memory in GPU and creates performance bottlenecks
Solution Approach 1:
The patent introduces shared memory as an intermediary storage mechanism between global memory and processor registers. Intermediate tensor results are stored in shared memory, which is significantly faster than global memory, allowing subsequent layers to access this intermediate data without the performance penalty of global memory transactions while maintaining data persistence across layer operations.
3Productivity
If a single GPU kernel executes multiple fused neural network layer operations, then global memory transactions are reduced, but the complexity of the kernel function increases
Solution Approach 1:
The patent segments the fused kernel execution into distinct operational phases corresponding to different neural network layers, with each phase processing specific input tensors and producing intermediate or final outputs. This segmentation allows the complex fused kernel to be organized into manageable, logically-separated operations that can be implemented systematically while maintaining the performance benefits of fusion.
Data Source
AI summary
The present disclosure provides a system and method for executing fused neural network layers using a graphics processing unit (GPU). The fused neural network layer combines multiple neural network operations into a single GPU kernel function, for efficient utilization of a GPU shared memory to reduce global memory transactions. The fused neural-network-layer system configures GPU thread blocks to iterate tiles in the GPU shared memory across portions of input tensors, and to perform a sequence of neural network layer operations using the tiles, before storing the results in an output tensor. The neural network layer operations can include element-wise, normalization, and pooling operations. The system also supports fused layers with nested traversals of input tensors, such as matrix multiplication, convolution, and attention mechanisms. The fused neural-network-layer system improves performance and reduces memory overhead compared to executing each layer as a separate GPU kernel, thereby enabling faster training and inference times.