ReLU Activation Compression for GPU Memory Bottlenecks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
GPU memory is insufficient for large neural network models, and data compression on GPUs is challenging due to limited parallelism, leading to slow data transfer between CPU and GPU.
Innovation Solution
A method for compressing Rectified Linear Unit (ReLU) data on GPUs using a parallel reduction algorithm and parallel prefix sum scan algorithm to generate a nonzero data bitmap and number table, allowing for efficient allocation and copying of nonzero data into a compressed format, reducing GPU memory usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data compression is performed on GPU, then GPU memory usage is reduced, but compression speed is slow due to limited parallelism
Solution Approach 1:
The patent segments the compression task into two distinct phases: a preprocessing phase that runs on CPU to generate compression indexes (bitmaps and position tables), and a compression phase that runs on GPU to perform the actual data compression using pre-computed indexes. This segmentation allows the CPU to handle the computationally intensive index generation while the GPU performs parallel compression operations, resolving the parallelism limitation for general compression algorithms.
Solution Approach 2:
The patent performs preliminary actions by pre-computing compression indexes (bitmaps indicating zero/non-zero positions and position tables mapping compressed data locations) on the CPU before transferring data to GPU. These pre-computed indexes enable the GPU to perform compression operations without needing to execute complex compression algorithms, thereby achieving fast parallel compression while maintaining high compression ratios.
2Reliability
If large neural network models are loaded, then model accuracy is improved, but GPU memory becomes insufficient
Solution Approach 1:
The patent extracts and removes zero values from ReLU activation outputs, creating a compressed representation that stores only non-zero values along with their positions. This extraction of unnecessary zero data reduces the memory footprint of neural network data while preserving all information needed for computation, enabling larger models to fit within limited GPU memory without compromising model accuracy.
3Productivity
If data is transferred between CPU and GPU, then computation can proceed, but transfer time increases due to slow bandwidth
Solution Approach 1:
The patent extracts zero values from the data before transfer, creating a compressed dataset that contains only non-zero values. This reduces the volume of data that needs to be transferred between CPU and GPU, thereby reducing transfer time while maintaining computation throughput. The compression indexes are generated on CPU and transferred once, enabling multiple compression operations without repeated full data transfers.
Data Source
AI summary
A method is presented for compressing data of a Rectified Linear Unit (ReLU) function on a graphical processing unit (GPU) employed in a learning process of a deep neural network. The method includes converting an initial data structure including nonzero data and zero data into a compressed data structure including only the nonzero data of the initial data structure as compressed data by generating a nonzero data bitmap region, generating a nonzero data number table region by employing a parallel reduction algorithm, calculating a nonzero data array index per block region of all blocks from the nonzero data number table region by employing a parallel prefix sum scan algorithm, allocating a buffer for the compressed data; and copying the nonzero data from the initial data structure into a nonzero data array region in a compressed data format in parallel.