GPU Virtual Matrix Multiplication Memory Bandwidth Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning algorithms, particularly in neural networks, face significant memory bandwidth issues due to the large image matrices generated during convolution operations, leading to increased memory access requests and consumption.
Innovation Solution
The implementation of a virtual GEMM space memory bandwidth reduction method, where convolution operations are assigned to compute units and duplicative data is fetched from an internal cache hierarchy, reducing the need for global memory access and minimizing memory bandwidth consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If image data is convolved with a filter matrix to perform convolution operations, then the neural network can perform image recognition tasks, but the resulting image matrices become much larger than the original image data, leading to increased memory bandwidth consumption
Solution Approach 1:
The patent divides the large image matrix into multiple smaller tiles or blocks that can be processed independently. Each compute unit processes a specific tile, loading only the necessary portion of the image data and filter into local memory. This segmentation reduces the memory bandwidth requirement for each compute unit while maintaining the overall convolution capability across the entire image.
Solution Approach 2:
The patent introduces a spatial dimension by organizing compute units in a grid layout that corresponds to the spatial structure of the image. Each compute unit is responsible for a specific spatial region (tile), and data is streamed through the compute unit grid in a systematic pattern. This dimensional organization allows efficient data reuse and reduces redundant memory accesses.
2Adaptability or versatility
If the image matrix size increases due to convolution with filter matrices, then the neural network processing capability is enhanced, but the number of memory access requests by compute units increases
Solution Approach 1:
The patent performs preliminary tiling and data organization before the convolution operation begins. Image data and filter matrices are pre-processed into tile formats and loaded into local memory of compute units in advance. This preliminary action ensures that during the actual convolution computation, compute units can operate with minimal additional memory access requests, as the necessary data is already staged in local memory.
Solution Approach 2:
The patent introduces local memory as an intermediary between global memory and compute units. Instead of compute units directly accessing global memory for every operation, data is first transferred to local memory which acts as a buffer. This intermediary allows compute units to reuse data from local memory multiple times without generating additional global memory access requests, thereby reducing overall memory access volume.
3Ease of operation
If duplicative data is fetched during convolution operations, then compute units can perform their calculations, but global memory access requests increase and memory bandwidth is consumed
Solution Approach 1:
The patent merges multiple memory access requests for the same data into a single global memory access. By coordinating compute units to share data through local memory and using cache coherence mechanisms, the system ensures that when multiple compute units need the same duplicative data, it is fetched from global memory only once and then distributed to the requesting compute units via local memory, eliminating redundant global memory accesses.
Data Source
AI summary
A processing system includes a central processing unit (CPU) and a graphics processing unit (GPU) that has a plurality of compute units. The GPU receives an image from the CPU and determines a total result area in a virtual-matrix-multiplication space of a virtual matrix-multiplication output matrix based on convolutional parameters associated with the image in an image space. The GPU partitions the total result area of the virtual matrix-multiplication output matrix into a plurality of virtual segments. The GPU allocates convolution operations to the plurality of compute units based on each virtual segment of the plurality of virtual segments.