GPU State Machine Unrolling for ML Convolution Memory
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current GPU systems are inefficient in accessing and processing three-dimensional (3D) matrices required for machine-learning tasks, leading to high storage and memory access overhead, especially during convolution operations in neural networks.
Innovation Solution
The implementation of a graphics processing unit (GPU) system that recasts convolution operations as matrix multiplication operations, utilizing a state machine to unroll data in a register file and control data movement, thereby reducing memory fetches and address calculations, and employing techniques like im2col and Memory-Efficient Convolution to optimize memory access.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If convolution operations are performed directly in GPU, then the operation can be executed, but memory access overhead and storage requirements become excessively high
Solution Approach 1:
The patent transforms the mathematical representation of convolution operations by changing parameters from direct spatial convolution to matrix multiplication formulation. This parameter transformation recasts the operation to leverage GPU's matrix processing capabilities while reducing memory requirements through efficient data layout and access patterns
Solution Approach 2:
The patent segments the convolution operation into distinct phases: data loading, matrix multiplication, and result computation. By dividing the operation into manageable segments and using a state machine to coordinate them, the system achieves efficient execution without requiring the entire input data to be stored in memory simultaneously
2Productivity
If data is loaded and processed in traditional manner, then processing can occur, but memory bandwidth requirements increase significantly
Solution Approach 1:
The patent performs preliminary actions by pre-computing and storing only the necessary portions of input data in a register file before the main processing operation. The state machine coordinates loading only required data slices, avoiding unnecessary memory bandwidth consumption while ensuring data is ready for efficient processing
Solution Approach 2:
The patent introduces a register file as an intermediary between main memory and processing units. This intermediary buffer holds pre-loaded data slices, allowing the processor to work with data in registers rather than continuously accessing main memory, thereby reducing memory bandwidth requirements
3Productivity
If state machine unrolls data in register file, then matrix multiplication efficiency improves, but device complexity increases
Solution Approach 1:
The state machine is designed to automatically manage data loading and coordinate between different processing units without external intervention. It self-regulates the unrolling process, managing register file access and data flow based on the convolution operation requirements, which simplifies the overall system architecture despite the added complexity
Data Source
AI summary
A graphics processing unit (GPU) and a method is disclosed that performs a convolution operation recast as a matrix multiplication operation. The GPU includes a register file, a processor and a state machine. The register file stores data of an input feature map and data of a filter weight kernel. The processor performs a convolution operation on data of the input feature map and data of the filter weight kernel as a matrix multiplication operation. The state machine facilitates performance of the convolution operation by unrolling the data of the input feature map and the data of the filter weight kernel in the register file. The state machine includes control registers that determine movement of data through the register file to perform the matrix multiplication operation on the data in the register file in an unrolled manner.