Graphics Processor Im2col Loading to Reduce Inference Time
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional graphics processors require two memory reads during General Matrix Multiplication (GEMM) operations, leading to increased bandwidth demand and longer calculation and inference times due to the im2col operation's necessity to prepare input data.
Innovation Solution
A graphics processor with a texel unit and execution unit, including a loading module and im2col module, that reads an original matrix from memory, expands it to an expansion matrix, and calculates feature coordinates to reduce memory access, thereby optimizing the im2col operation for faster convolution calculations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a traditional graphics processor reads the original matrix from memory and performs im2col operation to prepare input data for GEMM, then the convolution calculation can be executed, but the memory bandwidth demand increases and the calculation time becomes longer
Solution Approach 1:
The patent performs the im2col operation in advance during the data loading phase, transforming the original matrix into the expansion matrix before the GEMM operation begins. This preliminary action eliminates the need for separate im2col execution time during inference, as the data is pre-processed and ready for immediate multiplication.
Solution Approach 2:
The patent merges the im2col operation with the data loading process by integrating the expansion matrix generation into the memory read operation. The loading module simultaneously performs memory access and im2col transformation, combining two previously separate operations into one unified process that reduces overall execution time.
2Productivity
If a traditional graphics processor performs im2col operation and stores the result in memory before GEMM, then the input data is prepared for matrix multiplication, but the memory bandwidth demand increases due to two reads
Solution Approach 1:
The expansion matrix is generated in advance during the loading phase and kept in register files or on-chip memory, eliminating the need for a second memory read during GEMM execution. This preliminary preparation of data in faster memory hierarchies reduces memory bandwidth consumption.
Solution Approach 2:
The patent introduces an intermediate expansion matrix that serves as a mediator between the original matrix in memory and the GEMM operation. This intermediate representation is generated once during loading and then reused for multiple kernel multiplications, reducing repeated memory access and bandwidth consumption.
Data Source
AI summary
A graphics processor includes a texel unit and an execution unit. The texel unit includes a loading module. The execution unit includes an im2col module to execute an im2col algorithm to expand an original matrix to obtain an expansion matrix according to the size of a kernel. The execution unit multiplies the expansion matrix and the kernel to obtain a feature map matrix. The loading module calculates feature coordinates of each element of the feature map matrix according to the coordinates of the expansion matrix, and obtains the original coordinates of each element of the original matrix according to the feature coordinates, the size of the kernel, a stride, and padding. The loading module reads at least one of the memory blocks covered by the original coordinates of each element of the original matrix, and outputs data corresponding to the original coordinates in the memory blocks.