GPU Vector Signal Processors for Flexible Mixed-Precision Matrix Multiplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data-intensive applications such as deep learning and high-performance computing face limitations in processing power and memory bandwidth due to the precision of operands and deep learning operations, which restricts the performance of GPUs, especially in supporting multi-precision operations and double precision requirements for HPC applications.
Innovation Solution
The implementation of a processing system with a GPU that includes multiple vector signal processors (VSPs) which cooperatively fetch and rotate matrix portions from memory, using a crossbar switch to perform matrix multiplications efficiently, thereby reducing power consumption and increasing data reuse, and supporting flexible precision operations by extending the accumulation register file.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If flexible mixed-precision capabilities are implemented in GPUs, then adaptability for different precision operations is improved, but device complexity increases due to complex multiplexers and crossbar switches
Solution Approach 1:
The VSPs are designed to perform multiple precision operations (32-bit, 64-bit, 16-bit, 8-bit) using the same hardware structure. The accumulation register file can be configured to store different precision formats, and the VSPs can dynamically adapt to different operation types without requiring separate dedicated hardware for each precision level, thus achieving universality across different precision requirements
Solution Approach 2:
The system changes the precision parameter dynamically by configuring the accumulation register file width and the VSP operation mode. Instead of having fixed hardware for each precision level, the system allows the same hardware to operate at different precision levels by changing control parameters and register configurations, enabling flexible mixed-precision operations without increasing physical device complexity
2Productivity
If data is fetched from memory for matrix multiplication, then computation is enabled, but bandwidth consumption increases
Solution Approach 1:
The system performs preliminary action by pre-fetching and storing matrix portions in the VSP buffers before the actual multiplication operation. The VSPs load data into their internal buffers in advance, allowing subsequent multiply-accumulate operations to use this pre-loaded data without requiring additional memory bandwidth during the computation phase, thus reducing overall bandwidth consumption
Solution Approach 2:
The matrix multiplication is segmented into multiple iterations where different portions of matrices are processed in stages. The VSPs divide the computation into blocks and process them sequentially using the same loaded data, allowing one matrix portion to be reused across multiple VSPs and iterations, thereby reducing the total amount of data that needs to be fetched from memory
3Productivity
If multiple VSPs process matrix portions concurrently, then productivity increases, but device complexity increases due to coordination requirements
Solution Approach 1:
Multiple VSPs are merged into a unified matrix multiplication pipeline where they share common buffers and coordination logic. The VSPs work as a coordinated team processing different portions of the same matrix operation, sharing buffer resources and following a unified control scheme, which reduces the complexity that would arise from having completely independent processing units
Solution Approach 2:
The VSPs maintain continuous useful action by systematically rotating through different matrix portions in a coordinated sequence. Each VSP processes its assigned portion and then transitions to the next portion in a continuous cycle, ensuring that all VSPs remain actively engaged in computation without idle periods, maximizing productivity while maintaining simple rotational coordination
Data Source
AI summary
A processing unit such as a graphics processing unit (GPU) includes a plurality of vector signal processors (VSPs) that include multiply/accumulate elements. The processing unit also includes a plurality of registers associated with the plurality of VSPs. First portions of first and second matrices are fetched into the plurality of registers prior to a first round that includes a plurality of iterations. The multiply/accumulate elements perform matrix multiplication and accumulation on different combinations of subsets of the first portions of the first and second matrices in the plurality of iterations prior to fetching second portions of the first and second matrices into the plurality of registers for a second round. The accumulated results of multiplying the first portions of the first and second matrices are written into an output buffer in response to completing the plurality of iterations.