GeMM Computation Unit Decomposing Vectors for Lower Precision
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for deep learning, particularly in machine learning, face challenges in reducing memory footprint and energy consumption during training, with general matrix-matrix multiplication (GeMM) being inefficient due to large matrix sizes and limited parallelism in real-world applications, especially on GPUs.
Innovation Solution
A computationally efficient method for GeMM is implemented using a reconfigurable data processor with a computation unit that decomposes higher-precision multiplicand and multiplier vectors into lower-precision formats, enabling efficient execution through a multiplicand tensor encoder and product vector generator, reducing the number of GeMM operations required.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Use of energy by moving object
If general matrix-matrix multiplication is performed using conventional methods, then computational accuracy is maintained, but memory footprint and energy consumption increase significantly
Solution Approach 1:
The patent segments the matrix multiplication operation by dividing matrices into tiles and processing them in blocks. This segmentation enables the system to use lower-precision formats (bfloat16) for intermediate calculations while maintaining overall computational accuracy through careful tile-based processing and accumulation in higher-precision formats.
Solution Approach 2:
The patent changes the precision parameter dynamically during computation. It uses bfloat16 (16-bit floating point) for intermediate representations to reduce memory footprint and energy consumption, while maintaining FP32 (32-bit floating point) accuracy for final results. This parameter transformation is achieved through specialized conversion units that map between precision formats.
2Measurement precision
If higher-precision formats are used throughout computation, then computational accuracy is maintained, but memory footprint increases
Solution Approach 1:
The computation is segmented into stages where different precision formats are used at different levels. Lower-precision bfloat16 format is used for intermediate tile calculations to reduce memory requirements, while higher-precision FP32 is used for accumulation and final results to maintain accuracy.
Solution Approach 2:
The system dynamically changes the data precision parameter based on the computation stage. Input matrices are converted from FP32 to bfloat16 for processing, and intermediate results are accumulated back to FP32. This parameter transformation reduces memory footprint during computation while preserving final accuracy.
3Productivity
If reconfigurable architectures are used, then adaptability and efficiency are improved, but device complexity increases
Solution Approach 1:
The reconfigurable architecture implements multi-functional computation units that can perform various operations (matrix multiplication, vector operations, conversions) using a unified resource set. This universality improves productivity by reducing hardware resource requirements while managing complexity through software-based reconfiguration rather than dedicated hardware for each function.
Solution Approach 2:
The architecture uses dynamic reconfiguration where computation units can be programmed to perform different functions based on the specific computational task. This dynamic adaptability improves efficiency for specialized operations like GeMM while managing complexity through programmable rather than hardwired solutions.
4Loss of energy
If fewer GeMM operations are executed, then computational overhead and energy consumption are reduced, but implementation complexity increases
Solution Approach 1:
The patent segments the matrix multiplication into tile-based operations that can be processed in parallel. This segmentation enables the system to reduce the total number of sequential GeMM operations by executing multiple smaller tile operations simultaneously, reducing overall computational overhead and energy consumption.
Solution Approach 2:
The system performs preliminary actions by converting input matrices to lower-precision formats before computation and preparing tile data in advance. This preliminary processing reduces the number of high-cost operations needed during the main computation, lowering energy consumption while managing implementation complexity through automated conversion routines.
Data Source
AI summary
A computation unit that comprises (i) a multiplicand vector decomposer that generates a decomposed multiplicand vector which uses a sequence of first and second concatenated multiplicand sub-elements (1st2ndCMCSE) in a lower-precision format (LPF) to represent corresponding ones of multiplicand elements in a multiplicand vector in a higher-precision format (HPF), (ii) a multiplier vector decomposer that generates a decomposed multiplier vector which uses a sequence of first and second concatenated multiplier sub-elements (1st2ndCMLSE) in the LPF to represent corresponding ones of multiplier elements in a multiplier vector in the HPF, (iii) a multiplicand tensor encoder that encodes double reads of the sequence of the 1st2ndCMCSE in a decomposed multiplicand tensor, and (iv) a product vector generator that generates a product vector containing a sequence of first and second concatenated product sub-elements by executing general matrix-matrix multiplication (GeMM) operations between the double reads of the 1st2ndCMCSE and corresponding ones of the 1st2ndCMLSE.