Mixed-Precision Matrix Multiplication Decomposition
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
AI accelerators are optimized for reduced-precision operations, leading to degradation in AI workload quality and inefficiency in high-precision computations, limiting their use in applications requiring higher numerical precision like deep neural networks and high-performance computing.
Innovation Solution
A method for performing high-precision matrix multiplication using mixed-precision decomposition, where a high-precision matrix is broken down into multiple matrix multiplications of lower precision data types, with scaling factors applied to residue matrices to maintain accuracy, allowing for efficient processing using energy-efficient compute units.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If AI accelerators use reduced-precision operations (float16, int8) to achieve high throughput and energy efficiency, then processing speed and energy efficiency are improved, but numerical precision and result quality deteriorate
Solution Approach 1:
The patent segments high-precision matrix multiplication into multiple lower-precision matrix multiplications. Specifically, it divides the computation into several stages where intermediate results are computed using reduced precision (float16 or int8) and then combined through scaling and accumulation to achieve the final high-precision result. This segmentation allows the system to leverage fast low-precision compute units while maintaining overall numerical accuracy.
Solution Approach 2:
The patent changes the precision parameter dynamically during computation. It uses lower precision (float16/int8) for intermediate matrix multiplications to maximize throughput, then applies scaling factors and accumulation techniques to recover high precision in the final result. This parameter change strategy enables the system to adapt precision levels to computational needs at different stages.
2Measurement precision
If AI accelerators use double-precision operations to maintain high numerical precision, then result quality is improved, but processing speed and energy efficiency deteriorate significantly
Solution Approach 1:
The patent segments double-precision matrix multiplication into multiple reduced-precision operations. By dividing the computation into stages using float16 or int8 arithmetic with intermediate scaling and accumulation, it achieves double-precision accuracy without requiring dedicated double-precision hardware, thus maintaining high throughput on hardware optimized for lower precision operations.
Solution Approach 2:
The patent creates multiple copies of data at different precision levels. It maintains high-precision input matrices while generating lower-precision versions for computation, then reconstructs the high-precision result through scaling and accumulation. This copying strategy allows simultaneous use of high-precision data for accuracy and low-precision operations for speed.
3Use of energy by moving object
If AI accelerators use reduced-precision data types (float16, int8) for computation, then energy consumption is reduced and processing speed is increased, but the quality of AI workload results deteriorates
Solution Approach 1:
The patent segments the computation to use reduced precision only where and when it is sufficient, while preserving high precision for critical operations. By dividing matrix multiplication into stages with strategic use of low-precision arithmetic and scaling, it reduces energy consumption in intermediate steps while maintaining final result quality through careful accumulation and scaling operations.
Solution Approach 2:
The patent dynamically changes precision parameters during computation. It uses lower precision (float16/int8) for energy-efficient intermediate calculations, then applies scaling factors and accumulation to restore high precision in the final result. This parameter change enables energy efficiency in computation while preserving result quality.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
In some examples, a method for performing matrix multiplication using a mixed-precision decomposition of operands comprises converting a first matrix comprising multiple elements at a first precision to a second matrix, wherein the second matrix comprises multiple elements at a second precision that is lower than the first precision, generating a first output matrix comprising a product of the second matrix and a third matrix, wherein the third matrix comprises multiple elements at the second precision, and wherein elements of the first output matrix are stored at or converted to the first precision, calculating a residue value matrix using the first matrix and the second matrix, wherein the residue value matrix comprises elements representing respective differences in values between elements of the first matrix and elements of the second matrix, scaling the elements of the residue value matrix using a first scaling factor to generate a first scaled matrix, wherein the first scaling factor is determined on the basis of absolute values of the elements of the residue value matrix, generating a first intermediate matrix by calculating the element-wise product of the first scaled matrix using a fourth matrix, wherein the fourth matrix is determined using a version of the third matrix at the first precision that has been scaled using a second scaling factor, wherein the second scaling factor is determined on the basis of absolute values of the elements of the third matrix at the first precision, generating a fifth matrix from the element-wise quotient of the first intermediate matrix and the first scaling factor, calculating the product of the fifth matrix and the inverse of the second scaling factor to generate a second output matrix at the first precision, and generating a final output matrix at the first precision using the first output matrix and the second output matrix.