Floating-Point Dot Product Operator for Correctly Rounded Accumulation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current hardware operators for matrix multiplication in AI and deep learning, particularly those using FMA (Fused Multiply-Add) techniques, face challenges in maintaining precision during accumulation phases due to limited dynamic range and rounding errors, especially when dealing with mixed precision formats like binary16 and binary32, leading to loss of accuracy and increased hardware complexity.
Innovation Solution
The implementation of a mixed-precision FMA operator that converts operands into fixed-point numbers with sufficient dynamic range, allowing for lossless addition and correct rounding, using techniques that reduce the adder size to a reasonable value by leveraging guard bits and sticky bits for accurate mantissa calculation, and handling out-of-bounds cases to ensure precise results in binary32 format.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the adder size is increased to cover the entire dynamic range of binary32 format (277 bits), then measurement precision is improved, but device complexity increases
Solution Approach 1:
The patent segments the adder into multiple stages: an initial alignment stage that handles exponent comparison and mantissa alignment, followed by a main addition stage. This segmentation allows the adder to process binary32 operands efficiently without requiring a single excessively wide adder, thus reducing device complexity while maintaining precision.
Solution Approach 2:
The patent introduces a temporal dimension by using a pipeline architecture with multiple processing stages. Instead of performing all operations in a single wide adder, the computation is distributed across time stages, allowing narrower adders to achieve the same functional result, thereby reducing hardware complexity.
2Measurement precision
If the register size is increased to 277 bits to cover all dynamics of binary32 format, then measurement precision is improved, but device complexity increases
Solution Approach 1:
The register is segmented into multiple smaller registers that store intermediate results at different stages of the computation. Instead of requiring a single 277-bit register, the patent uses several smaller registers (e.g., 64-bit or 128-bit) to hold partial results, which are then combined in subsequent stages, reducing the complexity of individual register components.
Solution Approach 2:
The patent performs preliminary alignment and normalization of operands before they are loaded into registers. By pre-processing the input data to ensure proper formatting and scaling, the register size requirements are reduced, as the registers only need to accommodate the aligned mantissas rather than the full dynamic range of unprocessed binary32 values.
3Device complexity
If a window of significant bits is kept for addition and rounding, then device complexity is reduced, but measurement precision deteriorates due to truncation errors
Solution Approach 1:
The patent implements dynamic window adjustment where the size and position of the significant bit window are adapted based on the exponents of the input operands. When the exponent difference is large, a smaller window suffices; when exponents are closer, a larger window is used. This dynamic adaptation maintains precision for critical cases while reducing complexity for less critical cases.
Solution Approach 2:
The patent uses feedback mechanisms where the results of exponent comparison and alignment information are fed back to control the window selection and adder operation. This feedback ensures that the appropriate number of significant bits are processed in each case, maintaining precision without unnecessarily increasing adder size for all operations.
Data Source
Figure 1~4B
Figure 5
Figure 6
AI summary
The invention relates to a hardware dot product calculation operator, comprising several multipliers (10) each receiving two multiplicands (a, b) in the form of floating-point numbers coded in a first precision format (fpl6); an alignment circuit (12) associated with each multiplier, configured to, on the basis of the exponents of the corresponding multiplicands, convert the result of the multiplication into a respective fixed-point number having a sufficient number of bits (80) to cover the entire dynamics of the multiplication; and a multi-adder (30) configured to losslessly add the fixed-point numbers from the multipliers, providing a sum in the form of a fixed-point number.