Floating-Point Format with Dynamic Exponent Width for Mixed Precision
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional floating point data formats, such as IEEE 754, face challenges in balancing numerical range, precision, and bit width, leading to inefficient storage and transfer overheads, particularly in AI mixed precision training and high-performance computing applications.
Innovation Solution
A new floating point data format is introduced, which includes a Dot field indicating the bit width of the exponent field, allowing dynamic adjustment of the exponent and mantissa field bit widths within a fixed total bit width, thereby optimizing storage and transfer efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If conventional floating point data formats (IEEE 754) are used with fixed bit widths for exponent and mantissa fields, then the data format is simple and compatible, but the numerical range and precision cannot be flexibly adjusted, leading to inefficient storage and transfer overheads
Solution Approach 1:
The patent applies dynamics by making the bit widths of the exponent field and mantissa field adjustable rather than fixed. The floating point data structure allows dynamic allocation of bit widths based on computational requirements, enabling the system to adapt between different precision and range needs without changing the overall data format structure.
Solution Approach 2:
The patent changes the parameters of the floating point representation by allowing the exponent bit width and mantissa bit width to vary independently. This enables the system to optimize for either numerical range (by increasing exponent bits) or precision (by increasing mantissa bits) depending on the specific computational task, resolving the contradiction between adaptability and complexity.
2Measurement precision
If the bit width of the exponent field is increased to expand numerical range, then the representable range increases, but the bit width of the mantissa field decreases, reducing precision
Solution Approach 1:
The patent resolves this contradiction by making the bit width allocation dynamic. Instead of a fixed trade-off, the system can adjust the exponent and mantissa bit widths based on whether the current computational task prioritizes range or precision, allowing both characteristics to be optimized for different scenarios.
Solution Approach 2:
The patent enables independent adjustment of the exponent bit width parameter and mantissa bit width parameter. This allows the system to change these parameters based on computational requirements, so that when numerical range is needed, the exponent bits can be increased, and when precision is needed, the mantissa bits can be increased, without being constrained by a fixed allocation.
3Measurement precision
If the total bit width is increased to simultaneously improve both numerical range and precision, then the representation capability improves, but the storage and transfer overheads increase
Solution Approach 1:
The patent avoids increasing the total bit width by allowing dynamic redistribution of existing bits between the exponent and mantissa fields. This means the system can achieve both high range and high precision sequentially by adjusting parameters, but not simultaneously at full capacity, optimizing the trade-off and avoiding the storage overhead that would result from always using maximum bit widths.
Data Source
AI summary
Embodiments of this application disclose a method for processing a floating point number and a related device, which may be used in the fields of general-purpose computing, high performance computing, artificial intelligence training and inference, and the like. The method includes: obtaining a first floating point number, where the first floating point number includes a first sign field, an exponent bit width field, a first exponent field, and a first mantissa field, and the exponent bit width field is used for indicating a bit width D occupied by the first exponent field in a total bit width N of the first floating point number; and obtaining normalized data corresponding to the first floating point number based on the first sign field, the exponent bit width field, the first exponent field, and the first mantissa field.