Floating-Point Format With Dynamic Exponent Bit Allocation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional floating point data formats, such as IEEE 754, face challenges in balancing numerical range, precision, and bit width, leading to inefficiencies in data storage and transfer, particularly in AI mixed precision training and high-performance computing applications where fixed bit widths result in wasted storage space and increased overheads.
Innovation Solution
A new floating point data format is introduced, which includes an additional Dot field to dynamically indicate the bit width of the exponent field, allowing the bit widths of both the exponent and mantissa fields to adjust, thereby optimizing bit allocation without increasing the total bit width, thus enhancing flexibility and efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If fixed bit width floating point formats (IEEE 754) are used, then data storage and transfer are simplified, but storage space is wasted and overheads increase when numerical range or precision requirements vary
Solution Approach 1:
The patent introduces a dynamic floating point format where the exponent bit width field (Dot field) can take different values to dynamically adjust the bit width of the exponent field. When the Dot field value is D, the exponent field occupies D bits, and the mantissa field occupies the remaining bits. This dynamic allocation allows the system to adapt to different numerical range and precision requirements without wasting storage space, directly resolving the contradiction between adaptability and storage efficiency.
2Quantity of substance
If low-precision floating point formats (e.g., FP16, FP8) are used to reduce storage overhead, then data storage and transfer costs decrease, but numerical precision and convergence speed are affected
Solution Approach 1:
The patent changes the parameter of exponent bit width dynamically through the Dot field. By adjusting the Dot field value, the system can change the exponent field bit width from 1 to N bits, which directly controls the numerical range and precision. This allows the system to use smaller bit widths (reducing storage overhead) when precision requirements are modest, and increase bit width (improving precision) when needed, without being constrained to fixed precision levels like traditional FP16 or FP8 formats.
3Measurement precision
If high-precision floating point formats (e.g., FP64) are used to ensure numerical precision, then measurement precision is improved, but data storage and transfer overheads increase
Solution Approach 1:
The patent allows dynamic adjustment of the exponent field bit width parameter through the Dot field. When high precision is required, the Dot field can be set to a larger value D, increasing the exponent field bit width to N bits and thereby increasing numerical precision. When lower precision suffices, the Dot field value can be reduced, decreasing storage overhead. This dynamic parameter adjustment eliminates the need to always use full FP64 precision, reducing storage overhead while maintaining precision when needed.
4Adaptability or versatility
If larger bit width is allocated to exponent field to increase numerical range, then the representable numerical range is expanded, but the bit width available for mantissa field and overall precision decreases
Solution Approach 1:
The patent implements dynamic bit width allocation between exponent and mantissa fields through the Dot field mechanism. When the Dot field value is D, the exponent field occupies D bits and the mantissa field occupies the remaining bits. This dynamic allocation allows the system to adjust the balance between numerical range (controlled by exponent bit width) and precision (controlled by mantissa bit width) based on specific computational needs, resolving the contradiction between range and precision that exists in fixed-format floating point representations.
Data Source
Figure 1
Figure 2~3
Figure 4~5
AI summary
Embodiments of this application disclose a method for processing a floating point number and a related device, which may be used in the fields of general-purpose computing, high performance computing, artificial intelligence training and inference, and the like. The method includes: obtaining a first floating point number, where the first floating point number includes a first sign field, an exponent bit width field, a first exponent field, and a first mantissa field, and the exponent bit width field is used for indicating a bit width D occupied by the first exponent field in a total bit width N of the first floating point number; and obtaining normalized data corresponding to the first floating point number based on the first sign field, the exponent bit width field, the first exponent field, and the first mantissa field. In this application, an exponent bit width field for indicating a bit width of an exponent field is additionally defined in a conventional floating point number, so that the bit width of the exponent field and the bit width of the subsequent mantissa field may dynamically change with a value of the exponent bit width field to meet requirements for different numerical ranges and precision of floating point numbers in different scenarios. Embodiments of this application may optimize a floating point number format and improve use effect of a floating point number.