16-bit Floating-Point Format for Deep Learning Convergence
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current floating-point number formats, such as the 1/5/10 format, are inefficient for deep learning applications due to limited dynamic range and increased hardware complexity, leading to degraded accuracy and convergence issues in training deep neural networks.
Innovation Solution
An enhanced floating-point number format with a 1/6/9 structure, utilizing six bits for the exponent and nine bits for the mantissa, which eliminates subnormal numbers and simplifies handling of zero and NaN/infinity, allowing for faster convergence and lower error rates in machine and deep learning tasks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If current floating-point number formats (1/5/10 format) are used, then hardware complexity is reduced, but dynamic range is limited and accuracy degrades in deep learning applications
Solution Approach 1:
The patent changes the parameters of the floating-point number format from the conventional 1/5/10 structure to an enhanced 1/6/9 structure, specifically increasing the exponent field from 5 bits to 6 bits and adjusting the mantissa field accordingly. This parameter change expands the dynamic range while maintaining hardware efficiency, directly resolving the contradiction between hardware complexity and accuracy in deep learning applications
2Device complexity
If current floating-point number formats (1/5/10 format) are used, then hardware implementation is simpler, but convergence speed decreases in training deep neural networks
Solution Approach 1:
By modifying the exponent field size from 5 to 6 bits in the floating-point format, the patent enables representation of a wider range of values including smaller gradients and learning rates that are critical for efficient convergence in deep neural network training, thereby reducing convergence time without significantly increasing hardware complexity
3Quantity of substance
If current floating-point number formats are used, then hardware resource usage is lower, but error rates increase in machine learning tasks
Solution Approach 1:
The patent adjusts the mantissa field from 10 bits to 9 bits while increasing the exponent field to 6 bits, creating a balanced 1/6/9 format that maintains reasonable hardware resource usage while improving the representation precision and reducing rounding errors in machine learning computations through optimized parameter distribution
Data Source
AI summary
Techniques for operating on and calculating binary floating-point numbers using an enhanced floating-point number format are presented. The enhanced format can comprise a single sign bit, six bits for the exponent, and nine bits for the fraction. Using six bits for the exponent can provide an enhanced exponent range that facilitates desirably fast convergence of computing-intensive algorithms and low error rates for computing-intensive applications. The enhanced format can employ a specified definition for the lowest binade that enables the lowest binade to be used for zero and normal numbers; and a specified definition for the highest binade that enables it to be structured to have one data point used for a merged Not-a-Number (NaN)/infinity symbol and remaining data points used for finite numbers. The signs of zero and merged NaN/infinity can be “don't care” terms. The enhanced format employs only one rounding mode, which is for rounding toward nearest up.