Flexible Floating Point Tensor Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep learning neural network training is computationally intensive and requires significant resources, especially due to the use of FP32 which increases computational load, while FP16 may result in shallow gradients becoming '0', affecting training accuracy.
Innovation Solution
The use of a neural network training tensor with 16-bit floating point (FP16) numbers that share a common exponent, allowing for dynamic adjustment of the shared exponent and individual exponents to avoid overflow or underflow conditions, thereby enhancing training efficiency and accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If FP32 is used in neural network training, then training accuracy is maintained (gradients remain non-zero), but computational load and resource requirements significantly increase
Solution Approach 1:
The patent changes the precision parameter from FP32 to a custom FP16 format with modified exponent and mantissa bit allocations. Specifically, it uses an 8-bit exponent and 7-bit mantissa instead of the standard IEEE 754 FP16's 5-bit exponent and 10-bit mantissa, thereby expanding the representable range while maintaining half-precision benefits for reduced computational load
2Productivity
If FP16 is used in neural network training, then computational overhead and time are reduced, but gradients become shallow and become '0', affecting training accuracy
Solution Approach 1:
The patent modifies the FP16 format parameters by increasing the exponent bits from 5 to 8 and reducing mantissa bits from 10 to 7. This parameter change expands the dynamic range to prevent gradient underflow to zero, thereby maintaining training accuracy while preserving the computational efficiency of half-precision arithmetic
Solution Approach 2:
The patent introduces dynamic exponent adjustment mechanisms where the exponent field can be selectively modified or extended during different training phases. This dynamic adaptation allows the system to maintain precision where needed while exploiting the speed benefits of FP16 elsewhere in the computation pipeline
3Productivity
If standard IEEE 754 FP16 format is used, then computational efficiency is improved, but the range of representable values is limited causing overflow or underflow conditions
Solution Approach 1:
The patent changes the exponent bit width from 5 to 8 in the FP16 format, which exponentially increases the range of representable values. This parameter modification allows the system to handle both very small gradients and large activations without overflow or underflow, while maintaining the computational efficiency of half-precision operations
Solution Approach 2:
The patent segments the floating-point representation into differently weighted components, allocating more bits to the exponent field (8 bits) and fewer to the mantissa (7 bits). This segmentation strategy prioritizes range coverage over precision, which is appropriate for neural network training where dynamic range is more critical than fine-grained precision
Data Source
AI summary
Thus, the present disclosure is directed to systems and methods for training neural networks using a tensor that includes a plurality of FP16 values and a plurality of bits that define an exponent shared by some or all of the FP16 values included in the tensor. The FP16 values may include IEEE 754 format 16-bit floating point values and the tensor may include a plurality of bits defining the shared exponent. The tensor may include a shared exponent and FP16 values that include a variable bit-length mantissa and a variable bit-length exponent that may be dynamically set by processor circuitry. The tensor may include a shared exponent and FP16 values that include a variable bit-length mantissa; a variable bit-length exponent that may be dynamically set by processor circuitry; and a shared exponent switch set by the processor circuitry to selectively combine the FP16 value exponent with the shared exponent.