Tensor Rescaling for Float16 Neural Network Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The use of 16-bit float numbers in neural network training leads to overflow and underflow issues due to their narrower numerical range compared to single-precision floats, resulting in precision loss and network divergence.
Innovation Solution
A modified float16 format is introduced, where tensor values are rescaled using a float scale factor, allowing matrix operations to be performed while avoiding overflow and underflow by extending the dynamic range of values.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If float16 format is used for neural network training, then memory usage and computation time are reduced, but numerical range becomes too narrow causing overflow and underflow
Solution Approach 1:
The patent extends the dynamic range of float16 by introducing an additional dimension - a scale factor stored in the exponent field. This transforms the representation from a single fixed-range format to a multi-scale format where values can be scaled by powers of 2, effectively adding an extra dimension to the numerical representation space.
Solution Approach 2:
The patent changes the parameters of the float16 format by reinterpreting the exponent field to store scale factors rather than direct exponent values. This parameter change allows the same 16-bit structure to represent a much wider range of values by dynamically adjusting the scale based on the stored factor.
2Quantity of substance
If float16 format is used for data storage, then memory requirements are reduced, but precision is lost due to narrower numerical range
Solution Approach 1:
The patent adds a scale factor dimension to the float16 representation, allowing values to be extended beyond the traditional range while maintaining the same memory footprint. This dimensional extension enables precise representation of both very small and very large values without requiring additional memory bits.
Solution Approach 2:
By changing how the exponent field is interpreted to store scale factors, the patent enables the same 10-bit fraction field to maintain precision across a much wider range of values, effectively changing the precision parameters of the format without increasing memory usage.
3Speed
If standard float16 format is used, then computation is faster, but overflow and underflow cause network divergence
Solution Approach 1:
The patent introduces a scale factor dimension that allows dynamic adjustment of the numerical range during training. This additional dimension enables the computation to handle varying magnitudes of gradients and activations without overflowing or underflowing, maintaining training stability while preserving fast float16 computation.
Solution Approach 2:
The patent changes the interpretation of exponent bits to represent scale factors, allowing the computational range to be dynamically adjusted during training. This parameter change prevents overflow and underflow by accommodating the full range of values encountered during neural network training operations.
Data Source
AI summary
Aspects of the present invention are directed to computer-implemented techniques for improving the training of artificial neural networks using a reduced precision (e.g., float16) data format. Embodiments of the present invention rescale tensor values prior to performing matrix operations (such as matrix multiplication or matrix addition) to prevent overflow and underflow. To preserve accuracy throughout the performance of the matrix operations, the scale factors are defined using a novel data format to represent tensors, wherein a matrix is represented by the tuple X, where X=(a, v[.]), wherein a is a float scale factor and v[.] are scaled values stored in the float16 format. The value of any element X[i] according to this data format would be equal to a*v[i].