Scaled Half-Precision Floating Point Tensors for Deep Neural Network Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current machine learning processing systems face challenges in efficiently training deep neural networks using low-precision floating-point representations, particularly due to dynamic range limitations and the risk of information loss during half-precision training.
Innovation Solution
The implementation of a scaled half-precision floating-point representation that shifts floating-point values to a higher magnitude, utilizing the full range of half-precision, and dynamically adjusts the exponent bias based on the dynamic-range requirements at each neural network layer.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If half-precision floating-point representation is used for training deep neural networks, then memory bandwidth and computational efficiency are improved, but dynamic range limitations cause information loss
Solution Approach 1:
The patent implements dynamic scaling factors that are adjusted during the training process based on the observed gradient magnitudes. The scaling factors are not fixed but adaptively change to maintain optimal precision across different training stages, resolving the contradiction between using low-precision formats and preserving gradient information.
Solution Approach 2:
The patent changes the numerical parameters (scaling factors) of the half-precision representation system to match the dynamic range of gradient values at different training stages. By adjusting these parameters dynamically, the system maintains adequate precision while benefiting from the memory and speed advantages of half-precision arithmetic.
2Quantity of substance
If standard half-precision floating-point format is used, then memory usage is reduced, but the dynamic range is insufficient for representing gradient values
Solution Approach 1:
The patent introduces scaling factors as intermediary values that bridge the gap between the limited dynamic range of half-precision formats and the broader range of gradient values. These scaling factors act as mediators that allow gradient information to be represented accurately even when stored in compressed half-precision format.
Solution Approach 2:
The patent adds an additional dimensional aspect to the half-precision representation by incorporating per-channel or per-tensor scaling factors. This extra dimension allows the system to extend the effective dynamic range beyond what the base half-precision format can provide, while still maintaining memory efficiency.
3Speed
If low-precision training is implemented, then computational speed is increased, but the learning process integrity is compromised
Solution Approach 1:
The patent implements feedback mechanisms where the scaling factors are adjusted based on the observed behavior of gradient values during training. This feedback loop ensures that the low-precision computation remains faithful to the intended learning process by adapting to the actual data characteristics and preventing information loss that would compromise learning integrity.
Data Source
AI summary
One embodiment provides for a machine-learning accelerator device a multiprocessor to execute parallel threads of an instruction stream, the multiprocessor including a compute unit, the compute unit including a set of functional units, each functional unit to execute at least one of the parallel threads of the instruction stream. The compute unit includes compute logic configured to execute a single instruction to scale an input tensor associated with a layer of a neural network according to a scale factor, the input tensor stored in a floating-point data type, the compute logic to scale the input tensor to enable a data distribution of data of the input tensor to be represented by a 16-bit floating point data type.