Forward Tensor Scaling for Lower Precision Neural Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Lower precision floating-point formats, such as FP8, struggle to achieve comparable accuracy in training neural networks due to their smaller dynamic range, leading to inaccurate representations of numbers close to zero, which are often treated as zero, causing inaccuracy in neural network predictions.
Innovation Solution
The method involves scaling numbers represented by lower precision formats during forward propagation using static or dynamic scaling parameters to ensure they remain above the minimum absolute value of the dynamic range, preventing premature zeroing and improving accuracy by maintaining non-zero values.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If lower precision floating-point formats (e.g., FP8) are used, then processing speed and energy efficiency are improved, but accuracy deteriorates due to smaller dynamic range causing numbers close to zero to be treated as zero
Solution Approach 1:
The patent applies parameter changes by dynamically adjusting the scaling factor based on the statistical properties (mean and standard deviation) of the input tensor. This allows the system to adapt the precision parameters to match the actual data distribution, ensuring that numbers close to zero are preserved while maintaining the benefits of lower precision formats for overall processing efficiency.
Solution Approach 2:
The patent implements dynamics by using dynamic scaling factors that are computed based on the input data characteristics rather than using fixed scaling parameters. The scaling factor is adjusted dynamically according to the mean and standard deviation of the input tensor, allowing the system to optimize between speed and accuracy for different input distributions during neural network inference.
2Use of energy by moving object
If lower precision floating-point formats are used, then memory bandwidth and power consumption are reduced, but representation accuracy worsens due to limited dynamic range
Solution Approach 1:
The patent changes the precision parameters dynamically by computing scaling factors based on the input tensor's statistical properties. This allows the system to use lower precision formats efficiently while adapting the representation parameters to preserve important numerical information, particularly for values close to zero that would otherwise be lost in the limited dynamic range.
Solution Approach 2:
The patent introduces an intermediary scaling mechanism that bridges the gap between lower precision formats and full precision requirements. The dynamic scaling factor acts as a mediator that transforms the input data into a range suitable for lower precision representation while preserving the relative relationships and magnitude of numbers, especially those close to zero.
3Device complexity
If static scaling parameters are used, then computational complexity is reduced, but adaptability to different input distributions deteriorates
Solution Approach 1:
The patent implements dynamics by replacing static scaling parameters with dynamic scaling factors that are computed based on the input data's mean and standard deviation. This allows the system to adapt to different input distributions automatically while maintaining reasonable computational complexity through efficient statistical calculations.
Solution Approach 2:
The patent applies feedback by using the statistical properties (mean and standard deviation) of the input tensor to determine the scaling factor. This feedback mechanism allows the system to automatically adjust its scaling behavior based on the actual input data characteristics, improving adaptability without requiring complex manual tuning or multiple pre-configured scaling parameters.
Data Source
AI summary
A processing device is provided which comprises memory configured to store data and a processor configured to execute a forward activation of the neural network using a low precision floating point (FP) format, scale up values of numbers represented by the low precision FP format and process the scaled up values of the numbers as non-zero values for the numbers. The processor is configured to scale up the values of one or more numbers, via scaling parameters, to a scaled up value equal to or greater than a floor of a dynamic range of the low precision FP format. The scaling parameters are, for example, static parameters or alternatively, parameters determined during execution of the neural network.