Dynamic Directional Rounding for Gradient-Aligned Neural Network Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional rounding techniques, including IEEE standards and stochastic rounding, face challenges in efficiently implementing random rounding behaviors for floating-point numbers, particularly in applications like neural network training, leading to inaccuracies and increased circuitry and power consumption.
Innovation Solution
Dynamic directional rounding method that determines the rounding direction based on the sign of the operand, eliminating the need for generating random values, and is implemented in processing units like GPUs and CPUs to ensure floating-point results follow the direction of the operand's sign, enhancing accuracy and reducing circuitry requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If stochastic rounding techniques are used to round floating-point numbers, then rounding accuracy is improved, but circuitry complexity and power consumption increase due to the need for generating random values
Solution Approach 1:
The patent extracts the random value generation requirement from the rounding process entirely. Instead of using stochastic rounding that requires random number generators, the invention uses deterministic rounding based on the sign of the operand. This removes the complex random value generation circuitry while maintaining improved rounding accuracy through directional rounding.
Solution Approach 2:
The rounding process uses information already present in the floating-point number (its sign bit) to determine the rounding direction. The operand's own sign property serves the dual purpose of indicating its polarity and directing the rounding behavior, eliminating the need for external random value generation while achieving improved rounding accuracy.
2Measurement precision
If stochastic rounding techniques are used to round floating-point numbers, then rounding accuracy is improved, but power consumption increases due to the need for generating random values
Solution Approach 1:
The patent removes the random value generation component from the rounding process, thereby eliminating the associated power consumption. The deterministic approach based on operand sign requires no additional power-intensive random number generation hardware or operations, while still achieving improved rounding accuracy.
Solution Approach 2:
The operand's sign bit, which is already part of the floating-point representation and requires no additional energy to access, is used to control the rounding direction. This self-service approach eliminates the need for separate random value generation that would consume additional power, while maintaining improved rounding accuracy.
3Device complexity
If conventional IEEE rounding techniques are used, then circuitry complexity is reduced, but training accuracy in neural networks deteriorates due to lack of random rounding behavior
Solution Approach 1:
The patent introduces dynamic directional rounding where the rounding direction changes based on the sign of the operand. This dynamic behavior, controlled by the operand's own sign property, provides the variability needed for improved neural network training accuracy while keeping the circuitry relatively simple compared to full stochastic rounding.
Data Source
AI summary
A method, computer readable medium, and system are disclosed for rounding floating point values. Dynamic directional rounding is a rounding technique for floating point operations. A floating point operation (addition, subtraction, multiplication, etc.) is performed on an operand to compute a floating point result. A sign (positive or negative) of the operand is identified. In one embodiment, the sign determines a direction in which the floating point result is rounded (towards negative or positive infinity). When used for updating parameters of a neural network during backpropagation, dynamic directional rounding ensures that rounding is performed in the direction of the gradient.