Loss Scaling for Deep Neural Network Training Stability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks face numerical issues such as denormalization and zeroing of values when trained using reduced precision hardware, leading to instability and inefficiency in computations.
Innovation Solution
The solution involves scaling gradient values during training to shift denormal values into a normal range and compensating for scaling effects to prevent impact on weight updates, allowing for efficient reduced precision computations without affecting other training processes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If reduced precision (FP16) is used for training deep neural networks, then memory usage and computation time are reduced, but numerical stability deteriorates due to denormalization and zeroing of values
Solution Approach 1:
The patent applies loss scaling before the backward pass to prevent numerical issues during gradient computation. By pre-scaling the loss value, the gradient computations in reduced precision maintain better numerical stability, avoiding denormalization and zeroing problems that would otherwise occur during training.
Solution Approach 2:
The patent dynamically adjusts the loss scale factor based on the detected numerical conditions during training. When denormal or zero values are detected in gradients, the loss scale is modified to restore numerical stability, allowing continuous training in reduced precision without losing accuracy or stability.
2Productivity
If reduced precision arithmetic is used, then throughput and performance are improved, but computation accuracy deteriorates
Solution Approach 1:
The loss is scaled up before gradient computation to ensure that gradient values remain in a numerically stable range even when using reduced precision FP16 arithmetic. This preliminary scaling prevents loss of precision during the critical gradient calculation phase while maintaining the speed benefits of reduced precision hardware.
Solution Approach 2:
The patent introduces an FP32 accumulator as an intermediary for gradient computation. Gradients are computed in reduced precision but accumulated in higher precision, and the loss scale factor acts as a mediator to bridge between reduced precision computation and full precision weight updates, preserving accuracy while utilizing fast reduced precision hardware.
3Reliability
If loss scaling is applied to maintain numerical stability, then training stability is improved, but complexity of the training process increases
Solution Approach 1:
The training system automatically monitors gradient values for denormal or zero conditions and self-adjusts the loss scale factor without requiring external intervention. This automated monitoring and adjustment mechanism maintains training stability while minimizing the additional complexity burden on the user, as the system manages the scaling dynamically based on actual numerical conditions.
Data Source
AI summary
In training a deep neural network using reduced precision, gradient computation operates on larger values without affecting the rest of the training procedure. One technique trains the deep neural network to develop loss, scales the loss, computes gradients at a reduced precision, and reduces the magnitude of the computed gradients to compensate for scaling of the loss. In one example non-limiting arrangement, the training forward pass scales a loss value by some factor S and the weight update reduces the weight gradient contribution by 1/S. Several techniques can be used for selecting scaling factor S and adjusting the weight update.