Dynamic Exponent Bias for Neural Network Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training larger neural networks is becoming increasingly expensive in terms of energy and time to converge, and while quantization can help reduce costs by using lower precision data, maintaining accuracy in low-precision training is challenging.
Innovation Solution
The method involves converting tensor values from a higher-resolution format to a lower-resolution format by dynamically changing the bias, allowing for efficient use of fraction bits in the lower-resolution format, thereby reducing power consumption and convergence time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Use of energy by stationary object
If quantization is used to reduce training cost, then energy consumption and training time are reduced, but accuracy is compromised
Solution Approach 1:
The patent applies dynamics by making the bias value adaptive rather than fixed. The bias is dynamically adjusted based on the maximum exponent code found in the tensor data, allowing the system to optimize the representation range for each specific dataset. This dynamic adaptation enables accurate low-precision training by tailoring the bias to the actual data distribution, resolving the accuracy-loss contradiction in quantization.
Solution Approach 2:
The patent changes the bias parameter from a conventional fixed value to a dynamically computed value based on the maximum exponent code. By adjusting the bias parameter according to the specific tensor data characteristics, the system optimizes the mapping between exponent codes and actual values, maintaining accuracy while using lower precision formats.
2Ease of manufacture
If conventional fixed bias is used in quantization, then implementation is simple, but accuracy is difficult to maintain
Solution Approach 1:
The patent applies preliminary action by first analyzing the tensor data to find the maximum exponent code before training begins. This preliminary analysis allows the system to pre-compute the optimal bias value for the given data distribution. By performing this action before training starts, the system simplifies the overall process while ensuring accuracy, as the bias is determined in advance based on the actual data characteristics.
3Use of energy by stationary object
If lower precision data formats are used, then training cost is reduced, but representation range handling becomes challenging
Solution Approach 1:
The patent changes the bias parameter to adapt to the specific representation range requirements of different datasets. By computing the bias based on the maximum exponent code, the system automatically adjusts the mapping between available exponent codes and actual value ranges. This parameter change simplifies representation range handling in low-precision formats, as the optimal bias is automatically determined from the data itself rather than requiring manual configuration.
Data Source
AI summary
A method that may be computer implemented converts a tensor value from a first format to a second format and trains a neural network. The method determines a maximum exponent code in the first format and subtracts a first bias to obtain the highest needed exponent. It determines a second bias from the highest available code (HAC) in the second format and the HNE, and converts the tensor value from the first format to the second format by using the second bias instead of the first bias. The method uses the second format to train the neural network. The method may round the mantissa of the tensor value in the first format to obtain a rounded mantissa of the tensor value for the second format.