Neural Network Fixed-Point Precision Adjustment for Stable Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The precision of input neurons, weights, and output neuron gradients in neural network operations is insufficient, leading to errors and potential training failures.
Innovation Solution
A neural network operation module that dynamically adjusts the precision of input neurons, weights, and output neuron gradients by reducing their precision and increasing the bit width of their fixed-point data format when necessary, based on estimated error transfer precision calculations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the precision of fixed-point numbers is increased to reduce operation errors, then the representable numerical range decreases
Solution Approach 1:
The patent applies dynamic precision adjustment by changing the precision of fixed-point numbers during the training process based on the estimated error transfer precision. The controller unit dynamically modifies the precision parameter s in the fixed-point format Q(s) according to the calculated error precision requirements, allowing the system to adapt precision levels rather than using a fixed precision throughout training. This resolves the contradiction by making precision a dynamic parameter that increases when error accumulation is detected and decreases when error precision is sufficient, thereby maintaining both operational accuracy and numerical range as needed.
2Reliability
If the precision of fixed-point numbers is increased to ensure accurate training results, then the computational overhead increases
Solution Approach 1:
The patent changes the precision parameter s of the fixed-point number format based on the estimated error transfer precision. Instead of using a fixed high precision throughout training, the system adjusts the precision parameter dynamically according to the actual error precision requirements calculated from the relationship between adjacent layer precisions and error propagation. This resolves the contradiction by using parameter changes to match precision levels to actual computational needs, ensuring training accuracy when required while reducing overhead when lower precision suffices.
3Reliability
If the precision of output neuron gradients is increased to prevent training failure, then the resource consumption increases
Solution Approach 1:
The patent applies dynamic adjustment to the precision of output neuron gradients by modifying the precision parameter based on the estimated error transfer precision during training. The controller unit dynamically changes the precision level of gradient representations according to the calculated error precision requirements, rather than maintaining a fixed high precision for all gradient operations. This resolves the contradiction by making gradient precision dynamic, ensuring training stability when error precision indicates a risk of training failure while reducing resource consumption when the calculated error precision is sufficient.
Data Source
AI summary
A neural network operation module, which comprises a storage unit that stores output neurons, weight precision and output neuron gradient precision of a multi-layer neural network; a controller unit that obtains an average value Y1 of the absolute value of the output neuron before fixed-point and an average value Y2 of the absolute value of the output neuron after fixed-point; if Y1/Y2 is greater than a preset threshold K, obtaining the output neuron gradient precision of adjacent two layers of the multi-layer neural network, and obtaining an estimation value An of error transfer precision; when An is greater than a preset precision Ar, the output neuron gradient precision and weight precision of the adjacent two layers are increased; and an operation unit that represents the output neuron gradient and weight of the adjacent two layers according to the increased precision.