Neural Network Weight Updates With Low-Precision Gradient Accumulation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural network implementations require significant computational resources and processing time, leading to high power consumption and accuracy degradation when attempting to reduce processing requirements.
Innovation Solution
A neural network method that calculates individual gradient values, accumulates them to determine effective gradient values, and updates weights using a low-precision number system to reduce computational load while maintaining accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If standard precision gradient accumulation and weight update operations are used in neural network training, then accuracy and precision are maintained, but computational resources and processing time increase significantly leading to high power consumption
Solution Approach 1:
The patent changes the precision parameter of gradient values and weight updates from standard high precision (e.g., 32-bit floating point) to low precision (e.g., 8-bit integers or fixed-point representations). This parameter change reduces the computational complexity and power consumption of arithmetic operations while maintaining sufficient accuracy for effective neural network training through accumulated gradient values.
2Use of energy by moving object
If neural network processing requirements are reduced to lower power consumption, then power consumption decreases, but accuracy and precision degrade
Solution Approach 1:
The patent segments the weight update process into two parts: individual gradient calculations are performed with reduced precision to save power, while the accumulation of these gradients is maintained with higher precision through separate accumulation registers. This segmentation allows the system to benefit from low-power operations while preserving accuracy in the final weight updates.
Solution Approach 2:
The patent introduces an intermediary accumulation mechanism that stores gradient values before weight updates. This intermediary structure allows gradient values to be accumulated with higher precision even when individual gradient calculations use lower precision operations, thereby maintaining overall accuracy while enabling power-efficient processing.
3Productivity
If low-precision operations are used to reduce computational load, then processing speed and efficiency improve, but accuracy and precision deteriorate
Solution Approach 1:
The patent segments computational operations into gradient calculation phase (using low precision for speed) and weight update phase (using accumulated high precision values for accuracy). This segmentation enables the system to achieve high processing speed during gradient computation while maintaining accuracy during the critical weight update operations.
Data Source
AI summary
A neural network method and apparatus are provided. A processor implemented neural network includes calculating respective individual gradient values for updating a weight of a neural network, calculating a residual gradient value based on an accumulated gradient value obtained by accumulating the individual gradient values and a bit digit representing the weight, tuning the respective individual gradient values to correspond to a bit digit of the residual gradient value, summing the tuned respective individual gradient values, the residual gradient value, and the weight, and updating the weight and the residual gradient value based on a result of the summing to train the neural network.