Neural Network Weight Accumulation for Low-Precision Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural networks face challenges in reducing computational amount while minimizing precision loss, particularly when using low-precision number systems for training and updating weights.
Innovation Solution
A method and apparatus for neural network training that involves calculating individual update values, accumulating them in a buffer, and updating weights only when the accumulated update value exceeds a threshold, which corresponds to the least significant effective bit of the weight, while adjusting precision through decimal point positioning and bias adjustments.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If low-precision number systems are used for training neural networks, then computational amount is reduced, but precision loss occurs
Solution Approach 1:
The patent merges multiple small update values into an accumulated update value by summing them together before applying the weight update. This accumulation buffer stores the sum of gradients from multiple training iterations, allowing low-precision individual updates to be combined into a more precise cumulative update that reduces overall precision loss while maintaining computational efficiency.
Solution Approach 2:
The patent performs preliminary accumulation of update values in an accumulation buffer before actually updating the weight. By pre-summing multiple small updates and only applying the update when the accumulated value exceeds a threshold, the system prepares precise update values in advance, minimizing precision loss during the actual weight update operation.
2Reliability
If individual weight updates are performed frequently, then training accuracy is improved, but computational overhead increases
Solution Approach 1:
The patent implements periodic weight updates by accumulating update values over multiple training iterations and only applying the update when the accumulated value exceeds a threshold. This periodic update mechanism reduces computational overhead by avoiding frequent weight updates while maintaining training accuracy through the accumulation of meaningful gradient information.
Solution Approach 2:
The patent uses a threshold-based feedback mechanism that monitors the accumulated update value and only triggers a weight update when the accumulation exceeds the threshold. This feedback control ensures that updates are performed only when sufficiently meaningful, balancing training accuracy with computational efficiency by preventing unnecessary update operations.
Data Source
Figure 1
Figure 2
Figure 3A~3B
AI summary
A processor-implemented neural network method includes calculating individual update values for a weight assigned to a connection relationship between nodes included in a neural network; generating an accumulated update value by accumulating the individual update values in an accumulation buffer; and training the neural network by updating the weight using the accumulated update value in response to the accumulated update value being equal to or greater than a threshold value.