Integer Neural Network Weight Updates for Low-Memory Edge Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for updating neural network weights require significant memory resources and power consumption, especially when using floating-point formats, and are not feasible for resource-constrained hardware platforms like microcontrollers.
Innovation Solution
A method for updating neural network weights directly on the hardware platform using integer formats, involving error gradient calculations and stochastic rounding, followed by quantization to maintain accuracy and reduce memory requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If gradient backpropagation is performed using floating-point weights, then weight update accuracy is improved, but memory resource consumption increases significantly
Solution Approach 1:
The patent changes the parameter of weight representation from floating-point format to integer format. This parameter change allows the system to perform gradient backpropagation with reduced memory requirements while maintaining acceptable update accuracy through quantization-aware operations and appropriate scaling factors.
Solution Approach 2:
The patent employs lower-precision integer representations instead of high-precision floating-point numbers. This substitution uses 'cheaper' data types that consume less memory, enabling weight updates on resource-constrained devices without requiring the substantial memory resources that floating-point operations would demand.
2Measurement precision
If weights are converted to floating-point format for gradient backpropagation, then update precision is improved, but power consumption increases
Solution Approach 1:
The patent changes the numerical representation parameter from floating-point to integer format, which directly reduces the computational complexity and power consumption associated with arithmetic operations. Integer arithmetic requires fewer transistors and less energy per operation compared to floating-point arithmetic, thereby reducing overall power consumption during weight updates.
3Productivity
If neural network is quantized to integer format, then memory requirements and processing speed are improved, but weight update capability deteriorates
Solution Approach 1:
The patent performs preliminary quantization of weights to integer format before deployment, but prepares the system in advance to handle weight updates by implementing gradient backpropagation that operates directly on integer representations. This preliminary setup enables the system to maintain both the speed benefits of quantization and the adaptability needed for weight updates.
Solution Approach 2:
The patent introduces scaling factors and quantization parameters as intermediaries that bridge the gap between integer weight representations and the gradient update process. These intermediaries allow the system to perform accurate gradient calculations and weight updates using integer arithmetic, effectively mediating between the constraints of quantization and the requirements of learning.
4Adaptability or versatility
If updated neural network is transmitted via wireless transmission, then model adaptation is improved, but power consumption increases
Solution Approach 1:
The patent enables the neural network to perform weight updates autonomously on the edge device without requiring wireless transmission to remote servers. The system serves itself by implementing gradient backpropagation locally using integer arithmetic, thereby eliminating the need for power-intensive wireless communications while maintaining model adaptability through local learning.
Data Source
AI summary
According to one aspect, the disclosure proposes a method for updating an artificial neural network including initial weights stored in a memory at least in an integer format, which method includes: a processing unit determining the error gradients at the output of the layers of the neural network, the processing unit retrieving the initial weights from memory, the processing unit updating the initial weights comprising, for each initial weight, a first calculation of a corrected weight, in the integer format of this initial weight, the processing unit replacing the value of the initial weights stored in the memory by the value of the corrected weights.