Low-Bitwidth Neural Network Training With Moment Penalization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional deep learning algorithms require extensive computation time and specialized hardware due to inefficiencies in general-purpose processors, and low-bitwidth accelerators suffer from significant information loss during quantization, especially with distributions having long tails or high moments.
Innovation Solution
Implement moment-penalization functions to modify the distribution of neural network weights and hidden states, reducing information loss by penalizing higher-order moments, and utilize hybrid analog-digital processors with photonic accelerators for efficient computation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If low-bitwidth quantization is used to speed up computation, then processing speed improves, but information loss increases due to distribution characteristics
Solution Approach 1:
The patent applies moment-penalization during the training phase to pre-modify the weight and activation distributions before quantization. By penalizing higher-order moments (skewness, kurtosis) in advance, the distributions are transformed to be more compatible with low-bitwidth representation, thereby reducing information loss during subsequent quantization while maintaining fast processing speeds
2Ease of manufacture
If conventional quantization is applied without distribution modification, then implementation simplicity is maintained, but accuracy deteriorates due to long-tail distributions
Solution Approach 1:
The patent modifies the training objective function by adding moment-penalization terms that directly change the distribution parameters (mean, variance, skewness, kurtosis) of weights and activations. This transformation aligns the distributions with the uniform or discrete quantization levels, significantly improving quantization accuracy while maintaining the simplicity of the quantization process itself
3Measurement precision
If higher-order moments are penalized during training, then quantization accuracy improves, but training complexity increases
Solution Approach 1:
The patent segments the moment-penalization into separate, independently computable terms for different moments (first moment for mean, second for variance, third for skewness, fourth for kurtosis). Each moment term can be computed and optimized separately during training, reducing the overall training complexity compared to treating all moments simultaneously while still achieving improved quantization accuracy
Data Source
AI summary
Methods and systems for training neural networks using low-bitwidth accelerators are described. The methods described herein use moment-penalization functions. For example, a method comprises producing a modified data set by training a neural network using a moment-penalization function and the data set. The moment-penalization function is configured to penalize a moment associated with the neural network. Training the neural network in turn comprises quantizing the data set to obtain a fixed-point data set so that the fixed-point data set represents the data set in a fixed-point representation, and passing the fixed-point data set through an analog accelerator. The inventors have recognized that training a neural network using a modified objective function augments the accuracy and robustness of the neural network notwithstanding the use of low-bitwidth accelerators.