Smooth Quantized Neural Network Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep artificial neural networks require significant computational resources, leading to high energy consumption, latency, and storage challenges, especially in resource-constrained devices, and existing quantization methods often result in inaccurate neural networks during training.
Innovation Solution
A method for training neural networks with quantized inputs and weights using a scaling factor and uniform quantization ranges to reduce computational intensity while maintaining accuracy, involving incremental reduction of quantization function smoothness and regularization to stabilize the training process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Use of energy by moving object
If quantization is applied to reduce computational intensity, then energy consumption and hardware requirements are reduced, but training accuracy and stability deteriorate
Solution Approach 1:
The quantization function transitions from smooth to non-smooth dynamically during training. Initially, a smooth quantization function is used to maintain training stability and accuracy, then it transitions to a non-smooth function to achieve final quantization, resolving the contradiction between maintaining accuracy and achieving quantization
Solution Approach 2:
The smoothness parameter of the quantization function is changed during training. The function starts with high smoothness (using a small epsilon value in the sigmoid function) and gradually reduces smoothness (increasing epsilon) to reach the final non-smooth quantization state, allowing the system to balance accuracy and quantization effectiveness at different training stages
2Measurement precision
If full-precision neural networks are used, then prediction accuracy is maintained, but computational resources and hardware costs increase
Solution Approach 1:
The precision parameter is dynamically adjusted during training. The network starts with full-precision computations and gradually transitions to quantized computations as training progresses, allowing the system to achieve both high accuracy during learning and low precision for deployment
Solution Approach 2:
The precision of computations transitions from high (full-precision) to low (quantized) dynamically. The smooth quantization function enables this transition by providing continuous approximation early in training and discrete quantization later, resolving the contradiction between accuracy and hardware requirements
3Manufacturing precision
If non-smooth quantization functions are used, then final quantization accuracy is improved, but training stability deteriorates
Solution Approach 1:
The smoothness of the quantization function is dynamically adjusted during training. The function starts smooth to ensure training stability and gradually becomes non-smooth to achieve final quantization accuracy, resolving the contradiction between stability and precision
Solution Approach 2:
The smooth quantization function is applied preliminarily during early training stages to stabilize the training process. This preliminary smooth approximation prepares the model for the final non-smooth quantization, ensuring that training stability is maintained before achieving high quantization accuracy
Data Source
AI summary
Training a neural network, including applying a quantization function to a set of real-valued weights to generate quantized weights scaled to fall within a respective quantization range that is symmetrically centered at zero and comprises a defined number of uniform quantization levels corresponding to integer multiples of a scaling factor. A cost is computed based alignments of the quantized weights with the quantization levels. The real-valued weights and the scaling factor are adjusted with an objective of reducing the computed cost in one or more following training iterations. When performing a plurality training iterations, a smoothness of the quantization function is incrementally reduced for multiple training iterations. Alignment of quantized weights with quantization levels and decreasing smoothness of the quantization function can result in a trained neural network that can perform accurate predictions using relatively few computational resources.