Neural Network Quantization Training with Smooth Regularization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks (DNNs) require significant computational resources for training and inference, making them impractical for resource-limited edge devices, and existing quantization methods often degrade model accuracy and necessitate additional training or retraining, which is resource-intensive.
Innovation Solution
A smooth quantization regularization (SQR) function is used to constrain weight and activation values during training, allowing quantization to any bit-width precision without specialized implementation, using a differentiable periodic function to minimize a loss function and push values to discrete points.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If quantization is applied to reduce bit-width of weights and activations, then storage and compute requirements are reduced, but model accuracy deteriorates
Solution Approach 1:
The patent changes the parameter representation from continuous floating-point to discrete quantized values, and introduces a regularization term that dynamically adjusts quantization parameters during training to maintain accuracy while reducing bit-width
Solution Approach 2:
The patent introduces a regularization term as an intermediary mechanism that bridges the gap between full-precision training and quantized deployment, enabling the model to adapt to quantized representations without significant accuracy loss
2Use of energy by moving object
If quantization is applied to reduce bit-width of weights and activations, then computational resources are reduced, but model accuracy deteriorates
Solution Approach 1:
The patent transforms parameters from high-precision floating-point to low-precision quantized values, with a regularization term that optimizes the quantization process to maintain accuracy while reducing computational resource requirements
Solution Approach 2:
The regularization term serves as an intermediary that enables smooth transition from full-precision to quantized computation, allowing the model to achieve resource efficiency without significant accuracy degradation
3Adaptability or versatility
If quantization is applied to enable deployment on edge devices, then device resource requirements are reduced, but model accuracy deteriorates
Solution Approach 1:
The patent applies parameter quantization to make the model suitable for edge device deployment with limited resources, while the regularization term ensures that accuracy loss is minimized during the quantization process
Solution Approach 2:
The regularization term acts as an intermediary that facilitates the adaptation of the model to quantized representations, enabling successful deployment on resource-constrained edge devices while maintaining acceptable accuracy
4Manufacturing precision
If sinusoidal regularization is applied to mitigate accuracy loss from quantization, then model accuracy is improved, but training complexity increases
Solution Approach 1:
The sinusoidal regularization term serves as an intermediary that systematically addresses accuracy loss from quantization by providing a structured optimization signal during training, improving accuracy while keeping the additional complexity manageable
Data Source
Figure 1
Figure 2
Figure 3
AI summary
In some examples, a method for training an artificial neural network comprising multiple nodes each defining a quantized activation function configured to A model can therefore be quantized to any bit-width precision, and no special implementation measures are required in order to apply the method using existing architectures.