Logarithmic Quantization Training for Neural Network Accuracy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep Neural Networks (DNNs) face challenges in deployment due to increased memory and computational footprints, especially in constrained hardware environments, leading to accuracy drops when using Quantized Neural Networks (QNNs) with low precision representations.
Innovation Solution
A system that performs quantization training using a gradient descent optimization process to generate updated quantization log threshold parameters, allowing for the generation of quantized neural network parameters that improve accuracy by adapting to the dynamic range of network parameters.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If quantization is used to reduce memory and computational footprints, then hardware complexity is reduced, but accuracy drops significantly
Solution Approach 1:
The patent changes the parameter representation from linear quantization to logarithmic quantization. By representing quantization thresholds in the log domain rather than linear domain, the system achieves better accuracy preservation across different magnitude ranges while maintaining low-bit quantization benefits for hardware complexity reduction.
Solution Approach 2:
The patent introduces dynamic optimization of quantization parameters during training using gradient descent. The quantization thresholds are no longer static but are dynamically adjusted based on the training data distribution, allowing the system to adapt to different datasets and maintain accuracy while using low-precision representations.
2Ease of manufacture
If fixed quantization thresholds are used, then implementation is simpler, but accuracy drops on challenging datasets
Solution Approach 1:
The patent performs preliminary optimization of quantization thresholds during the training phase using gradient descent. By pre-optimizing the thresholds on training data before deployment, the system achieves better accuracy on challenging datasets while maintaining simple fixed-threshold implementation during actual inference, combining the benefits of both approaches.
3Measurement precision
If more training iterations are performed to improve accuracy, then model performance improves, but training time and computational cost increase
Solution Approach 1:
The patent changes the optimization approach by working in the log domain for quantization thresholds. This parameter transformation leads to faster convergence during training, allowing the model to achieve good accuracy with fewer training iterations, thus reducing training time while maintaining performance.
Data Source
AI summary
Systems and methods for training a neural network model includes providing a quantization function including a quantization log threshold parameter associated with a log value of a quantization threshold. A quantization training to a neural network model is performed to generate quantized neural network parameters. The quantization training includes: generating first values with a first precision for the neural network parameters; performing a first optimization process to generate an updated quantization log threshold parameter; and generating quantized values with a second precision lower than the first precision for the neural network parameters by applying the quantization function with the updated quantization log threshold parameter to the first values. The neural network model with the quantized values for the neural network parameters is provided for performing a task.