Neural Network Quantization With Dynamic Trust Region Scaling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing quantization techniques for neural networks require a large number of iterations to converge, leading to increased execution time and training time, as they often perform duplicate checks on bit width combinations, which is inefficient.
Innovation Solution
A method that calculates differences between trust region radius thresholds and quantization errors for each layer, updates the trust region radius using a minimum scaling coefficient, and optimizes bit widths efficiently to avoid duplicate checks, allowing all layers to be optimized in a shorter time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If the trust region radius is increased by multiplying by a constant greater than 1 when the loss function constraint is satisfied, then the convergence speed improves, but the bit width determination becomes less accurate due to excessive trust region radius
Solution Approach 1:
The trust region radius is made dynamic by adjusting its update rule: when the loss function constraint is satisfied, it is increased by multiplying by a constant greater than 1 to accelerate convergence; when the constraint is violated, it is decreased by multiplying by a constant less than 1 to maintain determination accuracy. This dynamic adjustment resolves the contradiction between convergence speed and accuracy.
2Measurement precision
If the trust region radius is decreased by multiplying by a constant less than 1 when the loss function constraint is violated, then the bit width determination accuracy is maintained, but the convergence speed deteriorates
Solution Approach 1:
The trust region radius adapts its behavior based on constraint satisfaction: it decreases when constraints are violated to maintain accuracy, and increases when constraints are satisfied to improve speed. This conditional dynamic behavior resolves the contradiction between accuracy maintenance and convergence speed.
Solution Approach 2:
The update rule uses feedback from the loss function constraint satisfaction result to control the trust region radius adjustment. The constraint satisfaction outcome feeds back into the radius update decision, creating a closed-loop control system that balances accuracy and speed.
3Productivity
If quantization is performed with a narrower bit width, then the calculation time and communication time are reduced, but the recognition rate deteriorates due to quantization error
Solution Approach 1:
The method changes the bit width parameter dynamically during quantization optimization. By temporarily setting different bit width values and evaluating the loss function, the system identifies the optimal bit width that achieves the desired balance between calculation efficiency and recognition rate, preventing deterioration of the recognition rate while reducing calculation time.
Data Source
AI summary
A non-transitory computer-readable storage medium storing a quantization program that causes at least one computer to execute a process, the process includes calculating, for all layers of a neural network, differences each between a trust region radius threshold and a quantization error of a first bit width that is narrower by one step than a second bit width, calculating, based on the differences, each scaling coefficient for all the layers, updating a trust region radius by using a smallest value among the scaling coefficients, and quantizing a parameter of the neural network by a third bit width set based on the trust region.