Quantization Parameter Optimization Preventing Neural Network Overfitting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for minimizing the cost function in neural network training with quantization error regularization fail to ensure the error function reaches zero when the regularization term is zero, leading to overfitting issues.
Innovation Solution
A quantization parameter optimization method that updates the quantization parameter using a cost function with a regularization term based on the quantization error, ensuring the function value derived from the regularization term is negatively correlated with inference accuracy, thereby optimizing the quantization parameter and preventing overfitting.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If a regularization term based on quantization error is added to the error function to optimize quantization parameters, then quantization error is reduced, but the error function may not reach zero when the regularization term is zero, causing overfitting
Solution Approach 1:
The patent changes the parameter form of the regularization term from a direct quantization error term to a function that incorporates both quantization error and its interaction with the error function. Specifically, it uses a regularization term of the form λ||w - w_q||^2 where the penalty is adjusted based on the relationship between quantization error and model performance, allowing the error function to reach zero while still preventing overfitting
Solution Approach 2:
The patent introduces a feedback mechanism where the regularization term is dynamically adjusted based on the error function value and quantization error. The optimization process continuously monitors the relationship between quantization error and inference accuracy, adjusting the regularization strength to maintain the error function at zero while preventing overfitting through adaptive penalty application
2Reliability
If the regularization term is increased to prevent overfitting, then model generalization improves, but the quantization error penalty increases causing the error function to not reach zero
Solution Approach 1:
The patent transforms the regularization parameter from a fixed value to a dynamic parameter that changes based on the optimization state. The regularization term is formulated as λ(||w - w_q||^2 + ε) where ε is a small positive value that ensures the error function can reach zero, and λ is adaptively adjusted during training to balance generalization and quantization accuracy
Solution Approach 2:
The patent applies partial regularization by using a softened penalty term that doesn't fully constrain the quantization error but provides sufficient guidance to prevent overfitting. The regularization term is designed to apply only when necessary, allowing the error function to reach zero in most cases while still providing generalization benefits
Data Source
AI summary
A quantization parameter optimization method includes: determining a cost function in which a regularization term is added to an error function, the regularization term being a function of a quantization error that is an error between a weight parameter of a neural network and a quantization parameter that is a quantized weight parameter; updating the quantization parameter by use of the cost function; and determining, as an optimized quantization parameter of a quantization neural network, the quantization parameter with which a function value derived from the cost function satisfies a predetermined condition, the optimized quantization parameter being obtained as a result of repeating the updating, the quantization neural network being the neural network, the weight parameter of which has been quantized, wherein the function value derived from the regularization term and an inference accuracy of the quantization neural network are negatively correlated.