Neural Network Quantization With Fixed Weight Range
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing quantization methods for deep neural networks face issues such as unstable training due to varying weight ranges and additional hardware processing requirements, especially at low quantization bits, leading to inconsistent results and increased power consumption.
Innovation Solution
A method that initializes a weight array, performs a quantization procedure using a nonlinear conversion function to fix the weight range, and includes a training process with a regularization term to prune weights, resulting in a stable and efficient quantization method for neural networks, and a deep learning accelerator design that stores and processes quantized weights with reduced hardware complexity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If weight range is not limited in quantization, then quantization flexibility is improved, but quantization training stability deteriorates
Solution Approach 1:
The patent applies parameter changes by introducing a weight range limitation parameter (fixed range) during quantization. This constraint transforms the weight values to fit within a predetermined range, ensuring consistent quantization intervals and stable training results regardless of the original weight distribution.
Solution Approach 2:
The patent implements preliminary action by pre-defining the fixed weight range before quantization training begins. This preliminary constraint setup ensures that all weight values are properly bounded in advance, preventing training instability caused by varying weight ranges during the training process.
2Quantity of substance
If quantization bit is reduced for compression, then model size is reduced, but quantization precision deteriorates
Solution Approach 1:
The patent changes the parameter of weight representation by enforcing a fixed range constraint during quantization. This allows lower bit-depth quantization to maintain better precision because the fixed range ensures consistent quantization intervals, reducing quantization error even with fewer bits.
Solution Approach 2:
The patent replaces the traditional approach of relying on higher bit-depth for precision with a constraint-based mechanism (fixed range limitation). This substitution allows low-bit quantization to achieve comparable or better precision by ensuring uniform weight distribution within the fixed range.
3Measurement precision
If bias term is added for quantization correction, then quantization accuracy is improved, but hardware complexity increases
Solution Approach 1:
The patent extracts and eliminates the bias term from the quantization process by using fixed range limitation instead. This removal simplifies the hardware architecture by eliminating the need for additional bias addition circuits while maintaining quantization accuracy through the fixed range constraint alone.
Solution Approach 2:
The fixed range limitation serves multiple functions simultaneously: it ensures consistent quantization intervals, bounds weight values for stable training, and eliminates the need for separate bias correction mechanisms. This multi-functionality reduces hardware complexity while maintaining or improving quantization accuracy.
Data Source
AI summary
A quantization method for neural network model includes following steps: initializing a weight array of a neural network model, wherein the weight array includes a plurality of initial weights; performing a quantization procedure to generate a quantized weight array according to the weight array, wherein the quantized weight array includes a plurality of quantized weights within a fixed range; performing a training procedure of the neural network model according to the quantized weight array; and determining whether a loss function is convergent in the training procedure and outputting a post-trained quantized weight array when the loss function is convergent.