Neural Network Quantization with Gradient-Based Lookup Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep learning models with floating-point operations incur high time and hardware costs, and quantization to integer operations, particularly for complex operators like exponential and sine operators, results in complex calculations and large storage pressure due to large quantization table entries.
Innovation Solution
Segment the quantization input data range into multiple preset data intervals based on the magnitudes of change gradients of output values, using these intervals for table lookup to reduce storage pressure and improve accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a quantization table with size 65536 is created to store all possible calculation results for int16 quantization, then complete coverage of input data range is achieved, but storage pressure increases significantly
Solution Approach 1:
The patent divides the quantization input data range into multiple preset data intervals based on the magnitudes of change gradients of output values. Instead of using a single large table covering the entire range, separate smaller tables are created for each interval. This segmentation reduces the storage requirements while maintaining accuracy by ensuring that each interval has appropriate table size based on its gradient characteristics.
Solution Approach 2:
The patent applies different table sizes and granularities to different data intervals based on their local characteristics. Intervals with large change gradients use finer-grained smaller tables to capture rapid variations, while intervals with small change gradients use coarser-grained larger tables. This local adaptation optimizes both storage efficiency and lookup accuracy for each region of the input range.
2Ease of manufacture
If equal-sized data intervals are used for table lookup segmentation, then implementation simplicity is maintained, but table lookup accuracy decreases for regions with significant output value fluctuations
Solution Approach 1:
The patent makes each data interval have different properties by determining their boundaries based on change gradient magnitudes. Regions with high gradients get finer segmentation with more intervals, while regions with low gradients get coarser segmentation with fewer intervals. This non-uniform interval design improves accuracy in critical regions while maintaining reasonable implementation complexity through automated gradient-based boundary determination.
3Measurement precision
If quantization bit width is increased to improve precision, then calculation accuracy improves, but table entry size and storage requirements increase
Solution Approach 1:
The patent segments the input range and creates separate tables for each segment, allowing each table to be optimized for its specific range. This enables using appropriate precision levels for each segment rather than uniformly high precision across the entire range, reducing total storage while maintaining overall accuracy.
Solution Approach 2:
The patent changes the parameters of the quantization tables by adjusting interval boundaries and table sizes based on change gradient magnitudes. This dynamic parameter adjustment allows the system to allocate storage resources efficiently, using more precision where needed (high gradient regions) and less precision where acceptable (low gradient regions).
Data Source
AI summary
Disclosed in the present disclosure are a quantization method for a neural network model, a medium, and a device. The method includes: determining, based on an operation operator for an operation of the neural network model and input data, quantization input data corresponding to the input data; determining a target data interval corresponding to the quantization input data from a plurality of preset data intervals, where the plurality of preset data intervals are determined based on magnitudes of change gradients of output values of a quantization operator relative to its input values; determining a target quantization output value corresponding to the quantization input data based on the quantization input data and index information corresponding to the target data interval; and determining, based on the target quantization output value, a quantization result of the input data calculated by the operation operator.