Neural Network Quantization Using Gradient-Based Interval Tables
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep learning models with floating-point operations incur high time and hardware costs, and quantization to integer operations, especially for complex operators like exponential and sine operators, results in complex calculations and large storage pressure due to large quantization table entries.
Innovation Solution
Segment the quantization input data range into multiple preset data intervals based on the magnitudes of change gradients of output values of the quantization operator, allowing for smaller table entries and improved accuracy by matching input data to specific intervals.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a quantization table with full input data range is created for complex operators, then table lookup accuracy is improved, but storage pressure increases significantly
Solution Approach 1:
The patent divides the quantization input data range into multiple preset data intervals based on the magnitudes of change gradients of the quantization operator. Each interval has its own smaller quantization table, reducing the overall storage requirement from a single large table to multiple smaller tables while maintaining lookup accuracy within each interval.
Solution Approach 2:
Different data intervals are treated with different table sizes and storage allocations based on their local characteristics (change gradient magnitudes). Intervals with larger change gradients receive more detailed tables, while intervals with smaller gradients use smaller tables, optimizing the balance between accuracy and storage.
2Measurement precision
If quantization bit width is increased to improve precision, then calculation accuracy is improved, but table entry size increases
Solution Approach 1:
Instead of creating one large table for the full input range with high precision, the patent segments the input range into multiple intervals, each with its own smaller table. This allows high precision to be maintained within each interval while the total table size remains manageable.
Solution Approach 2:
The patent changes the parameter of table organization from a single uniform table to multiple interval-specific tables. By adjusting the segmentation based on change gradient magnitudes, the system optimizes the trade-off between precision and table size dynamically.
3Ease of operation
If a single quantization table is used for all input data, then ease of operation is improved, but table lookup accuracy decreases for segmented data
Solution Approach 1:
The patent performs preliminary segmentation of the input data range into multiple intervals based on change gradient analysis before the actual quantization operation. This preliminary action organizes the data structure so that during runtime, the system can quickly determine which interval the input belongs to and perform accurate lookup without complex calculations.
Solution Approach 2:
The patent introduces an intermediary step of determining the target data interval based on change gradient magnitudes. This intermediary mechanism bridges the simplicity of single-table lookup and the accuracy of segmented tables by automatically routing inputs to the appropriate interval-specific table.
Data Source
Figure 1~3
Figure 4~5
Figure 6~7
AI summary
Disclosed in the present disclosure are a quantization method and apparatus for a neural network model, a medium, and a device. The method includes: determining, based on an operation operator for an operation of the neural network model and input data, quantization input data corresponding to the input data; determining a target data interval corresponding to the quantization input data from a plurality of preset data intervals, where the plurality of preset data intervals are determined based on magnitudes of change gradients of output values of a quantization operator relative to its input values; determining a target quantization output value corresponding to the quantization input data based on the quantization input data and index information corresponding to the target data interval; and determining, based on the target quantization output value, a quantization result of the input data calculated by the operation operator.