Neural Network Quantization Parameter Updates During Parallel Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network quantization techniques face issues with significant errors when input data distributions differ from calibration data, and Dynamic Quantization prolongs processing times due to sequential calculation of quantization parameters.
Innovation Solution
An information processing device with an arithmetic circuit that executes parallel modification of quantization parameters based on data distribution, reducing errors while maintaining processing efficiency by accumulating and analyzing data from previous neural network operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If quantization parameters are determined during calibration using calibration image data, then quantization reduces computational load, but significant errors occur when input data distribution differs from calibration data
Solution Approach 1:
The patent applies dynamic quantization by determining quantization parameters during the inference phase based on the actual distribution of input image data, rather than using fixed parameters from calibration. This allows the quantization process to adapt to varying data distributions, maintaining both computational efficiency and accuracy across different input conditions.
Solution Approach 2:
The patent changes the quantization parameters (scale and zero-point values) based on the statistical distribution of the actual input data. By calculating these parameters dynamically from the input image data's mean and standard deviation, the system adjusts quantization to match the current data distribution, reducing quantization errors while preserving computational speed.
2Measurement precision
If dynamic quantization is used to adapt to changing data distributions, then quantization accuracy improves, but processing time increases due to sequential calculation
Solution Approach 1:
The patent implements parallel processing where quantization parameter determination and image processing operations occur simultaneously rather than sequentially. Multiple quantization operations are performed in parallel on different regions or batches of data, eliminating the time penalty associated with dynamic parameter calculation and maintaining high processing throughput while adapting to data distribution changes.
Solution Approach 2:
The patent pre-calculates statistical properties (mean, standard deviation) of the input data distribution before performing quantization operations. By preparing these parameters in advance and using them for subsequent quantization steps, the system avoids repeated calculations during the main processing phase, reducing overall processing time while maintaining accurate quantization adaptation.
3Speed
If quantization parameters are fixed during inference phase, then processing speed is maintained, but quantization errors increase when input distribution differs from calibration data
Solution Approach 1:
The system transitions from static to dynamic quantization parameters during inference. By continuously adapting quantization parameters based on the actual input data distribution, the system maintains processing speed while eliminating the accuracy degradation that occurs with fixed parameters when data distributions change.
Data Source
AI summary
An information processing device includes a storage device configured to store a neural network and a quantization parameter, and an arithmetic circuit. The neural network includes a specific intermediate layer and a quantization layer that quantizes data, which is a set of values output by the specific intermediate layer, based on the quantization parameters. The arithmetic circuit executes repeated processing that repeatedly executes arithmetic processing that obtains an output result of the neural network by inputting image data into the neural network and stores the data output by the specific intermediate layer in the storage device, and in parallel with the repeated processing, executes modification processing that modifies the quantization parameter based on a distribution of values in the data stored in the storage device.