Dynamic Neural Network Quantization for Variable Input Ranges
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural networks trained using floating-point arithmetic require significant memory and computational resources, and existing quantization methods lead to inaccuracies when operated on inputs with dynamic ranges different from the expected range, affecting network output accuracy.
Innovation Solution
Dynamic quantization techniques that adjust neural networks on-the-fly to the input dynamic range without requiring prior knowledge of input statistics, involving graph traversal and scale factor propagation to minimize quantization error, while avoiding re-quantization of weight matrices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If neural networks are quantized to integer neural networks to reduce size and computational resources, then memory usage and computational requirements are reduced, but accuracy of the network outputs suffers when input dynamic range differs from expected range
Solution Approach 1:
The patent applies dynamic quantization where scale factors are adjusted in real-time based on the actual input data distribution. Instead of using fixed quantization parameters determined during training, the system dynamically recalculates scale factors at inference time to match the actual input dynamic range, thereby maintaining accuracy while using integer precision.
Solution Approach 2:
The patent changes the quantization parameters (scale factors) based on the observed input data characteristics. By monitoring the actual input dynamic range and adjusting the quantization scale factors accordingly, the system adapts the integer quantization to match the real-world data distribution, resolving the accuracy loss problem.
2Measurement precision
If elaborate iterative quantization procedures are used to reduce quantization error, then accuracy is improved for expected inputs, but the system becomes more complex and requires access to input features during quantization
Solution Approach 1:
The patent performs preliminary quantization during training with floating-point arithmetic, then applies a simpler dynamic adjustment procedure at inference time. The complex iterative optimization is replaced by a preliminary quantization step followed by lightweight dynamic scale factor adjustment based on actual input statistics.
Solution Approach 2:
The system uses the actual input data itself to determine the appropriate quantization parameters. By computing scale factors directly from the observed input distribution without requiring external annotation or complex iterative procedures, the quantization process serves itself using the data it processes.
3Productivity
If fixed quantization is applied during training, then the network can be deployed with reduced precision, but it cannot adapt to inputs with different dynamic ranges than those seen during training
Solution Approach 1:
The patent transforms fixed quantization into dynamic quantization by introducing runtime adjustment of scale factors. The system maintains the efficiency of pre-computed integer quantization while adding the ability to adapt to different input ranges through dynamic recalculation of quantization parameters based on actual input data characteristics.
Data Source
AI summary
An apparatus for applying dynamic quantization of a neural network is described herein. The apparatus includes a scaling unit and a quantizing unit. The scaling unit is to calculate an initial desired scale factors of a plurality of inputs, weights and a bias and apply the input scale factor to a summation node. Also, the scaling unit is to determine a scale factor for a multiplication node based on the desired scale factors of the inputs and select a scale factor for an activation function and an output node. The quantizing unit is to dynamically requantize the neural network by traversing a graph of the neural network.