Neural Network Inference Quantization With IQR Activation Clipping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Transformer-based neural networks like BERT and XLM-R achieve state-of-the-art results in NLP tasks but require costly run-time inference due to large computational demands, and direct integer quantization leads to significant accuracy drops without access to training data or calibration datasets.
Innovation Solution
A dynamic quantization method using Interquartile Range (IQR) clipping to adjust quantization thresholds during inference, ensuring at least 75% of token-wise extreme activations are not modified, allowing robust deployment on smaller devices without retraining or additional calibration.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If neural network quantization transforms 32-bit floating-point representations to 8-bit integers, then model size footprint is reduced by up to four times and inference throughput is increased, but accuracy drops significantly due to reduced numerical precision
Solution Approach 1:
The patent applies parameter changes by dynamically adjusting the clipping threshold based on the interquartile range (IQR) of activation values during inference. Instead of using a fixed clipping threshold, the system computes the IQR of the activation distribution and sets the clipping threshold at Q3 + 1.5×IQR, adapting to the actual data distribution. This dynamic parameter adjustment maintains numerical stability during quantization while preserving accuracy, resolving the contradiction between throughput improvement and accuracy maintenance.
2Adaptability or versatility
If existing quantization methods are applied, then model deployment on smaller devices becomes possible, but the methods require access to training data and pipelines, are complex, and unstable
Solution Approach 1:
The patent implements self-service by enabling the quantization process to automatically adapt to the specific neural network model and its activation distributions without requiring external training data or complex calibration pipelines. The IQR-based clipping threshold is computed directly from the model's own activation values during inference, making the quantization process self-contained, simple, and stable for deployment on various devices including smaller ones.
3Weight of stationary object
If aggressive quantization is applied to reduce model size for deployment on smaller devices, then model footprint is reduced, but computational requirements and accuracy are compromised
Solution Approach 1:
The patent uses parameter changes by dynamically adjusting the clipping threshold based on the interquartile range (IQR) of activation values during inference. Instead of using a fixed clipping threshold, the system computes the IQR of the activation distribution and sets the clipping threshold at Q3 + 1.5×IQR, adapting to the actual data distribution. This dynamic parameter adjustment maintains numerical stability during quantization while preserving accuracy, resolving the contradiction between throughput improvement and accuracy maintenance.
Solution Approach 2:
The patent applies dynamics by making the clipping threshold adaptive rather than static. The threshold is computed dynamically during inference based on the actual distribution of activation values in the model, allowing the quantization process to adapt to different inputs and model states. This dynamic approach ensures that the quantization maintains reliability across various operating conditions while keeping the model size reduced.
Data Source
AI summary
One or more computer processors responsive to neural network run-time, reduce one or more sets of maximum activations along a hidden dimension respectively associated with one or more activation tensors and one or more layers of a neural network. The one or more computer processors compute an interquartile range (IQR) clip threshold for each reduced set for each sequence dimension in the neural network. The one or more computer processors clip one or more activations based on respective computed IQR clip thresholds. The one or more computer processors quantize the clipped activations.