Transformer Quantization With Activation Clipping for Outlier Error
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Transformer networks quantized to fixed-point precision experience increased error and decreased accuracy due to outliers in floating-point weights and activations, which are not effectively managed by existing quantization techniques.
Innovation Solution
Implement outlier removal during model quantization by clipping activation values and determining quantization factors based on calibration data to reduce the range of values represented in fixed-point models, thereby improving accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If transformer networks are quantized to fixed-point precision for deployment on resource-constrained devices, then device compatibility and processing efficiency are improved, but model error increases and accuracy decreases due to outliers in floating-point weights and activations
Solution Approach 1:
The patent applies preliminary action by performing outlier detection and removal during the quantization process before deploying the model to fixed-point devices. By identifying and removing outliers from floating-point weights and activations during the transition phase, the method prepares cleaner data for fixed-point representation, thereby maintaining model accuracy while achieving device compatibility.
Solution Approach 2:
The patent employs parameter changes by dynamically adjusting quantization parameters based on the detected outlier characteristics. The system modifies quantization thresholds and ranges to accommodate the distribution of non-outlier values, optimizing the fixed-point representation to preserve accuracy while enabling deployment on resource-constrained devices.
2Adaptability or versatility
If existing quantization techniques are used to convert floating-point models to fixed-point, then deployment on embedded devices is enabled, but outliers are not effectively managed leading to increased model error
Solution Approach 1:
The patent applies the taking out principle by explicitly extracting and removing outliers from the floating-point weight and activation distributions during quantization. By separating the outlier components from the main data distribution, the method prevents these problematic values from degrading the fixed-point model's reliability, while still maintaining the ability to deploy on embedded devices.
Solution Approach 2:
The patent introduces an intermediary processing step between floating-point training and fixed-point deployment. This intermediary quantization process with outlier removal acts as a mediator that transforms the model representation while filtering out harmful outliers, thereby bridging the gap between high-accuracy floating-point training and reliable fixed-point deployment on embedded devices.
Data Source
AI summary
An example apparatus is to clip a value of an activation associated with a layer of a floating-point version of a machine learning model to determine a clipped value of the activation, the value of the activation based on calibration data applied to the floating-point version of the machine learning model. The example apparatus is also to determine, using the clipped value of the activation, a quantization factor to quantize activations associated with a corresponding layer of a fixed-point version of the machine learning model. The example apparatus is further to configure the fixed-point version of the machine learning model on a device using the quantization factor.