Power-of-2 Parametric Activation Clipping for CNN Quantization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep learning inference on low-power embedded devices faces challenges in achieving 8-bit fixed-point accuracy for convolutional neural networks (CNNs) due to severe constraints on quantization operations, particularly with symmetric, power-of-2, per-layer quantization, which results in significant accuracy loss.
Innovation Solution
The implementation of a power-of-2 parametric activation function (PACT2) that estimates and clips activation feature maps to suitable power-of-2 ranges, allowing for efficient 8-bit quantization without extensive back-propagation, using histogram-based methods and exponential moving averages to determine clipping values, and adaptive weight constraining for weights and biases.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If symmetric power-of-2 per-layer quantization is applied to CNNs for 8-bit fixed-point inference, then computational efficiency and hardware compatibility are improved, but accuracy is significantly degraded
Solution Approach 1:
The patent introduces a learnable clipping parameter α that dynamically adjusts the activation range for quantization. This parameter is optimized during training to find the optimal balance between quantization efficiency and accuracy preservation, allowing the system to adapt the clipping threshold based on the specific layer and data characteristics rather than using fixed symmetric power-of-2 ranges
Solution Approach 2:
The patent transforms the static quantization process into a dynamic one by making the clipping parameter α learnable and layer-specific. During training, α is continuously adjusted based on the activation distribution, enabling the quantization scheme to adapt to varying data characteristics across different layers and training stages, thus resolving the contradiction between fixed quantization efficiency and adaptive accuracy
2Manufacturing precision
If extensive back-propagation is used for quantization calibration, then accuracy is improved, but training time and computational overhead increase significantly
Solution Approach 1:
The patent performs quantization calibration during the main training process rather than as a separate post-training step. The clipping parameter α is learned concurrently with the network weights through back-propagation, allowing the quantization scheme to be optimized in advance as part of the training process itself, thus eliminating the need for extensive separate calibration procedures
Solution Approach 2:
The patent merges the quantization calibration process with the main training objective by formulating them as a unified optimization problem. The clipping parameter α and network weights are optimized simultaneously through a single back-propagation process, combining what were previously separate time-consuming steps into one efficient training routine
3Manufacturing precision
If additional quantization parameters are introduced for each layer, then accuracy is improved, but device complexity and memory requirements increase
Solution Approach 1:
The patent introduces layer-specific clipping parameters α_l that are tailored to the characteristics of each individual layer. Instead of using a single global quantization parameter for the entire network, each layer has its own optimized clipping threshold that adapts to the local activation distribution, thereby improving per-layer quantization accuracy without requiring complex per-parameter quantization schemes
4Productivity
If aggressive clipping is applied to constrain activations to power-of-2 ranges, then quantization efficiency is improved, but information loss increases
Solution Approach 1:
The patent uses the gradient of the clipping operation with respect to the clipping parameter α to provide feedback during training. This gradient information allows the system to learn the optimal clipping threshold that minimizes information loss while maintaining quantization efficiency, effectively balancing the trade-off between aggressive clipping benefits and information preservation through continuous optimization
Data Source
AI summary
In an example, a method includes executing, using one or more processors, a power-of-2 parametric activation (PACT2) function to quantize a set of data. The executing of the PACT2 function includes determining a distribution for the set of data; discarding a portion of the data corresponding to a tail of the distribution to form a remaining set of data; estimating a maximum value of the remaining set of data; determining a new maximum value of the remaining set of data using a moving average and at least one historical value of at least one prior remaining set of data; determining a clipping value by expanding the new maximum value to a nearest power of two value; and quantizing the set of data using the clipping value to form a quantized set of data.