Neural Network Quantization Using Offline and Online Data Distributions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current machine-learning model training and inference methods are inefficient due to the high bit-depth of 32-bit floating-point representations, which result in slower operations, higher memory utilization, and increased storage requirements, making it desirable to optimize AI models to lower bit-widths like 8/16-bits for faster performance and reduced resource usage.
Innovation Solution
The optimization method involves quantizing weights, activations, and feature maps of trained AI models using offline and online data distributions to generate low-bit representation models, employing symmetric or asymmetric quantization modes, and removing outliers to minimize precision loss, with the aid of data processing accelerators and neural network processing cores.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If 32-bit floating point representation is used in AI models, then measurement precision is maintained, but operation speed decreases and memory utilization increases
Solution Approach 1:
The patent applies parameter changes by transitioning from 32-bit floating point representation to lower bit-depth formats (8-bit or 16-bit integers). This changes the fundamental parameter of numerical representation precision to achieve faster operations and reduced memory usage while maintaining acceptable model accuracy through quantization techniques
2Measurement precision
If 32-bit floating point representation is used in AI models, then measurement precision is maintained, but memory utilization increases
Solution Approach 1:
The patent changes the bit-depth parameter from 32 bits to 8 or 16 bits, reducing memory utilization by 2-4 times. This parameter transformation allows the model to occupy significantly less memory space while maintaining functional accuracy through appropriate quantization methods
3Productivity
If quantization to low-bit representation is applied, then operation speed increases and memory utilization decreases, but manufacturing precision decreases
Solution Approach 1:
The patent applies local quality by using different quantization strategies for different parts of the model. Specifically, it uses asymmetric quantization for weights and symmetric quantization for activations, and applies per-channel or per-tensor quantization depending on the specific layer and data characteristics, thereby maintaining accuracy while achieving speedup
Solution Approach 2:
The patent employs preliminary action by performing quantization calibration and outlier detection before final model deployment. It identifies and removes outlier values that would disproportionately affect quantization accuracy, and pre-computes scale factors and zero points to minimize precision loss during the transition to low-bit representation
4Quantity of substance
If quantization to low-bit representation is applied, then memory utilization decreases, but manufacturing precision decreases
Solution Approach 1:
The patent changes the representation parameter from 32-bit to 8/16-bit while applying sophisticated quantization algorithms that adapt to the specific data distribution. This includes computing optimal scale factors and zero points, and using different quantization modes (symmetric/asymmetric) to preserve accuracy despite reduced bit-depth
Data Source
AI summary
Embodiments relate to systems and methods to optimize quantization of tensors of an AI model. According to one embodiment, a system receives an AI model having one or more layers. The system receives a number of input data for offline inferencing and applies offline inferencing to the AI model based on the input data to generate offline data distributions for the AI model. The system quantizes one or more tensors of the AI model based on the offline data distributions to generate a low-bit representation AI model, where each layer of the AI model includes the one or more tensors, where the one or more tensors include the one or more tensors. In one embodiment, the system applies online inferencing using the low-bit representation AI model to generate online data distributions for a feature map, and quantizes a feature map tensor based on the online data distributions.