Outlier Quantization for Neural Network Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training and inference processes for artificial neural networks (ANNs) are computationally intensive and require significant resources, leading to performance and energy usage inefficiencies, particularly due to the use of normal-precision floating-point formats which can result in increased storage and compute requirements, and may compromise accuracy when transitioning to quantized-precision formats.
Innovation Solution
The implementation of outlier quantization using block floating-point (BFP) formats, where vectors store floating-point values with a shared exponent and allow for outlier values to be represented by splitting mantissas and exponents, optimizing computations such as matrix multiplications and convolutions by enabling larger value ranges without increasing bit width, thereby improving efficiency and accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If normal-precision floating-point formats are used for ANN computations, then accuracy is maintained, but compute resource requirements and energy consumption increase significantly
Solution Approach 1:
The patent changes the precision parameter of floating-point computations from normal precision (e.g., 32-bit or 64-bit) to reduced precision (e.g., 8-bit or 16-bit) quantized formats. This parameter change reduces the computational complexity and energy consumption of matrix multiplications and convolutions while maintaining sufficient accuracy for neural network training and inference through carefully designed quantization schemes that preserve critical numerical properties.
2Use of energy by moving object
If quantized-precision floating-point formats are used to reduce storage and compute requirements, then energy consumption decreases, but accuracy is compromised
Solution Approach 1:
The patent segments the quantization process into multiple stages including separate quantization of weights and activations, different quantization strategies for different layers of the neural network, and staged conversion between precision formats during computation. This segmentation allows the system to use lower precision where acceptable (reducing energy consumption) while maintaining higher precision where critical for accuracy, thereby resolving the contradiction between energy efficiency and computational accuracy.
3Quantity of substance
If reduced bit width is used in quantized-precision formats, then storage requirements and compute resources are reduced, but the range of representable values is limited
Solution Approach 1:
The patent addresses the limited value range of reduced bit-width formats by introducing an additional dimension through dynamic scaling factors or exponent representations. Instead of relying solely on the mantissa bits to represent the full range of values, the system uses scaling parameters that can be adjusted per layer or per operation, effectively extending the representable value range without increasing the fundamental bit width of the quantized values themselves.
Data Source
AI summary
Machine learning may include training and drawing inference from artificial neural networks, processes which may include performing convolution and matrix multiplication operations. Convolution and matrix multiplication operations are performed using vectors of block floating-point (BFP) values that may include outliers. BFP format stores floating-point values using a plurality of mantissas of a fixed bit width and a shared exponent. Elements are outliers when they are too large to be represented precisely with the fixed bit width mantissa and shared exponent. Outlier values are split into two mantissas. One mantissa is stored in the vector with non-outliers, while the other mantissa is stored outside the vector. Operations, such as a dot product, may be performed on the vectors in part by combining the in-vector mantissa and exponent of an outlier value with the out-of-vector mantissa and exponent.