Fine-Grained Per-Vector Scaling for Low-Bit Neural Quantization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network quantization methods suffer from accuracy degradation due to the use of coarse-grained scale factors across multiple tensor dimensions, leading to increased quantization error and reduced precision.
Innovation Solution
Implementing fine-grained per-vector scaling, where a separate scale factor is used for each small vector of elements within a single dimension of a tensor, combined with a two-level scaling scheme that includes both per-vector and coarse-grained scale factors, to maintain accuracy while reducing energy consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If coarse-grained scale factors are shared across multiple tensor dimensions, then device complexity is reduced, but manufacturing precision (quantization accuracy) deteriorates
Solution Approach 1:
The patent segments the scale factor application into two distinct levels: coarse-grained scale factors applied to entire tensors and fine-grained per-vector scale factors applied to specific vectors within tensors. This segmentation allows the system to balance between storage efficiency (coarse level) and quantization accuracy (fine level), resolving the contradiction by operating at multiple granularity levels rather than a single level.
Solution Approach 2:
The patent applies local quality by introducing per-vector scale factors that provide localized precision adjustments for specific vectors within tensors, while maintaining coarse-grained scale factors for overall tensor-level compression. This allows different parts of the data structure to have different precision characteristics, optimizing both storage efficiency and quantization accuracy in their respective domains.
2Use of energy by moving object
If low-bitwidth integers are used for quantization, then energy consumption is reduced, but manufacturing precision (calculation accuracy) deteriorates
Solution Approach 1:
The patent changes the parameter of scale factor granularity from uniform coarse-grained to variable fine-grained per-vector scale factors. This parameter change enables the system to maintain higher calculation accuracy with low-bitwidth integers by adapting the scale factor precision to the specific needs of each vector, thereby improving energy efficiency without sacrificing overall calculation accuracy.
3Manufacturing precision
If per-vector scale factors are implemented, then manufacturing precision (quantization accuracy) is improved, but device complexity increases
Solution Approach 1:
The patent introduces another dimension of organization by structuring scale factors at two hierarchical levels: tensor-level coarse-grained scale factors and vector-level fine-grained scale factors. This dimensional organization allows the system to manage the complexity of per-vector scaling through a structured hierarchy, where the coarse level provides overall compression and the fine level provides localized precision, making the complexity manageable through hierarchical decomposition.
Data Source
AI summary
Today neural networks are used to enable autonomous vehicles and improve the quality of speech recognition, real-time language translation, and online search optimizations. However, operation of the neural networks for these applications consumes energy. Quantization of parameters used by the neural networks reduces the amount of memory needed to store the parameters while also reducing the power consumed during operation of the neural network. Matrix operations performed by the neural networks require many multiplication calculations, so reducing the number of bits that are multiplied reduces the energy that is consumed. Quantizing smaller sets of the parameters using a shared scale factor improves accuracy compared with quantizing larger sets of the parameters. Accuracy of the calculations may be maintained by quantizing and scaling the parameters using fine-grained per-vector scale factors. A vector includes one or more elements within a single dimension of a multi-dimensional matrix.