Quantization for neural network computation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Modern computing platforms, particularly those executing attention-based models for natural language understanding, face significant challenges with memory-bound execution due to their massive number of parameters, leading to high energy consumption and computational inefficiencies.
Innovation Solution
The GOBO quantization technique compresses the vast majority of 32-bit floating-point parameters of neural networks, such as BERT models, to 3 bits without requiring fine-tuning or retraining, using a novel centroid selection algorithm that maintains accuracy by separating weights into Gaussian and outlier groups.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If neural network parameters are stored in full precision (32-bit floating-point), then model accuracy is maintained, but memory footprint and energy consumption increase significantly
Solution Approach 1:
The patent segments parameters into two distinct groups: Gaussian parameters (stored as 3-bit references to representative values) and outlier parameters (stored in full precision). This segmentation allows the system to reduce memory footprint and energy consumption for the majority of parameters while maintaining model accuracy by preserving full precision for critical outlier parameters that deviate significantly from the Gaussian distribution.
Solution Approach 2:
The patent applies different storage precision strategies to different parameter subsets based on their statistical properties. Gaussian parameters use compressed 3-bit reference storage, while outlier parameters use full 32-bit floating-point storage. This local quality approach optimizes the balance between memory efficiency and model accuracy by tailoring the storage method to the specific characteristics of each parameter group.
2Quantity of substance
If neural network parameters are quantized to reduce memory footprint, then energy consumption decreases, but model accuracy may be sacrificed
Solution Approach 1:
The patent segments parameters into Gaussian and outlier groups, applying different quantization strategies to each. This allows aggressive compression (10× reduction) for Gaussian parameters while maintaining full precision for outlier parameters, achieving both memory footprint reduction and accuracy preservation simultaneously.
Solution Approach 2:
The patent changes the storage parameter representation from uniform 32-bit floating-point to a hybrid scheme where Gaussian parameters are represented by 3-bit references to representative values, and outlier parameters retain their original precision. This parameter change enables significant memory footprint reduction without sacrificing model accuracy.
3Productivity
If retraining or fine-tuning is performed to optimize model for compressed representation, then computational efficiency improves, but computational cost and time increase
Solution Approach 1:
The patent performs preliminary clustering to identify representative values and classify parameters into Gaussian and outlier groups before deployment. This preliminary action creates an optimized representation that can be directly used during inference without requiring retraining or fine-tuning, thus improving computational efficiency during operation while avoiding the time cost of additional training phases.
Solution Approach 2:
The patent makes the model self-optimizing by using the statistical properties of the parameters themselves to determine the compression strategy. The clustering algorithm automatically identifies representative values and outlier parameters based on the data distribution, eliminating the need for external retraining processes to adapt the model to the compressed representation.
4Quantity of substance
If all parameters are compressed to representative values, then memory bandwidth requirements decrease, but computation complexity increases due to lookup operations
Solution Approach 1:
The patent segments parameters into compressed Gaussian references and full-precision outlier values. During computation, the system checks whether each parameter is a reference or an outlier, applying the appropriate processing path. This segmentation reduces memory bandwidth requirements by compressing the majority of parameters while managing computation complexity through efficient classification and selective processing.
Data Source
AI summary
A method for memory storage including storing a neural network by storing values of the neural network each as a reference to a representative value; and, in some embodiments, storing additional values of the neural network. Each of the representative values can be generated by assigning each of the values of the neural network to a cluster; and for each cluster, selecting a centroid from the cluster. The method can include performing one or more multiply-accumulate operations A1B1+ . . . +AnBn on input vectors A and input vectors B, by accumulating input vectors A to an accumulated sum of input vectors A per input vector B having the same representative value and subsequently multiplying each of the accumulated sums of input vectors A by the representative value of the input vector B. A system is also described, as well as a method for configuring memory according to a data structure.