Neural Network Weight Quantization with Trainable Codebooks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The increasing scale and complexity of neural networks lead to high storage and computational demands due to large weights, necessitating a more effective method to reduce weights and computation while maintaining precision.
Innovation Solution
A data quantization method involving retraining the neural network with a weight codebook, using clustering algorithms like K-means or K-medoids to group weights into classes, and replacing them with center weights, encoded in a dictionary, allowing low-bit quantization and reducing storage and memory access overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If low-rank matrix decomposition or hash techniques are used to reduce weights, then storage capacity requirement is reduced, but the amount of computation that can be reduced is limited and neural network precision deteriorates
Solution Approach 1:
The patent applies quantization by changing the precision parameter of weight representation from high-precision floating-point to low-precision discrete levels. By clustering weights into multiple classes and representing them with fewer bits, the patent reduces storage requirements while maintaining acceptable computational accuracy through optimized quantization levels and clustering algorithms.
Solution Approach 2:
The patent creates a simplified copy of the original weight data through clustering and quantization. Instead of storing and processing exact weight values, the patent uses clustered center points and quantization tables that approximate the original weights, thereby reducing storage and computation while preserving the essential functional characteristics of the neural network.
2Measurement precision
If large-scale weight data is used to maintain neural network precision, then measurement precision is improved, but memory access energy consumption increases
Solution Approach 1:
The patent changes the precision parameter of weight representation from high-precision to low-precision formats. By quantizing weights to fewer bits and using clustering to represent weight distributions with compact codes, the patent significantly reduces memory access requirements and energy consumption while maintaining sufficient precision for accurate neural network operation.
3Power
If more operation units are added to handle large weights, then computing capability is improved, but computing time and energy consumption increase
Solution Approach 1:
The patent changes the computational parameter by quantizing weight values to discrete levels and using lookup tables for activation functions. This transformation enables the use of simpler, faster computation operations such as table lookups and integer arithmetic instead of complex floating-point operations, thereby reducing computing time and energy consumption while maintaining computational capability.
Data Source
Figure 1~2
Figure 3~4
Figure 5~6
AI summary
The present disclosure provides a data quantization configured to perform the following steps: grouping the weights of a neural network; performing a clustering operation on each group of weights by using a clustering algorithm, dividing a group of weights into m classes, computing a center weight for each class, and replacing all the weights in each class by the center weights, where m is a positive integer; encoding the center weight to get a weight codebook and a weight dictionary; and retraining the neural network, where only the weight codebook is trained, and the weight dictionary remains unchanged.