Neural Network Quantization via Residual Error Development
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep Neural Networks (DNNs) face challenges in efficiency due to high computational inference costs, limiting their deployment on edge devices, especially when real-time processing is required.
Innovation Solution
A computer-implemented method for neural network quantization that reduces the number of bits in the bit-wise representation of weights using a quantization operator, allowing for efficient encoding and inference by converting floating-point representations to lower bit representations like int8, int6, or ternary, while maintaining performance through residual error development and channel pruning.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If neural networks use high-precision floating-point representations for weights, then accuracy is maintained, but computational inference cost increases and deployment on edge devices becomes difficult
Solution Approach 1:
The patent changes the precision parameter of weight representations from high-precision floating-point to low-precision quantized formats (e.g., 8-bit, 4-bit, or ternary). This parameter transformation enables edge device deployment while maintaining acceptable accuracy through the development framework that compensates for quantization errors.
Solution Approach 2:
The patent introduces an intermediary development framework that acts as a mediator between the quantized weights and the original high-precision weights. This framework uses a series of development terms with increasing precision to approximate the original weights, allowing the system to benefit from both low-precision storage and high-precision computation when needed.
2Productivity
If neural networks are quantized to reduce bit-wise operations, then inference efficiency improves, but accuracy may deteriorate
Solution Approach 1:
The patent segments the weight representation into multiple precision levels through the development framework. Each term in the development series represents a segment with different precision, allowing the system to use low-precision segments for most computations while using higher-precision segments to correct errors and maintain overall accuracy.
Solution Approach 2:
The patent creates a composite weight representation that combines multiple quantized components with different precision levels. This composite structure allows the system to leverage the efficiency of low-precision arithmetic while incorporating high-precision elements to maintain accuracy, similar to how composite materials combine different properties to achieve superior performance.
3Quantity of substance
If quantization operator reduces number of bits in weight representation, then memory usage and computational cost decrease, but representation accuracy decreases
Solution Approach 1:
The patent adds an additional dimension to the weight representation by introducing the development series. Instead of representing weights with a single low-precision value, the system uses a sequence of values across different precision dimensions, where each dimension contributes to the overall accuracy while maintaining efficient low-precision storage for the majority of the representation.
Data Source
Figure 1A~1C
Figure 2A~2C
Figure 3
AI summary
The disclosure notably relates to a computer-implemented method for neural network quantization. The method comprises providing a trained neural network. The neural network has layers of weights. The method further comprises providing a quantization operator. The quantization operator reduces the number of bits of an input bit-wise representation. The method further comprises quantizing the neural network. The quantizing of the neural network includes, for each respective layer of one or more layers of the neural network, determining a respective development of the respective layer in a sum of quantized residual errors of the quantization operator. The method constitutes an improved solution for machine-learning and neural networks.