Cross-Layer Rescaling for Neural Network Quantization Error Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks trained with 32-bit floating point computations face significant biases and errors when converted to run on 8-bit processing architectures, due to rounding errors and varying weight ranges across layers.
Innovation Solution
The method involves scaling each output channel weight of a neural network layer by a corresponding scaling factor, and applying the inverse scaling factor to the input channel weights of the adjacent layer, to equalize weight tensor ranges and reduce quantization errors. This is achieved through various techniques such as heuristics, dynamic range equalization, and the use of black-box optimizers to minimize quantization error metrics.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If neural networks are converted from 32-bit floating point to 8-bit integer computations, then computational efficiency and hardware optimization are improved, but quantization errors and rounding biases increase
Solution Approach 1:
The patent applies preliminary rescaling operations before quantization to normalize weight ranges across different layers. By scaling output channel weights by a first factor and input channel weights by a second factor, the system prepares the neural network weights in advance to minimize quantization errors when converting from 32-bit floating point to 8-bit integer representations.
Solution Approach 2:
The system changes the scaling parameters of neural network weights dynamically based on layer-specific characteristics. Different scaling factors are applied to different layers (first scaling factor for output channel weights, second scaling factor for input channel weights) to optimize the distribution of weight values before quantization, thereby reducing rounding biases.
2Manufacturing precision
If weight ranges are equalized across neural network layers, then quantization performance is improved, but computational complexity increases
Solution Approach 1:
The patent applies local rescaling operations specific to each layer's weight distribution characteristics. Instead of using a uniform scaling approach, the system determines layer-specific scaling factors based on the actual weight ranges and distributions in each layer, applying appropriate scaling only where needed to equalize ranges effectively.
Solution Approach 2:
The system dynamically determines scaling factors based on the actual weight ranges detected in each layer during the rescaling process. The first scaling factor for output channel weights and second scaling factor for input channel weights are computed adaptively to equalize weight ranges, making the quantization process responsive to the specific characteristics of each layer.
Data Source
AI summary
Various embodiments include methods and neural network computing devices implementing the methods for performing quantization in neural networks. Various embodiments may include equalizing ranges of weight tensors or output channel weights within a first layer of the neural network by scaling each of the output channel weights of the first layer by a corresponding scaling factor, and scaling each of a second adjacent layer's corresponding input channel weights by applying an inverse of the corresponding scaling factor to the input channel weights. The corresponding scaling factor may be determined using a black-box optimizer on a quantization error metric or based on heuristics, equalization of dynamic ranges, equalization of range extrema (minima or maxima), differential learning using straight through estimator (STE) methods and a local or global loss, or using an error metric for the quantization error and a black-box optimizer that minimizes the error metric with respect to the scaling.