One-Sided Per-Kernel Weight Clipping for Neural Network Quantization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural networks face inefficiencies in memory and processing resources due to quantizing positively biased kernels into low-precision representations, leading to inefficient usage of available range and increased complexity.
Innovation Solution
Implement a per-kernel strategy of adjusting quantization bias followed by one-sided clipping to negate positively biased weights before quantization, using a global scale threshold for efficient utilization of the quantized representation range.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If positively biased kernels are quantized into low-precision representations (e.g., INT4), then memory size and computational complexity are reduced, but the available range is used inefficiently
Solution Approach 1:
The patent inverts the traditional quantization approach by applying a bias adjustment that flips the weight distribution. Instead of quantizing positively biased weights directly into a negatively biased format (which wastes range), the method transforms the weights to be negatively biased before quantization, thereby efficiently utilizing the available INT4 range from -8 to 7.
Solution Approach 2:
The patent changes the parameter distribution of kernel weights by applying a bias adjustment transformation. This parameter change converts positively biased weight distributions into negatively biased ones, allowing optimal mapping to the low-precision INT4 format and improving range utilization efficiency while maintaining computational accuracy.
2Productivity
If quantization is applied to reduce model size, then processing efficiency improves, but accuracy may be compromised due to range inefficiency
Solution Approach 1:
By inverting the bias of weight distributions before quantization, the patent ensures that the full dynamic range of low-precision formats is effectively utilized. This inversion prevents information loss that would occur with direct quantization, thereby maintaining computational accuracy while achieving the processing efficiency gains from quantization.
Solution Approach 2:
The patent performs a preliminary bias adjustment transformation on kernel weights before the quantization process. This preliminary action prepares the weight distribution to match the target low-precision format's characteristics, ensuring that the subsequent quantization step preserves maximum information and accuracy while enabling efficient processing.
3Device complexity
If standard quantization is used without bias adjustment, then implementation is simpler, but power consumption increases due to inefficient range usage
Solution Approach 1:
The patent modifies the weight distribution parameters through bias adjustment, transforming positively biased kernels into negatively biased ones. This parameter change enables efficient packing of weight values into the INT4 range, reducing the number of bits required to represent each weight and thereby reducing overall power consumption for storage and processing.
Solution Approach 2:
By inverting the bias orientation of weight distributions, the patent creates an optimal mapping to low-precision formats that minimizes the computational energy required. This inversion ensures that weight values are distributed across the full available range, reducing redundancy and lowering power consumption for both memory access and arithmetic operations.
Data Source
AI summary
Systems, apparatuses, and methods for implementing one-sided per-kernel clipping and weight transformation for neural networks are disclosed. Various parameters of a neural network are quantized from higher-bit representations to lower-bit representations to reduce memory utilization and power consumption. To exploit the effective range of quantized representations, positively biased weights are clipped and negated before convolution. Then, the results are rescaled back after convolution. A one-sided clipping technique is used for transforming weights to exploit the quantization range effectively, with the side chosen to be clipped being the biased side. This technique uses a global strategy for clipping without requiring skilled expertise. This approach allows the system to retain as much information as possible without losing unnecessary accuracy when quantizing parameters from higher-bit representations to lower-bit representations.