Neural Network Quantization Range Restriction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network quantization techniques face challenges in efficiently deploying transformer-based models on resource-constrained devices due to high latency, energy consumption, and large memory footprint, while also introducing noise that reduces model accuracy.
Innovation Solution
The approach involves determining a practical domain for the following function in a neural network layer, which restricts the domain to a one-to-one mapping, thereby narrowing the quantization range and improving quantization resolution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If quantization is applied to reduce memory footprint and computational resources, then resource efficiency is improved, but quantization error increases and model accuracy deteriorates
Solution Approach 1:
The patent changes the quantization parameters by determining practical domains for activation functions and setting quantization ranges based on actual data distributions rather than fixed or overly broad ranges. This involves analyzing the minimum and maximum values of activations in each layer and configuring quantization parameters (such as scale and zero-point) to match the actual data range, thereby reducing quantization error while maintaining low-bit representation
Solution Approach 2:
The patent performs preliminary analysis of activation function domains and ranges before applying quantization. By pre-determining the practical domains of activation functions and calculating appropriate quantization ranges in advance, the system prepares optimal quantization parameters that prevent excessive quantization error from occurring during model inference
2Adaptability or versatility
If quantization range is widened to cover all possible values, then coverage is improved, but quantization resolution deteriorates
Solution Approach 1:
The patent dynamically adjusts quantization parameters based on the actual data distribution in each layer. By changing the quantization range to match the practical domain of activation functions and the observed minimum-maximum values of activations, the system achieves fine quantization resolution without requiring an excessively wide range, thus avoiding the trade-off between coverage and precision
3Measurement precision
If inter-layer dependencies are considered for quantization optimization, then quantization accuracy is improved, but computational complexity increases
Solution Approach 1:
The patent performs preliminary analysis of inter-layer dependencies during the model preparation or training phase. By pre-determining the practical domains of activation functions and calculating appropriate quantization ranges in advance based on layer dependencies, the system avoids the need for complex real-time calculations during inference, thus achieving high quantization accuracy without excessive computational complexity during deployment
Data Source
AI summary
A processor-implemented method determines a practical domain for a following function in a following layer of an artificial neural network. The artificial neural network includes a leading function in a leading layer and the following function in the following layer, which is a subsequent consecutive layer of the artificial neural network. The method also sets a first quantization range of an output activation of the leading function based on the practical domain.