Inference Neural Network Quantization for Low-Precision Accuracy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks face challenges in deployment due to high computational complexity, with low-precision inference models lacking suitable accuracy and having excessive memory or bandwidth requirements.
Innovation Solution
Implement asymmetric quantization and per-input channel quantization techniques to optimize inference neural networks, converting high-precision data to low-precision integer values for efficient computation, and using weights accumulation tables for accurate restoration.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If high-precision neural network models are deployed, then computational accuracy is maintained, but computational complexity and memory requirements become excessive
Solution Approach 1:
The patent applies parameter changes by transitioning from high-precision floating-point arithmetic to low-precision integer arithmetic (e.g., INT8). This involves quantizing model weights and activations to reduce the number of bits used for representation, thereby decreasing computational complexity and memory requirements while maintaining acceptable accuracy through carefully designed quantization strategies that preserve critical numerical relationships in the neural network computations
2Device complexity
If low-precision inference models are used, then computational complexity is reduced, but accuracy becomes insufficient
Solution Approach 1:
The patent implements local quality by applying different quantization strategies to different parts of the neural network based on their specific requirements. Critical layers or weight matrices that demand higher precision are retained at higher precision levels, while less sensitive components are quantized to lower precision. This selective approach allows the model to achieve low overall computational complexity while preserving accuracy in regions where it matters most
3Quantity of substance
If low-precision integer values are used for computation, then memory requirements are reduced, but data restoration accuracy becomes challenging
Solution Approach 1:
The patent applies preliminary action by pre-computing and storing lookup tables (LUTs) during the model training phase. These LUTs contain pre-calculated quantization and de-quantization transformations that can be quickly retrieved during inference. This preliminary preparation eliminates the need for complex real-time calculations during data restoration, significantly reducing memory requirements while maintaining high restoration accuracy through the use of these pre-computed reference tables
Data Source
AI summary
Systems, apparatuses and methods may provide technology for optimizing an inference neural network model that performs asymmetric quantization by generating a quantized neural network, wherein model weights of the neural network are quantized as signed integer values, and wherein an input layer of the neural network is configured to quantize input values as unsigned integer values, generating a weights accumulation table based on the quantized model weights and a kernel size for the neural network, and generating an output restoration function for an output layer of the neural network based on the weights accumulation table and the kernel size. The technology may also perform per-input channel quantization. The technology may also perform mixed-precision auto-tuning.