DNN Quantization With Dynamic Bit Allocation for Edge Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing quantization techniques for Deep Neural Network (DNN) models are time-consuming, require access to training data, and are vendor-dependent, leading to inefficiencies and reduced accuracy on resource-constrained devices like smartphones and edge devices, especially when using INT8 quantization on Digital Signal Processor (DSP) and Neural Processing Unit (NPU) hardware.
Innovation Solution
A novel post-training quantization method that generates self-generated data to estimate activation ranges and sensitive indices for DNN models, allowing per-tensor and per-channel configurations, and flexible bit allocation, reducing model size by 4× while maintaining accuracy, and enabling execution on GPUs, DSPs, and NPUs without retraining.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If post-training quantization is performed using existing techniques, then model size is reduced, but accuracy deteriorates due to INT8 quantization limitations on DSP and NPU hardware
Solution Approach 1:
The patent changes the quantization parameters by implementing a hybrid quantization scheme that uses different precision levels (INT8 and INT16) for different parts of the neural network. Specifically, it applies INT8 quantization to convolutional layers while maintaining INT16 for fully connected layers, optimizing the balance between model size reduction and accuracy preservation for DSP and NPU hardware execution
Solution Approach 2:
The patent applies local quality by treating different layers of the neural network differently in terms of quantization precision. Instead of uniform quantization across all layers, it performs layer-wise analysis to determine which layers can tolerate INT8 quantization and which require INT16, thereby locally optimizing both size reduction and accuracy maintenance
2Reliability
If quantization aware training is performed to maintain accuracy, then accuracy is preserved, but training time increases significantly due to complete retraining requirement
Solution Approach 1:
The patent applies preliminary action by performing quantization calibration on a pre-trained model using a small calibration dataset before deployment. This calibration process estimates the quantization parameters (scale and zero-point) for each layer without requiring complete retraining, thereby preserving accuracy while significantly reducing the time investment compared to quantization-aware training
Solution Approach 2:
The patent uses partial action by applying quantization to only certain layers of the neural network rather than all layers. By selectively quantizing convolutional layers while keeping fully connected layers in higher precision, it achieves most of the compression benefits with minimal impact on accuracy and without the full time cost of complete model retraining
3Reliability
If INT16 quantization is used to maintain accuracy on DSP and NPU hardware, then accuracy is preserved, but model size reduction is insufficient
Solution Approach 1:
The patent applies local quality by differentiating quantization precision across different layer types. Convolutional layers, which can tolerate lower precision due to their local operation nature and hardware optimization, are quantized to INT8 for maximum size reduction. Fully connected layers, which have more global influence on output accuracy, are maintained at INT16 precision, achieving optimal balance between size and accuracy
4Ease of manufacture
If vendor-dependent quantization techniques are used, then quantization can be performed, but data sharing requirements create privacy concerns and vendor lock-in
Solution Approach 1:
The patent applies self-service by implementing a complete, independent quantization pipeline that can be executed locally on the user's device without requiring vendor-specific tools or data sharing. The system performs model analysis, calibration, and quantization using open-source libraries and algorithms, enabling privacy-preserving deployment while avoiding vendor lock-in
Solution Approach 2:
The patent creates a universal quantization framework that is vendor-agnostic and can be applied to any neural network model regardless of the original training framework. The hybrid quantization approach and calibration methodology are framework-independent, allowing the same pipeline to work with models from different vendors and platforms, thereby enhancing versatility and privacy compatibility
Data Source
AI summary
Various embodiments of the disclosure disclose a method for quantizing a Deep Neural Network (DNN) model in an electronic device. The method includes: estimating, by the electronic device, an activation range of each layer of the DNN model using self-generated data (e.g. retro image, audio, video, etc.) and/or a sensitive index of each layer of the DNN model; quantizing, by the electronic device, the DNN model based on the activation range and/or the sensitive index; and allocating, by the electronic device, a dynamic bit precision for each channel of each layer of the DNN model to quantize the DNN model.