Sensitivity-Based Quantized Training for Deep Neural Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
State-of-the-art deep neural network (DNN) models are inefficient for execution in limited usage environments like mobile devices due to their large size, and existing quantization methods often result in performance degradation and increased complexity in finding optimal precision settings.
Innovation Solution
An electronic device and method that applies sensitivity-based quantization by determining the sensitivity of layers and channels in a DNN model, selectively processing layers or channels with low sensitivity at lower precision and those with high sensitivity at higher precision, thereby optimizing model training and inference efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If quantization is applied to reduce model size for mobile devices, then model efficiency and size are improved, but performance degradation occurs
Solution Approach 1:
The patent applies different quantization precisions to different layers based on their sensitivity characteristics. High-sensitivity layers maintain higher precision (e.g., FP16 or INT32) to preserve accuracy, while low-sensitivity layers use lower precision (e.g., INT8 or INT4) to reduce computational burden. This selective approach resolves the contradiction by locally optimizing precision per layer rather than applying uniform quantization across the entire model.
Solution Approach 2:
The patent dynamically adjusts the precision parameter of each layer based on sensitivity analysis results. By changing the precision parameter adaptively - using higher precision for critical layers and lower precision for non-critical layers - the system achieves both performance preservation and efficiency improvement, resolving the trade-off between accuracy and execution speed.
2Reliability
If uniform high precision is used for all layers, then model accuracy is maintained, but training time and computational resources increase
Solution Approach 1:
Instead of uniformly applying high precision to all layers, the patent identifies and selectively applies high precision only to layers with high sensitivity scores. This local optimization reduces the overall computational workload during training while maintaining accuracy in critical pathways, thereby reducing training time without sacrificing model performance.
Solution Approach 2:
The patent segments the neural network into multiple groups based on layer sensitivity, assigning different precision levels to each segment. This segmentation allows the training process to focus computational resources on high-sensitivity layers while using lower precision for low-sensitivity layers, significantly reducing total training time while preserving overall model accuracy.
3Productivity
If mixed precision quantization is applied to optimize DNN model, then model efficiency is improved, but complexity in finding optimal precision settings increases
Solution Approach 1:
The patent implements an automated sensitivity analysis mechanism that self-determines the optimal precision configuration for each layer based on their contribution to model output. This self-service approach eliminates the need for manual precision tuning by automatically identifying which layers require high precision and which can tolerate lower precision, thereby reducing configuration complexity while maintaining execution efficiency.
Solution Approach 2:
The patent employs feedback loops during training and evaluation to automatically adjust precision settings based on performance metrics. By monitoring model accuracy and computational efficiency, the system iteratively refines precision assignments for different layers, automatically converging on optimal configurations without requiring complex manual intervention or extensive hyperparameter searching.
Data Source
AI summary
An electronic device for performing sensitivity-based quantized training and an operating method thereof is disclosed. The electronic device includes a processor, and a memory configured to store instructions executable by the processor, wherein the processor is configured to, in response to the instructions being executed by the processor, generate, based on a determination of sensitivity of layers in a model to be trained, sensitivity results, and train the model by applying quantization to a layer of the layers with a low sensitivity of the sensitivity results lower than a predetermined threshold.