Mixed-Precision Neural Network Training with Sensitivity-Based Quantization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks require significant memory and computational resources, making them unsuitable for resource-limited devices like smartphones and tablets, and existing methods for optimizing them, such as low-rank factorization and quantization, often result in suboptimal performance due to reliance on expert experience and manual thresholds.
Innovation Solution
A method for training a neural network model that involves pre-training with quantization units of varying bit widths, calculating sensitivity to adjust bit widths, and retraining to generate a mixed-precision model that meets computational constraints, using sensitivity measurements to iteratively refine the model.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If deep neural network models are deployed on resource-limited embedded devices, then recognition and generalization capabilities are achieved, but memory overhead and processor resource consumption become excessive
Solution Approach 1:
The patent applies quantization by changing the parameter precision of network weights and activations from high precision (32-bit floating point) to low precision (8-bit integer or lower), thereby reducing memory overhead and computational resource requirements while maintaining acceptable recognition performance on embedded devices
2Productivity
If quantization bit width is reduced to decrease memory footprint and increase throughput, then resource consumption is reduced, but quantization error increases and inference precision deteriorates
Solution Approach 1:
The patent implements mixed-precision quantization where different layers or channels of the neural network are assigned different quantization bit widths based on their importance and sensitivity to quantization, allowing critical layers to maintain higher precision while less critical layers use lower precision for resource efficiency
Solution Approach 2:
The patent introduces learnable scaling parameters that are dynamically adjusted during training to compensate for quantization effects, allowing the model to adaptively optimize precision allocation and maintain inference accuracy across varying quantization levels
3Measurement precision
If automatic determination of optimal hierarchical precision allocation is performed, then balance between efficiency and precision is achieved, but training complexity and computational overhead increase
Solution Approach 1:
The patent employs self-supervised learning mechanisms where the model automatically learns optimal precision allocation through internal consistency constraints and reconstruction losses, eliminating the need for manual hyperparameter tuning or complex external optimization algorithms
Data Source
AI summary
The present disclosure provides a training method and an application method of a neural network model, a training apparatus and an application apparatus of a neural network model, a storage medium, and a computer program product. The training method comprises: a pre-training step of pre-training the neural network model so that the neural network model includes at least one quantization unit, wherein each quantization unit contains a plurality of different quantization bit widths; a calculation step of calculating a sensitivity of the quantization unit, and updating the quantization bit width of each quantization unit based on the calculated sensitivity and updating a quantization parameter, thereby generating a mixed-precision neural network model, wherein the sensitivity indicates the extent to which the quantization bit width of the quantization unit affects a network output; and a retraining step of retraining the generated mixed-precision neural network model.