Method and storage medium for quantization aware retraining for graph-based neural network model using self-distillation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional neural network models face challenges such as high-power consumption, heat generation, memory bandwidth bottlenecks, latency, and computational inefficiencies, particularly in edge computing devices, due to large data sizes and increasing computational demands.
Innovation Solution
A method is developed to optimize neural network models by quantization-aware retraining, converting floating-point parameters to integer formats, and optimizing parameters of each layer to reduce power consumption and memory requirements, using a directed acyclic graph-based approach with markers and calibration data to determine scale and offset values.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If neural network models are trained with large data sizes and increased computational power to improve inference accuracy, then model performance is improved, but power consumption and heat generation increase
Solution Approach 1:
The patent applies parameter changes by converting floating-point model parameters to integer formats through quantization-aware retraining. This transformation changes the data representation from high-precision floating-point to lower-precision integer formats, reducing the computational power and memory bandwidth requirements while maintaining acceptable inference accuracy for edge device applications
2Measurement precision
If neural network models are trained with large data sizes and increased computational power to improve inference accuracy, then model performance is improved, but memory bandwidth requirements and latency increase
Solution Approach 1:
The patent changes the data type parameters of model weights and activations from floating-point to integer formats. This parameter transformation reduces the memory bandwidth requirements and decreases memory access latency, enabling faster data transfer between memory and processing units in edge devices
3Use of energy by moving object
If neural network models are optimized for low-power NPUs by reducing model size and using integer arithmetic, then power consumption is reduced, but computational precision may be compromised
Solution Approach 1:
The patent implements feedback through quantization-aware retraining, where the model is retrained using quantization loss functions that provide feedback on the impact of quantization on model performance. This feedback mechanism allows the system to adjust quantization parameters to maintain computational precision while using integer arithmetic for low-power processing
4Measurement precision
If neural network models are trained with large data sizes to improve performance, then inference accuracy is improved, but device complexity and computational demands increase
Solution Approach 1:
The patent changes the computational parameters by transforming the model to use integer arithmetic instead of floating-point operations. This parameter change simplifies the computational complexity and reduces the requirements for processing units and memory bandwidth, making the models suitable for edge devices with limited computational resources
Data Source
AI summary
A method may comprise: adding a plurality of markers to a plurality of graph modules in a first neural network (NN) model in a form of a directed acyclic graph (DAG); generating calibration data by collecting input values and output values of each of the plurality of graph modules using the plurality of markers; determining, based on the calibration data, a scale value and an offset value applicable to the first NN model; generating, based on the scale value and the offset value, a second NN model including a weight parameter in integer format through quantization; obtaining first output values of the first NN model with respect to a first retraining data; and updating, based on the first output values of the first NN model, at least one weight parameter included in the second NN model by performing a quantization-aware retraining technique on the second NN model.