Graph-Based Neural Network Quantization-Aware Retraining for Edge NPUs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional neural network models face challenges such as high-power consumption, heat generation, bottlenecks in processor operations due to low memory bandwidth, latency, and difficulties in optimizing neural processing units (NPUs) for edge devices, including selection, compatibility, and performance prediction.
Innovation Solution
A method and apparatus for optimizing neural network models by quantization-aware retraining, involving adding markers to graph modules, generating calibration data, determining scale and offset values, and updating parameters to minimize loss, enabling efficient operation on low-power NPUs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If neural network models are trained with large amounts of data to improve inference accuracy, then the performance of neural network inference services is improved, but the data size, computational power, and training data requirements increase exponentially
Solution Approach 1:
The patent applies parameter changes by transforming neural network parameters from floating-point format to integer format through quantization. This changes the data representation parameters to reduce data size and computational requirements while maintaining inference accuracy. The quantization process converts continuous floating-point values to discrete integer values with reduced precision, achieving compression of model parameters.
Solution Approach 2:
The patent creates a quantized copy of the neural network model that uses integer arithmetic instead of the original floating-point model. This copying approach allows the model to be retrained with reduced precision parameters, enabling deployment on edge devices with limited computational resources while preserving the essential inference capabilities.
2Measurement precision
If the number of hidden layers is increased to realize higher artificial intelligence, then the capabilities and accuracy of neural networks are improved, but the computational power and processing time requirements increase
Solution Approach 1:
The patent changes the computational parameters by converting floating-point operations to integer operations through quantization. This parameter transformation reduces the computational power requirements for deep neural networks with multiple hidden layers, making them feasible for execution on edge devices with limited processing capabilities.
Solution Approach 2:
The patent applies partial action by performing quantization-aware retraining that focuses on optimizing only the critical parameters and pathways in the neural network. This selective approach allows the model to achieve high accuracy with fewer computational resources by identifying and optimizing the most important computational paths rather than uniformly processing all layers and parameters.
3Adaptability or versatility
If conventional neural network models are used, then the models can operate on standard hardware, but they consume high power and generate heat
Solution Approach 1:
The patent changes the data type parameters from floating-point to integer format, which fundamentally alters how the neural network performs computations. This parameter change enables the model to operate efficiently on low-power NPUs by using integer arithmetic that consumes significantly less energy than floating-point operations while maintaining compatibility with standard hardware through the quantization process.
4Measurement precision
If neural network models are optimized for high performance, then inference accuracy is improved, but memory bandwidth and processor operation bottlenecks increase
Solution Approach 1:
The patent changes the memory data representation from floating-point to integer format, which reduces the memory bandwidth requirements. Integer data requires less memory bandwidth for transfer and processing compared to floating-point data, thereby reducing the bottleneck effects on memory bandwidth while preserving inference accuracy through quantization-aware retraining.
Data Source
AI summary
A method may comprise: adding a plurality of markers to a plurality of graph modules in a first neural network (NN) model in a form of a directed acyclic graph (DAG); generating calibration data by collecting input values and output values of each of the plurality of graph modules using the plurality of markers; determining, based on the calibration data, a scale value and an offset value applicable to the first NN model; generating, based on the scale value and the offset value, a second NN model including a weight parameter in integer format through quantization; and updating at least one parameter included in the second NN model by performing a quantization-aware retraining technique on the second NN model.