Graph-Based Neural Network Quantization for Low-Power NPUs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional neural network models face challenges such as high-power consumption, heat generation, bottlenecks in processor operations due to low memory bandwidth, latency, and difficulties in optimizing neural processing units (NPUs) for edge devices, which are exacerbated by the increasing data size and computational demands of AI inference services.
Innovation Solution
A method is provided to optimize neural network models by converting them into directed acyclic graphs, adding markers, generating calibration data, determining optimal scale and offset values through quantization simulations, and optimizing parameters to reduce power consumption and memory usage, enabling faster determination of optimal NPU selection and model optimization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional neural network models are used with increased data size and computational power, then AI inference performance is improved, but power consumption and heat generation increase significantly
Solution Approach 1:
The patent applies parameter changes by converting neural network parameters from floating-point format to integer format through quantization. This changes the numerical representation parameters, reducing the precision requirements while maintaining acceptable inference performance. The quantization process transforms high-precision floating-point numbers into lower-precision integers, directly addressing the power consumption issue by enabling efficient computation on low-power NPUs.
Solution Approach 2:
The patent extracts and optimizes specific components of the neural network model for deployment on resource-constrained devices. By separating the model optimization process into distinct stages (conversion to DAG, marker addition, calibration data generation, quantization simulation), the system can extract and apply optimizations tailored for low-power NPUs without retraining the entire model.
2Measurement precision
If conventional neural network models are used with increased data size, then AI inference accuracy is improved, but memory bandwidth bottlenecks and latency increase
Solution Approach 1:
The patent changes the data representation parameters from floating-point to integer format, which reduces memory bandwidth requirements. Integer operations require less memory bandwidth compared to floating-point operations, thereby alleviating the memory bandwidth bottleneck while maintaining inference accuracy through careful quantization calibration.
Solution Approach 2:
The patent performs preliminary actions by converting the neural network model to a directed acyclic graph (DAG) representation and adding markers before quantization. This preliminary structuring enables efficient memory access patterns and optimization opportunities that reduce latency during actual inference execution on NPUs.
3Use of energy by moving object
If neural network models are optimized for low-power NPUs, then power consumption is reduced, but model accuracy may deteriorate
Solution Approach 1:
The patent implements feedback mechanisms through calibration data generation and quantization simulation. The system generates calibration data from the optimized model, simulates quantization effects, and uses this feedback to adjust quantization parameters (scale and offset values) to minimize accuracy loss. This iterative feedback process ensures that the quantized model maintains high accuracy while running on low-power NPUs.
Solution Approach 2:
The patent applies partial quantization strategies, quantizing only certain layers or parameters of the neural network while keeping others in higher precision. This selective approach allows the system to achieve power consumption benefits from quantization in critical paths while maintaining accuracy in layers where precision is more important, balancing the trade-off between power efficiency and model accuracy.
4Productivity
If quantization optimization is performed for each NPU, then NPU-specific performance is improved, but optimization time and complexity increase
Solution Approach 1:
The patent performs preliminary actions by converting the neural network model to a directed acyclic graph (DAG) representation and adding markers before quantization. This preliminary structuring enables efficient memory access patterns and optimization opportunities that reduce latency during actual inference execution on NPUs.
Solution Approach 2:
The patent creates a universal optimization framework that can be applied across different NPU architectures. The quantization simulation and calibration process is designed to be architecture-agnostic, allowing the same optimization pipeline to generate NPU-specific optimizations without requiring complete re-optimization for each device, thereby reducing overall optimization time and complexity.
Data Source
AI summary
A method comprises: converting a plurality of functions or function call instructions of a first neural network (NN) model into a plurality of graph modules; analyzing a relationship between one or more inputs and one or more outputs of the plurality of graph modules; generating a second neural network (NN) model in a form of a directed acyclic graph (DAG) using the plurality of graph modules corresponding to the first NN model, by mapping the one or more inputs and the one or more outputs of the plurality of graph modules to each other based on the relationship; adding a plurality of markers to the plurality of graph modules in the second NN model; and generating calibration data by collecting input values and output values of each of the plurality of graph modules using the plurality of markers.