Neural Network DAG Conversion for Low-Power NPU Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network models face challenges in low-power neural processing units due to high computational demands, power consumption, and memory bottlenecks, particularly in edge devices requiring immediate and reliable performance.
Innovation Solution
The neural network model is updated and optimized for execution on a neural processing unit by converting it into a directed acyclic graph, applying quantization, outlier alleviation, parameter refinement, layer-wise training, and pruning, and compiling it for efficient execution on the unit, including operations like multiply-and-accumulation, convolution, and matrix multiplication.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If neural network models are used with high accuracy and complexity, then inference performance is improved, but power consumption and computational demands increase
Solution Approach 1:
The patent applies quantization to change the parameter precision of the neural network model from high-precision floating-point numbers to low-precision integers. This parameter transformation reduces the computational complexity and power consumption while maintaining acceptable inference accuracy. The compiler automatically determines quantization parameters and applies them to convert the model for efficient execution on low-power NPUs.
2Use of energy by moving object
If neural network models are optimized for low-power devices, then power consumption is reduced, but model performance and accuracy may deteriorate
Solution Approach 1:
The patent performs preliminary optimization actions during the compilation phase before the model is deployed to the low-power NPU. The compiler analyzes the model structure, determines appropriate quantization parameters, and applies optimizations in advance. This preliminary action ensures that the model is pre-adapted for low-power execution while maintaining accuracy through careful parameter selection and model transformation.
3Productivity
If neural network models are converted to directed acyclic graph format with quantization, then execution efficiency on NPU is improved, but model complexity increases
Solution Approach 1:
The patent introduces a compiler as an intermediary tool that automatically handles the complex transformation process. The compiler converts the neural network model into directed acyclic graph format, applies quantization, and optimizes the model structure. This intermediary automation reduces the manual complexity burden on developers while achieving improved execution efficiency on the NPU through systematic model transformation.
Data Source
AI summary
A method may comprise receiving a first neural network (NN) model including one or more functions; generating a second NN model in a form of directed acyclic graph (DAG) including one or more graph modules by converting the one or more functions; calculating one or more scale values by obtaining maximum and minimum values of parameters input to the one or more graph modules; updating the parameters based on the one or more scale values; and generating a third NN model, in a form of machine code executable on a particular neural processing unit, including the updated parameters.