Heterogeneous Neural Network Inference With GPGPU-to-NPU Weight Transfer
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The challenges of enhancing computational performance of neural network models are hindered by excessive power consumption, significant heat generation, processor bottlenecks, and memory latency, particularly in edge AI devices, and there is a lack of efficient methods to train and optimize neural network models for neural processing units (NPUs) due to differing calculation circuits and unpredictable performance.
Innovation Solution
A method is introduced to train a neural network model on a general-purpose graphics processing unit (GPGPU) and transfer weights to a neural processing unit (NPU) for inference, with online evaluation and optimization to improve accuracy and power efficiency, using compilation options like pruning, quantization, and AI-based optimization algorithms.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If neural network models are trained and executed on high-performance processors, then computational performance and accuracy are improved, but power consumption and heat generation increase excessively
Solution Approach 1:
The patent divides the neural network processing into two distinct stages: training phase executed on GPGPU and inference phase executed on NPU. This segmentation allows each processor to be optimized for its specific function, with the NPU consuming significantly less power during inference operations compared to running full training on high-performance processors.
Solution Approach 2:
The patent introduces weights as an intermediary data structure that is trained on the GPGPU and then transferred to the NPU for inference. This intermediary allows the computational heavy lifting to be done once on the power-intensive GPGPU, while the NPU performs lower-power operations using the pre-trained weights, thereby reducing overall power consumption.
2Use of energy by moving object
If neural network models are trained on GPGPU and deployed on NPU with different calculation circuits, then power efficiency is improved, but inference accuracy becomes unpredictable
Solution Approach 1:
The patent implements a feedback mechanism where inference results from the NPU are evaluated and used to update the weights back on the GPGPU. This closed-loop feedback allows the system to detect and correct any accuracy deviations caused by the different calculation circuits between GPGPU and NPU, ensuring consistent inference accuracy while maintaining power efficiency.
Solution Approach 2:
The patent performs preliminary training of the neural network weights on the GPGPU before deploying them to the NPU for inference. This preliminary action ensures that the weights are optimized for the target NPU's calculation circuit, preventing accuracy issues from arising during actual inference operations.
3Use of energy by moving object
If neural network models are optimized for NPU using compilation options like pruning and quantization, then power consumption is reduced, but model complexity increases
Solution Approach 1:
The patent applies compilation options such as pruning and quantization that change the parameters of the neural network model. Pruning removes unnecessary connections and quantization reduces the precision of weight representations, both of which reduce power consumption while the systematic application of these transformations keeps the model complexity manageable through automated optimization processes.
Data Source
AI summary
A method for training a neural network model based on heterogeneous processing units may comprise: training a neural network model with lightweight weights to be executable on a second processing unit by utilizing a training dataset on a first processing unit; and performing inference on an evaluation dataset using the lightweight weights on the second processing unit.