Neural Network Model Conversion to NPU Format
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network models face challenges in being converted to formats executable on neural processing units (NPUs), as different deep learning frameworks use specific formats, leading to inefficiencies and suboptimal performance due to varying representation and format compatibility issues.
Innovation Solution
A method and apparatus that utilize conversion tools and optimization techniques to convert neural network models from various formats to a format supported by NPUs, employing ONNX format and hardware instruction mapping, along with quantization and precision determination to optimize performance, and determining the optimal format and data quantity for execution on NPUs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If neural network models are converted to ONNX format as an intermediate representation, then format compatibility across different deep learning frameworks is improved, but conversion complexity and processing time increase
Solution Approach 1:
The patent employs ONNX as an intermediary format between various deep learning framework formats (PyTorch, TensorFlow, Caffe) and the target NPU executable format. This intermediary layer standardizes the conversion process, allowing different source formats to be uniformly transformed through a common intermediate representation before final conversion to NPU-specific format, thereby improving compatibility while managing complexity through standardization.
Solution Approach 2:
The conversion process is segmented into distinct stages: (1) conversion from source framework format to ONNX intermediate format, (2) conversion from ONNX to target NPU format. This segmentation allows each conversion stage to be optimized independently, with the ONNX stage handling framework-specific conversions and the second stage handling hardware-specific optimizations, reducing overall complexity.
2Loss of time
If neural network models are directly converted to NPU executable format without intermediate conversion, then processing time is reduced, but format compatibility and optimization capability deteriorate
Solution Approach 1:
The patent performs preliminary conversion to ONNX format before final conversion to NPU executable format. This preliminary action standardizes the model representation and enables subsequent optimization steps (quantization, format conversion, hardware instruction mapping) to be performed more efficiently, reducing total conversion time while improving compatibility.
Solution Approach 2:
ONNX serves as a mediator that enables efficient conversion from diverse framework formats to NPU format. The intermediary format allows for optimized conversion paths that leverage ONNX's standardization benefits, reducing conversion time compared to direct conversion while maintaining broad compatibility.
3Quantity of substance
If quantization is applied to reduce data precision, then memory consumption and calculation iterations are reduced, but measurement precision and model accuracy deteriorate
Solution Approach 1:
The patent applies quantization by changing the data representation parameters from floating-point to fixed-point formats with reduced precision. This parameter change reduces memory consumption and calculation iterations by using smaller data types (e.g., INT8 instead of FP32), while the impact on accuracy is managed through controlled quantization processes that minimize information loss.
Solution Approach 2:
The patent applies quantization selectively to different parts of the neural network model based on their importance and sensitivity to precision changes. Critical layers maintaining high accuracy are preserved with higher precision, while less critical layers are quantized more aggressively, achieving a balance between memory efficiency and accuracy.
4Adaptability or versatility
If multiple conversion formats are supported for different deep learning frameworks, then adaptability is improved, but device complexity and conversion overhead increase
Solution Approach 1:
The patent implements a universal conversion system where a single conversion pipeline handles multiple deep learning framework formats (PyTorch, TensorFlow, Caffe) by first converting them to the common ONNX intermediate format, then to the target NPU format. This multi-functional approach consolidates conversion logic and reduces system complexity compared to maintaining separate conversion paths for each framework.
Solution Approach 2:
ONNX acts as a universal intermediary that serves as a common language between different framework formats and the NPU executable format. This intermediary enables a single conversion system to handle multiple source formats through a standardized intermediate representation, improving adaptability while managing complexity through standardization.
Data Source
AI summary
Disclosed is a method of processing information in an electronic apparatus, the method including acquiring a neural network model, determining a reference format for conversion of the neural network model, and converting the neural network model to a model of the reference format, wherein the model converted into the reference format is executed in a neural processing unit (NPU).