Terminal Neural Network Deployment for Cross-Framework Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network model deployment methods face challenges in achieving high speed and real-time performance due to the need for coordination between terminal devices and servers, leading to network delays and hardware costs.
Innovation Solution
A neural network model deployment method that redefines each network layer of an initial model using a Layer class and connects them with a Net class, allowing deployment of models trained by different learning frameworks directly on terminal devices, eliminating the need for server coordination.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If neural network model prediction is performed on a server, then model training flexibility is maintained, but prediction speed and real-time performance deteriorate due to network coordination requirements
Solution Approach 1:
The patent segments the neural network model into multiple network layers, where the structure is defined by a Layer class and instantiated by specific layer types. This segmentation allows the model to be decomposed into independent, deployable units that can be efficiently executed on terminal devices while maintaining the original model's predictive capabilities.
Solution Approach 2:
The patent introduces an intermediary deployment process that converts models trained in frameworks like TensorFlow or PyTorch into a universal format suitable for terminal devices. This intermediary step includes generating a layer definition file that describes the network structure and a parameter file containing trained weights, enabling seamless migration from server-based training to device-based inference without losing model accuracy.
2Reliability
If neural network models are deployed on terminal devices, then real-time performance improves, but compatibility with different learning frameworks deteriorates
Solution Approach 1:
The patent creates a universal Layer class that can represent network layers from different learning frameworks (TensorFlow, PyTorch, Caffe, etc.). This universal representation allows models trained in various frameworks to be deployed on terminal devices with consistent performance. The layer definition file format is designed to be framework-agnostic, capturing essential layer characteristics without being tied to any specific framework's proprietary format.
3Adaptability or versatility
If models are converted to a universal format, then deployment universality improves, but conversion process complexity increases
Solution Approach 1:
The patent extracts only the essential information needed for inference from the original trained models, separating the layer structure definition from the parameter storage. The layer definition file contains only the architectural information (layer types, connections, configurations), while the parameter file stores the trained weights. This extraction simplifies the conversion process by focusing on what is truly necessary for deployment rather than trying to preserve all framework-specific details.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Embodiments of this application provide a neural network model deployment method, a prediction method and a device. The method includes: reading an initial neural network model, to obtain a layer definition of each network layer of the initial neural network model and an operation parameter of each network layer; implementing, by using a Layer class, a target network layer corresponding to each network layer in the terminal device separately according to the layer definition of each network layer, so that each target network layer is inherited from the Layer class; connecting the target network layers by using a Net class; converting the operation parameter of each network layer into a preset format, to obtain a target operation parameter of each network layer; and loading a corresponding target operation parameter in the target network layer corresponding to each network layer separately according to the target operation parameter of each network layer, to obtain a target neural network model deployed in the terminal device. The embodiments of this application can implement deployment of a neural network model to a terminal device, and improve the universality of the deployment of the neural network model to the terminal device.