Offline Neural Network Execution to Eliminate Repeated Compilation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deep learning systems face inefficiencies in processing speed and performance due to the need for repeated compilation and analysis of neural network models on processors like CPU or GPU, affecting the overall processing efficiency.
Innovation Solution
A neural network processing method that involves acquiring model datasets and structure parameters, determining operational attributes of compute nodes, and generating offline models for specific and general-purpose processors to enable direct execution without recompilation, utilizing nonvolatile memory for storing network weights and instructions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If neural network models are compiled and analyzed every time on CPU or GPU, then the system can execute various neural network models flexibly, but the processing speed decreases and processing efficiency becomes low
Solution Approach 1:
The patent pre-compiles and pre-analyzes neural network models into offline models during a preprocessing stage. These offline models are stored in nonvolatile memory and can be directly executed without repeated compilation and analysis, thereby resolving the contradiction between model execution flexibility and processing efficiency
Solution Approach 2:
The patent creates offline model copies from the original neural network models. These copies contain pre-processed computation instructions that can be directly executed on application-specific neural network processors, eliminating the need for repeated compilation while maintaining model execution capability
2Adaptability or versatility
If neural network models are compiled and analyzed every time, then the system can adapt to different model structures, but the time consumption increases
Solution Approach 1:
The patent performs model compilation and analysis in advance during an offline preprocessing stage. The computation instructions are generated and stored beforehand, eliminating the time-consuming compilation process during actual model execution while preserving adaptability to different model structures through the preprocessing stage
3Adaptability or versatility
If general purpose processors are used to execute neural network compute nodes, then the system has high versatility, but the processing speed is limited
Solution Approach 1:
The patent segments compute nodes into two categories: those that can be executed on application-specific neural network processors and those that must be executed on general-purpose processors. This segmentation allows the system to utilize specialized hardware for speed-critical operations while maintaining versatility through general-purpose processor support
Solution Approach 2:
The patent introduces offline models as an intermediary layer between general-purpose processors and application-specific neural network processors. The offline models contain pre-compiled instructions that enable efficient execution on specialized hardware while maintaining the ability to run on general-purpose processors when needed
Data Source
AI summary
A neural network processing method, comprising the following steps: obtaining a model dataset and model structure parameters of an original network (S100); obtaining an operational attribute of each compute node in the original network; operating the original network according to the model dataset and the model structure parameters of the original network and the operational attribute of each compute node, to obtain an instruction corresponding to each compute node in the original network (S200); and if the operational attribute of the current compute node is a first operational attribute, storing a network weight and the instruction corresponding to the current compute node into a first non-volatile memory, so as to obtain a first offline model corresponding to the original network (S300). Further provided are a computer system and a storage medium. The neural network processing method, the computer system, and the storage medium shorten the time for a processor to operate the same network, and improve the processing speed and efficiency of the processor.