Edge NPU Scheduling for Parallel Multi-ANN Execution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems face inefficiencies when using multiple artificial neural network (ANN) models due to idle NPU states and lack of an optimized operation sequence, leading to increased computation processing time.
Innovation Solution
A neural processing unit (NPU) with internal memory and scheduler that allocates operations of multiple ANN models in sequence or parallel based on operation sequence information, allowing for efficient resource utilization and reduced power consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a neural processing unit (NPU) is provided separately for each ANN model, then the processing capability for each model is improved, but the time that the NPU exists in an idle state is increased, which reduces efficiency
Solution Approach 1:
The patent combines multiple NPUs into a single integrated NPU that can process multiple ANN models. The NPU includes multiple processing elements (PEs) that can be dynamically allocated to different ANN models, allowing multiple models to share the same hardware resource and reducing idle time while maintaining processing capability.
Solution Approach 2:
The NPU is designed with multi-functionality to handle various ANN models using a single device. The processing elements can be configured to process different types of neural network operations, making the NPU universal rather than dedicated to a single model, thus improving resource utilization.
2Productivity
If computations of the plurality of ANN models are performed with one NPU, then resource utilization is improved, but absent the setting of an efficient operation sequence among the plurality of ANN models, a computation processing time is increased
Solution Approach 1:
The system performs preliminary actions by pre-determining the operation sequence of multiple ANN models before execution. The NPU scheduler analyzes the dependencies and characteristics of different ANN models to establish an optimal processing order, preventing unnecessary waiting time and improving overall computation efficiency.
Solution Approach 2:
The NPU employs dynamic scheduling to adjust the operation sequence of ANN models based on real-time conditions. The scheduler can dynamically allocate processing elements and adjust execution priorities, allowing the system to adapt to varying computational demands and minimize processing time.
3Speed
If multiple ANN models are processed in parallel, then processing speed is improved, but resource allocation complexity increases
Solution Approach 1:
The NPU is segmented into multiple independent processing elements (PEs) that can operate in parallel. Each PE can handle specific operations for different ANN models simultaneously, enabling parallel processing while keeping individual processing units simple and manageable.
Data Source
AI summary
A neural processing unit (NPU) includes an internal memory storing information on combinations of a plurality of artificial neural network (ANN) models, the plurality of ANN models including first and second ANN models; a plurality of processing elements (PEs) to process first operations and second operations of the plurality of ANN models in sequence or in parallel, the plurality of PEs including first and second groups of PEs; and a scheduler to allocate to the first group of PEs a part of the first operations for the first ANN model and to allocate to the second group of PEs a part of the second operations for the second ANN model, based on an instruction related to information on an operation sequence of the plurality of ANN models or further based on ANN data locality information. The first and second operations may be performed in parallel or in a time division.