NPU Scheduler for Parallel Multi-ANN Processing and Data Locality
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural processing units (NPUs) face inefficiencies when handling multiple artificial neural network (ANN) models, including increased idle time and prolonged computation processing times due to the lack of an optimized operation sequence.
Innovation Solution
A neural processing unit (NPU) with internal memory, processing elements, and a scheduler that allocates operations of multiple ANN models in sequence or parallel based on operation sequence information, allowing for efficient resource utilization and data locality.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a neural processing unit (NPU) is provided separately for each ANN model, then each ANN model can be processed independently, but the time that the NPU exists in an idle state is increased, which reduces efficiency
Solution Approach 1:
The patent combines multiple ANN model processing capabilities into a single NPU. The NPU is configured to store and execute multiple ANN models simultaneously, allowing different models to share the same hardware resources. This merging approach eliminates idle time between model executions while maintaining the ability to process each model independently when needed.
Solution Approach 2:
The NPU is designed as a universal processing unit capable of handling multiple different ANN models through a single device. The internal memory stores information on combinations of multiple ANN models, and the processing elements can be dynamically allocated to different models based on operational needs, making the NPU multi-functional rather than dedicated to a single model.
2Productivity
If computations of the plurality of ANN models are performed with one NPU, then resource utilization is improved, but absent the setting of an efficient operation sequence among the plurality of ANN models, a computation processing time is increased
Solution Approach 1:
The patent implements preliminary scheduling of ANN model operations before actual computation begins. The scheduler stores and manages operation sequence information for multiple ANN models in advance, determining the optimal execution order based on model dependencies and resource availability. This preliminary organization of computation sequences prevents idle waiting time during model execution.
Solution Approach 2:
The NPU employs dynamic scheduling capabilities that can adjust the execution sequence of ANN models based on real-time conditions. The scheduler can modify operation sequences dynamically, allocating processing elements to different models in an optimized order that minimizes total computation time while maintaining resource utilization efficiency.
3Speed
If multiple ANN models are processed in parallel, then processing speed is improved, but resource allocation complexity increases
Solution Approach 1:
The NPU divides its processing elements into multiple groups, with each group capable of independently processing different ANN models in parallel. This segmentation of processing resources allows simultaneous execution of multiple models while simplifying the allocation logic within each group, as each group operates semi-independently with its own scheduled task queue.
Data Source
AI summary
A neural processing unit (NPU) includes an internal memory storing information on combinations of a plurality of artificial neural network (ANN) models, the plurality of ANN models including first and second ANN models; a plurality of processing elements (PEs) to process first operations and second operations of the plurality of ANN models in sequence or in parallel, the plurality of PEs including first and second groups of PEs; and a scheduler to allocate to the first group of PEs a part of the first operations for the first ANN model and to allocate to the second group of PEs a part of the second operations for the second ANN model, based on an instruction related to information on an operation sequence of the plurality of ANN models or further based on ANN data locality information. The first and second operations may be performed in parallel or in a time division.