AI-ISP Neural Processing Scheduler for Multi-Model Parallel Execution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural processing units (NPUs) face inefficiencies when handling multiple artificial neural network (ANN) models, including increased idle time and prolonged computation processing times due to the lack of an optimized operation sequence.
Innovation Solution
A neural processing unit (NPU) with internal memory and scheduler that can store information on multiple ANN models, allowing parallel or time-division processing of operations across different groups of processing elements (PEs), and a central processing unit (CPU) that allocates tasks based on operation sequences.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a neural processing unit (NPU) is provided separately for each ANN model, then the processing capability for each model is improved, but the idle time of the NPU increases and efficiency is reduced
Solution Approach 1:
The patent combines multiple ANN models into a single NPU through a unified processing architecture. The NPU includes a control unit that manages multiple ANN models and a processing unit that can execute operations for different models, merging previously separate processing resources into one integrated system that handles multiple models simultaneously.
Solution Approach 2:
The NPU is designed as a universal processing unit capable of handling multiple different ANN models through a standardized interface. The control unit receives operation information for various ANN models and routes them to the processing unit, which can execute different types of neural network operations (convolution, fully connected, pooling) for different models using the same hardware resources.
2Productivity
If computations of the plurality of ANN models are performed with one NPU, then resource utilization is improved, but the computation processing time increases due to lack of efficient operation sequencing
Solution Approach 1:
The control unit performs preliminary scheduling of operations for multiple ANN models before execution. It analyzes the operation information from each model, determines the optimal execution sequence, and prepares the processing schedule in advance. This preliminary action allows the NPU to execute operations in an optimized sequence, reducing overall computation processing time while maintaining high resource utilization.
3Productivity
If operations of multiple ANN models are processed in sequence, then resource utilization is improved, but the processing speed decreases
Solution Approach 1:
The NPU implements periodic action by alternating between processing operations of different ANN models in a scheduled manner. The control unit divides the processing into periodic time slots, where the processing unit executes operations for one model during a specific period, then switches to another model in the next period. This periodic switching enables efficient resource utilization while maintaining acceptable processing speeds for all models.
Data Source
AI summary
A neural processing unit (NPU) includes an internal memory storing information on combinations of a plurality of artificial neural network (ANN) models, the plurality of ANN models including first and second ANN models; a plurality of processing elements (PEs) to process first operations and second operations of the plurality of ANN models in sequence or in parallel, the plurality of PEs including first and second groups of PEs; and a scheduler to allocate to the first group of PEs a part of the first operations for the first ANN model and to allocate to the second group of PEs a part of the second operations for the second ANN model, based on an instruction related to information on an operation sequence of the plurality of ANN models or further based on ANN data locality information. The first and second operations may be performed in parallel or in a time division.