Shared NPU Scheduling for Multi-Stage Video Quality Improvement
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural processing units (NPUs) for artificial neural network (ANN) models face inefficiencies due to idle time and increased computation processing time when handling multiple models without an optimized operation sequence.
Innovation Solution
A neural processing unit (NPU) with internal memory and scheduler to manage multiple ANN models, allowing parallel or time-division processing of operations based on operation sequence information, and a central processing unit (CPU) to allocate tasks efficiently across multiple NPUs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a neural processing unit (NPU) is provided separately for each ANN model, then each model can be processed independently, but the time that the NPU exists in an idle state is increased, which reduces efficiency
Solution Approach 1:
The patent merges multiple dedicated NPUs into a single shared NPU that can handle multiple ANN models. The NPU is configured to load and execute different model parameters from memory for different applications, allowing one physical NPU to perform the functions of multiple dedicated NPUs, thereby eliminating idle time and improving resource utilization.
Solution Approach 2:
The NPU is designed with universal functionality to support multiple different ANN models. It can dynamically load different model parameters (weights, biases, architecture configurations) from system memory to handle different applications such as object detection, image classification, and speech recognition, making a single NPU serve multiple purposes.
2Productivity
If computations of the plurality of ANN models are performed with one NPU, then resource utilization improves, but absent the setting of an efficient operation sequence among the plurality of ANN models, computation processing time is increased
Solution Approach 1:
The system performs preliminary actions by pre-loading model parameters into the NPU's buffer memory before computation is needed. The CPU prepares and transfers parameters for multiple models in advance, and the NPU schedules computations to minimize idle time by having parameters ready before each computation task begins.
Solution Approach 2:
The NPU employs dynamic scheduling to adapt its operation sequence based on current workload conditions. It can switch between different models and parameter sets dynamically, adjusting the computation sequence to optimize processing time while maintaining high resource utilization. The system adapts parameter loading and computation timing based on real-time demands.
3Measurement precision
If model parameters are repeatedly loaded from system memory during computation, then model accuracy is maintained, but power consumption increases due to frequent memory access
Solution Approach 1:
The memory system is segmented into hierarchical levels: system memory for long-term storage and NPU buffer memory for active computation. Model parameters are segmented between these memory levels based on usage frequency - frequently accessed parameters are kept in the faster NPU buffer memory, while less frequently used parameters remain in system memory.
Solution Approach 2:
The system performs preliminary loading of model parameters into the NPU's buffer memory before computation begins. This pre-loading action reduces the need for repeated accesses to system memory during computation, thereby reducing power consumption while maintaining model accuracy through consistent parameter access.
Data Source
AI summary
A neural processing unit (NPU) includes an internal memory storing information on combinations of a plurality of artificial neural network (ANN) models, the plurality of ANN models including first and second ANN models; a plurality of processing elements (PEs) to process first operations and second operations of the plurality of ANN models in sequence or in parallel, the plurality of PEs including first and second groups of PEs; and a scheduler to allocate to the first group of PEs a part of the first operations for the first ANN model and to allocate to the second group of PEs a part of the second operations for the second ANN model, based on an instruction related to information on an operation sequence of the plurality of ANN models or further based on ANN data locality information. The first and second operations may be performed in parallel or in a time division.