Heterogeneous Processor Scheduling for Concurrent DNN Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
When multiple deep neural networks are simultaneously inferred in a terminal using heterogeneous processors, scheduling is not supported, leading to interference and resource contention, and existing systems fail to efficiently utilize available processors.
Innovation Solution
A method for scheduling deep neural network operations using an analysis unit to partition requests into subgraphs, a profiling unit to predict operation execution times, and a scheduler to select subgraphs and processors based on frequency information and policy priorities, optimizing the use of heterogeneous processors like CPU, GPU, and NPU.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If multiple deep neural networks are simultaneously inferred in a terminal using heterogeneous processors, then computational capability is enhanced, but scheduling conflicts and resource contention occur
Solution Approach 1:
The patent segments the computational graph into multiple subgraphs that can be independently scheduled on different processors. The analysis unit partitions the computational graph, and the scheduler assigns subgraphs to appropriate processors based on availability and characteristics, enabling multiple DNNs to run simultaneously without interference.
Solution Approach 2:
The patent implements dynamic scheduling where the scheduler continuously monitors processor availability and adjusts task assignments in real-time. When a processor becomes available, the scheduler dynamically assigns new subgraphs, allowing flexible adaptation to changing system states and preventing resource contention.
2Device complexity
If only processors designated in the early stage are used, then system simplicity is maintained, but resource utilization is inefficient when current processor is occupied
Solution Approach 1:
The patent makes all heterogeneous processors (CPU, GPU, NPU) universally available for task assignment rather than restricting to predetermined processors. The scheduler considers all available processors and selects the most suitable one based on current workload and characteristics, improving resource utilization efficiency.
Solution Approach 2:
The patent changes the selection criterion from fixed processor designation to dynamic parameter-based selection. The scheduler evaluates processor availability, characteristics, and current workload to determine optimal assignments, allowing flexible adaptation to changing system conditions.
3Device complexity
If frequency information is not considered in scheduling, then scheduling simplicity is maintained, but processing efficiency is suboptimal
Solution Approach 1:
The patent incorporates frequency information as feedback to the scheduling decision process. The profiler provides execution time predictions based on processor frequency, and the scheduler uses this feedback to select optimal processor-frequency combinations, improving processing efficiency through informed scheduling decisions.
Data Source
AI summary
A single terminal performs scheduling of processing a request from a plurality of applications by using heterogeneous processors. The single terminal may include an analysis unit partitioning a request from an application in units and generating at least one subgraph, a profiling unit predicting an operation execution time for at least one frequency of at least one processor capable of processing the subgraph, and a scheduler performing scheduling based on a request from the application and the operation execution time.