Sparse Multi-DNN Inference Scheduling for Latency and ANTT
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing multi-DNN scheduling approaches neglect sparsity patterns and dynamicity, leading to sub-optimal performance in multi-DNN workloads by primarily optimizing for a single metric, either latency service-level objective (SLO) violations or average normalized turnaround time (ANTT), while degrading the other, and fail to consider fine-grained details crucial for further optimizations.
Innovation Solution
A bi-level dynamic and static scheduler, named Dysta, utilizes both static and dynamic sparsity information to optimize multi-DNN workloads by incorporating sparsity patterns and average latency for initial task priorities, and adaptively refines scheduling with runtime monitoring, using a lightweight hardware scheduler to minimize SLO violations and ANTT.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If existing multi-DNN scheduling approaches optimize for latency SLO violations, then latency service-level objective is improved, but average normalized turnaround time deteriorates
Solution Approach 1:
The patent implements a dynamic scheduler that adapts scheduling decisions based on real-time system state, including current queue length, processor availability, and model-specific characteristics. This dynamic approach allows the system to optimize for latency SLO when violations are detected while minimizing impact on overall turnaround time through context-aware scheduling adjustments.
Solution Approach 2:
The scheduler modifies scheduling parameters dynamically based on workload characteristics and system state. It adjusts priority assignments, time quantum allocations, and preemption thresholds according to the specific needs of each DNN inference request, enabling optimization of both latency SLO compliance and average turnaround time through parameter adaptation rather than fixed scheduling policies.
2Loss of time
If existing multi-DNN scheduling approaches optimize for average normalized turnaround time, then average normalized turnaround time is improved, but latency service-level objective deteriorates
Solution Approach 1:
The patent applies different scheduling strategies to different inference requests based on their specific characteristics. High-priority requests with strict latency SLOs receive preferential treatment through priority scheduling and preemption mechanisms, while less time-sensitive requests are scheduled to optimize overall system throughput and average turnaround time, achieving local optimization for different task types.
Solution Approach 2:
The dynamic scheduler continuously monitors system state and adjusts scheduling decisions in real-time. When latency SLO violations are detected or predicted, the scheduler dynamically shifts focus to prioritize SLO compliance. When the system is operating smoothly, it optimizes for average turnaround time, enabling adaptive optimization of both metrics based on current conditions.
3Device complexity
If sparsity information is neglected in scheduling approaches, then device complexity is reduced, but productivity deteriorates
Solution Approach 1:
The scheduler performs preliminary analysis of sparsity patterns and model characteristics before making scheduling decisions. By pre-computing sparsity metrics and storing them in lookup tables, the system prepares scheduling information in advance, enabling efficient runtime scheduling that leverages sparsity information without adding significant computational complexity during the actual scheduling process.
Solution Approach 2:
The patent uses simplified representations and lookup tables that store pre-computed sparsity information instead of performing complex real-time analysis. This copying approach allows the scheduler to efficiently access sparsity characteristics of different models without re-computing them, maintaining low complexity while improving productivity through sparsity-aware scheduling decisions.
Data Source
AI summary
Broadly speaking, embodiments of the present techniques relate to a method and apparatus for scheduling tasks performed by multiple machine learning, ML, models. In particular, the present techniques provide a method for scheduling the execution of inference requests that relate to a plurality of ML models, and which are all to be executed by the same apparatus or processing unit. In an embodiment, the present techniques use both static and dynamic sparsity information to optimise the processing of multiple ML inference requests.