Priority Queue Scheduling for Low-Latency Cloud Workloads
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing cloud-based computing systems often treat all workloads equally, leading to inefficiencies in handling latency-sensitive and latency-tolerant tasks, resulting in suboptimal performance and hardware utilization.
Innovation Solution
Implementing a high priority queue for latency-sensitive workloads and a low priority queue for latency-tolerant workloads, ensuring high priority workloads are executed first, with low priority workloads scheduled only when high priority queues are empty, thereby optimizing resource utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all workloads are served with the same priority using first-in-first-serve scheduling, then the scheduling system is simple to implement, but latency-sensitive workloads cannot be finished within desired latency budget
Solution Approach 1:
The patent segments the workload queue into multiple priority levels (high priority queue and low priority queue). High priority workloads such as online machine learning inference are separated from low priority workloads such as offline training tasks. This segmentation allows the system to guarantee latency budget for time-sensitive workloads while maintaining simplicity in implementation through clear priority-based scheduling rules.
2Loss of time
If high priority workloads are always executed first, then latency for high priority tasks is reduced, but computing resources remain idle during high priority queue emptiness
Solution Approach 1:
The scheduling system dynamically switches between high priority and low priority queues based on the state of the high priority queue. When the high priority queue is non-empty, the system executes high priority workloads. When the high priority queue becomes empty, the system automatically transitions to executing low priority workloads from the low priority queue. This dynamic adaptation ensures both low latency for high priority tasks and high hardware utilization by eliminating idle time.
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for scheduling workloads on computing resources using a high priority queue and a low priority queue. The high priority queue maintains pending high priority workloads to be scheduled for execution, and the low priority queue maintains pending low priority workloads to be scheduled for execution. The computing system as described in this specification schedules the pending low priority workloads for execution by utilizing computing resources provided by the system only when the high priority queue is empty.