Serverless Request Scheduling for Low-Latency Pod and Node Selection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Serverless platforms struggle to manage latency-sensitive requests, leading to violations of Service-Level Objectives (SLOs) due to inadequate consideration of request processing capabilities and underlying hardware resources, resulting in excessive queuing latencies and inefficient resource utilization.
Innovation Solution
A latency-aware-based serverless request scheduling apparatus and system that includes a first scheduling module and a second scheduling module to optimize pod and node selection based on execution and node creation conditions, using SLO models and cluster status information to ensure timely execution and reduce SLO violations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If traditional serverless scheduling is used, then resource allocation is simple, but request processing latency increases and SLO violations occur
Solution Approach 1:
The system performs preliminary actions by pre-evaluating node conditions and pod readiness before request arrival. The first scheduling module pre-identifies suitable target pods based on SLO requirements, and the second scheduling module pre-identifies suitable target nodes based on processing capabilities, so that when requests arrive, scheduling can be performed rapidly without violating SLOs.
Solution Approach 2:
The patent introduces intermediary scheduling modules that act as mediators between the request queue and the execution pods. The first scheduling module mediates between requests and pods by selecting target pods based on SLO compliance, while the second scheduling module mediates between pod creation requests and nodes by selecting target nodes based on processing capabilities, thereby reducing overall scheduling latency.
2Productivity
If pod creation is performed without node selection optimization, then resource utilization is inefficient, but scheduling speed is faster
Solution Approach 1:
The system changes parameters by dynamically evaluating node conditions including processing capabilities, resource availability, and historical performance metrics. The second scheduling module uses these parameter changes to identify target nodes with the strongest processing capabilities, thereby improving resource utilization while maintaining acceptable pod creation speeds through intelligent selection rather than exhaustive search.
3Reliability
If SLO constraints are strictly enforced, then service quality is improved, but request queuing latency increases
Solution Approach 1:
The patent applies local quality by differentiating scheduling strategies based on local conditions. The first scheduling module identifies target pods with local quality metrics that ensure SLO compliance, while the second scheduling module identifies target nodes with local processing capabilities optimized for speed. This localized optimization allows the system to enforce SLO constraints on critical paths while minimizing queuing latency through efficient resource selection.
Data Source
AI summary
The apparatus includes: a first scheduling module that determines a determined user request at a queue head of a current request queue as a target user request, and sends a pod creation request to a second scheduling module when determining that a target pod that meets an execution condition and that is to execute the target user request is absent; and the second scheduling module that, when determining, based on the pod creation request, that a new pod meets a node creation condition, selects a target node from a plurality of nodes, creates the new pod in the target node, and sends information about the new pod to the first scheduling module. The first scheduling module manages the new pod based on the received information about the new pod, determines the new pod as a target pod, and sends the target user request to the target pod for execution.