Hybrid Scheduler for Distributed Deep Learning Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The implementation of deep learning applications on serverless computing is limited due to latency-sensitive requirements and the 'stateless' nature of serverless computing, which poses challenges for resource allocation and cold start latency in distributed inference models.
Innovation Solution
A hybrid scheduler is implemented to optimize server resource allocation and container allocation using deep reinforcement learning, prioritizing candidate servers based on available resources and minimizing latency by evaluating similarity across model versions and optimizing resource utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If serverless computing is used for distributed inference deep learning models, then resource utilization flexibility and cost reduction are improved, but cold start latency and service response time worsen
Solution Approach 1:
The system performs preliminary actions by pre-loading deep learning model partitions into memory of serverless container instances before they are needed for inference requests. The hybrid scheduler proactively allocates and initializes container instances with model partitions in advance, reducing the cold start latency when inference requests arrive. This preliminary preparation of model partitions in memory eliminates the need to load models from storage during the actual inference process.
2Productivity
If model partitions are distributed across multiple serverless container instances, then resource utilization is improved, but coordination complexity and inference latency worsen
Solution Approach 1:
The system merges the scheduling functions into a unified hybrid scheduler that combines reinforcement learning-based resource allocation with round-robin-based container instance selection. This unified scheduler manages the distribution of inference requests across multiple serverless container instances holding different model partitions, coordinating the distributed inference process while optimizing resource utilization. The scheduler maintains a mapping between model partitions and container instances to facilitate efficient coordination.
Solution Approach 2:
The reinforcement learning component of the hybrid scheduler incorporates feedback mechanisms by monitoring resource utilization metrics, inference latency, and container instance performance. This feedback is used to dynamically adjust scheduling decisions, model partition allocations, and container instance selections. The system continuously learns from past scheduling outcomes to optimize future resource allocation and inference request routing across the distributed serverless environment.
3Reliability
If deep reinforcement learning is used for resource allocation, then service quality and resource optimization are improved, but computational overhead and system complexity worsen
Solution Approach 1:
The system segments the complex resource allocation problem into distinct components: a reinforcement learning-based resource allocator that handles high-level resource optimization decisions, and a round-robin-based container instance selector that handles low-level request routing. This segmentation allows the reinforcement learning component to focus on optimizing resource allocation based on feedback metrics, while the round-robin component provides simple, deterministic request distribution. The segmented architecture reduces the computational burden on any single component and simplifies the overall system complexity.
Data Source
AI summary
According to examples, a system for serving distributed inference deep learning (DL) models in serverless computing is described. The system may include a processor and a memory storing instructions. The processor, when executing the instructions, may cause the system to receive a request to initialize a container and request a first candidate server from an available resource finder and a second candidate server from a resource optimizer. The processor, when executing the instructions, may then implement the server allocator to prioritize use of one of the first candidate server and the second candidate server and provide feedback regarding the prioritized use of one of the first candidate server and the second candidate server.