LLM Request Scheduling Across Prefill and Decode Bottlenecks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The throughput of large-scale language model (LLM) service deployments is low due to inefficiencies in dynamic scheduling and resource allocation, particularly in the context of the prefill and decode phases, where computing power bottlenecks and memory access bottlenecks are not adequately addressed.
Innovation Solution
A request scheduling method that deploys devices in an independent separate manner, distinguishing between prefill and decode phases, and dynamically adjusts resource allocation based on load, allowing for concurrent processing of requests from different users without waiting for previous requests to be completed, thereby optimizing throughput.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a dynamic scheduling manner is used in LLM service deployment, then the system can handle multiple requests, but the throughput remains low due to bottlenecks in prefill and decode phases
Solution Approach 1:
The patent segments the LLM inference process into two distinct phases: prefill phase and decode phase. It further divides the system into multiple independent devices (first device, second device, third device) that can process different phases concurrently. This segmentation allows the prefill phase to be processed by one device while the decode phase is handled by another device, eliminating sequential processing bottlenecks and significantly improving throughput.
Solution Approach 2:
The patent implements dynamic scheduling where the first device can dynamically route requests to different devices based on phase requirements and system state. The scheduling mechanism adapts to varying workloads by dynamically allocating tasks to available devices, optimizing resource utilization and reducing processing time without compromising response quality.
2Productivity
If requests are processed sequentially in prefill and decode phases, then resource allocation is simple, but concurrent processing from multiple users is not enabled
Solution Approach 1:
The system segments processing responsibilities across multiple devices: the first device handles request reception and scheduling, the second device processes prefill phase requests, and the third device handles decode phase requests. This segmentation enables concurrent processing of requests from multiple users while maintaining clear separation of concerns and simplifying resource allocation through phase-based routing.
Solution Approach 2:
The first device acts as an intermediary between users and the processing devices. It receives requests from multiple users, determines the appropriate phase and target device, and coordinates the processing workflow. This intermediary role simplifies the overall system architecture by centralizing scheduling logic while enabling concurrent processing capabilities.
3Reliability
If computing resources are allocated uniformly across all phases, then resource management is simple, but computing power bottlenecks and memory access bottlenecks are not adequately addressed
Solution Approach 1:
The patent applies local quality by allocating resources differently to different phases based on their specific requirements. The prefill phase receives computing-intensive resource allocation while the decode phase receives optimized memory access resources. Each phase is served by dedicated devices with appropriately configured resources, addressing the specific bottlenecks of each phase rather than using uniform resource allocation.
Data Source
AI summary
In accordance with an embodiment, a method includes receiving, by a first device, a first request from a first user, where the first request is a request for a first service, and the first request comprises an identifier of the first service; scheduling, by the first device, the first request to a second device; and sending, by the second device, a second request to the first device, wherein the second request comprises the identifier of the first service and first output information corresponding to the first service, and the second request is associated with the first request.