Heterogeneous Core Scheduling for LLM Prefill-Decode Overlap
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing large language model (LLM) serving systems face a trade-off between serving latency and efficiency due to the increased compute needs when batch size is increased, particularly in the prefill phase, leading to underutilization of resources and inefficiencies in processing operations.
Innovation Solution
A heterogeneous processing core allocation and mapping system is employed, where separate processing cores are assigned for prefill and decode operations, utilizing core-level schedulers to balance shared hardware resources like high-bandwidth memory, chip-interconnect bandwidth, and power, allowing for parallel execution of operations and efficient resource utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If batch size is increased to improve decoding throughput, then serving efficiency is improved, but prefill phase compute needs increase significantly causing longer execution time
Solution Approach 1:
The system segments the LLM processing into two distinct phases (prefill and decode) and allocates separate processing cores to each phase. Decode cores handle decoding operations while prefill cores handle prefill operations, allowing independent optimization and parallel execution of each phase, thereby resolving the trade-off between decoding throughput and prefill execution time.
Solution Approach 2:
The system transitions from a single-phase sequential processing model to a two-phase parallel processing model by introducing separate processing cores for prefill and decode operations. This dimensional change in the processing architecture enables simultaneous execution of prefill and decode phases, eliminating the sequential bottleneck and improving overall throughput.
2Productivity
If separate processing cores are allocated for prefill and decode operations, then resource utilization is improved, but device complexity increases
Solution Approach 1:
The system implements dynamic core allocation and mapping that adapts to workload characteristics. The scheduler dynamically assigns workloads to appropriate cores based on real-time conditions, and the mapping can be adjusted per-workload to optimize performance. This dynamic approach allows the system to handle varying workload patterns efficiently while managing complexity through adaptive rather than static configurations.
Data Source
Figure 1
Figure 2A~2C
Figure 3A
AI summary
Methods, systems, and apparatus, including computer-readable storage media and computer program products for heterogeneous processing core allocation and mapping for accelerating artificial intelligence (AI) workload with prefill and decode operations. A fleet of processing devices can include separate processing cores for accelerating prefill and decode operations of an AI workload, respectively. Individual devices are assigned for one or both of prefill or decode operation execution and include core-level schedulers for balancing shared hardware resources, such as high-bandwidth memory, chip-interconnect bandwidth, and power, to increase utilization of resources for performing the assigned operations. The same device may be mapped to a logical allocation to a performing prefill operations, decode operations, or both prefill and decode operations on a workload-by-workload basis, using a scheduler that accounts for both the arithmetic intensity of pre-fill operations and the auto-regressive nature of decode operations for some AI workloads, such as executing large language models (LLMs).