Heterogeneous AI Core Mapping for LLM Prefill-Decode Overlap
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing large language model (LLM) serving systems face a trade-off between serving latency and efficiency due to the increased compute needs with larger batch sizes, particularly in the prefill phase, leading to underutilization of resources when operations are segregated by core type.
Innovation Solution
Implementing a heterogeneous processing core allocation and mapping system that separates prefill and decode operations on distinct cores, utilizing core-level schedulers to balance shared hardware resources such as high-bandwidth memory, chip-interconnect bandwidth, and power, allowing for parallel execution and efficient resource utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If batch size is increased to improve serving efficiency, then throughput increases, but prefill phase compute needs increase significantly causing longer execution time
Solution Approach 1:
The patent segments the LLM serving workflow into two distinct phases (prefill and decode) and assigns them to different processing core types. Prefill operations are executed on first processing cores optimized for high arithmetic intensity, while decode operations are executed on second processing cores optimized for throughput. This segmentation allows each phase to utilize the most appropriate hardware resources, enabling larger batch sizes to be processed efficiently without proportionally increasing prefill phase execution time.
2Loss of energy
If prefill and decode operations are segregated by core type, then resource utilization improves, but overall system efficiency decreases due to underutilization of processing capabilities
Solution Approach 1:
The patent implements a hybrid processing system where certain processing cores are designed to handle both prefill and decode operations, while other cores are specialized for specific operation types. The system dynamically allocates workloads to appropriate cores based on operational demands, allowing flexible resource utilization. This multi-functionality approach ensures that processing resources are fully utilized without being constrained by rigid segregation, thereby improving both resource utilization and system efficiency simultaneously.
Data Source
AI summary
Methods, systems, and apparatus, including computer-readable storage media and computer program products for heterogeneous processing core allocation and mapping for accelerating artificial intelligence (AI) workload with prefill and decode operations. A fleet of processing devices can include separate processing cores for accelerating prefill and decode operations of an AI workload, respectively. Individual devices are assigned for one or both of prefill or decode operation execution and include core-level schedulers for balancing shared hardware resources, such as high-bandwidth memory, chip-interconnect bandwidth, and power, to increase utilization of resources for performing the assigned operations. The same device may be mapped to a logical allocation to a performing prefill operations, decode operations, or both prefill and decode operations on a workload-by-workload basis, using a scheduler that accounts for both the arithmetic intensity of pre-fill operations and the auto-regressive nature of decode operations for some AI workloads, such as executing large language models (LLMs).