LLM Inference Phase Splitting Across Heterogeneous GPU VMs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The inefficient utilization of compute resources and high power consumption in generative large language model (LLM) inference tasks due to the inconsistent latency and power requirements of the prompt and token generation phases, leading to suboptimal performance and increased costs in GPU-based deployments.
Innovation Solution
Splitting the prompt and token generation phases of LLM inference onto separate virtual machines (VMs) with different hardware configurations, where the prompt phase runs on high-compute GPUs and the token phase on high-memory GPUs, optimizing resource allocation and reducing power consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the prompt and token generation phases are run on the same GPU, then the system structure is simple, but the resource utilization is inefficient and power consumption is high
Solution Approach 1:
The patent divides the LLM inference process into two separate phases: prompt processing phase and token generation phase. Each phase is assigned to a different GPU type based on its computational characteristics. The prompt processing phase runs on high-compute GPUs while the token generation phase runs on high-memory bandwidth GPUs, allowing each hardware component to be optimized for its specific function and improving overall resource utilization.
Solution Approach 2:
The patent applies different hardware configurations to different phases of the inference process based on their specific requirements. High-compute GPUs are used for the prompt processing phase which requires intensive computation, while high-memory bandwidth GPUs are used for the token generation phase which requires fast memory access. This localized optimization of hardware properties to match phase-specific requirements improves overall system efficiency.
2Power
If high-compute GPUs are used for both prompt and token generation, then compute power is sufficient, but power consumption increases
Solution Approach 1:
The patent changes the hardware configuration parameter based on the phase being executed. Instead of using high-compute GPUs for both phases, the system switches to high-memory bandwidth GPUs for the token generation phase. This parameter change in hardware selection reduces power consumption while maintaining sufficient compute power for each specific phase's requirements.
3Productivity
If separate VMs with different GPU configurations are used, then resource allocation is optimized, but system complexity increases
Solution Approach 1:
The patent creates a unified system where multiple VM types can coexist and be managed through a common orchestration layer. The first VMs and second VMs, despite having different GPU configurations, are managed within the same cloud computing environment, allowing the system to handle diverse workloads efficiently while maintaining a degree of management universality through standardized interfaces and orchestration.
Data Source
AI summary
A system and method for splitting a prompt and token generation phase in a generative large language model (LLM) inference onto separate virtual machines (VMs) is provided. Two separate pools of VMs for prompt and token processing are maintained. The VMs in each of the pools are pre-loaded with a model of choice. A scheduler allocates an inference to a prompt VM from a pool of prompt VMs and a token VM from a pool of token VMs. Context generated from layers of the generative LLM during the prompt computation is saved in a key-value (KV) cache that is transferred from the prompt VM to token VM as it is used for all the future token generation iterations.