Dynamic LLM Inference Clusters for Energy-SLO Workload Shaping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The exponential growth in the adoption of large language models (LLMs) has led to significant energy consumption and carbon emissions due to their reliance on power-hungry GPUs, with existing energy management techniques failing to consider the unique characteristics of LLM inference environments.
Innovation Solution
A dynamically reconfigurable LLM inference cluster that categorizes incoming requests based on input and output lengths, dynamically adjusts the number of LLM instances, model parallelism, and processor frequencies to optimize energy efficiency while meeting performance Service Level Objectives (SLOs).
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the number of LLM instances is increased to handle higher query volumes, then productivity is improved, but use of energy increases
Solution Approach 1:
The system dynamically adjusts the number of active LLM instances based on real-time query patterns and workload demands. During peak periods, more instances are activated to maintain high productivity, while during low-demand periods, instances are deactivated or put to sleep, significantly reducing energy consumption while maintaining the capability to handle high query volumes when needed
Solution Approach 2:
The system changes operational parameters of LLM instances including CPU/GPU frequencies, batch sizes, and memory allocation based on the type and volume of incoming queries. This allows the system to optimize the balance between processing capacity and energy consumption by adjusting parameters rather than simply adding or removing hardware resources
2Productivity
If processor frequency is increased to meet performance SLOs, then productivity is improved, but use of energy increases
Solution Approach 1:
The system dynamically adjusts processor frequencies based on the type of inference request, current workload, and performance requirements. For critical requests requiring guaranteed SLO compliance, higher frequencies are applied, while for less critical requests, lower frequencies reduce energy consumption. The adjustment is made continuously based on monitoring data
Solution Approach 2:
The system implements granular control over processor frequencies, allowing different CPU/GPU cores to operate at different frequency levels simultaneously. This enables optimization where only the necessary processing units run at high frequencies for meeting SLOs, while others run at lower frequencies or idle, reducing overall energy consumption
3Productivity
If model parallelism is increased to handle larger models, then productivity is improved, but device complexity increases
Solution Approach 1:
The system dynamically configures model parallelism strategies based on the specific model being deployed and current hardware resources. When deploying larger models, the system automatically adjusts parallelism settings to optimize both performance and resource utilization. The dynamic reconfiguration capability allows the same infrastructure to efficiently handle models of varying sizes without manual intervention
Solution Approach 2:
The system creates a universal platform that can handle multiple model sizes and types through a single unified architecture. By implementing flexible model parallelism configurations and a standardized instance management system, the same infrastructure can efficiently deploy and manage models ranging from small to large scales, reducing the need for separate specialized systems for different model types
Data Source
Figure 1
Figure 2
Figure 3~4
AI summary
Systems, methods, devices, and computer readable storage media described herein are directed to a dynamically reconfigurable large language model (LLM) inference cluster. The LLM inference cluster receives an inference request that includes a prompt. An input length is determined for the prompt, and an output length is predicted for the inference request based on the prompt. A request type of the inference request is determined based on the predicted output length and the input length, and an LLM instance is selected from a plurality of LLM instances based at least on the request type. The inference request is provided to the selected LLM instance for processing.