LLM Inference Cluster Reconfiguration for Energy-Aware Request Routing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The exponential growth in the adoption of generative large language models (LLMs) has led to significant energy consumption and carbon emissions due to their reliance on power-hungry GPUs, with existing energy management techniques failing to consider the unique characteristics of LLM inference environments.
Innovation Solution
A dynamically reconfigurable LLM inference cluster that categorizes incoming requests based on input and output lengths, dynamically adjusts the number of LLM instances, model parallelism, and processor frequencies to optimize energy efficiency while meeting performance Service Level Objectives (SLOs).
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the number of LLM instances is increased to handle higher request volumes, then productivity is improved, but use of energy increases
Solution Approach 1:
The system dynamically adjusts the number of active LLM instances based on real-time request patterns. During high-demand periods, more instances are activated to maintain productivity, while during low-demand periods, instances are deactivated or put to sleep, reducing energy consumption without compromising service level objectives.
Solution Approach 2:
The system changes operational parameters of LLM instances including CPU/GPU frequencies, batch sizes, and instance counts based on workload characteristics. By adjusting these parameters dynamically, the system optimizes the balance between processing capacity and energy consumption for different request patterns.
2Speed
If LLM instances operate at higher frequencies to meet performance SLOs, then speed is improved, but use of energy increases
Solution Approach 1:
The system dynamically adjusts processor frequencies and other operational parameters based on current workload demands and SLO requirements. During peak demand, frequencies are increased to meet performance targets, while during lower demand periods, frequencies are reduced to minimize energy consumption while maintaining acceptable service levels.
Solution Approach 2:
The system implements dynamic frequency adjustment where LLM instances can operate at different frequency levels depending on the current inference workload. This allows the system to match computational resources more closely with actual demand, reducing wasted energy during low-demand periods while ensuring performance SLOs are met when needed.
3Reliability
If a fixed infrastructure is used to ensure reliability, then reliability is improved, but adaptability deteriorates
Solution Approach 1:
The system transitions from a fixed infrastructure to a dynamically reconfigurable one that can automatically adjust the number of active instances, their configurations, and resource allocation in real-time based on incoming request patterns. This maintains reliability by ensuring sufficient capacity is always available while improving adaptability to varying workload characteristics.
Solution Approach 2:
The system continuously monitors performance metrics and SLO compliance, using this feedback to automatically adjust infrastructure configuration. When SLOs are at risk, the system scales up or reconfigures instances to maintain service reliability, while during stable periods it optimizes resource utilization to improve adaptability to different workload patterns.
Data Source
AI summary
Systems, methods, devices, and computer readable storage media described herein are directed to a dynamically reconfigurable large language model (LLM) inference cluster. The LLM inference cluster receives an inference request that includes a prompt. An input length is determined for the prompt, and an output length is predicted for the inference request based on the prompt. A request type of the inference request is determined based on the predicted output length and the input length, and an LLM instance is selected from a plurality of LLM instances based at least on the request type. The inference request is provided to the selected LLM instance for processing.