GPU Power Oversubscription for LLM Inference Clusters
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The rapid increase in compute capacity demand for large language models (LLMs) has led to a GPU capacity crunch in datacenters, with existing datacenters facing power, space, and cooling bottlenecks, and building new datacenters is expensive and carbon-intensive, while existing power management techniques are unreliable and inefficient for GPU clusters.
Innovation Solution
Implementing a power oversubscription framework with double threshold rules for frequency throttling in GPU clusters, using out-of-band interfaces to manage power consumption, prioritizing high and low priority workloads, and employing configurable policies to maintain service level objectives (SLOs) in existing infrastructure.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If more GPU servers are added to existing datacenter to meet LLM demand, then compute capacity is improved, but power consumption exceeds the fixed power budget
Solution Approach 1:
The system dynamically adjusts GPU clock frequencies based on real-time power consumption monitoring. When power thresholds are exceeded, the system throttles GPU frequencies to different levels (first frequency level for low-priority workloads, second frequency level for high-priority workloads) to bring power consumption back within the budget while maintaining service level objectives.
Solution Approach 2:
The system applies differentiated frequency throttling to different workload priorities. Low-priority inference workloads are throttled to the first frequency level, while high-priority inference workloads are throttled to the second frequency level, allowing selective power management that preserves critical services while reducing overall power consumption.
2Use of energy by moving object
If existing power management techniques are used for GPU clusters, then power consumption is controlled, but reliability and efficiency are insufficient
Solution Approach 1:
The system continuously monitors actual power consumption of GPU servers and uses this feedback to dynamically adjust frequencies. The monitoring and adjustment loops ensure that power consumption remains within the budget while adapting to changing workload conditions, thereby improving both reliability and efficiency compared to static power management techniques.
Solution Approach 2:
The system changes operational parameters (GPU clock frequencies) based on power consumption levels. By transitioning between different frequency levels and adjusting power management policies dynamically, the system achieves more reliable and efficient power management that adapts to varying conditions rather than relying on fixed, unreliable techniques.
3Productivity
If new datacenters are built to address GPU capacity crunch, then compute capacity is improved, but cost and carbon footprint increase
Solution Approach 1:
The system performs preliminary frequency throttling and power optimization actions before building new datacenters. By maximizing the utilization of existing datacenter capacity through dynamic power management, the system delays or eliminates the need for new infrastructure, thereby reducing upfront costs and carbon footprint associated with construction and deployment.
Solution Approach 2:
The system recovers and reuses existing datacenter infrastructure by optimizing power consumption patterns. Instead of discarding existing servers and building new ones, the system optimizes the power usage of existing hardware through frequency throttling and workload prioritization, thereby avoiding the carbon-intensive process of building new datacenters while maintaining necessary compute capacity.
Data Source
AI summary
Systems and methods for implementing power oversubscription in graphic processing unit (GPU) servers are provided. An increase to a quantity of servers allocated to a group of GPU servers in an inference cluster is applied. Based on the power consumption of the group of GPU servers exceeding a first threshold, a frequency of low priority inference workloads is capped, and based on the power consumption of the group of GPU servers exceeding a second threshold, the frequency of the low priority inference workloads are capped and a frequency of high priority inference workloads are capped, enabling an increase in allocated server capacity in the existing inference clusters while maintaining service level objectives (SLOs).