Federated VM Placement for NUMA-Aware Socket Allocation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing virtual machine allocation systems struggle with efficient placement on hosts with NUMA architectures, leading to performance issues due to vCPU threads crossing NUMA boundaries, which are not accurately reflected in the platform allocator's view, and require complex data management across diverse hardware configurations.
Innovation Solution
A federated approach involving a platform allocator and local managers on each host, where the local managers collect and export socket-level data to optimize VM placement, allowing the platform allocator to make informed decisions while local managers modify placements based on real-time events and resource usage, minimizing vNUMA node splits.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the platform allocator uses a centralized approach to manage VM placement across all hosts, then it can make global optimization decisions, but it becomes difficult and expensive to gather accurate socket-level NUMA data from diverse hardware configurations
Solution Approach 1:
The system divides the centralized allocation function into distributed components by deploying local managers on each host. Each local manager independently collects socket-level NUMA data and makes placement decisions for its own host, eliminating the need for the platform allocator to gather and manage complex hardware data from all hosts centrally.
Solution Approach 2:
Each host's local manager autonomously collects its own socket-level NUMA topology data and uses this information to make VM placement decisions. The system serves itself by having distributed agents gather and process hardware information locally, rather than requiring centralized data collection from diverse hardware configurations.
2Ease of operation
If vCPUs are assigned without considering NUMA topology, then allocation is simpler and faster, but vCPU threads cross NUMA boundaries causing performance degradation
Solution Approach 1:
The system performs preliminary collection of socket-level NUMA topology data before VM allocation decisions are made. Local managers gather hardware configuration information in advance and use it to pre-determine optimal vCPU-to-socket mappings, ensuring that vCPUs are assigned to the correct NUMA node before the VM starts running, thus avoiding performance degradation from cross-boundary access.
3Productivity
If the platform allocator makes all placement decisions centrally, then global resource optimization is achieved, but it cannot adapt to real-time local events and resource usage changes
Solution Approach 1:
The system transitions from static centralized decision-making to dynamic distributed decision-making. Local managers continuously monitor real-time events and resource usage on their respective hosts and can dynamically adjust VM placements in response to changing conditions, while still contributing to overall resource utilization optimization across the platform.
4Measurement precision
If socket-level NUMA data is collected from all hosts, then accurate VM placement decisions can be made, but the data management burden on the platform allocator becomes excessive
Solution Approach 1:
The system extracts the data collection and processing function from the platform allocator and places it locally on each host through local managers. Each local manager collects and manages socket-level NUMA data for its own host, extracting this management burden from the centralized platform allocator while maintaining accurate hardware configuration data for placement decisions.
Data Source
AI summary
Improvements in allocating virtual machines (VMs) to hosts can be accomplished using an federated approach that involves coordination between a platform allocator and local managers that are run on each host. The platform allocator can be made NUMA-aware with socket level information about current VM placement exported from the hosts by their resident local manager. Accordingly, the platform allocator can place newly requested VMs on specific sockets within hosts, attempting to optimize packing and performance. A local manager, however, may override the socket placement based on its more detailed view of subsequent local events (such as VMs being deleted or migrating away), performance data, and how the placed VMs have been using resources.