Live Migration of Virtual Machines with Peripheral Devices via Non-Transparent Bridges
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing cloud computing technologies face challenges in live migrating virtual machines with attached peripheral devices, such as GPUs, due to the lack of a clean method to pause and resume operations, hindering the ability to transparently move guest virtual machines across hosts.
Innovation Solution
The use of hardware transactions and custom PCIe handling devices, coupled through non-transparent bridges (NTBs), allows for atomic movement of pages from one host's memory to another, enabling live migration while maintaining availability of memory and devices across hosts, allowing each host to reboot independently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional memory pausing mechanisms are used for live migration, then CPU memory access can be paused and resumed cleanly, but assigned peripheral devices cannot be paused and resumed
Solution Approach 1:
The patent introduces NTB (Non-Transparent Bridge) devices as intermediaries between the source and target hosts. These NTB devices intercept and manage memory access requests from peripheral devices during migration, allowing the system to handle device memory accesses without requiring the devices themselves to support pausing mechanisms. The NTB acts as a mediator that translates and routes memory requests appropriately during the transition phase.
Solution Approach 2:
The patent segments the memory access path into distinct components: guest virtual addresses, host page tables, NTB translation layers, and physical memory. By segmenting the address translation and access paths, the system can independently manage different stages of memory access, allowing peripheral devices to continue operating while their memory targets are being migrated behind the NTB boundary.
2Productivity
If memory pages are copied during live migration, then guest memory is transferred from source to target host, but peripheral devices may access incorrect memory locations
Solution Approach 1:
The patent sets up NTB devices and their address translation structures in advance before the actual memory page copying begins. The NTB translation tables are pre-configured to intercept memory requests, ensuring that even during active copying, any peripheral device accesses are correctly routed to the current valid memory location rather than to stale or incorrect addresses.
Solution Approach 2:
The system implements feedback mechanisms where the NTB devices monitor and track memory access patterns from peripheral devices during migration. This feedback information is used to dynamically adjust translation entries and ensure that memory requests are always directed to the correct, up-to-date memory locations, preventing access to incorrect memory during the copy operation.
3Adaptability or versatility
If hosts are connected through NTBs for memory sharing, then memory and devices become available to both hosts, but system complexity increases
Solution Approach 1:
The NTB devices are designed with multi-functionality to handle various operations: memory address translation, request routing, migration coordination, and device access management. This universal design allows a single NTB component to perform multiple critical functions that would otherwise require separate specialized devices, thereby reducing overall system complexity while enabling resource sharing between hosts.
Data Source
AI summary
Hardware transactions or other techniques, such as custom PCIe handling devices, are used to atomically move pages from one host's memory to another host's memory. The hosts are connected by one or two non-transparent bridges (NTBs), which make each host's memory and devices available to the other, while allowing each host to reboot independently.