Live Migration of VMs with Passthrough PCI Devices
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Live migration of virtual machines (VMs) with passthrough PCI devices is incompatible due to challenges in tracking VM memory pages modified by direct memory access (DMA) and handling device-specific state, existing solutions requiring guest software modifications or being non-scalable.
Innovation Solution
Implementing a double buffering technique to remap the DMA ring buffer into two separate buffers for accurate dirty memory tracking and a generic callback framework for device-specific state management, allowing for seamless live migration without modifying guest software or increasing hypervisor engineering effort.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If PCI passthrough is used to allow direct access to physical PCI devices, then performance of VM operations is improved, but compatibility with live VM migration is lost
Solution Approach 1:
The DMA ring buffer is segmented into two separate buffers: a device buffer accessible by the passthrough PCI device and a vCPU buffer accessible by the VM. This segmentation allows the hypervisor to track DMA operations and dirty memory pages while maintaining PCI passthrough performance, thereby enabling live migration compatibility without sacrificing productivity.
2Device complexity
If a single DMA ring buffer is used for VM and passthrough PCI device, then device complexity is reduced, but ability to track dirty memory pages is lost
Solution Approach 1:
The single DMA ring buffer is divided into two separate buffers: a device buffer for PCI device access and a vCPU buffer for VM access. The hypervisor synchronizes these buffers and intercepts DMA work requests, enabling precise tracking of dirty memory pages while maintaining manageable device complexity through structured buffer management.
Solution Approach 2:
The hypervisor acts as an intermediary between the passthrough PCI device and the VM by synchronizing the device buffer and vCPU buffer. It intercepts DMA work requests submitted by the VM and monitors completions from the device, enabling accurate dirty page tracking without requiring complex modifications to the device or guest software.
3Measurement precision
If hypervisor intercepts all DMA work requests to track dirty pages, then measurement precision of memory tracking is improved, but processing overhead increases
Solution Approach 1:
The hypervisor performs preliminary actions by pre-configuring the dual buffer structure and establishing synchronization mechanisms before live migration begins. DMA work requests are intercepted and processed in the context of the existing DMA flow, allowing accurate dirty page tracking without significant additional processing overhead during critical migration operations.
Data Source
AI summary
Techniques for enabling live migration of VMs with passthrough PCI devices are provided. In one set of embodiments, a hypervisor of a host system can create a copy of a DMA buffer used by a VM of the host system and a passthrough PCI device of the VM. The hypervisor can further designate one of the DMA buffer or the copy of the DMA buffer as a vCPU buffer that is accessible by the VM, and designate the other of the DMA buffer or the copy of the DMA buffer as a device buffer that is accessible by the passthrough PCI device. The hypervisor can then synchronize the vCPU buffer and the device buffer with each other as the VM and passthrough PCI device interact with their respective buffers, and as part of the synchronization can intercept DMA work requests submitted by the VM/completed by the passthrough PCI device.