IOMMU Dirty Bit Tracking for VM Live Migration Passthrough

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Device passthrough technology, which improves VM performance by removing the hypervisor intermediary, is incompatible with live migration due to the hypervisor's inability to track DMA writes from passthrough physical devices, leading to incomplete memory page transfers during VM migration.

Innovation Solution

Utilizing the IOMMU's dirty bit tracking feature to identify and transfer guest memory pages dirtied by passthrough physical devices during live migration, through an enhanced live migration module that enables and manages dirty bit tracking within the hypervisor, allowing for both non-iterative and iterative tracking approaches.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Speed

If device passthrough is used to improve VM performance by removing hypervisor intermediary, then VM-to-physical device communication latency is reduced and CPU overhead is eliminated, but the hypervisor loses ability to track DMA writes from passthrough devices

Engineering Contradiction:

ImproveVM-to-physical device communication latencyVSAvoidHypervisor visibility of DMA writes

Core Design Contradiction:

SpeedVSLoss of information

Solution Approach 1:

The IOMMU acts as an intermediary between the passthrough device and guest memory. It intercepts DMA write operations and sets dirty bits in the IOMMU page tables, providing the hypervisor with visibility into device-induced memory modifications without requiring hypervisor mediation of the actual data transfer.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The patent replaces software-based tracking mechanisms with hardware-based dirty bit tracking in the IOMMU. This substitution enables automatic tracking of DMA writes through hardware-set bits, eliminating the need for complex software polling or device driver modifications.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Loss of information

If traditional hypervisor-based device emulation is used to maintain hypervisor control, then the hypervisor can mediate communication and track memory access, but VM performance deteriorates due to CPU overhead and latency

Engineering Contradiction:

ImproveHypervisor visibility of memory accessVSAvoidVM-to-physical device communication latency

Core Design Contradiction:

Loss of informationVSSpeed

Solution Approach 1:

The IOMMU serves as a hardware intermediary that enables the hypervisor to track memory access patterns without being involved in the actual data path. The hypervisor queries the IOMMU for dirty pages rather than intercepting individual DMA operations, reducing latency while maintaining visibility.

Inventive Principle:

Principle #24Intermediary (Mediator)

3Reliability

If device-specific implementations are used to track DMA writes for live migration, then tracking capability is achieved for specific devices, but device complexity increases and scalability is limited

Engineering Contradiction:

ImproveDMA write tracking capabilityVSAvoidDevice-specific implementation complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The IOMMU provides a universal tracking mechanism that works with all passthrough devices regardless of device type or vendor. The dirty bit tracking functionality is built into the IOMMU hardware itself, eliminating the need for device-specific tracking implementations and enabling broad scalability.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Solution Approach 2:

The IOMMU automatically tracks DMA writes through hardware-set dirty bits without requiring device driver modifications or special device capabilities. The system self-services the tracking function through existing IOMMU functionality, reducing complexity and improving reliability.

Inventive Principle:

Principle #25Self-service

4Measurement precision

If iterative tracking approaches are used to improve accuracy of dirty page identification, then completeness of memory page transfer is improved, but migration time increases

Engineering Contradiction:

ImproveAccuracy of dirty page identificationVSAvoidLive migration time

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The IOMMU continuously tracks dirty pages throughout the migration process, maintaining accurate information about modified memory regions. This continuous tracking enables the hypervisor to identify all dirty pages in a single pass, eliminating the need for multiple iterative scans and reducing migration time.

Inventive Principle:

Principle #20Continuity of useful action

Data Source

PatentUS12086084B2IOMMU-based direct memory access (DMA) tracking for enabling live migration of virtual machines (VMS) using passthrough physical devices

Publication Date: 2024.09.10 VMWARE INC

AI summary

Techniques for implementing IOMMU-based DMA tracking for enabling live migration of VMs that use passthrough physical devices are provided. In one set of embodiments, these techniques leverage an IOMMU feature known as dirty bit tracking which is available in most, if not all, modern IOMMU implementations. The use of this feature allows for the tracking of passthrough DMA in a manner that is device/vendor/driver agnostic, resulting in a solution that is universally applicable to all passthrough physical devices.