SR-IOV Dirty Page Tracking via Hypervisor Introspection

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

SR-IOV is incompatible with virtualization features like live VM migration due to the hypervisor's inability to track VM memory pages modified by VF-initiated DMA writes, leading to inefficiencies and broken migration processes.

Innovation Solution

A hypervisor-based solution that involves a hardware interrupt from the SR-IOV device to invoke a PF driver's introspection function to identify and mark dirty memory pages in the hypervisor's page table, without requiring changes to the guest OS or VF driver, allowing real-time tracking and compatibility with standard OS/driver distributions.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If SR-IOV device directly writes data to VM guest memory via DMA, then I/O performance is improved, but the hypervisor cannot track modified memory pages

Engineering Contradiction:

ImproveI/O performanceVSAvoiddirty page tracking information

Core Design Contradiction:

ProductivityVSLoss of information

Solution Approach 1:

The patent introduces an intermediary mechanism (EPT violation handler and introspection function) that mediates between the SR-IOV device's DMA writes and the hypervisor's dirty page tracking. When the VF writes to guest memory via DMA, it triggers an EPT violation that the hypervisor catches, allowing the hypervisor to inspect the write and mark the appropriate pages as dirty without blocking the direct DMA path.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The patent replaces the traditional CPU-mediated memory write mechanism with a direct DMA write mechanism from the VF to guest memory. This substitution improves I/O performance by eliminating CPU involvement in the data path, while the EPT violation handler provides the necessary tracking capability that would otherwise require CPU intervention.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Measurement precision

If hypervisor tracks CPU-initiated memory writes via EPT, then dirty page tracking works for CPU writes, but VF-initiated DMA writes cannot be tracked

Engineering Contradiction:

Improvedirty page tracking accuracyVSAvoidcompatibility with SR-IOV

Core Design Contradiction:

Measurement precisionVSAdaptability or versatility

Solution Approach 1:

Instead of having the VF driver or guest OS notify the hypervisor about dirty pages (the traditional approach), the patent inverts the approach by having the hypervisor actively detect dirty pages through EPT violations triggered by VF DMA writes. This inversion allows the hypervisor to track both CPU-initiated and VF-initiated writes uniformly.

Inventive Principle:

Principle #13The other way round (Inversion)

Solution Approach 2:

The EPT mechanism, originally designed only for CPU-initiated memory accesses, is extended to also track VF-initiated DMA writes. The same EPT structure and violation handling mechanism that tracks CPU writes now also tracks VF writes, providing a universal tracking solution for both types of memory accesses.

Inventive Principle:

Principle #6Universality (Multi-functionality)

3Reliability

If para-virtualized VF driver and guest OS are used to notify hypervisor of dirty pages, then dirty page tracking is enabled, but standard OS/driver distributions cannot be used

Engineering Contradiction:

Improvedirty page tracking reliabilityVSAvoidcompatibility with standard distributions

Core Design Contradiction:

ReliabilityVSEase of manufacture

Solution Approach 1:

The system enables the VF and guest memory to self-report dirty pages through the EPT violation mechanism. When the VF writes to guest memory, the write automatically triggers an EPT violation that the hypervisor detects and processes, eliminating the need for the VF driver or guest OS to actively notify the hypervisor about dirty pages.

Inventive Principle:

Principle #25Self-service

4Measurement precision

If hypervisor inspects VF state to identify modified memory portions, then accurate dirty page identification is achieved, but additional processing overhead is introduced

Engineering Contradiction:

Improvedirty page identification accuracyVSAvoidprocessing time for dirty page identification

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The patent sets up the EPT structure in advance with the appropriate mappings between guest physical memory and host physical memory. This preliminary configuration allows the hypervisor to efficiently intercept and inspect VF DMA writes as they occur, without needing to perform complex analysis of VF state for each write operation. The introspection function leverages the pre-configured EPT to quickly identify the affected memory portions.

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS9898430B2Tracking virtual machine memory modified by a single root I/O virtualization (SR-IOV) device

Publication Date: 2018.02.20 VMWARE INC

AI summary

Techniques for tracking, by a host system, virtual machine (VM) memory modified by a physical input/output (I/O) device that supports I/O virtualization are provided. In one embodiment, a hypervisor of the host system can receive a hardware interrupt from the physical I/O device, where the hardware interrupt indicates that a virtual function (VF) of the physical I/O device has completed a direct memory access (DMA) write to a guest memory space of a VM running on the host system. In response to the hardware interrupt, the hypervisor can invoke a function implemented by a physical function (PF) driver of the physical I/O device, where the function is configured to inspect the VF's state in order to identify memory portions modified by the DMA write. The hypervisor can then mark, in a hypervisor-level page table, one or more memory pages corresponding to the identified memory portions as dirty pages.