Hardware-Assisted RDMA Write Tracing for PVRDMA Live Migration

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

The live migration of Paravirtual RDMA (PVRDMA) virtual machines is challenging due to the inability of the hypervisor to capture internal runtime state and track RDMA writes from remote endpoints, especially when interacting with bare-metal endpoints, as existing workarounds are ineffective in such scenarios.

Innovation Solution

The solution involves generating write traces for dirtied memory portions and suspending queue pairs to capture their internal runtime state, allowing for the recreation of RDMA resources on a destination host system, enabling independent tracking and migration of PVRDMA VMs even when interacting with bare-metal endpoints.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Loss of information

If hardware-assisted write tracing is implemented to track RDMA writes from bare-metal endpoints, then the ability to capture complete memory state during live migration is improved, but the device complexity increases

Engineering Contradiction:

Improvememory state captureVSAvoidHCA hardware complexity

Core Design Contradiction:

Loss of informationVSDevice complexity

Solution Approach 1:

The patent introduces an intermediary trace buffer mechanism within the HCA that captures write trace information from RDMA operations. This intermediary structure allows the hypervisor to retrieve and process memory state information without direct intervention in the RDMA data path, thus improving memory state capture while managing hardware complexity through a dedicated intermediate component.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The patent replaces software-based memory tracking mechanisms with hardware-assisted write tracing capabilities in the HCA. By substituting mechanical/software polling and copying methods with hardware-level trace generation, the system achieves more complete and accurate memory state capture during live migration without proportionally increasing overall system complexity.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Measurement precision

If queue pairs are suspended to capture internal runtime state, then the accuracy of RDMA resource state transfer is improved, but the migration time increases

Engineering Contradiction:

Improveruntime state capture accuracyVSAvoidmigration time

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The patent applies preliminary action by suspending queue pairs before capturing their runtime state during live migration. This ensures that the state captured is consistent and accurate, as no new operations are interleaved during the capture process. The suspension is temporary and controlled, allowing precise state measurement while minimizing the overall migration time impact.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent rapidly suspends queue pairs, captures their state, and resumes them in a streamlined sequence. By minimizing the duration of suspension and efficiently processing the state capture operation, the system achieves high measurement precision for runtime state while reducing the time penalty associated with suspending and resuming queue pairs during migration.

Inventive Principle:

Principle #21Skipping (Rushing through)

3Reliability

If hardware-assisted tracking is used to monitor RDMA writes during live migration, then the reliability of data transfer is improved, but the use of energy increases

Engineering Contradiction:

Improvedata transfer reliabilityVSAvoidHCA energy consumption

Core Design Contradiction:

ReliabilityVSUse of energy by moving object

Solution Approach 1:

The patent implements periodic activation of hardware-assisted write tracing during live migration operations. Instead of continuously monitoring all RDMA writes, the trace buffer is activated during critical migration phases when memory state capture is needed and deactivated during normal operation. This periodic action maintains data transfer reliability during migration while reducing overall energy consumption of the HCA.

Inventive Principle:

Principle #19Periodic action

Data Source

PatentUS11822509B2Hardware-assisted tracking of remote direct memory access (RDMA) writes to virtual machine memory

Publication Date: 2023.11.21 VMWARE INC

AI summary

Techniques for live migrating a paravirtual remote direct memory access (PVRDMA) virtual machine (VM) from a source host system to a destination host system are provided. In one set of embodiments, during a pre-copy phase of the live migration process, a source hypervisor of the source host system can invoke an application programming interface (API) exposed by a source host channel adapter (HCA) of the source host system for initiating write tracing of remote direct memory access (RDMA) writes/atomic operations received from remote endpoints and can retrieve a write trace element generated by the source HCA that identifies a memory region of the PVRDMA VM written to as a result of an RDMA write/atomic operation and a write location within the memory region. The source hypervisor can then identify one or more guest memory pages of the PVRDMA VM dirtied per the write trace element and transmit data contents of those pages to the destination host system.