TLB Synchronization via T Bit in Virtualized Systems

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Modern CPUs with hardware-assisted virtualization introduce non-determinism due to memory management units (MMUs) and translation lookaside buffers (TLBs), which can lead to inconsistencies in address translation mappings, affecting deterministic replay in virtualized systems.

Innovation Solution

The introduction of a flag bit, referred to as the 'T' bit, in Extended Page Tables (EPT) entries to synchronize cached address translation mappings with guest translation tables, ensuring that TLB entries remain in sync with modifications to guest page tables, thereby eliminating non-determinism.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If hardware-assisted virtualization with MMU and TLB is used, then virtualization performance is improved, but non-determinism is introduced due to TLB eviction and page table modifications

Engineering Contradiction:

Improvevirtualization performanceVSAvoiddeterminism

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

The system performs preliminary actions by flushing the TLB before recording execution and clearing page table modifications before replay. This ensures that the TLB starts in a known state for both recording and replay, eliminating non-determinism while maintaining virtualization performance through hardware-assisted virtualization.

Inventive Principle:

Principle #10Preliminary action

2Speed

If TLB caching is used for address translation, then address translation speed is improved, but synchronization with guest page tables becomes problematic

Engineering Contradiction:

Improveaddress translation speedVSAvoidsynchronization

Core Design Contradiction:

SpeedVSStability of the object's composition

Solution Approach 1:

The system uses feedback mechanisms through bitmaps that track which TLB entries correspond to modified guest page table entries. When page tables are modified, the system checks these bitmaps to determine which TLB entries need to be flushed, ensuring synchronization while maintaining high address translation speed through selective invalidation rather than complete flushing.

Inventive Principle:

Principle #23Feedback

3Reliability

If complete TLB flushing is performed to ensure synchronization, then determinism is improved, but performance overhead increases

Engineering Contradiction:

ImprovedeterminismVSAvoidperformance

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

Instead of performing complete TLB flushing, the system applies local quality by selectively invalidating only those specific TLB entries that correspond to modified guest page table entries. This is achieved through bitmaps that track modifications, allowing the system to maintain determinism while minimizing performance overhead by affecting only the necessary local portions of the TLB rather than the entire structure.

Inventive Principle:

Principle #3Local quality

Data Source

PatentUS9928180B2Synchronizing a translation lookaside buffer with page tables

Publication Date: 2018.03.27 VMWARE INC

AI summary

The translation lookaside buffer (TLB) of a processor is kept in synchronization with a guest page table by use of an indicator referred to as a “T” bit. The T bit of the NPT/EPT entries mapping the guest page table are set when a page walk is performed on the NPT/EPT. When modifications are made to pages mapped by NPT/EPT entries with their T bit set, changes to the TLB are made so that the TLB remains in synchronization with the guest page table. Accordingly, record/replay of virtual machines of virtualized computer systems may be performed reliably with no non-determinism introduced by stale TLBs that fall out of synchronization with the guest page table.