Persistent Main Memory Log Tail Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing event logging systems face significant latency issues due to the need for threads to wait for confirmation that log cache contents have been written to a persistent store, leading to increased I/O operation latency and overhead.
Innovation Solution
Implementing a system that uses persistent main memory to write events, allowing immediate thread release after successful writing to persistent main memory, with optional volatile log caches mirroring persistent log caches to minimize infrastructure impact and facilitate recovery.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If threads wait for confirmation that log cache contents have been written to persistent store, then data reliability is improved, but I/O operation latency increases significantly
Solution Approach 1:
The patent applies preliminary action by writing event data to persistent main memory (PMEM) before the log cache is full or before system failure. This allows threads to be released immediately after writing to PMEM rather than waiting for log cache serialization and disk I/O completion. The PMEM serves as an intermediate persistent storage that guarantees data survival while enabling faster thread turnover.
Solution Approach 2:
The patent introduces persistent main memory (PMEM) as an intermediary between the volatile log cache and the disk persistent store. PMEM acts as a mediator that accepts event writes from threads, provides immediate persistence guarantees, and allows async serialization to disk later. This intermediary layer decouples the thread release timing from the slow disk I/O operations.
2Productivity
If log cache is sized to be serialized in single I/O operation, then I/O channel utilization is improved, but thread latency increases due to waiting for log cache to fill
Solution Approach 1:
The patent applies preliminary action by performing the persistence operation to PMEM immediately when events are written, rather than waiting for the log cache to fill. This allows threads to be released early while the system asynchronously manages log cache serialization to disk in the background, decoupling thread latency from I/O batching requirements.
3Productivity
If threads are released immediately after writing to persistent main memory, then thread turnover and productivity are improved, but data may be lost if persistent main memory is not properly preserved
Solution Approach 1:
The patent applies beforehand cushioning by using persistent main memory (PMEM) as a protective buffer that guarantees data survival. PMEM provides non-volatile storage characteristics that protect event data from loss during system failures, power outages, or crashes. This cushioning layer allows aggressive thread release policies while maintaining data safety, as PMEM preserves data even when volatile memory is lost.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A system that uses a persistent main memory to preserve events that await logging in a persistent store. Each event is written into the persistent main memory so as to be loggable in case of recovery. For instance, the event may be written into a log cache structure, along with other state which identifies that the event is in the particular log cache structure, the location of the event within the particular log cache structure, and the order of the event. To recover, the log in the persistent store is evaluated to identify the end of the stored log. The tail of the log is identified in the persistent main memory by identifying any log cache structures that are after the end of the stored log and which are validly recoverable. The log cache structure contents are then serialized one log cache at a time, earliest first.