Ephemeral-Backed Journaling With Persistent Flushes for Rehosting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Virtual storage systems in cloud environments lack access to non-volatile memory or persistent storage with NVRAM-like performance, leading to data loss during shutdown or reboot scenarios, especially when rehosted on new hosts.
Innovation Solution
Flush operation log journals to both ephemeral and persistent storage during shutdown sequences, ensuring data durability by using persistent storage as a backup to ephemeral storage, particularly in scenarios involving rehosting.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If ephemeral storage is used as backing storage for journaling, then write performance is improved, but data durability deteriorates during shutdown or rehosting scenarios
Solution Approach 1:
The system performs preliminary flushing of the operation log to both ephemeral storage and persistent storage before shutdown occurs. This advance action ensures that journaling data is preserved in persistent storage even if the compute instance is rehosted, preventing data loss while maintaining the performance benefits of using ephemeral storage for active journaling operations.
Solution Approach 2:
The system establishes persistent storage as a backup cushion for the operation log before failure scenarios occur. By maintaining copies in both ephemeral and persistent storage, the system creates a safety buffer that protects against data loss during shutdown or rehosting events, while the primary ephemeral storage continues to provide high-speed write performance during normal operation.
2Reliability
If operation log is flushed to both ephemeral storage and persistent storage during shutdown, then data durability is improved, but device complexity increases
Solution Approach 1:
The system dynamically adapts its storage behavior based on operational context. During normal operation, it uses ephemeral storage for fast journaling. Upon detecting shutdown conditions, it transitions to flushing both ephemeral and persistent storage. This dynamic approach allows the system to maintain simplicity during most operations while providing enhanced durability only when needed, managing complexity through conditional behavior rather than always-on dual-storage mode.
Data Source
AI summary
Systems and methods for flushing an operation log journal to both ephemeral storage and persistent storage during a shutdown sequence of a virtual storage system to minimize data-loss scenarios are provided. According to one embodiment, the shutdown or reboot scenarios that result in loss of data are minimized by using persistent storage as a backup to ephemeral storage when the scenario results in rehosting of virtual storage system. For example, responsive to an event indicative of an imminent shutdown or reboot of the virtual storage system, vNVRAM memory may be flushed to both ephemeral storage and persistent storage (e.g., a boot disk). In this manner, when the virtual storage system is rehosted after an unplanned shutdown or reboot resulting from an unrecoverable host error (other than an unrecoverable hardware failure), the operation log journal may be recovered from persistent storage to facilitate vNVRAM replay and avoid data loss.