Storage System Failure Recovery via Non-Volatile Log
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage systems experience delays and timeouts during system failure recovery and takeover procedures, as they cannot process client-initiated requests until operations logs are replayed and system memory is flushed to disk, which can lead to client failures or malfunctions, especially in time-sensitive applications.
Innovation Solution
The storage system processes operations from the non-volatile operations log during boot-up or takeover to place system memory in a consistent state with the failure event, then resumes processing client requests before flushing data to disk, utilizing an unused portion of the operations log to handle new requests.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the storage system replays operations from the operations log and flushes system memory to disk during recovery, then data reliability is ensured, but the time to process client requests increases
Solution Approach 1:
The system pre-loads operations from the operations log into an in-memory buffer during system startup or takeover, before any client requests arrive. This preliminary action ensures that when clients do connect, their requests can be processed immediately without waiting for the time-consuming disk flush operation to complete. The buffer is pre-populated with recovered operations, separating the recovery timeline from the request processing timeline.
Solution Approach 2:
The recovery process is segmented into distinct phases: (1) loading operations from the operations log into memory buffer, (2) accepting and processing client requests against the in-memory buffer, and (3) asynchronously flushing the buffer to disk storage. This segmentation allows request processing to occur independently and concurrently with the final persistence operation, eliminating the sequential bottleneck.
2Stability of the object's composition
If the storage system waits to process client requests until after operations log replay and memory flush, then data consistency is maintained, but system productivity decreases
Solution Approach 1:
An in-memory buffer acts as an intermediary layer between client requests and the final disk storage. The buffer receives and holds operations during the recovery period, allowing clients to interact with a responsive system. The buffer is then flushed to disk as a single batch operation, ensuring data consistency while maintaining high productivity during the client interaction phase.
Solution Approach 2:
The system maintains continuous request processing capability throughout the recovery period by keeping the in-memory buffer populated with recovered operations. Instead of halting productivity entirely during recovery, the system continuously accepts and processes requests against the buffer, ensuring uninterrupted useful action while background operations complete the persistence to disk.
3Productivity
If the storage system uses volatile system memory for temporary data storage, then performance increases, but data loss risk increases during system failure
Solution Approach 1:
The system performs preliminary recovery actions by loading operations from the non-volatile operations log into the volatile memory buffer before resuming normal operation. This pre-loading ensures that even though system memory is volatile, the critical recovery data is already in place and ready for immediate processing, eliminating data loss risk while maintaining performance benefits.
Solution Approach 2:
The operations log serves as a cushion or safety net that protects against data loss from volatile memory. By maintaining a complete record of all operations in non-volatile storage and systematically replaying them into memory during recovery, the system cushions against any potential data loss events while preserving the high-performance characteristics of volatile memory during normal operation.
Data Source
AI summary
On or more techniques and/or systems are provided for rapidly resuming processing of client requests after a system failure event. Accordingly, during a boot-up process, a storage system, upon detecting a system failure event of the storage system, conditions its system memory to reflect a state of the system memory at a time of the failure by processing client requests or commands stored in a non-volatile operations log. The storage system can resume processing client requests after processing the operations in the nonvolatile operations log and prior to flushing data to storage devices.