Key-Value Store Recovery via Checkpoint Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Key-value stores face challenges in providing fast recovery and continuous read-only replicas in log-structured storage systems, particularly in maintaining system integrity and data availability during failures.
Innovation Solution
The implementation of a storage engine with a key-value engine that employs a page mapping data structure and system checkpoints to manage page mapping changes, user and system transactions, and garbage collection, ensuring data integrity and availability through periodic checkpointing and replaying transaction logs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If log-structured storage is used for key-value stores, then data persistence and write performance are improved, but recovery speed after system failures deteriorates
Solution Approach 1:
The patent applies preliminary action by creating system checkpoints periodically during normal operation, capturing the state of the key-value store at specific intervals. These checkpoints are stored in advance and can be used during recovery without requiring reprocessing of all transactions, thus reducing recovery time while maintaining the benefits of log-structured storage.
Solution Approach 2:
The patent segments the transaction log into manageable units by introducing system checkpoints that divide the continuous log into discrete segments. Each checkpoint represents a consistent state boundary, allowing the recovery process to resume from the most recent checkpoint rather than processing the entire log from the beginning, thereby improving recovery speed.
2Productivity
If tree structure index is used for page mapping, then data organization and lookup efficiency are improved, but system complexity increases
Solution Approach 1:
The patent introduces system checkpoints as intermediary structures that simplify the relationship between the tree structure index and the underlying storage. Checkpoints act as mediators that capture state information at specific points in time, reducing the complexity of maintaining continuous consistency while preserving the efficient lookup capabilities provided by the tree structure.
3Productivity
If continuous replication is implemented, then data availability and read performance are improved, but data consistency and integrity deteriorate
Solution Approach 1:
The patent applies preliminary action by establishing system checkpoints as predefined consistent states before replication occurs. These checkpoints ensure that data is in a known consistent state at the time of replication, allowing continuous replication to proceed while maintaining data consistency and integrity through the checkpointed state boundaries.
Data Source
AI summary
A key-value engine of a storage system may perform a restart recovery after a system failure. The key-value engine may read a metadata log to locate a latest system checkpoint, and load a page mapping table from the latest system checkpoint. The key-value engine may replay to apply changes to the page mapping table from a system transaction log starting from a system transaction replay starting point. The key-value engine may further form one or more read-only replicas using an underlying file stream opened in a read-only mode during the recovery after the system failure to further facilitate fast recovery and provide fast response to user transactions that conduct read only transactions after the system failure.