Persistent Key-Value Store With Journaling for Crash Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage solutions in distributed systems face challenges with high latency and suboptimal performance due to data being distributed across multiple storage devices, leading to increased network hops and write amplification, especially when using non-persistent caches and non-volatile logs.
Innovation Solution
Implementing a persistent key-value store as a primary cache backed by fast storage media like NVRAM, using key-value record pairs, and a two-phase commit process to ensure data integrity and reduce write amplification, with journaling and crash recovery mechanisms for efficient data management.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is distributed across multiple storage devices in a distributed file system, then data capacity and reliability are improved, but latency increases and performance deteriorates due to increased network hops and write amplification
Solution Approach 1:
The patent segments the write operation into two distinct phases: first writing data to the persistent key-value store (local cache), then asynchronously replicating to distributed storage. This segmentation allows the client to receive immediate acknowledgment, reducing latency, while maintaining data reliability through the two-phase commit mechanism that ensures eventual consistency across the distributed system.
Solution Approach 2:
The persistent key-value store acts as an intermediary layer between the client and the distributed storage system. It buffers write operations locally, providing fast read/write access while asynchronously synchronizing with the distributed storage backend. This intermediary reduces network hops for frequent operations and manages the complexity of distributed consistency transparently.
2Reliability
If data is distributed across multiple storage devices, then data capacity and reliability are improved, but write amplification increases leading to suboptimal performance
Solution Approach 1:
The patent performs preliminary writing of data to the persistent key-value store before initiating the distributed replication process. This preliminary action ensures that the data is immediately available locally, reducing the need for repeated writes to distributed storage and thereby reducing write amplification. The two-phase commit process then completes the operation by replicating to the distributed backend only when necessary.
3Speed
If a non-persistent cache is used for fast access, then read speed is improved, but data integrity is compromised and crash recovery becomes difficult
Solution Approach 1:
The patent changes the persistence parameter of the cache by using a persistent key-value store instead of a volatile cache. This persistent store maintains data on non-volatile memory, ensuring that data survives crashes and can be recovered. The system maintains fast read/write performance by keeping the persistent store in memory or on fast storage, while the persistence property ensures data integrity and enables crash recovery through journaling and chain reconstruction.
4Reliability
If a non-volatile log is used for journaling, then crash recovery capability is improved, but write amplification increases and performance deteriorates
Solution Approach 1:
The patent merges the journaling function with the persistent key-value store by using the same underlying storage mechanism for both caching and journaling. The key-value store inherently provides persistence and can be equipped with journaling capabilities, combining the benefits of fast access, crash recovery, and reduced write amplification into a single unified system rather than requiring separate cache and log structures.
Data Source
AI summary
Techniques are provided for implementing a persistent key-value store for caching client data, journaling, and/or crash recovery. The persistent key-value store may be hosted as a primary cache that provides read and write access to key-value record pairs stored within the persistent key-value store. The key-value record pairs are stored within multiple chains in the persistent key-value store. Journaling is provided for the persistent key-value store such that incoming key-value record pairs are stored within active chains, and data within frozen chains is written in a distributed manner across distributed storage of a distributed cluster of nodes. If there is a failure within the distributed cluster of nodes, then the persistent key-value store may be reconstructed and used for crash recovery.