Global Journal for Cross-Store Consistency in Key-Value Databases
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Key-value stores face challenges in maintaining cross-store consistency and data integrity during system crashes or storage failures, as existing journaling mechanisms can lead to inconsistent states and data loss due to the lack of a unified journaling system across multiple KV stores.
Innovation Solution
Implementing a global journal that combines sub-entries from multiple KV stores into a single atomic write operation, ensuring that all modifications are persisted before transactions can commit, and replaying uncommitted sub-entries upon system restart to maintain consistency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If separate journaling mechanisms are used for each KV store, then each KV store can independently manage its own transactions, but cross-store consistency cannot be guaranteed during system crashes or storage failures
Solution Approach 1:
The patent merges separate journaling mechanisms into a single global journal that serves multiple KV stores. The global journal writes combined journal entries containing sub-entries from multiple KV stores in a single atomic write operation, ensuring that all modifications are persisted together or none at all, thereby guaranteeing cross-store consistency during crashes.
Solution Approach 2:
The global journal acts as an intermediary between multiple KV stores and the persistent storage system. It receives modifications from multiple KV stores, combines them into a single atomic write operation, and manages the replay process during recovery, simplifying the complexity while ensuring consistency.
2Reliability
If a global journal combines sub-entries from multiple KV stores into a single atomic write operation, then cross-store consistency is ensured, but the write operation becomes more complex and requires coordination
Solution Approach 1:
The global journal serves multiple functions: it acts as a transaction coordinator, a persistent storage buffer, and a recovery mechanism for multiple KV stores. By making the journal universal across all stores, the system ensures data integrity without requiring separate coordination mechanisms for each store.
3Reliability
If journal entries are written to disk before transactions commit, then data loss is prevented during crashes, but the time to complete transactions increases
Solution Approach 1:
The system performs preliminary writing of journal entries to the global journal before transactions officially commit. This preliminary action ensures that modifications are persisted to disk before the transaction is considered committed, preventing data loss during crashes while allowing the transaction to proceed without waiting for full commit processing.
4Reliability
If the system replays uncommitted sub-entries upon restart, then data consistency is recovered, but the recovery process takes additional time
Solution Approach 1:
The global journal enables self-service recovery by automatically replaying uncommitted sub-entries during system restart. The journal contains all necessary information about uncommitted transactions, allowing the system to recover consistency without external intervention or complex recovery procedures, reducing recovery time while ensuring reliability.
Data Source
Figure 1
Figure 2
Figure 3~4
AI summary
A method is provided for a transaction manager in a database system to provide consistency across multiple persistent key-value (KV) stores, each having its own non-persistent transactions, for modifications to the persistent KV stores that must all persist or none at all. The method includes joining a first KV store's transaction and joining a second KV store's transaction. For a set of modifications including a first modification to the first KV store and a second modification to the second KV store that must all persist or none at all, the method includes inserting the first modification into the first KV store's transaction, inserting the second modification into the second KV store's transaction, and creating and writing a combined journal entry including first and second sub-entries describing the first and the second modifications in a persistent global journal, respectively.