Journal Forking for Distributed Storage Transaction Consistency

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Distributed computing environments face challenges in managing transactions across multiple data stores due to lack of support for ACID properties, varying data models, and network delays, leading to complex and unreliable transaction management mechanisms.

Innovation Solution

A journal-based multi-data-store database system that implements partitioning policies, optimistic concurrency control, and journal forking to manage transactions across diverse data stores, ensuring scalability and robustness by registering data stores as materialization nodes, using horizontal and vertical partitioning, and creating snapshots for efficient synchronization.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If ad-hoc transaction management mechanisms are implemented, then transaction support across data store boundaries is achieved, but maintenance complexity increases and reliability decreases

Engineering Contradiction:

Improvetransaction supportVSAvoidtransaction reliability

Core Design Contradiction:

Adaptability or versatilityVSReliability

Solution Approach 1:

The patent introduces a journal manager as an intermediary component that coordinates transactions across multiple data stores. The journal manager maintains a sequential journal of all data store operations, acting as a mediator that ensures atomicity and consistency without requiring complex ad-hoc transaction management at each data store boundary.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The system segments transaction management into distinct components: the journal manager handles coordination and conflict detection, while individual data store managers handle their respective data stores. This segmentation allows each component to focus on specific tasks, improving reliability without sacrificing adaptability.

Inventive Principle:

Principle #1Segmentation

2Reliability

If journal size increases to support more transactions, then transaction history is more complete, but commit analysis time increases

Engineering Contradiction:

Improvetransaction history completenessVSAvoidcommit analysis time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The system performs preliminary actions by maintaining the journal in sequential order with pre-analyzed conflict information. The journal manager prepares conflict detection data in advance during journal entry creation, so that commit analysis can quickly reference pre-computed information rather than analyzing the entire journal history each time.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent extracts conflict detection information from the full journal entries and stores it in a separate, optimized format. This extraction allows the system to maintain complete transaction history in the journal while using a streamlined representation for rapid conflict analysis during commit operations.

Inventive Principle:

Principle #2Taking out (Extraction)

3Productivity

If data stores are distributed across multiple locations, then system scalability is improved, but network delays and failures increase

Engineering Contradiction:

Improvesystem scalabilityVSAvoidtransaction robustness

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

The system implements beforehand cushioning by maintaining a complete journal of all transactions before committing changes to distributed data stores. This journal serves as a cushion that can be replayed in case of network failures or data loss, ensuring transaction robustness while allowing data stores to be distributed across multiple locations for scalability.

Inventive Principle:

Principle #11Beforehand cushioning (Prior cushioning)

4Adaptability or versatility

If different data manipulation operations are supported across data stores, then system versatility is improved, but transaction management complexity increases

Engineering Contradiction:

Improvedata manipulation versatilityVSAvoidtransaction management complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The journal manager implements a universal interface that handles diverse data manipulation operations across different data stores through a common conflict detection and coordination mechanism. This multi-functional approach allows the system to support various data store types and operations without increasing transaction management complexity, as the journal manager handles all coordination uniformly.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS10235407B1Distributed storage system journal forking

Publication Date: 2019.03.19 AMAZON TECH INC

AI summary

A forking coordinator of a journal-based multi-data-store database determines that a forking criterion for a first journal has been met. The coordinator stores an indication that processing of transaction requests directed to a particular subset of the database is suspended. The coordinator instantiates a second journal for storing committed transaction entries for the particular subset. Metadata of the second journal indicates that transaction entries with sequence numbers in a particular range are stored in the first journal. The coordinator stores an indication that transaction processing associated with the particular subset of the system is to be resumed using the second journal.