Single-Page Transaction Journal for Distributed Storage Recovery

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

The existing networked storage systems face performance degradation and increased recovery time due to the growing size of write ahead logs (WAL) and disk footprints, which are proportional to the number of extent groups, leading to inefficiencies in data processing and crash recovery.

Innovation Solution

Maintaining a journal for a distributed storage system on a single memory page, storing only basic information about pending transactions, allows for a constant journal size and reduces recovery time by using a single block for tentative updates, facilitating atomic read-modify-write operations and eliminating checkpointing.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If a write ahead log is used to record transactions, then data integrity and crash recovery capability are improved, but the journal size and in-memory footprint grow continuously as more transactions are processed

Engineering Contradiction:

Improvecrash recovery capabilityVSAvoidjournal size

Core Design Contradiction:

ReliabilityVSQuantity of substance

Solution Approach 1:

The patent segments the journal into fixed-size pages (e.g., 4KB) that are processed in units. Each page represents a discrete transaction boundary, allowing the system to manage journal growth in manageable segments rather than continuously expanding memory structures. This segmentation enables efficient page-level operations while maintaining crash recovery capability.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent implements a discard policy where processed journal pages are removed from memory after their transactions are committed to the extent group. This allows the journal to maintain a bounded size by discarding old pages while recovering only the necessary information for crash recovery, preventing unbounded memory growth.

Inventive Principle:

Principle #34Discarding and recovering

2Quantity of substance

If checkpoint operations are performed to reduce WAL size, then journal footprint is reduced, but system performance suffers due to processing overhead and recovery time increases

Engineering Contradiction:

Improvejournal footprintVSAvoidsystem performance

Core Design Contradiction:

Quantity of substanceVSProductivity

Solution Approach 1:

The patent employs periodic flushing of journal pages to the extent group at predetermined intervals or when pages are full, rather than continuous or event-driven checkpointing. This periodic action reduces the frequency of expensive checkpoint operations while maintaining acceptable journal footprint, thereby preserving system performance.

Inventive Principle:

Principle #19Periodic action

Solution Approach 2:

The patent treats journal pages as temporary, disposable structures that serve their purpose of recording transactions and are then discarded after flushing. This approach eliminates the need for persistent checkpoint files and complex recovery mechanisms, reducing overhead and improving performance.

Inventive Principle:

Principle #27Cheap short-living objects (Disposable)

3Reliability

If the journal grows with more extent groups, then crash recovery completeness is improved, but recovery time becomes proportional to the amount of data stored

Engineering Contradiction:

Improverecovery completenessVSAvoidcrash recovery time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The patent performs preliminary actions by maintaining journal pages in memory with transaction information already organized and ready for recovery. When a crash occurs, the system can immediately replay only the pending pages without scanning entire data sets, significantly reducing recovery time while ensuring completeness.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

By segmenting the journal into discrete pages associated with specific extent groups, the patent enables targeted recovery operations. Only the relevant pages needing recovery are processed, rather than replaying the entire journal, thus reducing recovery time proportionally to the segmented structure.

Inventive Principle:

Principle #1Segmentation

4Device complexity

If atomic read-modify-write operations are used on a single page, then journal updating simplicity is improved, but handling concurrent transactions becomes more challenging

Engineering Contradiction:

Improvejournal updating complexityVSAvoidconcurrent transaction handling

Core Design Contradiction:

Device complexityVSAdaptability or versatility

Solution Approach 1:

The patent implements dynamic journal page management where pages can be allocated, flushed, and reallocated based on transaction load and system state. This dynamic approach allows the journal structure to adapt to concurrent transaction patterns while maintaining simple atomic operations on each page, balancing simplicity with concurrency capability.

Inventive Principle:

Principle #15Dynamics

Data Source

PatentUS20250307214A1Distributed storage system journal on a single memory page

Publication Date: 2025.10.02 NUTANIX INC

AI summary

Techniques for storing metadata include receiving a write operation directed to at least one data block in a first location on a storage device, and storing write data associated with the write operation in a key-value store associated with the first location of the storage device. The techniques also include storing a first pending transaction record associated with the write operation in a pending transaction journal, wherein the pending transaction journal comprises a single storage page that further includes all other pending transaction records associated with other write operations associated with the storage device.