Persistent Metadata Change Logs for Faster Storage Recovery

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing data storage systems face inefficiencies in managing metadata changes and recovering from failures, leading to high write amplification and reduced drive life due to the reliance on volatile memory for metadata logs.

Innovation Solution

A metadata change log is maintained in persistent memory, with transaction entries that are processed atomically to ensure efficient application of metadata changes, and recovery is facilitated by battery-backed persistent memory mirroring, reducing the need for full log reconstruction after failures.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Speed

If metadata changes are stored in volatile memory, then write operations are fast, but data loss occurs during power failure and full log reconstruction is required

Engineering Contradiction:

Improvewrite speedVSAvoiddata persistence

Core Design Contradiction:

SpeedVSReliability

Solution Approach 1:

The metadata change log is segmented into two distinct storage layers: volatile memory for active logging during operations and persistent memory for durability. This segmentation allows the system to maintain high write speeds in volatile memory while ensuring data persistence through periodic or trigger-based flushing to persistent memory, thereby resolving the contradiction between speed and reliability.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system performs preliminary actions by pre-allocating and pre-configuring the persistent memory region before it is needed for failure recovery. The persistent memory is prepared to receive metadata logs in advance, and the system establishes the dual-memory architecture beforehand, so that when a power failure occurs, the recovery process can immediately utilize the pre-prepared persistent storage without needing to reconstruct the entire log from scratch.

Inventive Principle:

Principle #10Preliminary action

2Reliability

If persistent memory is used for metadata change log, then data persistence is improved, but write amplification increases and drive life reduces

Engineering Contradiction:

Improvedata persistenceVSAvoidwrite amplification

Core Design Contradiction:

ReliabilityVSLoss of substance

Solution Approach 1:

The system merges the benefits of both volatile and persistent memory by combining them into a unified metadata logging system. The volatile memory provides fast writes while the persistent memory provides durability, and they work together as an integrated solution. This merging eliminates the need to choose one over the other, allowing the system to achieve data persistence without the full penalty of write amplification that would occur if only persistent memory were used.

Inventive Principle:

Principle #5Merging (Combining)

Solution Approach 2:

The volatile memory acts as an intermediary buffer between the host system and the persistent memory. Instead of writing directly to persistent memory for every metadata change, the system first writes to the faster volatile memory, which then periodically or selectively flushes to persistent memory. This intermediary approach reduces the direct write amplification on persistent storage while maintaining data persistence guarantees.

Inventive Principle:

Principle #24Intermediary (Mediator)

3Reliability

If full log reconstruction is performed after failure, then data consistency is ensured, but recovery time increases

Engineering Contradiction:

Improvedata consistencyVSAvoidrecovery time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The system performs preliminary actions by maintaining a ready-to-use persistent memory copy of the metadata change log before any failure occurs. This pre-prepared persistent storage contains the latest committed metadata changes, so when a failure happens, the system can immediately restore from this pre-configured persistent copy rather than needing to reconstruct the entire log from scratch, thereby significantly reducing recovery time while maintaining data consistency.

Inventive Principle:

Principle #10Preliminary action

4Reliability

If transaction entries are processed atomically, then data integrity is improved, but processing complexity increases

Engineering Contradiction:

Improvedata integrityVSAvoidprocessing complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The transaction processing system is segmented into distinct components: transaction log generation, atomic commit protocol, and recovery management. Each component handles a specific aspect of atomic transaction processing, which modularizes the complexity and makes it more manageable. This segmentation allows the system to implement robust atomic processing with data integrity guarantees while keeping the overall system architecture organized and maintainable.

Inventive Principle:

Principle #1Segmentation

Data Source

PatentUS12430210B2Managing changes to metadata in a data storage system

Publication Date: 2025.09.30 DELL PROD LP

AI summary

A searchable metadata change log is maintained in a persistent memory of a data storage system, and stores metadata changes to be aggregated and applied to data storage system metadata located in non-volatile data storage. New metadata changes are added to the metadata change log by storing the new metadata changes into the persistent memory, storing structural modifications to the metadata change log needed to add the new metadata changes to the metadata change log into a transaction entry located in the persistent memory, and processing the resulting transaction entry to perform the structural modifications to the metadata change log that were stored in the transaction entry.