Bulk Metadata Page Flush Bypassing Delta Log Overload

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Clustered storage systems face inefficiencies and potential overload when handling highly amortized metadata page updates, as writing these updates to delta log infrastructure can lead to reduced efficiency and incomplete application of recent changes during recovery from disasters or data corruption.

Innovation Solution

Implementing a transaction commit operation that writes bulk metadata pages to a transactional buffer area, persists raw delta updates and drop-delta flags to a metadata delta log, synchronizes these updates, and flushes the bulk metadata pages directly to the storage array, bypassing the delta log to avoid overloading and ensure efficient data recovery.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If bulk metadata pages with large numbers of delta updates are written to the delta log infrastructure, then data consistency is maintained, but the delta log infrastructure becomes overloaded and efficiency is reduced

Engineering Contradiction:

Improvedata consistencyVSAvoiddelta log efficiency

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The patent segments the metadata update path into two distinct channels: (1) a delta log path for small, incremental updates that maintain consistency, and (2) a direct path for bulk metadata pages that bypass the delta log infrastructure. This segmentation allows small updates to continue using the delta log for consistency while bulk updates use a separate path that doesn't overload the delta log, thereby resolving the contradiction between maintaining data consistency and preserving delta log efficiency.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces a metadata page cache as an intermediary component between the host and the storage array. This cache holds bulk metadata pages and manages their lifecycle, allowing the system to apply delta updates to cached pages in memory and only flush to storage when necessary. The intermediary absorbs the burden of handling bulk updates, preventing direct overload of the delta log infrastructure while maintaining data consistency through controlled flush operations.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Productivity

If bulk metadata pages are flushed directly to storage array bypassing delta log, then delta log infrastructure overload is prevented, but recent delta updates may not be appropriately applied

Engineering Contradiction:

Improvedelta log efficiencyVSAvoiddelta update application

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

The patent applies preliminary action by pre-loading bulk metadata pages into the metadata page cache before they are needed, and pre-applying delta updates to these cached pages in memory. This allows the system to prepare the metadata state in advance, so when bulk pages are flushed to storage, the delta updates have already been incorporated into the cache, ensuring consistency without requiring the delta log infrastructure to track every individual update.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent implements a feedback mechanism through the metadata page cache that tracks which delta updates have been applied to cached bulk metadata pages. The cache maintains information about the state of each metadata page and coordinates with the delta log infrastructure to ensure that when bulk pages are flushed to storage, all relevant delta updates have been appropriately applied. This feedback loop ensures reliability while allowing the direct flush path to maintain productivity.

Inventive Principle:

Principle #23Feedback

3Quantity of substance

If small delta updates are aggregated and stored in amortized fashion, then storage efficiency is improved, but handling highly amortized pages becomes problematic

Engineering Contradiction:

Improvestorage efficiencyVSAvoiddelta log infrastructure complexity

Core Design Contradiction:

Quantity of substanceVSDevice complexity

Solution Approach 1:

The patent applies parameter changes by introducing a threshold parameter that determines when bulk metadata pages should be created versus when small delta updates should continue to be aggregated. When the number of delta updates to a metadata page exceeds this threshold, the system transitions from aggregation mode to bulk page creation mode. This parameter-based approach allows the system to optimize for storage efficiency through aggregation while avoiding the complexity of managing highly amortized pages in the delta log infrastructure by switching to a different handling mechanism.

Inventive Principle:

Principle #35Parameter changes

Data Source

PatentUS11347725B2Efficient handling of highly amortized metadata page updates in storage clusters with delta log-based architectures

Publication Date: 2022.05.31 EMC IP HLDG CO LLC

AI summary

Techniques for handling highly amortized metadata page updates in storage clusters with delta log-based architectures. In response to performance of a transaction commit operation, bulk metadata pages are written in a transactional area of persistent memory in association with a unique transaction identifier. Further, raw delta updates to metadata pages and, for each bulk metadata page, a specialized flag, are written to a metadata delta log in association with the unique transaction ID. The specialized flag provides an indication that all delta updates that have occurred previous to the writing of a bulk metadata page are no longer relevant and should be dropped. The raw delta updates to metadata pages and specialized flags are updated in a volatile memory structure. The bulk metadata pages are flushed from the transactional buffer area directly to a metadata store of a storage array in a manner that bypasses the metadata delta log.