Dense Tree Metadata Update Logging and Checkpointing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing storage systems face inefficiencies in updating and managing metadata across a distributed storage cluster, particularly in determining the location of storage container data, which leads to cumbersome updates and suboptimal performance.
Innovation Solution
A multi-level dense tree metadata structure is implemented, where each level includes volume metadata entries, with a top level maintained in-core for efficient merging and checkpointing to solid-state drives, and lower levels updated batch-wise, utilizing a two-level append-only logging structure to record changes and facilitate efficient logging and checkpointing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If metadata is updated every time storage container data location changes, then data location tracking is accurate, but update overhead and system complexity increase significantly
Solution Approach 1:
The patent segments metadata into multiple levels (first level, second level, third level) with different update frequencies. The first level metadata is updated frequently with current data locations, while second and third level metadata are updated less frequently, reducing overall update complexity while maintaining accurate current location tracking through the first level.
Solution Approach 2:
The patent performs preliminary actions by pre-calculating and storing aggregated metadata information in higher levels before it is needed. This allows the system to quickly resolve data locations without performing complex calculations at query time, reducing the apparent complexity of location tracking.
2Reliability
If metadata is updated frequently to reflect data location changes, then data location information remains current, but write amplification and storage operations increase
Solution Approach 1:
The patent divides metadata update operations into segments based on hierarchy levels. Only the first level metadata requires frequent updates with current location information, while second and third levels are updated less frequently. This segmentation reduces the total volume of data that must be written frequently, thereby reducing write amplification while maintaining reliable current location tracking.
Solution Approach 2:
The patent merges metadata updates across multiple levels in a hierarchical manner. Changes are propagated from the first level to higher levels only when necessary, combining update operations to minimize redundant writes. This merging strategy reduces write amplification by eliminating duplicate update operations across the metadata hierarchy.
3Productivity
If a simple metadata structure is used, then update operations are fast, but data location determination becomes inefficient
Solution Approach 1:
The patent segments the metadata structure into a hierarchical hierarchy where each level serves a specific function. The first level stores current location information for quick updates, while higher levels store aggregated data for efficient range queries. This segmentation allows both fast updates (affecting only the first level) and efficient location determination (utilizing the hierarchical structure).
Solution Approach 2:
The patent introduces a hierarchical dimension to the metadata structure, organizing data across multiple levels rather than using a flat structure. This dimensional change enables the system to optimize for both update speed (by localizing updates to specific levels) and query efficiency (by leveraging the hierarchical organization for range queries).
Data Source
AI summary
The embodiments described herein are directed to efficient merging of metadata managed by a volume layer of a storage input/output (I/O) stack executing on one or more nodes of a cluster. The metadata managed by the volume layer, i.e., the volume metadata, is illustratively organized as a multi-level dense tree metadata structure, wherein each level of the dense tree metadata structure (dense tree) includes volume metadata entries for storing the volume metadata. The volume metadata entries of an upper level of the dense tree metadata structure are merged with the volume metadata entries of a next lower level of the dense tree metadata structure when the upper level is full. The volume metadata entries of the merged levels are organized as metadata pages and stored as one or more files on the SSDs.