Dense Tree Metadata Structure for Distributed Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing storage systems face inefficiencies in updating metadata when storage container data locations change within a distributed storage cluster, leading to cumbersome metadata updates and suboptimal performance.
Innovation Solution
A dense tree metadata structure is implemented, organizing metadata as a multi-level data structure where recent changes are stored in memory and older changes are stored on storage devices, enabling efficient access and reducing metadata overhead by using durable extent keys and volume metadata entries.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If metadata is maintained in a traditional data structure to track storage container data locations, then the system can identify data locations, but metadata updates become cumbersome every time data location changes
Solution Approach 1:
The metadata structure is segmented into multiple levels (root level, intermediate levels, and leaf level) forming a tree structure. Each level manages a portion of the address space, allowing incremental updates without modifying the entire metadata structure when data locations change.
Solution Approach 2:
The patent transitions from a flat, single-level metadata structure to a multi-dimensional hierarchical tree structure. This adds depth to the metadata organization, enabling efficient localization of changes and reducing update complexity by affecting only specific branches of the tree rather than the entire structure.
2Reliability
If metadata is updated frequently to reflect changing data locations, then data location accuracy is maintained, but system performance degrades due to update overhead
Solution Approach 1:
By segmenting metadata into hierarchical levels, the patent enables selective updates at specific levels rather than updating the entire metadata structure. This reduces the amount of metadata that needs to be modified and written back when data locations change, thereby maintaining accuracy while improving throughput.
Solution Approach 2:
The patent applies partial action by updating only the necessary portions of the metadata tree rather than the complete structure. When a data location changes, only the affected leaf node and its ancestor paths need updating, not the entire metadata structure, thus reducing update overhead while maintaining location accuracy.
3Measurement precision
If a detailed metadata structure is used to precisely track data locations, then data access accuracy improves, but metadata overhead increases
Solution Approach 1:
The hierarchical tree structure segments metadata management across multiple levels, where each level stores aggregated information about its subtree. This segmentation allows precise data location identification at the leaf level while reducing overall overhead by avoiding redundant storage of complete path information at each level.
Solution Approach 2:
The patent merges multiple metadata functions into the hierarchical structure, where internal nodes store aggregated information about their child nodes. This merging reduces redundancy and optimizes the metadata-to-data ratio by combining location tracking, allocation management, and structural organization into a unified efficient structure.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
In one embodiment, a node coupled to one or more storage devices executes a storage input/output (I/O) stack having a volume layer. The volume layer manages volume metadata embodied as mappings from offsets of a logical unit (LUN) to extent keys associated with storage locations for extents on the one or more storage devices. Volume metadata is maintained as a dense tree metadata structure representing successive points in time. The dense tree metadata structure has multiple levels, wherein a top level of the dense tree metadata structure represents newer volume metadata changes and descending levels of the dense tree metadata structure represent older volume metadata changes. The node accesses a latest version of changes to the volume metadata by searching from the top level to the descending levels in the dense tree metadata structure.