Dual Metadata Tree for Storage Object Location Management
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Enterprise storage systems face challenges in managing large numbers of clones or snapshots, leading to metadata bloat and inefficient data storage due to each copy managing its own metadata structure, which complicates lookups, data access, and modifications.
Innovation Solution
A dual-metadata tree system is implemented, where a primary metadata tree provides a virtual address for locating primary object data, and sub-trees from leaf nodes manage derived objects, with an ancestry graph linking derived object identifiers to facilitate quick lookups and reduce metadata overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If each copy manages its own metadata structure, then data access and modifications can be performed independently, but metadata bloat occurs and system complexity increases
Solution Approach 1:
Multiple independent metadata structures managed by each copy are merged into a single shared metadata structure. This allows all copies to access and modify data independently while avoiding the complexity of managing separate metadata structures for each copy.
Solution Approach 2:
The shared metadata structure serves multiple copies simultaneously, making it a universal structure that handles data for all copies. This multi-functional approach eliminates the need for separate metadata structures while preserving independent access capabilities through copy-specific lookup paths.
2Adaptability or versatility
If multiple clones or snapshots are created, then data protection and development capabilities are enhanced, but storage space is consumed by excess data copies
Solution Approach 1:
Multiple copies share common data blocks through the shared metadata structure, merging their data storage requirements. This allows many clones and snapshots to coexist without consuming proportional storage space for each copy's data, as they all reference the same underlying data blocks.
Solution Approach 2:
Instead of creating physical copies of data for each clone or snapshot, the system creates references to the same data blocks. This logical copying approach enables unlimited copies to be created while using minimal additional storage space, as only the metadata references are duplicated, not the actual data.
3Quantity of substance
If data deduplication and change bitmaps are used, then data bloat from copies is reduced, but lookup and access operations become more complex
Solution Approach 1:
The complex data deduplication and change bitmap mechanisms are extracted and replaced with a simpler shared metadata structure. This extraction eliminates the need for complex deduplication algorithms while maintaining efficient data access through direct lookup paths in the shared metadata tree.
Solution Approach 2:
The metadata structure is segmented into a shared common metadata structure and copy-specific lookup paths. This segmentation allows simple, direct access for each copy while the shared structure handles deduplication implicitly, reducing operational complexity compared to managing complex deduplication logic for each copy.
Data Source
AI summary
Described are techniques for managing data locations of derived storage objects. The techniques include accessing a primary metadata tree providing a lookup path using a virtual address to locate data for a primary object in physical storage pointed to by a leaf node of the primary metadata tree and providing a sub-tree from a leaf node of the primary metadata tree associated with the primary object, the sub-tree providing lookup paths using virtual addresses for derived objects. Wherein a derived object is a view of the primary object at a specific point in time, with copy leaf nodes of the sub-tree pointing to stored data of the derived objects. The techniques further include maintaining an ancestry graph associated with the sub-tree with the ancestry graph linking derived object identifiers to provide an ancestral chain lookup to identify derived object identifiers of the closest stored physical data of a derived object.