Variable Size Metadata Pages for Sparse Storage Utilization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage systems face inefficiencies in metadata storage, particularly when logical volumes are sparse, leading to wasted space due to fixed-size metadata pages that store both valid and invalid data, and this issue is exacerbated by snapshotting operations.
Innovation Solution
The implementation of a log-structured system (LSS) approach and architecture that uses variable size metadata pages, storing only valid pointers and updates in a metadata log, eliminating the need for physical 'in place' updates and allowing for more efficient storage utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If fixed-size metadata pages are used to store both valid and invalid data, then storage structure simplicity is maintained, but storage space is wasted especially in sparse logical volume scenarios
Solution Approach 1:
The patent segments the metadata storage by separating valid pointers from invalid pointers within metadata pages. Instead of storing all pointers uniformly, the system divides metadata pages into regions for valid entries and invalid entries, allowing sparse logical volumes to occupy only necessary storage space while maintaining structured organization.
Solution Approach 2:
The patent implements dynamic metadata page sizing and structure based on the actual data density. Metadata pages can adapt their layout to accommodate varying proportions of valid and invalid pointers, transitioning between dense and sparse representations depending on the logical volume's fill status, thereby optimizing storage utilization.
2Ease of operation
If physical in-place updates are performed on metadata pages, then update simplicity is maintained, but storage requirements increase due to need for overwrite capacity
Solution Approach 1:
The patent extracts the update operation from the metadata pages themselves and relocates it to a separate log structure. Updates are recorded as append-only entries in the log rather than being performed in-place on metadata pages, separating the update mechanism from the storage structure and reducing storage requirements.
Solution Approach 2:
The patent introduces a log structure as an intermediary between update operations and metadata pages. The log acts as a buffer that captures updates sequentially, allowing the system to defer and batch metadata page updates, thereby reducing the need for immediate in-place overwrite capacity while maintaining data consistency.
3Device complexity
If fixed-size metadata pages are used, then page management simplicity is maintained, but storage utilization efficiency decreases in sparse scenarios
Solution Approach 1:
The patent applies local quality by allowing different regions within metadata pages to have different characteristics. Valid pointer regions and invalid pointer regions can be managed differently, with the system selectively activating or deactivating regions based on the logical volume's actual data density, thereby improving storage utilization without complicating overall page management.
4Reliability
If snapshotting operations are performed on fixed-size metadata pages, then snapshot consistency is maintained, but storage overhead increases
Solution Approach 1:
The patent performs preliminary actions by pre-allocating and initializing metadata pages with a standard structure before snapshotting operations. This preliminary setup ensures that when snapshots are taken, the metadata pages are already in a consistent state, reducing the need for additional storage overhead to maintain snapshot consistency.
Data Source
AI summary
In at least one embodiment, a log-structured system for metadata (MD) updates can be leveraged to enable use of variable size MD pages with multiple different encoding formats. Processing can include: generating an updated MD page by performing one or more updates to an existing version of the MD page; selecting a first page type to be used for encoding the updated MD page, wherein said selecting selects the first page type from a plurality of predefined page types, wherein each of the plurality of predefined page types identifies a different metadata layout or format for storing MD pages; encoding the updated MD page in accordance with the first page type and accordingly generating a first encoded version of the updated MD page; and storing the first encoded version of the updated MD page at a first physical storage location of a non-volatile storage device.