Metadata Structure for Partial Page Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data deduplication methods are not scalable and incur overhead costs due to increased IO access and metadata complexity, especially when dealing with partial page deduplication, as they require saving multiple instances of pages with only minor differences.
Innovation Solution
A metadata structure with multiple levels of indirection is implemented, including a logical level pointing to a virtual level and a physical level, where a reference counter tracks instances of shared sectors, allowing for efficient partial deduplication by storing a single instance of common sectors at the physical level and maintaining pointers between virtual level address entries.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If traditional data deduplication methods are used to eliminate duplicate data, then storage space is saved, but input/output access overhead and metadata complexity increase
Solution Approach 1:
The patent segments data into fixed or variable-sized segments and assigns identifiers to each segment. The metadata structure is segmented into multiple levels (logical level, virtual level, physical level), allowing the system to manage deduplication metadata in a structured manner that reduces overall complexity while maintaining storage efficiency.
Solution Approach 2:
The patent introduces multiple levels of indirection (logical level pointing to virtual level, which points to physical level) to transform the metadata management problem from a single-dimension structure into a multi-dimensional hierarchy. This dimensional transformation allows the system to reduce metadata complexity by distributing information across multiple levels rather than maintaining all metadata in a single complex structure.
2Reliability
If multiple instances of pages with minor differences are stored to maintain data integrity, then data reliability is preserved, but storage efficiency decreases
Solution Approach 1:
The patent applies local quality by making different parts of the data structure serve different functions. The metadata structure contains specific fields for tracking differences (such as offset and length information) while sharing common data segments. This allows the system to maintain data integrity for pages with minor differences by storing only the unique portions locally while referencing shared common sectors, thereby preserving reliability without sacrificing storage efficiency.
Solution Approach 2:
The patent merges multiple pages that share common sectors by storing a single instance of the common data at the physical level and using pointers at the virtual level to reference it. Pages with minor differences are merged by sharing the common portion while maintaining separate metadata entries for the unique portions, thus combining storage efficiency with data integrity.
3Quantity of substance
If partial page deduplication is implemented to improve storage efficiency, then capacity utilization increases, but IO access overhead increases
Solution Approach 1:
The patent segments pages into common sectors and unique portions, allowing partial deduplication at the sector level rather than requiring full page-level deduplication. This segmentation enables the system to improve capacity utilization by storing only unique portions of pages while maintaining pointers to shared common sectors, reducing the IO access overhead compared to traditional full-page deduplication approaches.
Solution Approach 2:
The patent introduces a virtual level as an intermediary between the logical level and physical level, which contains pointers to physical data locations. This intermediary layer allows the system to manage partial page deduplication efficiently by resolving pointers to common sectors without requiring direct access to the underlying physical storage, thereby reducing IO access overhead while maintaining high capacity utilization.
Data Source
AI summary
An aspect includes providing a metadata structure having a logical level that points to a virtual level and a physical level to which the virtual level points. The method also includes storing, at the virtual level, a reference counter for each of a plurality of virtual-level type storage address entries in the metadata structure, and providing a pointer in the metadata structure between each pair of a number of pairs of virtual level address entries in which corresponding pages share a set of common sectors. The reference counter tracks a number of instances in which a corresponding pointer points to a corresponding virtual level address entry. An aspect further includes storing a single instance of the common sectors at the physical level.