Globally Optimized Partial Page Deduplication Storage Objects
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Partial deduplication techniques in storage systems lead to performance reductions due to increased latency and I/O overhead, as they are sensitive to the order of page arrival and local optimization decisions, resulting in suboptimal data reduction ratios and read performance.
Innovation Solution
A method for globally optimized partial deduplication that groups pages sharing common features, assigns representative dedupe pages, and writes them to corresponding physical containers, optimizing both capacity reduction and read performance by aggregating similar pages into the same metadata and physical containers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If partial deduplication is implemented, then storage capacity is reduced, but read performance and latency are degraded
Solution Approach 1:
The patent segments pages into fixed-size chunks and processes them independently through hash functions. Each page is divided into multiple segments that can be hashed and compared separately, enabling parallel processing and reducing the computational overhead that degrades read performance while maintaining effective deduplication
Solution Approach 2:
The patent performs preliminary hashing of page segments during the write operation to create a deduplication index. This preliminary action stores hash values and metadata in advance, so that subsequent read operations can quickly verify deduplication status without performing complex comparisons, thereby improving read performance while maintaining storage reduction
2Quantity of substance
If partial deduplication is implemented, then storage overhead is reduced, but I/O access overhead increases
Solution Approach 1:
The patent extracts only the essential hash values and metadata from full page content for storage in the deduplication index. By taking out only the critical identifying features (hashes of fixed-size segments) rather than storing complete page copies, the system reduces storage overhead while enabling fast I/O access through efficient hash-based lookup
Solution Approach 2:
The patent changes the parameter of data representation from storing complete page content to storing compact hash values and metadata. This parameter change transforms the storage format into a more space-efficient representation that also enables faster comparison and retrieval operations, reducing both storage overhead and I/O access time
Data Source
AI summary
An aspect of implementing globally optimized partial deduplication of storage objects includes gathering pages that share a common feature, dividing the pages into groups based on commonality with corresponding representative pages, where each is assigned as a representative dedupe page for the corresponding groups. For each group in the groups of pages, an aspect also includes writing the pages to a corresponding physical container.