Partitioned Metadata Management for Data Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data deduplication methods struggle to efficiently manage frequently updated data, leading to high resource occupation and reduced deduplication ratios, which in turn causes infrequently updated data to be evicted prematurely.
Innovation Solution
The proposed method actively partitions a metadata management structure based on the features of data blocks, such as fingerprints and address information, to prevent infrequently updated data from being evicted by allocating sufficient resources to frequently updated data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If data blocks are stored in a unified metadata management structure without partitioning, then the structure is simple to implement, but frequently updated data occupies resources and causes infrequently updated data to be evicted, reducing deduplication ratio
Solution Approach 1:
The metadata management structure is divided into multiple partitions based on data block features (such as fingerprint hash values). This segmentation allows different types of data to be stored in separate partitions, preventing resource competition between frequently updated and infrequently updated data, thereby maintaining high deduplication ratio without excessive complexity
Solution Approach 2:
Different partitions are created with specific characteristics tailored to different data types. Each partition can have optimized parameters for its specific data characteristics, allowing infrequently updated data to be stored in partitions with appropriate resource allocation, preventing eviction and improving overall deduplication effectiveness
2Device complexity
If resources are allocated uniformly to all data blocks, then the allocation is simple, but frequently updated data occupies excessive resources causing infrequently updated data to be evicted
Solution Approach 1:
Resource allocation is made non-uniform based on data block characteristics. Partitions storing infrequently updated data are allocated sufficient resources to prevent eviction, while partitions for frequently updated data have appropriate resource limits. This local differentiation optimizes resource distribution without complex centralized control
Solution Approach 2:
The resource allocation problem is solved by segmenting the metadata management structure into partitions. Each partition manages its own resources independently, allowing infrequently updated data to secure adequate resources without being displaced by frequently updated data, thus preventing resource-wasting eviction
Data Source
AI summary
A data deduplication method includes: receiving a write request with first data block; writing the first data block into a storage device; writing metadata of the first data block into a first partition that is in a plurality of partitions of a metadata management structure and that is determined based on a feature of the first data block; and deleting the metadata of the first data block in the first partition and deleting the first data block from the storage device based on address information of the first data block when a fingerprint that is the same as the fingerprint of the first data block exists in the first partition. The method can prevent infrequently updated data from being evicted because resources are occupied by frequently updated data, thereby improving a deduplication ratio.