Sub-block Data Deduplication via Shared Mapping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data deduplication techniques fail to detect redundant data when blocks are similar but not identical, missing opportunities for deduplication by only considering full block matches, and not accounting for partial similarities between candidate and target blocks.
Innovation Solution
Implementing a shared sub-block mapping (SSM) structure that compares sub-blocks of candidate and target blocks using digests and entropy values to identify matching sub-blocks, allowing for partial deduplication and storing metadata to indicate shared sub-blocks, thereby enabling flexible partial block deduplication.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If full block deduplication is used, then storage efficiency is improved for identical blocks, but redundant data detection fails for similar but not identical blocks
Solution Approach 1:
The patent divides data blocks into smaller sub-blocks (e.g., 512-byte sectors) and performs deduplication at the sub-block level rather than requiring full block matches. This segmentation allows the system to identify and deduplicate similar data even when blocks differ slightly, resolving the contradiction between storage efficiency and detection accuracy.
Solution Approach 2:
The patent changes the granularity parameter from full block level to sub-block level, and introduces a similarity threshold parameter (e.g., 70% or 80% match) to determine deduplication eligibility. This allows flexible control over deduplication behavior, improving both storage efficiency and detection reliability.
2Reliability
If sub-block level comparison is implemented, then detection accuracy for similar blocks is improved, but processing complexity increases
Solution Approach 1:
The patent performs preliminary actions by first computing digests for all sub-blocks of candidate and target blocks before performing actual comparisons. This preliminary digest computation and indexing allows for rapid similarity assessment, reducing the complexity of subsequent detailed sub-block comparisons.
Solution Approach 2:
The patent introduces an intermediary data structure (SSM - Shared Sub-block Mapping) that mediates between candidate blocks and target blocks. This SSM structure stores mapping relationships between matching sub-blocks, simplifying the comparison process and reducing processing complexity while maintaining high detection accuracy.
3Loss of substance
If partial deduplication is enabled, then storage efficiency is improved for similar blocks, but metadata structure complexity increases
Solution Approach 1:
The patent segments the metadata structure into multiple components: sub-block digests, SSM mappings, and deduplication indicators. This segmentation allows the system to efficiently represent partial deduplication relationships without creating overly complex monolithic metadata structures.
Solution Approach 2:
The patent uses copying by storing references and mappings (SSM) rather than duplicating entire block data. The SSM structure contains pointers and mapping information that references original sub-blocks, reducing metadata overhead while enabling partial deduplication.
Data Source
AI summary
Techniques for data processing may include: determining one or more sub-blocks of a target block that match one or more sub-blocks of a candidate block; creating a shared sub-block mapping (SSM) structure having a plurality of entries, wherein each of the plurality of entries corresponds to a different one of the sub-blocks in the candidate block and wherein a value stored in said each entry, corresponding to one of the sub-blocks of the candidate block, identifies a sub-block of the target block matching said one sub-block of the candidate block; and storing the candidate block as a deduplicated block sharing at least one sub-block with the target block. The SSM structure may be stored as a metadata structure of the candidate block to identify deduplicated sub-blocks of the candidate block and to identify sub-blocks of the target block providing content for the deduplicated sub-blocks of the candidate block.