Hybrid Deduplication With Variable-Size Detection And Fixed-Size Segments
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data deduplication techniques face inefficiencies as they either operate on fixed-sized units, missing opportunities for deduplication of content aligned differently, or on variable-sized items, requiring excessive overhead for metadata management.
Innovation Solution
A hybrid deduplication system that calculates unique identifiers for variable-sized data sections, compares them to existing stored identifiers, and creates references for matching sections, while writing non-matching sections to storage, thereby optimizing storage space without unnecessary duplication.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If data deduplication operates on fixed-size units of storage, then storage organization is simplified, but opportunities to deduplicate content aligned differently within fixed-sized units are missed
Solution Approach 1:
The patent segments data into fixed-size units for storage organization while simultaneously enabling variable-size deduplication operations by allowing flexible alignment and comparison across segment boundaries. This segmentation approach maintains organizational simplicity while improving deduplication effectiveness through multi-granularity processing.
2Productivity
If data deduplication operates on variable-sized items of data, then deduplication coverage is improved, but overhead for metadata management increases
Solution Approach 1:
The patent implements a dynamic deduplication system that adapts the granularity of operation based on data characteristics and storage requirements. The system can switch between fixed-size and variable-size operations, dynamically adjusting metadata management complexity to match deduplication coverage needs while optimizing resource utilization.
3Loss of substance
If variable-sized data sections are deduplicated, then storage space optimization is improved, but alignment with fixed-size storage segments becomes complex
Solution Approach 1:
The patent introduces a hierarchical dimension to data organization, operating simultaneously at multiple levels: variable-size logical data sections for deduplication purposes and fixed-size physical storage segments for implementation. This dimensional approach allows storage space optimization through variable-size matching while managing alignment complexity through fixed-size physical boundaries.
Data Source
AI summary
A hybrid deduplication system operates to detect variable-sized deduplication matches, while performing the storage deduplication on fixed-size segments of data. The hybrid deduplication system calculates unique identifiers for variable-sized sections of data within a data stream being written to a deduplicated data store. The hybrid deduplication system then compares those newly-calculated identifiers to identifiers of variable-sized sections of data that have already been stored within the deduplicated data store. If a match is found, the hybrid deduplication system identifies the location of each of the fixed-size data segment(s), already stored in the deduplicated data store, that include the identified variable-sized section of data. Instead of writing the sections that match already-existing sections to the deduplicated data store, the hybrid deduplication system simply causes the creation of a reference to the identified storage locations, indicating that the data stream being written includes the data in these pre-existing storage locations.