Data Deduplication via Digest-Based Block Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data deduplication systems face challenges in efficiently processing large sets of data due to the need for extensive index tables that consume storage resources and memory, leading to increased processing time and potential data staleness, especially when dealing with limited resources.
Innovation Solution
The method involves selecting a pair of data blocks based on digest information for deduplication processing, storing only the digest information for these blocks in an index table, and applying deduplication techniques to the associated data blocks, allowing for reduced resource utilization and improved performance by minimizing the size of the index table and processing only a subset of data blocks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If digest information for every data block is stored in an index table, then duplicate detection accuracy is improved, but storage resource consumption and memory usage increase
Solution Approach 1:
The patent segments the data blocks into different sizes (e.g., 4KB, 8KB, 16KB) and processes them separately. By dividing the data into manageable chunks, the system can maintain index tables for smaller segments without requiring memory for entire large files, thus reducing peak memory consumption while still achieving comprehensive duplicate detection across all segments.
Solution Approach 2:
The patent applies partial action by processing only a subset of data blocks at any given time rather than loading all blocks into memory simultaneously. The system processes data blocks in batches or streams, maintaining index tables only for the current batch being processed, which significantly reduces memory requirements while still achieving complete deduplication coverage across the entire dataset.
2Measurement precision
If extensive index tables are used for deduplication processing, then duplicate detection completeness is improved, but processing time increases
Solution Approach 1:
The patent divides the deduplication process into segments based on data block sizes and processes them in parallel or sequential batches. This segmentation allows the system to build and query index tables for smaller subsets of data multiple times rather than building one large index table once, reducing the time complexity from O(n) to approximately O(n/k) where k is the number of segments.
Solution Approach 2:
The system performs partial deduplication passes on subsets of data blocks rather than processing the entire dataset in a single pass. By repeatedly processing smaller portions of data with refreshed index tables, the system achieves complete duplicate detection while avoiding the memory and time overhead of maintaining a single comprehensive index table for all data.
3Measurement precision
If large index tables are maintained in memory, then deduplication accuracy is improved, but system performance and responsiveness deteriorate
Solution Approach 1:
The patent implements segmentation by processing data in fixed-size blocks (e.g., 4KB, 8KB, 16KB) and maintaining index tables only for the current block being processed. This approach keeps memory usage constant and predictable regardless of total data size, allowing the system to maintain high deduplication accuracy while preserving system responsiveness and overall productivity.
Solution Approach 2:
The system applies partial action by repeatedly processing small portions of data with fresh index tables rather than maintaining a large persistent index table. This strategy ensures that memory resources remain available for other system operations, preventing performance degradation while still achieving accurate duplicate detection through multiple passes over the data.
Data Source
AI summary
A method is used in deduplicating sets of data blocks. A collection of data is selected from a set of storage extents. The collection of data is selected based on evaluation of digest information for first and second data blocks of the collection of data. A deduplicating technique is applied to the collection of data.