Deduplicating Data Blocks Using Digest Subset Evaluation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data deduplication systems face challenges in efficiently processing large sets of data due to resource constraints, leading to increased storage and CPU utilization, and the need for large index tables that consume memory and time, potentially resulting in stale data.
Innovation Solution
A method that selects a subset of data blocks based on digest information evaluation, storing only a subset of digest information in an index table to deduplicate ranges of data blocks, reducing the size of the index table and processing overhead, and allowing deduplication without storing digest information for every data block.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If digest information for every data block is stored in an index table, then deduplication accuracy is improved, but memory consumption and processing time increase significantly
Solution Approach 1:
The patent segments the data storage space into multiple extents and processes deduplication in smaller chunks rather than loading all data into memory at once. This allows the system to handle large datasets by dividing them into manageable portions, reducing peak memory consumption while maintaining deduplication functionality across the entire dataset.
Solution Approach 2:
The patent implements partial action by selecting a subset of data blocks for digest calculation and index table population based on criteria such as data patterns, access frequency, or extent boundaries. This partial processing approach reduces memory usage and processing time while still achieving effective deduplication for the most relevant data portions.
2Reliability
If a large index table is created to store all digest information, then complete deduplication coverage is achieved, but processing speed decreases due to increased search time
Solution Approach 1:
The index table is segmented into multiple smaller index structures corresponding to different extents or data ranges. Each segment handles a specific portion of the data, allowing for faster local searches and reduced overall processing time compared to a single large index table, while maintaining complete coverage through the combination of all segments.
Solution Approach 2:
The system performs partial indexing by creating index entries only for selected data blocks that meet specific criteria, such as being part of a deduplication candidate set or falling within certain extent boundaries. This selective indexing approach maintains adequate deduplication coverage while significantly reducing the size of the index table and associated processing overhead.
3Loss of substance
If all data blocks are processed for deduplication, then maximum space savings are achieved, but CPU utilization increases leading to system performance degradation
Solution Approach 1:
The patent extracts and processes only the most relevant data blocks for deduplication based on specific criteria such as data patterns, redundancy indicators, or business rules. By taking out only the necessary portions of data for processing rather than analyzing everything, the system achieves significant space savings in duplicate data while minimizing CPU utilization and avoiding system performance degradation.
4Reliability
If the entire dataset is loaded into memory for processing, then deduplication completeness is improved, but system resources are exhausted and performance slows down
Solution Approach 1:
The dataset is divided into multiple smaller segments or extents that are processed sequentially or in parallel batches. Each segment is loaded into memory independently, processed for deduplication, and then released before moving to the next segment. This segmentation approach ensures complete deduplication across the entire dataset while maintaining system performance by keeping memory usage within acceptable limits at all times.
Data Source
AI summary
A method is used in deduplicating ranges of data blocks. A collection of data is selected from a set of storage extents. The collection of data is selected based on evaluation of digest information for a subset of the collection of data. A deduplicating technique is applied to the collection of data.