Data Deduplication via Hash Table Reference Counts
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data deduplication techniques in data storage systems face inefficiencies in identifying and managing duplicate data blocks across multiple logical storage entities, leading to suboptimal storage utilization and performance.
Innovation Solution
A method that involves deriving hash values for data blocks, maintaining a table with reference counts for each hash value, and determining deduplication processing based on a threshold rate, allowing for selective caching and deduplication across multiple logical storage entities, thereby optimizing storage and performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If deduplication processing is performed for all data blocks, then storage efficiency is improved, but processing time and system performance deteriorate
Solution Approach 1:
The patent applies partial action by performing deduplication processing selectively rather than universally. It uses hash value tables and reference counts to identify data blocks that meet specific deduplication criteria, processing only those blocks where deduplication is likely to be effective. This selective approach reduces overall processing time while maintaining storage efficiency for the most beneficial cases.
Solution Approach 2:
The patent implements preliminary action by pre-computing and storing hash values for data blocks in a hash value table before actual deduplication processing. This pre-processing step allows the system to quickly identify candidate blocks for deduplication without performing full comparison operations on every block, thereby reducing the time required for the actual deduplication process.
2Measurement precision
If a comprehensive deduplication data store is maintained, then duplicate detection accuracy is improved, but memory usage and system complexity increase
Solution Approach 1:
The patent segments the deduplication data store into multiple hash value tables, each organized by hash value ranges or prefixes. This segmentation allows the system to maintain comprehensive deduplication coverage while reducing the complexity of any single table. The segmented structure enables more efficient memory management and simpler lookup operations compared to a single monolithic data store.
Solution Approach 2:
The patent introduces hash values as intermediaries between raw data blocks and the deduplication data store. Instead of storing and comparing entire data blocks directly, the system uses compact hash values as proxies for identification and comparison. This intermediary approach maintains high duplicate detection accuracy while significantly reducing memory usage and system complexity.
3Reliability
If reference counting is performed for all data blocks, then data integrity is improved, but processing overhead increases
Solution Approach 1:
The patent applies partial action to reference counting by maintaining reference counts only for data blocks that are stored in the deduplication data store, rather than for all data blocks in the system. This selective reference counting approach ensures data integrity for deduplicated blocks while reducing the processing overhead associated with tracking references across the entire data set.
Data Source
AI summary
Techniques for processing data may include: receiving a data block stored in a data set, wherein a hash value is derived from the data block; determining, in accordance with selection criteria, whether the hash value is included in a subset; responsive to determining the hash value is included in the subset, performing processing that updates a table in accordance with the hash value and the data set, and determining, in accordance with the information in the table, whether to perform deduplication processing for the data block to determine whether the data block is a duplicate of another stored data block. The table may include an entry for the hash value. The entry may include information identifying data sets referencing the data block and, for each of the data sets, may specify a reference count denoting a number of times the data set references the data block.