Deduplication Tables Using Hash Segmentation to Reduce CPU Overhead
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data deduplication solutions face performance issues due to high CPU overhead, false positives, and significant disk I/O operations, particularly when dealing with large datasets and maintaining persistent databases of signatures, leading to mediocre performance unless substantial computing and DRAM resources are used.
Innovation Solution
A system utilizing a Known Duplicates (KD) Table and a Possible Duplicates (PD) Table, along with a KD Filter, to efficiently index and manage deduplicated data blocks, reducing the need for persistent database lookups and minimizing unnecessary comparisons, thereby improving deduplication performance and reducing CPU and disk I/O costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If cryptographic signatures (SHA-1, SHA-2) are used for deduplication, then data integrity is guaranteed, but CPU overhead becomes excessive
Solution Approach 1:
The patent segments the signature verification process into two stages: first using a small non-cryptographic hash (like Murmur or FarmHash) as a quick filter, and only using cryptographic signatures for blocks that pass the filter. This segmentation reduces CPU overhead by avoiding expensive cryptographic operations on all blocks while maintaining data integrity through selective verification.
Solution Approach 2:
The patent introduces non-cryptographic hash functions as intermediary filters between the deduplication system and the cryptographic signature verification. These intermediaries quickly eliminate non-duplicate blocks without requiring CPU-intensive cryptographic operations, while still allowing the system to guarantee data integrity through subsequent cryptographic verification of potential duplicates.
2Use of energy by moving object
If small non-cryptographic signatures are used, then CPU overhead is reduced, but false positives increase requiring additional block comparisons
Solution Approach 1:
The patent segments the duplicate detection process into two phases: Phase 1 uses small non-cryptographic hashes for quick filtering with acceptable false positives, and Phase 2 performs definitive verification only on blocks that passed Phase 1. This segmentation allows the system to benefit from low CPU overhead while managing false positives through structured multi-phase verification.
Solution Approach 2:
The patent performs preliminary filtering using small non-cryptographic hashes before committing to full block comparisons. This preliminary action eliminates the majority of non-duplicate blocks early in the process, reducing the number of expensive full comparisons needed while accepting that some false positives will require verification in subsequent phases.
3Measurement precision
If a persistent database of all signatures is maintained, then duplicate detection accuracy is improved, but disk I/O operations increase significantly
Solution Approach 1:
The patent extracts only the essential filtering function from the persistent database, using small non-cryptographic hashes that can be stored in memory rather than on disk. This extraction allows the system to maintain high duplicate detection accuracy for the filtering stage without the disk I/O overhead of maintaining a complete persistent signature database, while cryptographic verification provides the final accuracy guarantee.
Solution Approach 2:
The patent uses inexpensive in-memory hash tables with small non-cryptographic hashes as temporary filtering structures, replacing the need for expensive persistent database storage. These in-memory structures are discarded or updated periodically, providing accurate duplicate detection during their lifetime without the energy cost of persistent disk I/O operations.
4Measurement precision
If direct block comparison is performed, then duplicate detection accuracy is maximized, but processing time becomes infeasible for large datasets
Solution Approach 1:
The patent segments the comparison process into hierarchical stages: first comparing small non-cryptographic hash values, then only performing full block comparisons on blocks with matching hashes. This segmentation makes large-scale deduplication feasible by reducing the number of full block comparisons from potentially billions to a manageable subset, while still achieving maximum accuracy through final cryptographic verification.
Solution Approach 2:
The patent performs preliminary hash comparison using small non-cryptographic signatures before undertaking full block comparisons. This preliminary action filters out the vast majority of non-duplicate blocks, making the subsequent full comparison process feasible for large datasets while ensuring that no duplicates are missed through the multi-phase verification approach.
Data Source
AI summary
A system and associated methodology for controlling block duplicates when deduplicating data (Dedup Blocks) to a storage space. The system includes a persistent database of known duplicates stored in the storage space (KD Table), and a non-persistent database of possible duplicates stored in the storage space (PD Table). Computer logic executes programming instructions stored in memory that are configured to index the KD Table according to a value derived from bits of a Dedup Block's hash signature, to index the PD Table according to another value derived by other bits of the Dedup Block's hash signature, to demote known duplicates from the KD Table to the PD Table, and to promote possible duplicates from the PD Table to the KD Table.