Hash Collision Reduction in Data Deduplication Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data archiving methods face challenges in reducing hash collisions, which can lead to false matches between non-identical datasets, compromising data integrity and efficiency in storage and backup systems.
Innovation Solution
The implementation of a system that randomly selects a hash function from a family of hash functions for each data chunk, generating a unique bit vector to determine true matches, thereby reducing the probability of hash collisions and ensuring data integrity without the need for byte-by-byte comparisons.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional hash functions are used for data deduplication, then storage efficiency is improved through duplicate detection, but hash collisions occur causing false matches between non-identical datasets
Solution Approach 1:
The patent segments the hash verification process into multiple independent stages: first computing a primary hash value for quick comparison, then selectively applying secondary verification methods only when hash collisions are detected. This segmentation allows the system to maintain high reliability while minimizing false matches caused by hash collisions.
Solution Approach 2:
The patent introduces an intermediary verification mechanism that acts as a mediator between hash comparison and final duplicate confirmation. When hash values match, the system employs additional verification steps (such as comparing actual data chunks or using cryptographic signatures) to confirm true identity, thereby eliminating false positives while maintaining storage efficiency.
2Reliability
If byte-by-byte comparisons are performed to verify data matches, then hash collision false matches are eliminated, but computing costs and processing time increase significantly
Solution Approach 1:
The patent applies partial verification action by performing byte-by-byte comparisons only when necessary - specifically when hash collisions are detected. For the majority of cases where hash values differ, the system accepts the hash comparison result without further verification. This selective approach maintains high accuracy for duplicate detection while preserving overall processing speed.
Solution Approach 2:
The patent implements local quality verification by applying different verification intensities to different data scenarios. High-precision byte-by-byte comparison is applied locally only to suspicious cases (hash collisions), while standard hash comparison is used for the bulk of data. This localized approach optimizes the balance between reliability and productivity.
3Reliability
If multiple hash functions are used to reduce hash collisions, then data integrity is improved, but device complexity and computational overhead increase
Solution Approach 1:
The patent employs dynamic hash function selection where the system adaptively chooses which hash functions to apply based on the specific data being processed and the detected collision patterns. Rather than statically applying multiple hash functions to all data, the system dynamically adjusts its verification strategy, applying additional hash functions only when collision resistance is particularly needed, thereby managing complexity while maintaining reliability.
Data Source
AI summary
A system obtains a first data chunk and a second data chunk of a plurality of data chunks associated with a first data snapshot of a computing system. A hash record is assigned to a data chunk, and used to create a hash value that is written to a first lookup table. The hash function is selected from a plurality of hash functions. The system creates a first archive by saving the plurality of data chunks and the first lookup table to a datastore. The system writes a second hash record for the individual data chunks to a second lookup table using the same hash functions that were used for the first lookup table. Dissimilar hash values between the first lookup table and the second lookup table are identified, and a second archive that includes data chunks with different data from the corresponding data chunks from the first data snapshot is created based on the data chunks with the dissimilar hash values.