Keyed Hash Values for Secure Data Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data storage systems face a tradeoff between weak and strong hash values for deduplication, where strong hash values are computationally expensive and require more memory space, limiting the number of entries in the index table and impacting performance.
Innovation Solution
The use of a keyed hash function generates smaller hash values (128 to 160 bits) for data blocks, reducing memory usage while maintaining an acceptable probability of accidental collision, and can be implemented as a keyed universal hash function to reduce processor cycles.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If strong hash values (cryptographic hash values) are used for data deduplication, then the probability of accidental collision is reduced to statistically negligible levels, but the number of bits required to represent them increases (256 to 512 bits), which increases memory usage and limits the number of entries in the index table
Solution Approach 1:
The patent changes the parameter of hash value size from conventional large sizes (256-512 bits) to a smaller size (128-160 bits) while maintaining security through the use of a keyed hash function. This parameter change reduces memory footprint in the index table while preserving collision resistance through the cryptographic key mechanism.
Solution Approach 2:
The patent introduces a cryptographic key as an intermediary element in the hash function (keyed hash function H(k, m)). This key acts as a mediator that enhances the security of the smaller hash values, allowing them to maintain cryptographic strength despite reduced bit length. The key prevents accidental and intentional collisions without requiring the full 256-512 bits of traditional cryptographic hash values.
2Reliability
If strong hash values are used for data deduplication, then collision resistance is improved, but the computational cost increases and more processor cycles are required
Solution Approach 1:
The patent changes the computational parameter by using a keyed hash function with a fixed-size key (128-160 bits) rather than processing full cryptographic hash values (256-512 bits). This reduces the computational burden while maintaining security, as the keyed hash function requires fewer processor cycles to evaluate while providing equivalent collision resistance.
3Reliability
If strong hash values are used for data deduplication, then security against collision attacks is improved, but the memory space available for index table entries is reduced
Solution Approach 1:
The patent changes the memory usage parameter by reducing hash value size from 256-512 bits to 128-160 bits. This parameter change directly increases the free space available in main memory for index table entries, allowing more data blocks to be tracked for deduplication while maintaining security through the keyed hash mechanism.
Solution Approach 2:
The cryptographic key serves as an intermediary that compensates for the reduced hash value size. By introducing this key into the hash function, the system maintains security against collision attacks despite using smaller hash values that consume less memory space in the index table.
Data Source
AI summary
Techniques for providing secure deduplication in a data storage system using smaller hash values. The techniques employ a keyed hash function to generate keyed hash values for data blocks received at the data storage system. The keyed hash function can generate keyed hash values having an acceptable probability of accidental collision, in which each keyed hash value requires a reduced number of bits to represent them. By representing each keyed hash value with a number of bits less than the number required for a cryptographic hash value, the impact that an index table in main memory has on the amount of free memory space available can be reduced, while still providing an acceptable probability of accidental collision. The keyed hash function can be implemented as a keyed universal hash function, which can reduce the number of processor cycles required to generate a keyed hash value for each received data block.