Hash Collision Detection in Content Addressable Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Content addressable storage (CAS) systems face issues with data collisions due to the possibility of hash collisions, which can lead to data corruption and reduced user confidence, as attackers can create malicious data blocks with equivalent hash values, causing substitution or corruption of existing data.
Innovation Solution
The implementation of a unique hash generator that produces 2-tuple or 3-tuple hash values, including a cryptographic hash ID, domain ID, and hash number, to distinguish between data blocks with the same hash ID, and a collision detection mechanism that performs bit-by-bit comparisons to identify and prevent collisions by generating new unique hash values when necessary.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a cryptographic hash function is used to identify data blocks in a CAS system, then storage efficiency and deduplication are improved, but the risk of hash collisions increases, leading to data corruption
Solution Approach 1:
The patent segments the hash identification into multiple components: a primary cryptographic hash value and additional metadata (hash number, domain ID). This segmentation allows the system to maintain the efficiency of hash-based identification while adding layers of differentiation to prevent collisions from causing data corruption.
Solution Approach 2:
The patent adds additional dimensions to the hash identification space by incorporating hash numbers and domain IDs alongside the cryptographic hash value. This multi-dimensional approach expands the identification space, making collisions significantly less likely to result in data corruption while maintaining storage efficiency.
2Loss of substance
If hash values are used to fingerprint data in CAS systems, then storage space is reduced through deduplication, but collisions can occur resulting in corrupt data
Solution Approach 1:
The patent implements a feedback mechanism where the system actively monitors for hash collisions and responds by generating unique hash numbers for colliding data blocks. This feedback loop ensures that even when collisions occur, the system can distinguish between different data blocks and maintain data correctness while still achieving deduplication benefits.
Solution Approach 2:
The patent changes the parameters used for data identification by moving from a single hash value to a composite identifier including hash value, hash number, and domain ID. This parameter expansion maintains deduplication efficiency while significantly reducing the probability of collisions leading to data corruption.
3Speed
If a simple hash function is used for data identification, then computation speed is improved, but collision resistance deteriorates, allowing attackers to create malicious data blocks
Solution Approach 1:
The patent performs preliminary actions by maintaining additional metadata (hash numbers, domain IDs) alongside cryptographic hash values before collisions can be exploited. This preliminary preparation ensures that even if an attacker creates a collision, the system has already established additional differentiation mechanisms to prevent successful attacks.
Data Source
AI summary
The subject matter described herein is generally directed to detecting and managing collisions in storage. A hash identifier (ID) for a first block of data is calculated and a determination is made whether the calculated hash ID matches hash IDs associated with a storage. If the calculated hash ID matches at least one of the hash IDs, the first block of data is compared with a second block of data, associated with the hash IDs, in the storage. If the first block of data is different from the second block of data based on the comparison, a hash number is associated with the calculated hash ID and the first block of data is stored in storage using the calculated hash ID and associated hash number as an index to the first block of data in the storage. In this manner, collision between data blocks is detected and prevented.