Hash-Based Data Transfer in Distributed Deduplication Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed deduplication storage systems, naive data transfer methods result in inefficiencies due to the transfer of duplicate data blocks, leading to excessive network and processing resource usage.
Innovation Solution
Implementing hash-based data transfers where source nodes transmit content-based hashes to destination nodes, which respond with a bitmap indicating duplicate blocks, allowing only unique data blocks to be transmitted, and using a write buffer to prevent data corruption by incrementing reference counts and writing new blocks as needed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If naive data transfer is used in deduplicated storage systems, then data can be transferred between nodes, but duplicate data blocks are transferred unnecessarily, increasing network and processing resource usage
Solution Approach 1:
The source node calculates hashes of data blocks before transfer and sends these hashes to the destination node in advance. The destination node uses these hashes to check its deduplication index and determine which blocks are duplicates before the actual data transfer occurs, preventing unnecessary network transmission
Solution Approach 2:
The patent extracts only the hash values (metadata) from the actual data blocks for preliminary comparison. By separating the hash verification step from the data transfer step, the system identifies and excludes duplicate blocks without transferring the full data, reducing network bandwidth consumption
2Quantity of substance
If naive data transfer is used, then all data blocks are transferred from source to destination, but this results in transferring significantly more data than required, wasting network bandwidth
Solution Approach 1:
The destination node creates a bitmap copy representing the transfer status of all data blocks. This bitmap is sent back to the source node to indicate which blocks were duplicates and which need transfer, enabling the source to send only the necessary data blocks rather than all blocks
Solution Approach 2:
The patent replaces the mechanical approach of transferring all data blocks with a computational approach using hash-based comparison. By substituting physical data transfer with hash value comparison and bitmap communication, the system dramatically reduces the actual data movement while maintaining transfer completeness
3Productivity
If naive data transfer is used, then reading and writing operations are performed on all data blocks, but this increases processing overhead and storage operations beyond what is actually needed
Solution Approach 1:
The source node performs hash calculations and the destination node performs hash lookups in its deduplication index before initiating data transfer. This preliminary processing identifies duplicate blocks upfront, allowing the system to skip reading, transferring, and writing operations for duplicate blocks entirely
Solution Approach 2:
The patent extracts only the necessary data blocks (non-duplicates) for transfer by comparing hashes against the destination's deduplication index. By separating duplicate identification from the transfer process, the system performs minimal read and write operations, reducing processing overhead and storage I/O
4Loss of energy
If hash-based data transfer is implemented, then duplicate data blocks are identified and omitted from transfer, but additional processing is required to calculate and compare hashes
Solution Approach 1:
The patent transforms the data representation from full data blocks to compact hash values for comparison purposes. By changing the parameter being transmitted and compared from bytes of data to fixed-size hash values, the system reduces network overhead while the computational cost of hash calculation is minimal compared to transferring actual data
Data Source
AI summary
Exemplary methods, apparatuses, and systems include a destination node receiving, from a source node, a hash of data stored by the source node and designated for transfer from the source node to the destination node. In response to determining the hash received from the source node maps to data stored by the destination node, the destination node reads the data into a write buffer and transmits an indication to the source node that the data is present at the source node. The destination node reads the data from write buffer and, in response to determining the data read from the write buffer remains unchanged as stored by the destination node, increases a reference count for the data as stored by the destination node.