Storage Replication Using Similarity Hashes to Reduce Bandwidth
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data replication methods in distributed storage systems consume excessive bandwidth due to the transfer of redundant data blocks, especially in limited network conditions, which undermines system reliability and increases deployment costs.
Innovation Solution
The method employs a similarity hash and a cryptographic hash to determine if a data block has multiple instances in the storage system, allowing the source system to transmit only these hashes to the target system, which then verifies and replicates the data block from a deduplication table, reducing the need for redundant data transfer.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional data replication methods are used to ensure system reliability, then data redundancy is improved, but bandwidth consumption increases excessively
Solution Approach 1:
The patent extracts only the essential identification information (hash values) from the complete data blocks for replication purposes. Instead of transmitting entire data blocks, the system computes and transmits only hash values (e.g., 16-byte or 32-byte identifiers) that uniquely represent the data blocks, thereby dramatically reducing bandwidth consumption while maintaining replication reliability.
Solution Approach 2:
The patent uses hash values as simplified copies or representations of the actual data blocks. These hash copies serve as unique identifiers that enable the target system to verify data integrity and detect duplicates without requiring transmission of the full data block content, thus resolving the contradiction between reliability and bandwidth usage.
2Loss of information
If redundant data blocks are transferred during replication, then data completeness is improved, but network resource consumption increases
Solution Approach 1:
The patent extracts the essential identifying characteristic (hash value) from each data block and transmits only this extracted information. The target system uses these extracted identifiers to determine which data blocks need replication, eliminating the need to transmit redundant complete data blocks while ensuring data completeness through hash-based verification.
Solution Approach 2:
The patent changes the parameter being transmitted from the full data block (large size) to a hash value (small size). This parameter transformation maintains the ability to identify and verify data blocks while dramatically reducing the quantity of network resources consumed during replication.
3Reliability
If all data blocks are replicated to ensure backup reliability, then backup completeness is improved, but deployment cost increases
Solution Approach 1:
The patent extracts hash values as minimal sufficient information to determine backup requirements. The target system compares these extracted identifiers against its existing deduplication table to identify which data blocks already exist, thereby avoiding redundant replication operations and reducing deployment costs while maintaining backup completeness.
Solution Approach 2:
The target system uses its own deduplication table and hash comparison capability to self-determine which data blocks need replication. This self-service mechanism eliminates the need for expensive centralized coordination or transmission of complete data block metadata, reducing deployment costs while ensuring backup reliability.
Data Source
AI summary
Techniques are used for replication in a storage system. The techniques may be used to provide, among other things, the determination that a data block has more than one instance in a storage system. A similarity hash for the data block is determined, and an instruction is transmitted from a source system to a target system to replicate the data block based on the similarity hash.