Deduplication via Synchronized Hash Tables for Data Replication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data replication techniques face inefficiencies in handling duplicate data across distributed storage systems, leading to increased network bandwidth usage and computational overhead, particularly in scenarios where data deduplication is not effectively implemented.
Innovation Solution
The method involves using hash tables to identify and manage duplicate data at a per-write operation level, where a hash value is computed for each write operation, and by comparing it to existing entries, determining if it's a duplicate or unique, and transmitting only metadata for duplicates, thus eliminating redundant data transfer and maintaining synchronized hash tables across systems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional data replication techniques are used to ensure data consistency across distributed storage systems, then data reliability is improved, but network bandwidth usage increases due to transmission of redundant duplicate data
Solution Approach 1:
The system performs preliminary deduplication processing at the source storage system before data transmission. By computing hash values and identifying duplicate data blocks in advance, the system prepares only unique data for transmission, eliminating redundant network traffic while ensuring the destination system receives complete and consistent data sets
Solution Approach 2:
The invention extracts and transmits only the essential components for replication - specifically metadata containing hash values and deduplication information - while leaving out redundant duplicate data blocks. This extraction approach allows the destination system to reconstruct complete data sets using the transmitted metadata and its own deduplication capabilities
2Loss of energy
If comprehensive data deduplication processing is performed on all write operations, then network bandwidth usage is reduced by eliminating redundant transmissions, but computational overhead increases
Solution Approach 1:
The system performs partial deduplication processing by focusing computational efforts only on computing hash values for deduplication identification, rather than performing complete data comparison and analysis. This partial action approach significantly reduces computational overhead while still achieving effective duplicate detection and elimination
Solution Approach 2:
The invention changes the parameter of data processing from examining entire data blocks to computing condensed hash values. This parameter transformation reduces the computational complexity from O(n) data comparison to O(1) hash computation, dramatically lowering computational overhead while maintaining deduplication effectiveness
3Productivity
If hash tables are synchronized across distributed storage systems to enable efficient duplicate detection, then data processing efficiency is improved, but device complexity increases
Solution Approach 1:
The system creates and maintains simplified copies of hash table information across distributed storage systems. Instead of synchronizing entire data sets or complex metadata structures, the invention transmits and replicates only the essential hash value information needed for duplicate detection, reducing synchronization complexity while maintaining processing efficiency
Solution Approach 2:
The hash table structure serves multiple functions simultaneously: it acts as a deduplication index, a replication tracking mechanism, and a data verification tool. This multi-functionality eliminates the need for separate systems for each purpose, reducing overall device complexity while enhancing data processing efficiency
4Loss of energy
If metadata is transmitted for duplicate data blocks instead of the actual data, then network bandwidth usage is reduced, but data transmission complexity increases
Solution Approach 1:
The system changes the transmission parameter from sending actual data blocks to transmitting condensed metadata containing hash values. This parameter transformation reduces transmission size from potentially large data blocks to compact metadata structures, significantly lowering network bandwidth usage while the standardized metadata format keeps transmission complexity manageable
Data Source
AI summary
Techniques for processing write operations may include: receiving, at a first data storage system, a first write operation that writes first data to a first device, wherein the first device is configured for replication on a second device of a second data storage system; performing first processing that determines whether the first data written by the first write operation is a duplicate of an existing entry in a first hash table of the first data storage system; responsive to determining the first data written by the first write operation is a duplicate of an existing entry in the first hash table, performing second processing; responsive to determining the first data written by the first write operation is unique and is not a duplicate of an existing entry in the first hash table, performing third processing; and transmitting the final buffer to the second data storage system.