Data Replication Using Binary Search Tree Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage systems face inefficiencies in data replication and deduplication processes, particularly in handling write operations across multiple data storage systems, leading to increased I/O latency and resource utilization due to existing compression and deduplication methods.
Innovation Solution
The method involves converting write data into a deduplicated representation using a binary search tree to identify unique data blocks and create a deduplication bit vector, allowing for efficient transmission and restoration of data across systems, thereby reducing redundant data transfer and improving replication efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional compression and deduplication methods are used for data replication, then data transfer efficiency is improved, but I/O latency and resource utilization increase
Solution Approach 1:
The data is divided into fixed-size blocks, and a binary search tree is constructed where each node represents a data block. This segmentation allows for efficient identification and deduplication of individual blocks without processing entire data sets, reducing computational overhead and I/O latency while maintaining deduplication effectiveness.
Solution Approach 2:
The binary search tree is pre-populated with data block identifiers from the source storage system before replication begins. This preliminary action enables the target storage system to quickly identify duplicate blocks during transmission without performing complex deduplication operations in real-time, thereby reducing I/O latency.
2Loss of substance
If traditional deduplication methods are used, then redundant data transfer is reduced, but system resource consumption increases
Solution Approach 1:
Instead of transferring actual data blocks, the system transfers references (pointers) to data blocks that already exist in the target storage system's binary search tree. This copying approach eliminates redundant data transfer while minimizing resource consumption, as only small reference identifiers need to be transmitted rather than entire data blocks.
Solution Approach 2:
The system changes the parameter being transmitted from actual data blocks to compact reference identifiers. This parameter transformation dramatically reduces the amount of data that needs to be processed and transmitted, lowering resource consumption while achieving the same deduplication goal.
3Reliability
If data is replicated across multiple storage systems, then data reliability is improved, but data transfer size increases
Solution Approach 1:
The binary search tree structures from source and target storage systems are merged, allowing the system to identify and eliminate duplicate data blocks across multiple systems. This merging approach enables reliable data replication while minimizing transfer size by only transmitting unique blocks that don't already exist in the target system.
Data Source
AI summary
Techniques for processing write operations may include: receiving, at a first data storage system, a write operation that writes first data to a first device, wherein the first device is configured for replication on a second device of a second data storage system; writing the first data to the first device of the first data storage system; performing first processing that converts the first data to a deduplicated data representation; transmitting the deduplicated data representation from the first data storage system to the second data storage system; and responsive to receiving the deduplicated data representation, performing second processing on the second data storage system that converts the deduplicated data representation to its original form in accordance with the first data.