Asynchronous Data Replication Buffer with Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data replication methods are inefficient in terms of network bandwidth usage and processing overhead, particularly during asynchronous replication, leading to issues like data collisions, duplicate blocks, and security risks due to unencrypted data transfer.
Innovation Solution
A data storage system with a replication buffer logically positioned between the source and target data storage units, utilizing deduplication, compression, and encryption circuitry to transform and transfer only unique data blocks, optimizing asynchronous data replication by scheduling data transfers based on real-time and predicted system conditions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all data blocks are transferred during asynchronous replication, then data completeness is ensured, but network bandwidth consumption increases and duplicate blocks are created
Solution Approach 1:
The patent extracts and identifies duplicate data blocks using hash functions and compares them against a sliding window of previously transferred blocks. Only unique blocks are selected for transfer, eliminating redundant data transmission while maintaining data completeness at the target.
Solution Approach 2:
The system changes the parameter of data selection from transferring all blocks to transferring only unique blocks identified through hash comparison. This parameter change reduces bandwidth consumption while ensuring data integrity through selective transfer based on uniqueness verification.
2Reliability
If data is transferred synchronously, then data consistency is maintained, but system performance and productivity decrease
Solution Approach 1:
The patent performs preliminary actions by computing hash functions and identifying unique blocks before data transfer. The sliding window mechanism pre-processes block comparisons, allowing asynchronous transfer of only necessary blocks while maintaining consistency without synchronous blocking.
Solution Approach 2:
The sliding window acts as an intermediary mechanism that mediates between source and target systems. It tracks transferred blocks and determines uniqueness without requiring synchronous coordination, enabling asynchronous operation while maintaining data consistency through the intermediary's state management.
3Speed
If data is transferred without filtering, then transfer speed is maintained, but data collisions and security risks increase
Solution Approach 1:
The system performs preliminary filtering using hash functions and sliding window comparison before data transfer. This pre-processing identifies unique blocks, preventing data collisions at the target while maintaining efficient transfer speeds by avoiding redundant block transmissions.
Solution Approach 2:
The patent extracts unique data blocks from the source using hash-based identification and removes duplicates through sliding window comparison. Only the essential unique blocks are transferred, eliminating security risks and collisions associated with transferring redundant or duplicate data.
Data Source
AI summary
Asynchronous data replication between source and target data storage devices of a data storage system can be directed by a connected network controller. A source data set to be replicated from the source data storage device to the target data storage device may be stored in a staging cache while portions of the source data set are parsed by the network controller so that less than all of the source data set is transferred to the target data storage device.