Source Deduplication for Logical Block Mirroring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In large-scale storage systems, logical mirroring with deduplication leads to unnecessary bandwidth consumption and storage space usage due to the transmission of duplicate data blocks, as the mirroring process is unaware of deduplication at the source, resulting in inefficient data replication and increased processing resources at the destination.
Innovation Solution
Implement a method where deduplication is performed only at the source storage system, using unique references for each data block, such as physical volume block numbers or virtual volume block numbers, to avoid sending duplicate blocks during logical block mirroring, and maintain a history buffer at the destination to track received blocks, ensuring only modified blocks are resent if needed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If logical block mirroring is performed without deduplication awareness, then data replication is simplified, but bandwidth consumption increases due to transmitting duplicate blocks
Solution Approach 1:
The source storage system performs deduplication before data replication, identifying and eliminating duplicate blocks in advance. This preliminary action ensures that only unique data blocks are transmitted to the destination, reducing bandwidth consumption while maintaining the simplicity of logical block mirroring operations.
Solution Approach 2:
The destination storage system sends feedback information about received blocks to the source storage system. This feedback mechanism enables the source to track which blocks have been successfully replicated and avoids retransmitting duplicate blocks, thereby reducing unnecessary bandwidth consumption while maintaining replication simplicity.
2Quantity of substance
If deduplication is performed at the destination storage system, then storage space is saved, but processing resources are consumed
Solution Approach 1:
The source storage system performs deduplication before replication, eliminating duplicate blocks in advance. This shifts the processing workload from the destination to the source, reducing the processing resources required at the destination while still achieving storage space savings through deduplication.
Solution Approach 2:
The source storage system acts as an intermediary that performs deduplication before data is transmitted to the destination. This intermediary role consolidates the processing function at the source, allowing the destination to receive only unique blocks and reducing its processing resource requirements.
3Reliability
If duplicate data blocks are transmitted during mirroring, then data completeness is ensured, but storage space at destination increases
Solution Approach 1:
The source storage system performs deduplication before replication, identifying and eliminating duplicate blocks in advance. This ensures that only unique data blocks are transmitted to the destination, maintaining data completeness while preventing unnecessary storage space consumption at the destination.
Solution Approach 2:
The destination storage system sends feedback about received blocks to the source, enabling the source to track successfully replicated blocks. This feedback mechanism ensures data completeness by confirming receipt while avoiding retransmission of duplicates, thereby preventing storage space waste at the destination.
Data Source
AI summary
Bandwidth consumption between a data replication source and destination and storage consumption at the destination are reduced, when logical block mirroring is used with source deduplication, by eliminating repeated transmission of data blocks from source to destination. A reference is created for each data block at the source, the reference being unique within a storage aggregate of the source. During a mirror update, the source initially sends only the references of modified data blocks to the destination. The destination compares those references against a data structure to determine whether the destination already has any of those data blocks stored. If the destination determines that it already has a data block stored, it does not request or receive that data block again from the source. Only if the destination determines that it has not yet received the referenced data block does it request and receive that data block from the source.