Content-Based Replication Checksum Validation for Scale-Out Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing network storage systems face challenges in verifying the correct transfer of large data volumes during replication, leading to potential errors and resource-intensive retransmissions, especially when only a small portion of the data has changed or if a common base snapshot is lost.
Innovation Solution
The implementation of a Content-Based Replication (CBR) method that validates data by comparing checksums across upstream and downstream systems, identifying and correcting errors at the chunk or slice level, thereby minimizing the data needed to be resent and ensuring accurate replication.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is replicated from upstream system to downstream system, then data redundancy and availability are improved, but verification of correct transfer requires extensive processor and network resources
Solution Approach 1:
The patent divides data into hierarchical segments (volumes, bins, slices, chunks, blocks) and applies checksum validation at multiple levels. This segmentation allows the system to identify and retransmit only the specific faulty segments rather than validating entire data sets, significantly reducing processor and network resources required for verification.
Solution Approach 2:
The patent replaces exhaustive data retransmission and comparison with a checksum-based validation mechanism. Instead of mechanically comparing large volumes of data, the system uses cryptographic checksums to verify data integrity, substituting heavy mechanical processing with efficient mathematical validation.
2Reliability
If all data is retransmitted when replication errors are detected, then data accuracy is ensured, but network bandwidth and time are excessively consumed
Solution Approach 1:
The patent segments data into hierarchical units (chunks, slices, bins) and identifies the specific location of errors. When validation fails, only the affected segments are retransmitted rather than the entire data set, dramatically reducing network bandwidth consumption while maintaining data accuracy.
Solution Approach 2:
The patent extracts and identifies only the specific faulty data segments through checksum validation. Instead of retransmitting all data, the system extracts the minimal necessary portion (only failed chunks/slices) for retransmission, optimizing network efficiency.
3Measurement precision
If checksum validation is performed at the chunk level instead of bin level, then error detection precision is improved, but computational overhead increases
Solution Approach 1:
The patent implements a hierarchical segmentation strategy with checksums at multiple levels (bin, slice, chunk). This allows the system to first perform coarse-grained validation at the bin level, then drill down to finer-grained chunk-level validation only when necessary, balancing precision with computational efficiency.
Solution Approach 2:
The patent applies partial validation by performing checksum operations selectively at different hierarchical levels based on error detection needs. Full chunk-level validation is performed only when bin-level validation fails, avoiding unnecessary computational overhead while maintaining precision when required.
Data Source
AI summary
Methods, systems, and programs are presented for replicating data across scale-out storage systems. One method includes replicating, from an upstream to a downstream system, a volume snapshot having one or more bins. Locations for the bins of the snapshot are identified, the location for each bin including the upstream array storing the bin and the downstream array storing a replicated version of the bin. Each bin is validated by comparing an upstream bin checksum of the bin with a downstream bin checksum of the replicated version of the bin. When the checksums are different, a plurality of chunks are defined in the bin, and for each chunk in the bin an upstream chunk checksum calculated by the upstream array is compared with a downstream chunk checksum calculated by the downstream array. The chunk is sent from the upstream to the downstream array when the chunk checksums are different.