Storage Re-synchronization via Content-Based Signatures
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data replication methods face challenges in maintaining target replica consistency during synchronous replication, particularly due to communication link failures, which can lead to time-consuming full data re-synchronization, undermining system performance.
Innovation Solution
Implementing a clustered content addressable storage system with a distributed storage controller that receives content-based signatures from the source storage system, compares them with an address-to-signature table, and provides indications of successful re-synchronization without updating reference counts, thereby optimizing re-synchronization by avoiding unnecessary metadata updates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional full data re-synchronization is performed after communication link failures during synchronous replication, then target replica consistency is preserved, but re-synchronization time and system resources increase significantly
Solution Approach 1:
The system performs preliminary actions by maintaining content-based signatures and address-to-signature tables continuously during normal operation, so that when re-synchronization is needed, the comparison can be performed efficiently without needing to scan all data. The signatures are prepared in advance and updated as data changes occur.
Solution Approach 2:
Instead of transferring complete data pages during re-synchronization, the system uses content-based signatures as copies or references to identify matching data. The signature acts as a pointer or reference that allows the system to verify consistency without moving the actual data, significantly reducing transfer time and bandwidth requirements.
2Reliability
If conventional full data re-synchronization is performed after communication link failures during synchronous replication, then target replica consistency is preserved, but system resources are consumed excessively
Solution Approach 1:
The system uses content-based signatures as lightweight copies that represent data pages. Instead of transferring and processing complete data pages during re-synchronization, the system compares signatures which are much smaller and less resource-intensive. This dramatically reduces CPU, memory, and bandwidth consumption.
Solution Approach 2:
The system extracts only the essential identification information (content-based signatures) from the full data pages for the re-synchronization process. By taking out just the signature portion rather than the complete data, the system achieves consistency verification with minimal resource expenditure.
3Reliability
If reference counts and metadata are updated during re-synchronization, then data consistency is maintained, but unnecessary metadata updates increase processing overhead
Solution Approach 1:
The system uses content-based signatures as references or copies that point to actual data pages. When comparing signatures during re-synchronization, the system is comparing references rather than updating actual data structures. This approach maintains consistency information without triggering unnecessary metadata updates and reference count changes.
Solution Approach 2:
The content-based signature comparison mechanism is self-sufficient for verifying data consistency during re-synchronization. The signatures themselves carry the necessary identification and comparison information, eliminating the need for additional metadata updates or reference count adjustments to maintain consistency.
Data Source
AI summary
A target storage system in one embodiment receives from a source storage system a plurality of content-based signatures of respective data pages of a storage object that is subject to replication from the source storage system to the target storage system. For a given one of the received content-based signatures having a particular logical address, the target storage system compares at least a portion of the received content-based signature with a particular one of a plurality of entries of an address-to-signature table maintained by the target storage system. Responsive to a match between at least a portion of the received content-based signature and the particular one of the entries of the address-to-signature table, the received content-based signature is compared with a full content-based signature corresponding to the particular entry, and responsive to a match between the received content-based signature and the full content-based signature corresponding to the particular entry, an indication of successful re-synchronization is provided to the source storage system for the storage object data page having the received content-based signature.