Distributed Storage Snapshot Merging for Split-Brain Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed replicated storage systems, network failures can lead to 'split brain' scenarios where both active and backup servers continue to serve clients independently, resulting in inconsistent data across servers.
Innovation Solution
Implement a method that includes detecting a connection loss, writing data to a first block of persistent storage, storing a snapshot of this region, overwriting the second block with data from the second server, and mounting the snapshot via a file system to resolve data inconsistencies.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the backup server becomes active autonomously when connection is lost, then service continuity is maintained, but data consistency deteriorates
Solution Approach 1:
The system performs preliminary actions by creating snapshots of persistent storage data before potential conflicts occur. When connection loss is detected, the snapshot mechanism is already in place to capture the state of data, enabling later resolution of inconsistencies without compromising service continuity.
Solution Approach 2:
The snapshot acts as an intermediary mechanism between the active server and backup server. It captures the state of persistent storage at a specific point in time, serving as a reference for resolving data inconsistencies that arise when both servers operate independently during network failures.
2Productivity
If both servers service requests independently during network failure, then service availability is improved, but data inconsistency increases
Solution Approach 1:
The system creates a copy of the persistent storage state through snapshots. This copying mechanism allows the backup server to operate independently using its own data copy while the active server continues servicing requests, and later enables comparison and resolution of any inconsistencies that arose during independent operation.
3Quantity of substance
If data is overwritten without snapshot, then storage space is conserved, but data recovery capability is reduced
Solution Approach 1:
The system performs preliminary action by creating snapshots before overwriting data. This ensures that if data inconsistencies are detected after connection restoration, the original state can be recovered from the snapshot, maintaining data recovery capability while still allowing normal overwrite operations to conserve storage space.
Data Source
AI summary
In a distributed replicated storage system, an active server may, during normal operation, service requests from clients, and back up its persistent storage to a backup server. The backup server may be configured to become an active server autonomously if the active server fails. If a network connection between the active server and the backup server fails, severing the connection between the active server and the backup server, the backup server may become a second active server and begin to service requests from clients, and the original active server may continue to service requests from clients. This may result in inconsistencies in the persistent storage of the original active server and of the second active server. As such, a system and method for intelligent merging of primary and backup data in a distributed replicated storage system are provided.