Quarantine and Repair of Data Replicas in Quorum Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In quorum-based data storage systems, repairing missing replicas is challenging, especially when a node failure or communication issues lead to a shard being inaccessible, risking data loss and unavailability.
Innovation Solution
Implementing a repair task mechanism that distinguishes between 'quarantined' and 'trash-bin' replicas, where unavailable replicas are marked 'quarantined' if a quorum is not available, to prevent data loss and ensure continuous availability, and a 'resolve quarantine' task to synchronize data once replicas become available again.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If missing replicas are repaired quickly to maintain data availability, then data availability is improved, but data loss risk increases when quorum is not available
Solution Approach 1:
The system performs preliminary actions by creating repair tasks and marking replicas as 'quarantined' before actual data repair occurs. This preliminary marking prevents premature data overwriting while preparing the repair process, allowing safe data recovery once quorum is restored.
Solution Approach 2:
The 'quarantined' state acts as an intermediary status between 'available' and 'repaired'. This intermediate state holds replicas in a suspended condition, preventing data loss while allowing the repair process to proceed safely once quorum requirements are met.
2Productivity
If repair tasks are initiated immediately when replicas are missing, then repair speed is improved, but system complexity increases due to quarantine management
Solution Approach 1:
The repair process is segmented into distinct phases: task creation, quarantine marking, quorum verification, and actual repair execution. This segmentation allows each phase to be managed independently, improving repair speed while keeping complexity manageable through clear phase boundaries.
Solution Approach 2:
The system dynamically adjusts the repair process based on quorum availability. When quorum is available, repair proceeds immediately; when not available, replicas are quarantined and repair waits. This dynamic behavior optimizes repair speed while managing complexity through conditional logic.
3Reliability
If replicas are marked as available for repair, then data persistence is improved, but data coherence deteriorates without quorum verification
Solution Approach 1:
The system uses feedback from quorum verification to control the repair process. Repair tasks wait for quorum confirmation before proceeding, ensuring that data coherence is maintained. The feedback loop checks quorum status and only allows repair when coherence can be guaranteed.
Solution Approach 2:
The quarantine mechanism provides beforehand cushioning by preventing premature repair actions that could compromise data coherence. This protective measure ensures that repair only occurs when quorum conditions are satisfied, maintaining data stability.
Data Source
AI summary
A data storage system with quorum-based commits sometimes experiences replica failure, due to unavailability of a replica-hosting node, for example. Described herein are methods and systems for improving data persistence and availability in a distributed data store where data is stored in a plurality of shards and a given shard is replicated across a plurality of nodes so as to create a plurality of replicas, and a quorum of replicas is needed for access to the given shard. Among other things, the methods and systems generally involve determining whether to quarantine or delete unavailable replicas in a given shard, and how to handle purge requests related to the shard when there are quarantined replicas.