Synchronous Dataset Replication With Cloud Mediation for Split-Brain Failover
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing storage systems face challenges in efficiently synchronizing datasets and managed objects to cloud-based storage systems, particularly in maintaining data integrity and availability during failures and power outages, and optimizing data management operations.
Innovation Solution
Implementing a direct-mapped flash storage system with a centralized operating system that directly addresses data blocks without translation by storage controllers, utilizing non-volatile RAM for quick data buffering, and employing dual storage array controllers for failover and distributed data management, along with erasure coding and mirroring for redundancy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If synchronous replication is implemented between storage systems, then data integrity is improved, but write performance deteriorates due to redundant writes
Solution Approach 1:
The patent implements a write-ahead logging mechanism where data is first written to a journal or log structure before being replicated to remote storage systems. This preliminary action allows the system to acknowledge writes locally before completing remote replication, improving write performance while maintaining data integrity through the logged transaction record.
Solution Approach 2:
The patent introduces an intermediary caching layer or buffer between the write operation and the remote replication process. This intermediary allows writes to be queued and batched for efficient bulk replication, reducing the performance impact of synchronous replication while ensuring data integrity through controlled transfer.
2Adaptability or versatility
If cloud-based storage systems are used for replication, then storage capacity and accessibility are improved, but data availability during network failures deteriorates
Solution Approach 1:
The patent implements local caching and buffering mechanisms at each storage system node, allowing data to be served from local storage during network failures. This local quality ensures data availability continues even when cloud connectivity is lost, while cloud-based replication provides extended storage capacity and accessibility during normal operation.
Solution Approach 2:
The patent implements local redundancy and error correction codes at each storage node as a cushion against network failures. By preparing local backup capacity and error correction mechanisms beforehand, the system can maintain data availability during cloud connectivity issues while still benefiting from cloud storage expansion.
3Reliability
If multiple storage array controllers are used for failover, then system reliability is improved, but system complexity deteriorates
Solution Approach 1:
The patent merges the control logic of multiple storage array controllers into a unified failover management system. By combining controller functions and using centralized coordination for failover decisions, the system achieves high reliability through redundancy while reducing operational complexity through standardized management procedures.
Solution Approach 2:
The patent uses controller copying or cloning techniques where a standby controller is maintained as an exact copy of the active controller. This copying approach simplifies failover by allowing immediate substitution of the active controller with its identical copy, ensuring reliability while keeping the failover mechanism simple and straightforward.
4Reliability
If erasure coding is implemented for redundancy, then data protection is improved, but computational overhead deteriorates
Solution Approach 1:
The patent implements erasure coding selectively on specific data blocks or portions rather than applying it universally to all data. This partial action approach provides adequate data protection for critical data while reducing computational overhead for less critical data, optimizing the balance between reliability and energy consumption.
Solution Approach 2:
The patent implements erasure coding operations periodically or in batches rather than continuously for every write operation. By consolidating erasure coding computations into periodic background processes, the system maintains strong data protection while significantly reducing the immediate computational overhead and energy consumption during active operations.
Data Source
Figure 1A
Figure 1B
Figure 1C
AI summary
A method and apparatus are disclosed for mediating between multiple storage systems that synchronously replicate a dataset. Upon detecting a communication fault between two storage systems, each system waits for a predefined period before requesting mediation from a mediation service. The system with the shorter wait time is designated as the preferred storage system. The mediation service determines which system will continue processing data storage requests for the shared dataset. The preferred storage system, upon winning mediation, processes incoming storage requests and detaches the other system from an in-sync list that tracks active storage systems. This approach ensures data consistency and coordinated failover in distributed storage environments. The mediation service may be implemented as a cloud-based database or other shared resource, and the process guarantees that only one storage system at a time can update membership information, preventing split-brain scenarios and maintaining system integrity