Replicating Storage Tables for Cloud Resource Management
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cloud computing systems face downtime and data loss due to storage account outages, as existing replication methods fail to ensure high durability and availability across multiple data centers, leading to inconsistent recovery and potential data loss.
Innovation Solution
A replication table system that synchronously replicates storage tables across multiple data centers, using a chain replication protocol to maintain an ordered set of replicas, ensuring data durability and availability by writing sequentially from the first to the last replica and confirming write operations, while also generating new replicas and performing repair operations in the background to maintain data integrity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If data is stored in a single storage account for cloud resource management, then the system is simple to operate and manage, but the system suffers from single point of failure and global downtime when the storage account fails
Solution Approach 1:
The storage account is segmented into multiple replicas distributed across different data centers. Each replica maintains a subset of the storage table data, allowing the system to operate even when individual replicas or data centers fail. This segmentation eliminates the single point of failure while maintaining operational simplicity through automated replication management.
Solution Approach 2:
A replication management system acts as an intermediary between the storage account and multiple replicas. This intermediary automatically handles data synchronization, failover detection, and replica management, shielding users from the complexity of distributed storage while ensuring high availability through automated failover mechanisms.
2Reliability
If multiple replicas of storage tables are created across different data centers, then the system achieves high availability and durability, but the system complexity increases due to replication management
Solution Approach 1:
The replication system implements self-service mechanisms where replicas automatically synchronize data, detect failures, and perform failover without manual intervention. The system autonomously manages replication state, handles conflict resolution, and maintains consistency across distributed replicas, reducing operational complexity while ensuring high availability.
Solution Approach 2:
The system dynamically adjusts replication parameters such as synchronization frequency, consistency models, and failover thresholds based on system conditions. By changing operational parameters rather than structural complexity, the system maintains simplified management while adapting to different availability and durability requirements across data centers.
3Manufacturing precision
If synchronous replication is used across multiple replicas, then data consistency is maintained, but the write operation latency increases due to sequential writing requirements
Solution Approach 1:
The replication system dynamically adjusts the synchronization model based on operational context. For time-critical writes, the system uses asynchronous replication with eventual consistency, while for accuracy-critical operations, it switches to synchronous replication. This dynamic adaptation maintains data consistency when needed while minimizing write latency through selective use of replication modes.
Solution Approach 2:
The system implements partial synchronous replication where only critical data elements require immediate consistency across all replicas, while non-critical data can be replicated asynchronously. This partial action approach maintains necessary data consistency for essential operations while reducing overall write latency by not requiring full synchronization for all data types.
4Manufacturing precision
If repair operations are performed synchronously to maintain data integrity, then data consistency is ensured, but the system availability decreases during repair operations
Solution Approach 1:
The system performs preliminary validation and consistency checks before initiating repair operations. By pre-identifying corrupted data and preparing repair strategies in advance, the system can execute repairs with minimal disruption. Critical consistency checks are performed beforehand, allowing repair operations to proceed quickly while maintaining data integrity without prolonged system unavailability.
Solution Approach 2:
The system implements periodic consistency verification and background repair operations that do not block write access. By scheduling repair operations periodically and using background processing for non-critical repairs, the system maintains data integrity through regular checks while preserving system availability during repair activities through asynchronous execution.
Data Source
Figure 1~2
Figure 3
Figure 4~5
AI summary
A storage account is replicated across multiple data centers so as to withstand an outage of the storage account. A front end and applications use data stored in the storage accounts to manage resources of a cloud computing system. A client includes an interface used by the front end to access the storage accounts, in addition to an interface that is used by the applications to access the storage accounts. Other features, which ensure that a read operation survives even if a head replica or a tail replica is down, include reading from a head replica instead of from a tail replica, a two phase prepare-commit operation to propagate changes from head to tail replicas, and provisioning a back end repair service if a failure occurs in the middle of the prepare-commit operation.