Cluster Data Replication via Segmented Sync Sets
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Shared-nothing clusters face challenges in managing popular data items across multiple nodes, particularly in ensuring data consistency and reducing latency and overhead associated with synchronous replication, as the size of the cluster increases.
Innovation Solution
Implementing a technique where changes to popular data items are synchronously applied to a subset of replicas (sync set) in a single atomic operation, followed by asynchronous propagation to the remaining replicas, allowing for reduced latency and overhead by optimizing the size and selection of the sync set based on factors like node relationships, availability, and data criticality.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If changes are synchronously propagated to all replicas of a popular data item, then data consistency is ensured, but latency and overhead increase significantly as cluster size grows
Solution Approach 1:
The patent segments the set of all replicas into multiple groups, where changes are synchronously propagated within each group but asynchronously across groups. This segmentation allows the system to maintain data consistency within segments while reducing overall propagation latency by processing segments in parallel rather than sequentially across all replicas.
Solution Approach 2:
The patent applies partial synchronous action by performing synchronous propagation only to a subset of replicas (those in the same segment) rather than all replicas. The remaining replicas receive updates asynchronously, which reduces the critical path latency while accepting that not all replicas are updated synchronously.
2Reliability
If changes are propagated to all replicas in a single atomic operation, then data consistency is maintained, but the overhead and complexity of distributed transactions increase
Solution Approach 1:
The patent divides the atomic update operation into multiple smaller atomic operations, each confined to a segment of replicas. This reduces the scope of distributed transaction coordination required for each operation, thereby lowering the overhead and complexity of the two-phase commit protocol while maintaining consistency within each segment.
Solution Approach 2:
The patent performs atomic operations only on a subset of replicas (those in the same segment) rather than all replicas simultaneously. This partial action reduces the coordination overhead of distributed transactions while accepting that full atomic propagation across all replicas is not achieved in a single operation.
3Reliability
If a larger subset of replicas is included in synchronous updates, then data consistency is improved, but latency and overhead increase
Solution Approach 1:
The patent segments replicas into multiple groups, allowing synchronous updates to be performed on multiple segments in parallel. This increases overall update throughput while maintaining data consistency within each segment, as each segment can be updated independently without blocking other segments.
Solution Approach 2:
The patent performs synchronous updates on only a portion of replicas (those in the same segment) at a time, rather than all replicas. This partial action reduces the resource contention and coordination overhead associated with updating all replicas synchronously, thereby improving update throughput while maintaining adequate consistency within segments.
Data Source
AI summary
Techniques are provided for managing replicated data items in a cluster of shared nothing nodes. The techniques include synchronously propagating changes to one or more replicas, while asynchronously propagating changes to the remaining replicas. Because the replicas to which the change is synchronously replicated may be a relatively small percentage of the total number of replicas, the overhead associated with performing an update is reduced. The process of selecting which nodes are involved in the synchronous update operation may take into account a variety of factors, including which replicas are available at the time the update operation is being performed.