Master Replica Management in Distributed Data Stores
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage systems face challenges in providing seamless scaling, high durability, and low latency while supporting flexible schema and multiple consistency models, especially in handling large datasets and varying workloads across multi-tier e-commerce systems.
Innovation Solution
A Web-based data storage service that enables automatic live repartitioning, supports flexible schema, multiple consistency models, and low latency through a distributed system architecture with replicated partitions across multiple storage nodes, utilizing a single master failover protocol and fault-tolerant log shipping based replication mechanism.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is distributed and replicated across multiple machines for security and availability, then reliability and availability improve, but system complexity increases
Solution Approach 1:
The patent divides the data store into multiple partitions that can be independently replicated across different machines. Each partition is a self-contained unit that can be managed, replicated, and recovered independently, which reduces the overall system complexity while maintaining high availability and reliability through distributed replication.
Solution Approach 2:
The patent introduces a coordination service that acts as an intermediary to manage partition assignments, replication, and failover across the distributed system. This intermediary abstracts the complexity of distributed coordination from individual nodes, enabling reliable data availability without requiring each node to directly manage complex inter-node relationships.
2Reliability
If read operations are served from multiple slave replicas for high availability, then availability improves, but data consistency becomes more difficult to maintain
Solution Approach 1:
The patent implements synchronous replication where slave replicas are updated with data changes before they can serve read operations. This preliminary action ensures that replicas are consistently synchronized with the master before becoming readable, preventing data inconsistency while maintaining high availability through the use of multiple updated replicas.
Solution Approach 2:
The patent employs acknowledgment mechanisms where the master waits for confirmation that slave replicas have successfully received and applied data changes before confirming write operations to clients. This feedback loop ensures data consistency is maintained across replicas while still allowing the system to serve reads from multiple available replicas once they are synchronized.
3Adaptability or versatility
If the system supports flexible schema and multiple consistency models, then adaptability improves, but system complexity increases
Solution Approach 1:
The patent implements dynamic consistency models that allow different partitions or operations to use different consistency levels (strong, eventual, causal) based on specific requirements. This dynamic approach enables the system to adapt to varying workload demands and flexibility requirements without requiring a completely different system architecture for each consistency model, thus managing complexity while maintaining versatility.
Solution Approach 2:
The patent allows schema evolution where data structures can be modified dynamically without requiring system-wide reconfiguration. Partitions can have different schemas tailored to their specific needs, and these schema parameters can be changed independently, providing high adaptability while containing complexity within individual partitions rather than across the entire distributed system.
4Quantity of substance
If the system handles large datasets with seamless scaling, then capacity improves, but latency increases
Solution Approach 1:
The patent divides large datasets into smaller partitions that can be independently managed and scaled. This segmentation allows the system to scale capacity by adding more partitions across additional machines without requiring operations on the entire dataset, thereby maintaining low latency for individual partition operations while achieving large overall capacity through distributed parallelism.
Solution Approach 2:
The patent scales the system by adding horizontal dimensions (more machines, more partitions) rather than vertically increasing the capacity of individual nodes. This dimensional approach to scaling allows the system to handle large datasets by distributing them across multiple dimensions of the distributed architecture, maintaining low latency through parallel processing while achieving massive aggregate capacity.
Data Source
AI summary
A system that implements a data storage service may store data on behalf of clients in multiple replicas on respective computing nodes. The system may employ an external service to select a master replica for a replica group. The master replica may service consistent read operations and/or write operations that are directed to the replica group (or to a data partition stored by the replica group). The master replica may employ a quorum based mechanism for performing replicated write operations, and a local lease mechanism for determining the replica authorized to perform consistent reads, even when the external service is unavailable. The master replica may propagate local leases to replica group members as replicated writes. If another replica assumes mastership for the replica group, it may not begin servicing consistent read operations that are directed to the replica group until the lease period for a current local lease expires.


