Master Replica Management in Distributed Data Stores

Resolve Bottlenecks,
Find Innovative Solutions
Generate Solutions

Solution Overview

Problem

Current data storage systems face challenges in providing seamless scaling, high durability, and low latency while supporting flexible schema and multiple consistency models, especially in handling large datasets and varying workloads across multi-tier e-commerce systems.

Innovation Solution

A Web-based data storage service that enables automatic live repartitioning, supports flexible schema, multiple consistency models, and low latency through a distributed system architecture with replicated partitions across multiple storage nodes, utilizing a single master failover protocol and fault-tolerant log shipping based replication mechanism.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If data is distributed and replicated across multiple machines for security and availability, then reliability and availability improve, but system complexity increases

Engineering Contradiction:
Improvedata availabilityVSAvoidsystem complexity
Core Design Contradiction:
ReliabilityVSDevice complexity

Solution Approach 1:

The patent divides the data store into multiple partitions that can be independently replicated across different machines. Each partition is a self-contained unit that can be managed, replicated, and recovered independently, which reduces the overall system complexity while maintaining high availability and reliability through distributed replication.

Inventive Principle:
Principle #1Segmentation

Solution Approach 2:

The patent introduces a coordination service that acts as an intermediary to manage partition assignments, replication, and failover across the distributed system. This intermediary abstracts the complexity of distributed coordination from individual nodes, enabling reliable data availability without requiring each node to directly manage complex inter-node relationships.

Inventive Principle:
Principle #24Intermediary (Mediator)

2Reliability

If read operations are served from multiple slave replicas for high availability, then availability improves, but data consistency becomes more difficult to maintain

Engineering Contradiction:
ImproveavailabilityVSAvoiddata consistency
Core Design Contradiction:
ReliabilityVSLoss of information

Solution Approach 1:

The patent implements synchronous replication where slave replicas are updated with data changes before they can serve read operations. This preliminary action ensures that replicas are consistently synchronized with the master before becoming readable, preventing data inconsistency while maintaining high availability through the use of multiple updated replicas.

Inventive Principle:
Principle #10Preliminary action

Solution Approach 2:

The patent employs acknowledgment mechanisms where the master waits for confirmation that slave replicas have successfully received and applied data changes before confirming write operations to clients. This feedback loop ensures data consistency is maintained across replicas while still allowing the system to serve reads from multiple available replicas once they are synchronized.

Inventive Principle:
Principle #23Feedback

3Adaptability or versatility

If the system supports flexible schema and multiple consistency models, then adaptability improves, but system complexity increases

Engineering Contradiction:
ImproveflexibilityVSAvoidsystem complexity
Core Design Contradiction:
Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The patent implements dynamic consistency models that allow different partitions or operations to use different consistency levels (strong, eventual, causal) based on specific requirements. This dynamic approach enables the system to adapt to varying workload demands and flexibility requirements without requiring a completely different system architecture for each consistency model, thus managing complexity while maintaining versatility.

Inventive Principle:
Principle #15Dynamics

Solution Approach 2:

The patent allows schema evolution where data structures can be modified dynamically without requiring system-wide reconfiguration. Partitions can have different schemas tailored to their specific needs, and these schema parameters can be changed independently, providing high adaptability while containing complexity within individual partitions rather than across the entire distributed system.

Inventive Principle:
Principle #35Parameter changes

4Quantity of substance

If the system handles large datasets with seamless scaling, then capacity improves, but latency increases

Engineering Contradiction:
Improvedata capacityVSAvoidlatency
Core Design Contradiction:
Quantity of substanceVSLoss of time

Solution Approach 1:

The patent divides large datasets into smaller partitions that can be independently managed and scaled. This segmentation allows the system to scale capacity by adding more partitions across additional machines without requiring operations on the entire dataset, thereby maintaining low latency for individual partition operations while achieving large overall capacity through distributed parallelism.

Inventive Principle:
Principle #1Segmentation

Solution Approach 2:

The patent scales the system by adding horizontal dimensions (more machines, more partitions) rather than vertically increasing the capacity of individual nodes. This dimensional approach to scaling allows the system to handle large datasets by distributing them across multiple dimensions of the distributed architecture, maintaining low latency through parallel processing while achieving massive aggregate capacity.

Inventive Principle:
Principle #17Another dimension (Dimensionality change)

Data Source

PatentUS11899684B2System and method for maintaining a master replica for reads and writes in a data store
Publication Date: 2024.02.13 AMAZON TECH INC
  • US11899684B2 patent drawing
  • US11899684B2 patent drawing
  • US11899684B2 patent drawing

AI summary

A system that implements a data storage service may store data on behalf of clients in multiple replicas on respective computing nodes. The system may employ an external service to select a master replica for a replica group. The master replica may service consistent read operations and/or write operations that are directed to the replica group (or to a data partition stored by the replica group). The master replica may employ a quorum based mechanism for performing replicated write operations, and a local lease mechanism for determining the replica authorized to perform consistent reads, even when the external service is unavailable. The master replica may propagate local leases to replica group members as replicated writes. If another replica assumes mastership for the replica group, it may not begin servicing consistent read operations that are directed to the replica group until the lease period for a current local lease expires.