Replication Units for Low-Latency Strong-Consistency Database Failover
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current database replication solutions fail to meet the requirements of high performance, low overhead replication with strong consistency, fast failover, zero data loss, and full SQL support while ensuring horizontal scalability and symmetric shards.
Innovation Solution
Implement asynchronous database transaction replication using a consensus protocol like Raft, where the leader server performs operations and creates replication log records without waiting for follower consensus, and synchronously replicates commit log records, ensuring immediate results for clients and efficient replication.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If synchronous database replication is implemented to ensure strong consistency, then data consistency is improved, but user transaction response time increases
Solution Approach 1:
The patent segments the replication process into two distinct parts: (1) asynchronous application of transactions to follower replicas for fast reads, and (2) synchronous commit logging to ensure consistency. This segmentation allows different operations to use different replication modes, resolving the contradiction between speed and consistency.
Solution Approach 2:
The patent applies partial synchronous replication only where necessary (commit logs and metadata) while allowing asynchronous replication for data operations. This partial application of synchronicity maintains consistency for critical operations without slowing down all transactions, thus resolving the time-consistency tradeoff.
2Productivity
If more shards are used to balance workload and achieve horizontal scalability, then system capacity is improved, but replication overhead increases
Solution Approach 1:
The patent implements dynamic replication group management where replication groups can be created, modified, and destroyed on-the-fly based on workload demands. This dynamic approach allows the system to optimize replication overhead by adjusting the number and composition of replication groups as shards are added or removed, rather than maintaining fixed replication structures.
Solution Approach 2:
The system performs preliminary actions by pre-configuring replication groups and leaders before workload increases, allowing for efficient scaling. When new shards are added, the system proactively establishes their replication relationships in advance, reducing the overhead of ad-hoc replication setup and enabling smoother horizontal scaling.
3Reliability
If automatic fast failover is implemented to improve availability, then system availability is improved, but complexity of failover management increases
Solution Approach 1:
The patent implements self-service automatic failover where the replication manager autonomously monitors replica health, detects failures, and promotes appropriate followers to leader status without manual intervention. The system automatically manages the complexity of failover scenarios including split-brain detection, quorum calculation, and coordinated state transitions, reducing operational complexity while maintaining high availability.
4Productivity
If replication units are optimized for efficient data management, then data management efficiency is improved, but replication configuration complexity increases
Solution Approach 1:
The patent creates a universal replication manager that handles multiple functions including replication group management, leader election, failover coordination, and performance optimization through a single integrated component. This multi-functional approach consolidates what would otherwise be separate complex configuration tasks into a unified system that automatically manages replication units, reducing configuration complexity while maintaining data management efficiency.
Data Source
AI summary
A consensus protocol-based replication approach is provided. Chunks are grouped into replication units (RUs) to optimize replication efficiency. Chunks may be assigned to RUs based on load and replication throughput. Splitting and merging RUs do not interrupt concurrent user workload or require routing changes. Transactions spanning chunks within an RU do not require distributed transaction processing. Each replication unit has a replication factor (RF), which refers to the number of copies/replicas of the replication unit, and an associated distribution factor (DF), which refers to the number of servers taking over the workload from a failed leader server. RUs may be placed in rings of servers, where the number of servers in a ring is equal to the replication factor, and quiescing the workload can be restricted to a ring of servers instead of the entire database.