Leader Election and Synchronous Replication in Distributed Region Servers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current distributed computing systems face inefficiencies in failover mechanisms due to asynchronous data replication, leading to potential data loss and prolonged failover times, especially when a server or network fails, as there is no guarantee of when data is replicated across clusters, and there is no efficient method to select a master server.
Innovation Solution
Implementing a distributed computing system where data is managed by region servers with a designated leader and followers, using synchronous replication protocols like Raft, and assigning ranks to region servers for leader election, allowing for quick failover with active and shadow witness servers to minimize data loss and optimize failover times.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If asynchronous replication is used to replicate data from master server to slave servers, then the system can operate with higher throughput and lower latency during normal operation, but when a server fails there is no guarantee of when data is replicated leading to potential data loss and prolonged failover times
Solution Approach 1:
The patent implements pre-synchronization of data between master and slave servers before actual failover events occur. This preliminary action ensures that slave servers have up-to-date data ready, eliminating the uncertainty of replication timing during failover and preventing data loss while maintaining high throughput during normal operation
Solution Approach 2:
The system dynamically adjusts the replication mode between asynchronous during normal operation and synchronous upon detection of server failure. This dynamic switching allows the system to optimize for throughput during stable operation while ensuring data availability and minimizing failover time when failures occur
2Reliability
If the system waits for all servers to replicate data before failover to ensure no data loss, then data availability is maintained, but significant time is consumed during the failover process
Solution Approach 1:
Data synchronization is performed in advance during normal operation, so when failover is needed, the slave server already has the necessary data and can immediately take over as master, eliminating the need to wait for replication completion and reducing failover time to minimal levels
3Device complexity
If traditional master-slave replication is used without efficient selection mechanism, then the system structure is simple, but there is no efficient way of selecting one of the servers as a master server during failover
Solution Approach 1:
Slave servers are pre-designated and prepared with data synchronization capabilities before failover events. This preliminary preparation allows for efficient automatic selection of the next master server without complex real-time decision-making, maintaining simple system structure while improving selection efficiency
Solution Approach 2:
The system implements automatic failover where slave servers autonomously detect master server failures and promote themselves to master status based on pre-established protocols, eliminating the need for complex manual selection processes or centralized coordination while maintaining simplicity
Data Source
AI summary
The disclosure is directed to failover mechanisms in a distributed computing system. A region of data is managed by multiple region servers. One of the region servers is elected as a “leader” and the remaining are “followers.” The leader serves the read/write requests from a client. The leader writes the data received from the client into the in-memory store and a local write-ahead log (“WAL”), and synchronously replicates the WAL to the followers. A region server designated as an “active” region server synchronizes a distributed data store with the data from the WAL. Active witness followers apply the data from the WAL to their in-memory store while shadow witness followers do not. Different types of servers provide failover mechanisms with different characteristics. A leader is elected based on their associated ranks—higher the rank, higher the likelihood of electing itself as a leader.