Checkpoint-Based Data Replication for Server Cluster Failover
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Ensuring the safety and security of data while optimizing data processing across server clusters and data centers is challenging, particularly in maintaining consistency and coherence of distributed database records, especially when failures occur and data needs to be replicated across multiple systems.
Innovation Solution
Implementing a checkpoint-based data replication method where a computing platform initiates data processing jobs, directs server nodes to execute processing steps, updates a checkpoint table, and replicates results across data centers, allowing for failover capabilities in case of failures by resuming data processing from the last completed checkpoint.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is replicated across multiple data centers, then data safety and security are improved, but system complexity increases
Solution Approach 1:
The patent implements checkpoint tables that record the state of data processing at regular intervals before failures can occur. These checkpoints are pre-established during normal operation, enabling the system to quickly resume from a known good state without complex real-time coordination between data centers during failure recovery
Solution Approach 2:
The patent divides the distributed data processing system into independent segments with local checkpoint tables at each data center. Each data center maintains its own checkpoint state and can independently resume processing, reducing the complexity of inter-data-center coordination while maintaining data safety through replication
2Stability of the object's composition
If data processing jobs are restarted from the beginning after failure, then data consistency is maintained, but processing time increases
Solution Approach 1:
Checkpoint tables are continuously updated during normal data processing operations to record the state after each processing step. When a failure occurs, these pre-recorded checkpoints enable immediate resumption from the last known good state, eliminating the time loss associated with restarting from the beginning while maintaining data consistency through the recorded checkpoint states
3Reliability
If checkpoint tables are replicated across data centers, then failover capability is improved, but network bandwidth consumption increases
Solution Approach 1:
Each data center maintains its own local checkpoint table with the processing state specific to that data center. This localization reduces the amount of data that needs to be replicated across the network, as each data center only needs to replicate its own checkpoint state rather than receiving complete checkpoint information from all other data centers, thereby reducing network bandwidth consumption while maintaining failover capability
Data Source
AI summary
Aspects of the disclosure relate to optimizing data processing across server clusters and data centers using checkpoint-based data replication. A computing platform may determine to initiate a data processing job associated with identifying one or more features of a source dataset, and the data processing job may include multiple processing steps. Based on determining to initiate the data processing job, the computing platform may generate one or more commands directing one or more cluster server nodes associated with a data center to execute the multiple processing steps. The one or more commands may direct the one or more cluster server nodes to update a checkpoint table as each processing step is completed, and may further direct the one or more cluster server nodes to replicate processing results data to at least one other data center. Subsequently, the computing platform may send the generated commands to the cluster server nodes.