Distributed Data Processing with Two-Phase Cross-Cluster Replication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage systems in distributed environments face issues with data loss and inconsistency due to node failures, particularly in high-availability scenarios like multi-region active-active setups, where synchronous replication is required but current methods fail to ensure strict data consistency across clusters.
Innovation Solution
A data processing method and system that employs a cross-cluster synchronous replication solution based on a consensus protocol, using a two-phase apply approach to ensure data preprocessing and update requests are processed sequentially across clusters, ensuring data consistency and preventing data loss by setting storage units to an inaccessible 'Prep' state during updates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is stored in multiple data storage nodes of a distributed system to avoid data loss, then data security is improved, but data inconsistency and system complexity increase when any node fails
Solution Approach 1:
The patent divides the data storage system into multiple independent data storage nodes (first data storage node, second data storage node, etc.), each capable of storing complete copies of data. This segmentation allows the system to distribute data across multiple nodes, improving reliability while maintaining manageable complexity through modular architecture.
Solution Approach 2:
The patent implements preliminary data replication before any failure occurs. The request processing module sends data processing requests to multiple data storage nodes in advance, ensuring that data is already replicated across nodes before a failure event. This preliminary action eliminates the need for complex real-time failover decisions during failures.
2Reliability
If synchronous replication is implemented across clusters to ensure data consistency, then data consistency is improved, but processing time and system complexity increase
Solution Approach 1:
The patent performs preliminary actions by first sending data processing requests to multiple data storage nodes before actual data updates. The request processing module waits for all nodes to acknowledge receipt of the preprocessing request before initiating the actual data update operation. This preliminary coordination ensures data consistency without requiring complex synchronous replication during the update phase.
Solution Approach 2:
The patent segments the data processing operation into distinct phases: a preprocessing phase where requests are sent to multiple nodes independently, and an execution phase where data updates are applied. This segmentation allows the system to achieve data consistency across clusters while reducing processing time by parallelizing the preprocessing operations.
3Reliability
If data storage units are set to inaccessible state during updates to prevent data loss, then data security is improved, but data availability and ease of operation decrease
Solution Approach 1:
The patent sets data storage units to an inaccessible state as a preliminary action before actual data updates occur. The request processing module sends preprocessing requests to all data storage nodes, waits for confirmation that the units are set to the inaccessible state, and only then proceeds to send data processing requests. This preliminary state change prevents data loss while minimizing impact on availability since the state is temporary and automatic.
Solution Approach 2:
The patent implements a feedback mechanism where the request processing module waits for preprocessing completion notifications from all data storage nodes before proceeding to data updates. This feedback loop ensures that data storage units are properly prepared in the inaccessible state before updates, and automatically transitions to accessible state after successful updates, maintaining data availability without compromising security.
Data Source
AI summary
A data processing method and apparatus are disclosed. The data processing method comprises: receiving a data processing request, wherein the data processing request carries target data; generating, based on the target data, a data preprocessing request for the target data; sending the data preprocessing request to at least two data storage modules respectively; in a case where a preprocessing completion notification returned by each data storage module in response to the data preprocessing request is received, sending the data processing request to each data storage module; and receiving a data processing completion notification returned by each data storage module in response to the data processing request. The method ensures the data consistency of each data storage module, further avoiding the problem of data loss in a case where any data storage module of at least two data storage modules fails, thereby ensuring the security of data.