Asynchronous Replication Data Loss Mitigation via History Replay
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Asynchronous log shipping in large messaging solutions can result in data loss during failover due to its asynchronous nature, which does not guarantee immediate replication of changes across servers, forcing a tradeoff between recovery characteristics and costs.
Innovation Solution
A computer-implemented data replication system that maintains a short history of recent changes and automatically resubmits this history upon failover, using a feedback loop to coordinate replication progress and reduce data loss by pruning outdated changes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If asynchronous log shipping is used for replication, then cost and complexity are reduced, but data loss occurs during failover
Solution Approach 1:
The system performs preliminary actions by maintaining a short-term history of recent changes on frontend servers before failover occurs. This history is kept ready for quick resubmission to backend servers, ensuring that data can be recovered without requiring complex synchronous replication mechanisms.
Solution Approach 2:
The frontend server acts as an intermediary by maintaining and managing the short-term history of changes. This intermediary stores the history and facilitates its resubmission to backend servers after failover, bridging the gap between asynchronous replication and data integrity requirements.
2Reliability
If synchronous replication is used to prevent data loss, then data integrity is improved, but cost and system complexity increase
Solution Approach 1:
Instead of using complex synchronous replication, the system performs preliminary action by pre-maintaining a short-term history of changes on frontend servers. This history is readily available for immediate resubmission after failover, achieving data integrity without the complexity of synchronous replication protocols.
Solution Approach 2:
The system uses a disposable short-term history mechanism rather than expensive synchronous replication. The history is maintained for a short duration just long enough to cover the failover scenario, then discarded, providing an economical solution to data integrity.
3Reliability
If a longer history of changes is maintained for resubmission, then data recovery completeness is improved, but storage burden and processing overhead increase
Solution Approach 1:
The system applies local quality by maintaining history with a specific, optimized time duration that is just sufficient for failover recovery. This localized approach to history retention balances data recovery completeness with storage efficiency, avoiding the need to maintain extensive historical data.
Solution Approach 2:
The system changes the parameter of history retention duration to an optimized value that provides adequate recovery capability while minimizing storage burden. By adjusting this temporal parameter, the system achieves the right balance between recovery completeness and resource consumption.
Data Source
AI summary
Architecture that reduces data loss resulting from failover in an asynchronous log shipping deployment, but leveraging mid-tier and frontend servers to fill in lost data. In an asynchronous log shipping operation, a replication component asynchronously replicates messaging data to a backend server in accordance with one or more replication operations, which can be updates to databases on the backend server. These databases can include messaging data, such as email address books, mailboxes, etc. A history component maintains a history of replication operations on a frontend server. In the event of a lossy failover, a replay component is used for replaying the replication operations from the history to the backend server.