Resilient Transaction Pipeline Across Data Centers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cloud-based deployments of financial transaction processing systems face challenges in minimizing data loss and performance impact due to communication and processing latency variations, making it difficult to ensure zero transaction loss and rapid recovery in case of failures.
Innovation Solution
Implementing a system that intercepts incoming transactions and forwards copies to a backup instance for processing, ensuring that the primary instance only proceeds with transaction processing after receiving an acknowledgment from the backup, thereby minimizing data loss and performance degradation by regulating the communication of outputs based on successful receipt and commitment by the backup instance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If transactions are forwarded to backup instance for acknowledgment before processing, then data loss is minimized, but processing latency increases
Solution Approach 1:
The system performs preliminary action by forwarding transaction copies to the backup instance and obtaining acknowledgments before the primary instance processes the transactions. This ensures the backup is ready to take over if failure occurs, minimizing data loss while managing latency through parallel preparation.
Solution Approach 2:
The system creates a copy of the transaction and sends it to the backup instance while the primary instance retains the original for processing. This copying mechanism allows the backup to prepare independently without blocking the primary processing path, resolving the contradiction between reliability and latency.
2Reliability
If multiple data centers are used for backup, then system resiliency is improved, but communication latency variations increase
Solution Approach 1:
The system segments the backup function across multiple geographically distributed data centers, each handling portions of the backup workload. This segmentation provides resiliency through distribution while managing latency variations by allowing flexible selection of backup centers based on current communication conditions.
Solution Approach 2:
The system dynamically selects which backup data center to use based on current communication latency conditions. When latency variations occur, the system can switch between different backup centers, maintaining resiliency while adapting to changing network conditions to minimize communication delays.
3Reliability
If synchronous replication is implemented, then data consistency is ensured, but system performance degradation occurs
Solution Approach 1:
The system implements asynchronous copying where the primary instance processes transactions using the original data while the backup instance receives and processes copies. This ensures data consistency eventually without blocking the primary system's performance, as the copy operation occurs in parallel without synchronous waiting.
Solution Approach 2:
The system uses an intermediary acknowledgment mechanism where the backup instance sends notifications when it has successfully processed transaction copies. This intermediary feedback allows the primary instance to continue processing without waiting for backup completion, maintaining performance while ensuring eventual data consistency through the acknowledgment trail.
Data Source
AI summary
The disclosed embodiments relate to a system/process which can be used to provide redundancy and error/failure recovery to a System of Record (SOR), such as an electronic financial transaction processing system or sub-system thereof, which processes transactions received from one or more sources and generates results/outputs based thereon for communication to one or more recipients/destinations/consumers. The disclosed embodiments minimize, if not eliminate, both the amount of data, i.e., the number of transactions, which can be lost during a failure and the performance impact on the SOR. Generally, the disclosed embodiments are implemented so as to receive/intercept each incoming transaction communicated to the primary instance prior to receipt thereby and intercept or otherwise regulate the communication of the outputs of the primary instance, generated based on the primary instance's processing of the incoming transactions, to the destinations/recipients/consumers thereof.