Cloud Data Change Event Capture via Transaction Log
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current cloud computing systems face challenges in efficiently capturing and managing data change events across a cluster of application servers, leading to potential data loss and inconsistencies in replication and integration processes.
Innovation Solution
A system where a selected application server stores transaction data and corresponding keys in its memory buffer, transfers copies to other servers, and creates records in a transaction log, ensuring data consistency and redundancy across the cluster.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data change events are captured and stored in memory buffers of multiple application servers, then data redundancy and reliability are improved, but system complexity and memory resource consumption increase
Solution Approach 1:
The system divides the cluster into different roles: producer application servers that generate data change events and consumer application servers that process them. This segmentation allows data to be captured at source (improving reliability) while processing is distributed to specific servers, managing system complexity through role differentiation
Solution Approach 2:
A transaction log is introduced as an intermediary component that records data change events centrally. This mediator allows multiple application servers to access and process the same events without direct peer-to-peer coordination, reducing system complexity while maintaining data reliability across the cluster
2Reliability
If data change events are captured and stored in memory buffers of multiple application servers, then data redundancy is improved, but memory resource consumption increases
Solution Approach 1:
The system implements selective replication where only necessary data change events are copied to consumer servers' memory buffers, not all possible combinations. This partial action approach provides sufficient redundancy for reliability while avoiding the excessive memory consumption that would result from universal replication
Solution Approach 2:
Data change events are copied from producer servers to consumer servers in a controlled manner. The copying is selective and targeted, creating sufficient redundancy for reliability while managing memory resource consumption through intelligent copy decisions based on event type and consumer needs
3Stability of the object's composition
If a transaction log is implemented to track data changes, then data consistency is improved, but device complexity increases
Solution Approach 1:
The transaction log serves multiple functions: it records data change events, provides a sequence for processing, enables recovery operations, and facilitates auditing. This multi-functionality improves data consistency through centralized tracking while avoiding additional complexity by consolidating multiple needs into a single component
Data Source
AI summary
A software application executing on a selected application server in a cluster of application servers stores in a buffer in a memory of the selected application server a payload of data and a corresponding key. The application transfers a copy of the payload of data and the corresponding key to a selected second application server in the cluster, and creates a record in an entry in a transaction log stored in a database to which the cluster of applications servers has access, the record including: the corresponding key, a first value identifying the selected second application server to which the copy of the payload of data and corresponding key were transferred, and a second value identifying the selected first application server in which the payload of data and corresponding key were stored.