Message Throttle for Real-Time Data Replication Sequencing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data replication methods, particularly batch data replication, are cumbersome, slow, and inefficient as they do not operate in real-time and handle data in large segments, leading to potential data integrity issues due to out-of-sequence processing.
Innovation Solution
A data replicator system comprising a system exit, message broker, and sequence engine that captures and re-sequences data in real-time, ensuring chronological order and using a message throttle to control data flow, preventing data loss by managing the channel between the message broker and sequence engine based on message queue thresholds.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If batch data replication is used, then data can be replicated in large segments, but the process becomes cumbersome, slow, and inefficient
Solution Approach 1:
The patent segments data replication into individual messages captured by system exits, processed by a message broker, and delivered to the sequence engine. This segmentation enables real-time processing of small data units rather than large batch segments, significantly improving replication efficiency and reducing time loss.
Solution Approach 2:
The patent implements continuous data replication through real-time capture of individual messages by system exits, continuous conversion by the message broker, and continuous sequencing by the sequence engine. This continuous processing eliminates the interruptions and delays inherent in batch processing, maintaining productive action throughout the replication process.
2Reliability
If real-time data capture is implemented, then data integrity is improved, but the system complexity increases
Solution Approach 1:
The patent introduces a message broker as an intermediary component that receives individual messages from system exits, converts them to the destination database format, and forwards them to the sequence engine. This intermediary layer manages the complexity of real-time data capture and transformation, ensuring data integrity while isolating the complexity from the overall system architecture.
3Productivity
If data flow is not controlled, then replication speed increases, but data loss occurs due to message queue overflow
Solution Approach 1:
The patent implements a feedback mechanism where the message throttle monitors the number of messages in the message queue and dynamically adjusts the channel state between the message broker and sequence engine. When the queue reaches a threshold, the throttle closes the channel to prevent overflow and data loss; when the queue is cleared, the channel is reopened to resume replication, maintaining both speed and integrity.
4Productivity
If messages are processed out of sequence, then processing efficiency improves, but data integrity deteriorates
Solution Approach 1:
The patent performs preliminary sequencing of individual messages by the sequence engine before they are delivered to the destination database. The sequence engine receives messages in any order, reorders them chronologically based on their sequence numbers, and then delivers them in the correct sequence. This preliminary sequencing action ensures data integrity is maintained while allowing efficient parallel processing during message capture and conversion.
Data Source
AI summary
According to one aspect, embodiments of the invention provide a data replicator comprising a system exit configured to capture data received at a source, the captured data including a plurality of individual messages, a message broker configured to receive the plurality of individual messages from the configuration control system and to convert the plurality of individual messages to a format of a destination, and a sequence engine in communication with the message broker via a channel, the sequence engine configured to receive and store the converted plurality of individual messages, to re-sequence the converted plurality of individual messages into a desired sequence, and to provide the re-sequenced messages to a destination database in the desired sequence, wherein the sequence engine includes a message throttle which is configured to close the channel in response to a determination by the message throttle that the sequence engine cannot receive additional converted individual messages.