Multi-constrained Event Batching for Database Replication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The inefficiency in database replication due to excessive overhead in change data capture events during database migration, particularly in multi-tenant systems, where frequent transactions lead to high resource consumption and synchronization issues.
Innovation Solution
Implementing a multi-constrained event batching system with multiple buffers and asynchronous processing, which dynamically creates batches based on prioritized constraints and uses tenant scores to optimize resource allocation, decouples event extraction and transmission, and splits events into partial attributes for independent processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If change data capture events are transmitted frequently during database replication, then data synchronization accuracy is improved, but network resource consumption and system overhead increase
Solution Approach 1:
The patent combines multiple change data capture events into batches before transmission. The event batcher aggregates events from multiple sources and combines them into single transmission batches, reducing the number of network transmissions while maintaining data synchronization accuracy.
Solution Approach 2:
The patent performs preliminary batching and filtering of events before transmission. The system prepares event batches in advance, applies filtering criteria, and optimizes the batch composition before sending to the target system, reducing unnecessary network traffic.
2Reliability
If change data capture events are transmitted frequently during database replication, then data synchronization accuracy is improved, but system overhead and processing time increase
Solution Approach 1:
Multiple events are merged into single batches for processing and transmission. The event batcher consolidates individual events into batches that are processed together, reducing the total number of processing operations and time spent on event handling.
Solution Approach 2:
The system implements periodic batching intervals where events are accumulated over defined time periods before being processed and transmitted. This periodic approach optimizes processing time by avoiding continuous individual event handling while maintaining synchronization accuracy.
3Reliability
If multiple subsystems process the same event, then data consistency across subsystems is improved, but synchronization overhead increases
Solution Approach 1:
The patent segments the event processing into independent parallel paths. Different subsystems can process different aspects of the batched events independently, reducing synchronization points and overhead while maintaining data consistency through the batched processing model.
Solution Approach 2:
The event batcher acts as an intermediary layer between event sources and multiple subsystems. It standardizes event formatting and provides a unified interface, reducing the complexity of direct synchronization between multiple subsystems while maintaining data consistency.
4Use of energy by moving object
If event batching is implemented during database replication, then network resource efficiency is improved, but event processing complexity increases
Solution Approach 1:
The event batcher implements dynamic batching behavior that adapts to changing conditions. It dynamically adjusts batch sizes, timing, and composition based on event types, system load, and network conditions, optimizing network efficiency while managing processing complexity through adaptive rather than static rules.
Data Source
AI summary
An online system performs replication of database from one system to another. A source database is replicated to a target database by performing an initial bulk copy followed by repeated change data capture. The system performs batching to group events generated by the change data capture events. The system performs a multi-constrained analysis to determine when a batch is ready for sending. The system uses multiple buffers for transmitting events to decouple the receiving and sending of events. The online system may be a multi-tenant system the source and target databases stores data of tenants. The system determines a tenant score to adjust the resources allocated during event transfer. The system splits an event into partial events, to avoid dependency between the partial events so that tasks can be executed independent of each other. Therefore, the system performs replication with reduced synchronization overhead.