Log Read Thread for Low-Latency DBMS Replication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database replication systems face challenges in maintaining predictable and constant latency during asynchronous log-based replication, particularly during periods of low or high activity, due to variable transaction sizes and inter-dependencies, which can lead to blocking issues and exceed maximum tolerable data staleness.
Innovation Solution
A system comprising a log read module, a log flush module, an update module, a reading module, and a notification module, which initializes a log read thread to read a predefined number of DBMS log records in a single API call, generates loggable transactions to unblock the log read thread if it times out, and notifies the log flush thread upon successful read completion, ensuring that the log read thread remains unblocked and continues reading log records.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the system waits for log records to be available before reading, then data accuracy is improved, but latency increases during low activity periods
Solution Approach 1:
The system performs preliminary actions by initializing a log read thread that proactively monitors and reads log records before they are strictly needed. The thread is prepared in advance to read predefined numbers of log records, reducing the waiting time when logs are needed while maintaining data accuracy through controlled reading mechanisms.
Solution Approach 2:
The log read thread operates periodically, reading predefined numbers of log records at scheduled intervals rather than waiting continuously for data. This periodic operation maintains data accuracy by systematically processing logs while reducing latency by establishing predictable reading cycles that prevent indefinite blocking.
2Reliability
If the log read thread blocks until logs are available, then data completeness is improved, but system responsiveness deteriorates
Solution Approach 1:
The system applies partial action by reading a predefined number of log records rather than waiting for complete transaction sets. This approach ensures sufficient data completeness for replication purposes while preventing the thread from blocking indefinitely, thus maintaining system responsiveness during low activity periods.
Solution Approach 2:
The log read thread acts as an intermediary that mediates between the blocking log read API and the replication process. It buffers and processes log records in controlled batches, ensuring data completeness for replication while preventing direct blocking of the main system, thereby maintaining responsiveness.
3Productivity
If multiple log records are read in a single API call, then throughput is improved, but complexity of error handling increases
Solution Approach 1:
The system segments the log reading process into controlled batches by reading predefined numbers of log records in single API calls. Each batch is processed independently with its own error handling logic, which simplifies error management compared to handling large unsegmented batches while still improving throughput through batched operations.
Solution Approach 2:
The system implements feedback mechanisms that monitor the success of batched log reads and adjust subsequent reading operations accordingly. When errors occur in batch processing, the feedback loop enables targeted error handling and recovery without compromising overall throughput, managing complexity through adaptive control.
Data Source
AI summary
A system and method to ensure a low-latency read of log records from a Database Management System (“DBMS”) in asynchronous log-based database replication capture from a blocking log read Application Programming Interface (“API”). The system may include a replication server with a log read module to initialize a log read thread. The reading module of the log read thread may read a predefined number of DBMS log records. A log flush module of the log flush thread may be executed by the log read thread and waits for a read complete notification. The update module, in response to a log flush thread time out, may also generate a plurality of loggable transactions such that a number of log records in an internal API log satisfies a DBMS threshold and unblocks the DBMS API. Therefore, the latency time may be constant, low, and predictable ensuring an accurate replication capture.