Read-Replica Parallel Recovery with LSN-Ordered WAL Batching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database management systems (DBMS) struggle to keep up with the processing of a high volume of write ahead log (WAL) records, leading to lag in read-replica databases and inability to serve the latest data.
Innovation Solution
Implementing a parallel recovery mode that allows for batch processing of log records in log sequence number (LSN) order, while still serving read queries, and managing dependencies between log records to maintain transactional consistency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If serial log record processing is used, then transactional consistency is maintained, but processing speed cannot keep up with high volume WAL records
Solution Approach 1:
The patent segments the batch of log records into multiple subsets, each assigned to a different parallel process for simultaneous processing. This segmentation enables the system to process high volumes of WAL records in parallel while maintaining consistency through careful management of dependencies between segments.
Solution Approach 2:
The patent implements dynamic parallel processing where multiple processes can simultaneously apply log records to different data blocks. The system dynamically manages the parallel execution of processes while ensuring that transactional consistency is maintained through dependency tracking and read transaction coordination.
2Productivity
If parallel processing is used, then processing speed increases, but maintaining LSN order and transactional consistency becomes complex
Solution Approach 1:
The patent performs preliminary actions by receiving and batching log records before parallel processing begins. The system prepares the log records in advance, organizing them into batches that can be efficiently distributed to parallel processes while preserving LSN order within each process's assigned subset.
Solution Approach 2:
The patent implements feedback mechanisms where the system monitors the state of parallel processing and coordinates with read transactions. When a read transaction accesses a data block being modified, the system provides feedback to ensure the read waits for appropriate log records to be applied, maintaining consistency without blocking parallel processing.
3Productivity
If log records are applied in batches, then processing efficiency improves, but read queries may access inconsistent data
Solution Approach 1:
The patent applies local quality by ensuring that each parallel process maintains LSN order for its assigned subset of log records. This local ordering guarantee ensures that within each process's scope, data consistency is maintained while allowing different processes to operate independently on different data blocks.
Solution Approach 2:
The patent uses an intermediary mechanism to coordinate between parallel processing and read queries. The system acts as an intermediary by managing the interaction between log record application and read transactions, ensuring that reads access consistent data snapshots while parallel processing continues uninterrupted.
Data Source
AI summary
Aspects of the disclosure are directed to a parallel recovery mode that applies log records while allowing read queries on read-replica databases. The parallel recovery mode can include applying log records in log sequence number (LSN) order for a block or for multiple blocks, and managing log records affecting multiple blocks. The parallel recovery mode can further manage dependency between different log records and maintain transactional consistency on read queries.