Lock-Free Parallel Log Replay for Database Replication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Legacy replication solutions in database systems lock tables during the replication process, leading to query performance degradation, deadlocking, and blocking, which is unacceptable in high-availability systems employing dynamic tiering.
Innovation Solution
A lock-free parallel log replay and synchronization scheme is implemented to support asynchronous table replication, allowing the dynamic tiering engine to process queries involving in-memory and disk-based data without locking the source table, using a replication manager, synchronizer, and replayer to maintain data integrity and availability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If legacy replication solutions lock tables during replication, then data integrity is maintained, but query performance degrades and blocking occurs
Solution Approach 1:
The replication process is segmented into distinct phases: initial data copy, log replay, and incremental updates. The synchronizer copies initial data while the replayer applies transaction logs in the background, allowing queries to proceed without blocking. This segmentation enables concurrent operations that maintain data integrity while improving query performance.
Solution Approach 2:
A replication manager acts as an intermediary between the source database and replica tables, coordinating the synchronization process. The manager orchestrates the synchronizer and replayer components, managing transaction logs and ensuring data consistency without requiring table locks during query operations.
2Stability of the object's composition
If table locking is used during replication, then data consistency is ensured, but deadlocking and blocking occur
Solution Approach 1:
The synchronizer performs preliminary data copying before the replication process fully begins, establishing an initial consistent state. Transaction logs are captured and queued in advance, allowing the replayer to apply changes sequentially without requiring locks on the source tables during query operations.
Solution Approach 2:
The replication process maintains continuous operation through background log replay and incremental updates. The source tables remain accessible for queries throughout the replication process, with data consistency maintained through continuous synchronization rather than periodic locking.
3Productivity
If in-memory storage is used for all data, then query performance improves, but storage costs increase
Solution Approach 1:
Different data are stored in different locations based on their access patterns. Frequently accessed data (hot data) are stored in in-memory replica tables for fast query performance, while less frequently accessed data remain in disk-based source tables. This local quality differentiation optimizes both performance and cost by placing data in the most appropriate storage medium.
Solution Approach 2:
The system adds a spatial dimension to data storage by creating replica tables in memory that mirror the structure of disk-based source tables. This dimensional addition allows the system to provide fast in-memory access for frequently queried data while maintaining the cost-effective disk storage for the complete dataset.
Data Source
AI summary
Disclosed herein are system, method, and computer program product embodiments for providing a lock-free parallel log replay and synchronization scheme to support asynchronous table replication. By synchronizing a replica table with the server-side data and conducting subsequent updates using transaction logs via a replayer, locking of tables may be avoided. A consistent transactional state may be maintained by employing a replayer to mark the table as enabled instead of a synchronizer. The replayer may also deduce transitive closures among transactions and replay the transactions in parallel based on the deduced transitive closures to optimize playback. These techniques provide enhanced data availability and minimize database blocking and deadlocking while improving query performance.