Latch-Free Shared Queue for Database Change Data Replication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current database systems face inefficiencies in distributed information sharing through log mining due to the need for manual configuration and error-prone processes when handling multiple source instances, particularly in using the combined capture and apply mechanism with latch mechanisms that slow down concurrent access.
Innovation Solution
A latch-free, single publisher, multiple subscriber shared queue system is implemented, where an enqueuing process mines redo logs to produce logical change records and places them in a shared queue, allowing subscribers to read and dequeue messages without latches, enabling efficient distribution of change information across multiple apply processes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If latch mechanisms are used to coordinate concurrent access to shared queue, then data consistency is maintained, but system performance and throughput deteriorate due to access overhead and blocking
Solution Approach 1:
The patent removes the latch mechanism entirely from the shared queue implementation. Instead of using traditional locking mechanisms to ensure data consistency, the system employs a latch-free architecture where multiple subscribers can concurrently access and dequeue messages from the shared queue without blocking each other, thus eliminating the performance bottleneck while maintaining reliability through alternative synchronization methods
Solution Approach 2:
The patent replaces the mechanical latch-based synchronization system with a software-based concurrent access mechanism. The shared queue uses atomic operations and memory barriers at the hardware level combined with careful software design to ensure thread-safe access without requiring latches, substituting the mechanical locking system with a more efficient software-hardware integrated approach
2Device complexity
If manual configuration is used for combined capture and apply mechanism, then system complexity is reduced, but error rate increases and automation capability deteriorates
Solution Approach 1:
The patent implements automatic configuration capabilities where the system can self-configure the capture and apply processes without requiring manual intervention. The system automatically discovers source instances, configures subscription relationships, and sets up the shared queue architecture, thereby reducing human error while maintaining manageable system complexity through automated routines
Solution Approach 2:
The patent incorporates feedback mechanisms that allow the system to monitor its own configuration state and automatically adjust settings. The system provides feedback loops that detect configuration errors, validate settings, and automatically correct issues, reducing the error rate associated with manual configuration while keeping the overall system complexity manageable through intelligent automation
3Reliability
If latches are used to ensure thread-safe access to shared queue, then data integrity is maintained, but access speed and concurrency performance deteriorate
Solution Approach 1:
The patent extracts and removes the latch mechanism from the system entirely. The shared queue is designed to be inherently thread-safe without requiring latches, using atomic operations for enqueue and dequeue operations. This allows multiple threads to access the queue simultaneously at full speed while data integrity is maintained through the atomic nature of the operations and memory consistency models
Solution Approach 2:
The patent changes the fundamental parameter of synchronization from latch-based blocking to latch-free atomic operations. By changing the synchronization mechanism from software latches that cause blocking to hardware-level atomic operations that don't block, the system achieves both data integrity and high-speed concurrent access simultaneously
Data Source
AI summary
A method for sharing information between a publisher and multiple subscribers is provided. The publisher uses a latch-free, single publisher, multiple subscriber shared queue to share information. Logical change records representing changes made to a database are enqueued in the shared queue as messages in a stream of messages, and subscribers read the logical change records. Subscribers may filter logical change records before sending to apply processes for processing. An identifying property of the source instance of a change encapsulated in a logical change record may be included with each message enqueued.