Multi-level replication filtering for distributed database log records
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed databases face challenges in scaling network bandwidth efficiently, particularly with multiple writers, leading to bottlenecks and performance issues due to the need to send log records to all replicas and writers, which exhausts network capacity and causes replication to lag behind, resulting in storage nodes running out of memory.
Innovation Solution
Implementing multi-level replication filtering by having primary nodes selectively send log records only to read replicas that need them, using page-level and database object-level filtering to determine which records are applicable, and sending invalidation records to maintain consistency, thereby reducing unnecessary network traffic and optimizing resource usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the entire log of changes is sent to all different processes implemented at separate computer systems, then data consistency is maintained across the distributed system, but network bandwidth is consumed and processing resources are wasted
Solution Approach 1:
The patent implements filtering mechanisms that identify and send only the specific log records relevant to each individual reader process. Each reader process receives a customized subset of log records based on its specific needs and the changes that actually affect it, rather than receiving the entire log. This localizes the quality of information received to match the specific requirements of each process.
Solution Approach 2:
The patent extracts and filters out only the necessary log records from the entire change log that are applicable to each reader process. By using filtering criteria based on the identified portion of data stored at each reader, the system extracts only the relevant subset of log records, leaving out the unnecessary ones that would consume network bandwidth without providing value.
2Productivity
If additional hardware or software resources are deployed to increase service capacity, then the computing service can meet growing demand, but the complexity of the distributed system increases
Solution Approach 1:
The patent segments the log record distribution task by implementing filtering mechanisms at the primary node that divide the entire log stream into smaller, targeted subsets for each reader process. This segmentation occurs based on the identified portions of data stored at each reader, allowing the system to scale to multiple readers without proportionally increasing the complexity of log management, as each reader receives only its relevant segment.
Solution Approach 2:
The patent performs preliminary filtering and identification of relevant log records at the primary node before distribution to reader processes. By pre-processing the log stream and identifying which log records are applicable to each reader's stored data portions, the system reduces the complexity that would otherwise arise at the reader nodes, centralizing the filtering logic in advance.
Data Source
AI summary
This application relates to multi-level replication filtering for a distributed database. A primary node of the database can generate log records that describe write requests performed to the database. The primary node can, in some embodiments, identify a portion of the data of the database that is stored at a second node, such as data stored by a read replica node that can be used in responding to read requests from clients. The primary node can select log records that are applicable to this identified portion of the data to send to the second node. Log records are not selected based on a determination that those unselected log records are not applicable to the identified portion of the data. The primary node can then send the selected log records to the second node, for the second node to apply the log records to its stored database data.