Database Replication Scaling via Segmented Log Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data replication in distributed database management systems can lead to data corruption due to incorrect processing order of updates, especially when parallel processing is employed without proper synchronization of related tables.
Innovation Solution
The system identifies related tables through access patterns and forms replication groups, processing related tables serially while allowing unrelated tables to be processed in parallel, with mechanisms to adjust and refine parallelization based on drift parameters and outlier handling.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data replication uses serial transmission and processing of log entries, then data consistency is maintained, but replication performance is limited
Solution Approach 1:
The patent segments the replication process by dividing log entries into different categories (committed entries vs. uncommitted entries) and routing them through different processing paths. Committed entries are replicated immediately in parallel, while uncommitted entries are held for later processing, enabling partial parallelization without compromising data consistency.
Solution Approach 2:
The system dynamically adjusts replication behavior based on transaction commit status. Log entries are processed differently depending on whether their transactions have committed, allowing the system to optimize performance for safe operations while maintaining consistency for dependent operations through conditional parallel processing.
2Productivity
If parallel processing is used to increase replication performance, then productivity improves, but data corruption risk increases due to incorrect processing order
Solution Approach 1:
The patent divides replicated data into independent segments (committed transactions) that can be processed in parallel without affecting each other, while maintaining serial processing for dependent segments (uncommitted transactions), thus achieving parallel performance improvement without data corruption.
Solution Approach 2:
The system introduces an intermediary mechanism (transaction commit status tracking) that mediates between parallel processing demands and consistency requirements, allowing the system to safely parallelize operations that have completed transactions while deferring those that haven't.
3Reliability
If all tables are processed serially to ensure correct update order, then data consistency is maintained, but replication time increases
Solution Approach 1:
The patent segments tables into groups based on their transactional relationships, allowing independent table groups to be processed in parallel while maintaining serial processing within each group, thereby reducing overall replication time without sacrificing update order correctness.
Solution Approach 2:
The system performs preliminary analysis of transaction commits and table relationships before replication, pre-identifying which tables can be safely processed in parallel, thus avoiding full serial processing while ensuring correctness through advance preparation.
4Productivity
If parallel replication streams are used, then productivity improves, but system complexity increases due to synchronization requirements
Solution Approach 1:
The patent simplifies the parallel replication system by segmenting streams into committed and uncommitted categories, each with simple, well-defined processing rules, avoiding the need for complex synchronization protocols while still achieving parallel throughput improvement.
Data Source
AI summary
A hosted database service may receive a request to replicate data between two locations. Data may be replicated serially during an initial observation period. During the observation period, replication data may be analyzed to detect access patterns involving transaction delimiters and sequential or closely-timed access to data. Related tables may be assigned to replication groups. Replication data for the replication groups may be sent and processed in parallel. During parallel replication, effectiveness of the replication data may be monitored and the replication groups may be periodically readjusted.