Consistent Backups Using Local Node Clocks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data backup systems for large-scale databases face challenges in achieving causal consistency during backups, leading to interruptions in production systems and inefficiencies due to the need for central coordination, clock skew issues, and the inability to handle distributed systems effectively.
Innovation Solution
A consistent data backup system that uses a central coordinator to broadcast a backup time to nodes, allowing each node to perform backups based on its local clock, ensuring causal consistency by blocking updates for a duration that accounts for maximum clock skew, and utilizing transaction logs to create point-in-time backups without interrupting production systems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If scan-based backup solutions are used to backup database data, then backup capability is provided, but production workloads are interrupted and read throughput is reduced due to resource competition
Solution Approach 1:
The backup system segments the database into multiple independent partitions that can be backed up concurrently. Each partition is backed up by a separate backup thread without requiring global locks, allowing production workloads to continue on other partitions. This eliminates the resource competition that reduces read throughput while maintaining backup capability.
2Reliability
If centralized coordination is used to ensure causal consistency during backups, then consistency is achieved, but system complexity increases and performance decreases
Solution Approach 1:
Each storage node autonomously determines when to perform backups using its local wall-clock time, without requiring centralized coordination. The system achieves causal consistency through self-service mechanisms where nodes independently block updates for durations accounting for maximum clock skew, eliminating complex centralized scheduling while maintaining consistency.
Solution Approach 2:
The system changes the approach from centralized time coordination to using local wall-clock times with compensation for clock skew. By adjusting backup timing parameters based on maximum expected clock skew between nodes, the system achieves causal consistency without centralized coordination, reducing system complexity.
3Reliability
If scan-based solutions are used for backups, then data can be copied to storage, but the backup process takes too much time and interrupts customer workflow
Solution Approach 1:
The database is divided into multiple partitions that can be backed up in parallel by different threads. This segmentation allows the backup process to complete much faster than sequential scanning while ensuring complete data capture through coordinated partition-level backups with update blocking.
Solution Approach 2:
The system performs preliminary actions by blocking updates on each partition before the backup scan begins, and maintains this blocking until the backup is complete. This preliminary and continuous action ensures backup completeness without requiring lengthy sequential scanning, as the blocking duration is minimized through parallel processing.
4Productivity
If local clocks are used for backup timing without coordination, then system efficiency improves, but causal consistency cannot be guaranteed due to clock skew
Solution Approach 1:
The system adjusts the backup timing parameters by blocking updates for durations that account for the maximum expected clock skew between nodes. This parameter adjustment ensures that even with unsynchronized local clocks, the backup captures all relevant updates maintainin g causal consistency while preserving system efficiency through decentralized operation.
Data Source
AI summary
Disclosed are systems and methods for generating consistent backups. A central coordinator informs each node storing a partition of the time to perform a backup. At the designated time, each node blocks updates for a corresponding time interval measured by its local clock. Each node performs the backup operation according to its own local clock. Consistent backups may be generated in spite of clock skew between the local clocks as long as the time interval is at least as long as a maximum local clock skew (among the nodes performing the backup). In some systems the maximum local clock skew may be reduced, by a round-trip update latency from a client, for example.