Optimistic Deduplication for Consistent Distributed Database Snapshots
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Generating consistent deduplicated snapshots of distributed databases is challenging due to inconsistencies across nodes and the need for multiple scans, which is time and processor intensive, especially since data changes are not synchronized and freezing the database is impractical.
Innovation Solution
The method involves optimistic deduplication, where each node's data is scanned once to generate summaries, identify and correct mistakes based on a quorum, and create a consistent deduplicated snapshot, reducing the need for multiple scans and improving processing efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If multiple scans are performed on each node to ensure consistency during snapshot generation, then data consistency is improved, but time consumption and processing power requirements increase significantly
Solution Approach 1:
The system performs preliminary actions by recording data version information and generating summaries during normal database operations before snapshot generation is needed. This preliminary indexing and version tracking enables the snapshot process to quickly identify consistent data without requiring multiple verification scans, thus resolving the contradiction between ensuring data consistency and reducing time consumption
Solution Approach 2:
The system implements feedback mechanisms where each node provides information about its data state and version to a coordination service. This feedback allows the system to determine snapshot consistency without repeatedly scanning nodes, as the coordination service can aggregate version information from all nodes to verify consistency in a single pass
2Reliability
If multiple scans are performed on each node to ensure consistency during snapshot generation, then data consistency is improved, but processing power requirements increase significantly
Solution Approach 1:
Data version information and summaries are prepared in advance during normal database operations, creating an index structure that records which data items are consistent across nodes. This preliminary action eliminates the need for computationally intensive multiple scans during snapshot generation, as the system can directly query the pre-computed version information
Solution Approach 2:
Instead of performing multiple actual data scans across distributed nodes, the system creates and uses copies of version metadata and summary information stored locally at each node and centrally at the coordination service. These copies enable fast consistency verification without the computational overhead of repeatedly accessing actual data files
3Reliability
If the database is frozen to capture all node states at the same time, then snapshot consistency is improved, but database availability and operational continuity deteriorate
Solution Approach 1:
The system performs preliminary version recording and data item tracking during normal database operations before snapshot generation. Each node continuously updates and maintains version information about its data items, so when snapshot generation is triggered, the system can immediately determine consistency using these pre-recorded versions without freezing the database
Solution Approach 2:
A coordination service acts as an intermediary between distributed nodes, aggregating version information and determining snapshot consistency without requiring the actual data to be frozen. The coordination service receives version reports from nodes and computes consistency, allowing the database to remain operational while snapshot generation proceeds
Data Source
AI summary
Embodiments disclosed herein provide systems, methods, and computer readable media for generating a consistent deduplicated snapshot of a distributed database using optimistic deduplication. In a particular embodiment, a method provides, for each node of a plurality of nodes in the distributed database, deduplicating data items stored on the node that are owned by the node and generating a summary that describes a file in which the data items are located. The method further provides identifying from the summaries for each of the nodes whether mistakes occurred during deduplication and, upon identifying one or more mistakes, determining one or more corrections for the one or more mistakes. Also, the method provides generating a consistent deduplicated snapshot for the distributed database comprising the deduplicated data items from each node and the one or more corrections.