Point-in-time Snapshot Generation for Eventually-Consistent Data Stores
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems for creating a point-in-time snapshot of an eventually-consistent data store often disrupt normal operations and require significant resources, as they either force consistency across all nodes or necessitate additional reconciliation services, making it difficult to generate a snapshot without impacting data availability.
Innovation Solution
A method that collects inconsistent snapshots from multiple nodes, resolves inconsistencies to create a consistent point-in-time snapshot, allowing for uninterrupted data access by processing these snapshots on a related system without overloading the data store.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a consistent read is forced across all nodes to retrieve a consistent snapshot, then data consistency is improved, but system availability and processing capacity deteriorate due to the large load and prevention of normal requests
Solution Approach 1:
The system divides the data store into multiple independent nodes that can operate autonomously. Each node maintains its own snapshot independently, allowing the system to retrieve consistent snapshots without forcing a coordinated read across all nodes, thus maintaining high availability while ensuring consistency at the node level
Solution Approach 2:
The system pre-generates and stores snapshots at each node before they are needed. When a consistent snapshot is required, the system can retrieve pre-computed snapshots without disrupting normal operations, as the snapshot generation has already been performed in advance during periods of lower load
2Reliability
If additional services are implemented to generate and reconcile snapshots from individual nodes, then snapshot consistency is improved, but device complexity increases
Solution Approach 1:
Each node autonomously generates and maintains its own snapshots without requiring external coordination or reconciliation services. The nodes self-manage their snapshot creation and storage, eliminating the need for complex centralized reconciliation mechanisms while ensuring consistency through independent operation
Data Source
AI summary
A method and system for building a point-in-time snapshot of an eventually-consistent data store. The data store includes key-value pairs stored on a plurality of storage nodes. In one embodiment, the data store is implemented as an Apache® Cassandra database running in the “cloud.” The data store includes a journaling mechanism that stores journals (i.e., inconsistent snapshots) of the data store on each node at various intervals. In Cassandra, these snapshots are sorted string tables that may be copied to a back-up storage location. A cluster of processing nodes may retrieve and resolve the inconsistent snapshots to generate a point-in-time snapshot of the data store corresponding to a lagging consistency point. In addition, the point-in-time snapshot may be updated as any new inconsistent snapshots are generated by the data store such that the lagging consistency point associated with the updated point-in-time snapshot is more recent.