Clustered Database Node Consistency via Versioned Snapshots
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Clustered databases face challenges in maintaining node and shard consistency during topology changes, such as additions or removals of nodes or shards, which can lead to inconsistencies in data synchronization and versioning.
Innovation Solution
A control computer monitors the clustered database, identifies topology changes, and configures node agents to stream data images and operation logs to a secondary storage platform, versioning the database with timestamps that ensure consistency across all shards, even during synchronization backups.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If topology changes are made to the clustered database (adding or removing nodes or shards), then the database can adapt to changing requirements and scale, but inconsistencies in data synchronization and versioning occur across nodes
Solution Approach 1:
The system performs preliminary actions by creating consistent snapshots of the database state before topology changes occur. The snapshot mechanism captures the database state at a specific point in time, ensuring that subsequent additions or removals of nodes or shards do not compromise data consistency. This preliminary snapshot creation allows the system to adapt to topology changes while maintaining reliability through versioned, time-stamped database states.
Solution Approach 2:
The system creates copies of the database state through snapshots that are stored with version identifiers and timestamps. These snapshots serve as immutable copies that can be referenced to restore consistency if inconsistencies arise during topology changes. The copying mechanism ensures that the original database state is preserved while allowing flexible topology modifications.
2Reliability
If data synchronization is continuously maintained across all nodes, then node consistency is preserved, but system complexity and resource consumption increase during topology changes
Solution Approach 1:
Instead of continuous synchronization, the system employs periodic snapshot creation at defined intervals or trigger points. This periodic approach maintains data synchronization reliability by capturing consistent database states without requiring constant communication and coordination between nodes during topology changes, thereby reducing system complexity and resource consumption while preserving data integrity.
3Reliability
If versioning is implemented with timestamps for each database state, then data integrity and recovery capability are enhanced, but storage requirements and processing overhead increase
Solution Approach 1:
The system implements a nested versioning structure where snapshots are organized hierarchically with parent-child relationships. Each snapshot contains a version identifier and timestamp, and can reference previous snapshots, creating a nested structure that efficiently manages storage. This nesting allows the system to maintain multiple versions for data integrity and recovery while optimizing storage space by avoiding redundant data storage across all versions.
Data Source
AI summary
Systems and methods for node consistency in a clustered database are described. The system automatically identifies an addition of a first node to a clustered database on a source storage platform. The system initializes the first node on the source storage platform communicating a first node agent to the first node. The first node agent is configured to execute on the first node to extract the operation log at the source storage platform and stream the operation log to the secondary storage platform. The system versions a first version of the clustered database. The versioning of the first version including not storing a version of the operation log associated with the first node at the secondary storage platform responsive to identifying, during the first interval of time, the synchronizing of the operation log at the first node having not been synchronized, in a prior version of the clustered database, with the operation log at the storage platform.