Distributed Database Scaling via Physical Standby Nodes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database scaling methods require extended downtime for repartitioning and data relocation when adding new physical data stores, which is impractical for maintaining performance and capacity.
Innovation Solution
Implementing a physical standby approach where block-by-block copying of database portions is used, allowing for replication of operations and minimizing downtime, followed by even distribution of buckets across new nodes, enabling seamless integration of additional data stores without shutting down the cluster.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If traditional database scaling methods are used to add new physical data stores, then cluster capacity is increased, but extended downtime is required for repartitioning and data relocation
Solution Approach 1:
The patent creates physical standby copies of data buckets before the actual scaling operation. These standbys are pre-prepared and can be quickly swapped in when needed, avoiding the need to create them during the scaling process. This preliminary preparation significantly reduces the downtime required for cluster expansion.
Solution Approach 2:
The patent uses physical standby copies (bitwise copies) of data buckets that can be rapidly deployed during scaling operations. Instead of copying data during the scaling process (which causes downtime), pre-made copies are used, enabling capacity expansion without interrupting service.
2Quantity of substance
If data is copied row-by-row during scaling operations, then data is relocated to new nodes, but the process is slow and causes extended downtime
Solution Approach 1:
The patent uses block-by-block copying at the storage level instead of row-by-row copying at the application level. This lower-level copying approach is much faster because it operates directly on storage blocks rather than processing individual database rows through application logic, dramatically increasing scaling speed.
Solution Approach 2:
The patent replaces the application-level row-by-row copying mechanism with a storage-level block copying mechanism. This substitution moves the copying operation from the software/application layer to the storage layer, where it can be performed more efficiently and rapidly.
3Productivity
If physical standbys are used for block-by-block copying, then copying speed is improved and downtime is reduced, but system complexity increases
Solution Approach 1:
The patent makes physical standbys multi-functional: they serve both as backup copies for fault tolerance and as pre-prepared data sources for rapid scaling operations. This universal use of standbys justifies the additional complexity by providing dual benefits from a single architectural feature.
Solution Approach 2:
The physical standbys act as intermediaries between the existing data buckets and the new cluster nodes during scaling. They buffer the data transfer process, allowing the system to scale without direct real-time copying from production buckets, thereby simplifying the overall scaling workflow despite the added standby layer.
Data Source
AI summary
Disclosed are various embodiments for increasing distributed database capacity by adding new nodes while minimizing downtime. A database is partitioned among multiple nodes in a cluster. Each node stores a respective portion of the database. The portions of the database are replicated to corresponding physical standbys. For each portion, a first portion is mapped to an existing node, while a remaining portion is mapped to a physical standby. The cluster is updated to include the physical standbys as additional nodes among which the database is partitioned.