Live Resharding Cloud Database Shards Without Downtime
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Centrally managed cloud resource metadata databases face challenges in scaling due to physical storage capacity limitations, leading to significant downtime and latency during data migration to new partitions.
Innovation Solution
Implementing a method for live resharding, where a computer system receives a resharding request, prepares resource metadata for migration, migrates a subset of metadata while serving requests, and installs a new database configuration, allowing for ongoing request servicing without downtime.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is migrated to newly added database partitions to scale capacity, then storage capacity is improved, but downtime and latency increase
Solution Approach 1:
The system creates a new database partition and pre-migrates a subset of data to it before the partition becomes active. This preliminary action allows the system to prepare the infrastructure in advance, so when the migration completes, the new partition is ready to receive data without causing downtime. The resharding operation is prepared and executed in a way that maintains continuous availability.
Solution Approach 2:
The system dynamically switches between database partitions during the migration process. While migrating a subset of data to a new partition, the system continues to serve requests on the original partition. After migration, it dynamically redirects traffic to the new partition. This dynamic switching enables capacity scaling without requiring the entire database to be taken offline.
2Quantity of substance
If data is migrated to newly added database partitions to scale capacity, then storage capacity is improved, but latency increases
Solution Approach 1:
By pre-creating the new partition and preparing the migration infrastructure in advance, the system eliminates the need for lengthy setup operations during the migration process. This preliminary preparation ensures that when data migration occurs, the infrastructure is already optimized, minimizing latency spikes that would otherwise occur during partition creation and data transfer.
3Quantity of substance
If the entire database is migrated at once to new partitions, then storage capacity is improved, but system performance deteriorates
Solution Approach 1:
The system segments the database into multiple partitions and migrates only a subset of data at a time rather than the entire database at once. This segmentation allows the migration process to be divided into manageable chunks, enabling continuous service operation while progressively transferring data to the new partition. The resharding operation processes data in batches, maintaining system performance throughout the migration process.
Solution Approach 2:
The system migrates only a subset of data to the new partition in each migration wave rather than attempting to migrate everything at once. This partial action approach allows the system to maintain performance by processing data in controlled increments, avoiding the performance deterioration that would result from attempting a complete database migration in a single operation.
Data Source
AI summary
Techniques for live migration of cloud resource infrastructure metadata are provided. A method may include receiving a resharding request associated with a database configured to store resource metadata according to a first database configuration. The resource metadata may be identified by a first mapping. The method may include preparing the resource metadata for migration to a second database configuration corresponding with a second mapping. The method may include migrating a subset of the resource metadata within the database in accordance with the second mapping. The method may include while migrating the subset of the resource metadata, serving requests on the resource metadata according to the first database configuration. The method may include after migrating the subset of the resource metadata, installing the second database configuration. The method may also include removing the subset of the resource metadata remaining in the database according to the first database configuration.