Decoupled Database Scaling with Copy-on-Write Shard Creation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database scaling methods, particularly through sharding, face challenges in implementing dynamic scaling efficiently due to slow rebalancing processes caused by mismatched storage capacity and bandwidth, leading to network transmission times, excessive load on source nodes, and potential downtime.
Innovation Solution
Implementing a decoupled storage and compute architecture with a two-tier system, where the storage tier manages data durability and consistency, utilizing copy-on-write snapshots and lightweight head nodes to facilitate rapid shard creation without immediate data copying, and leveraging a proxy layer for traffic redirection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional sharding with rebalancing is used for database scaling, then data can be distributed across multiple shards, but the scaling process becomes slow due to network transmission times and excessive load on source nodes
Solution Approach 1:
The system separates storage and compute functions into distinct tiers. Storage nodes handle data persistence while compute nodes handle query processing and shard management. This segmentation allows new shards to be created by simply allocating storage resources without requiring data rebalancing operations, dramatically accelerating the scaling process.
Solution Approach 2:
A proxy layer is introduced as an intermediary between clients and database nodes. The proxy handles client connections, query routing, and result aggregation, allowing new shards to be integrated into the system without disrupting existing client connections or requiring coordinated rebalancing across the entire system.
2Productivity
If data rebalancing is performed during scaling, then new shards can be populated with data, but excessive load is placed on source nodes causing potential downtime
Solution Approach 1:
New shards are pre-configured with empty storage structures before needing to serve data. When scaling occurs, the storage infrastructure is prepared in advance with allocated space and proper indexing structures, so that data can be streamed to new shards without blocking client operations or causing system downtime.
Solution Approach 2:
The system maintains continuous data flow from source shards to destination shards during scaling operations. Data is streamed continuously using efficient copy mechanisms, and the proxy layer ensures client queries continue to be served without interruption, maintaining system availability throughout the scaling process.
3Reliability
If complete data copying is performed during shard creation, then data consistency is maintained, but network bandwidth is consumed and storage capacity requirements increase
Solution Approach 1:
Instead of traditional data copying that duplicates entire datasets, the system uses storage-level copying mechanisms at the block or file level. The storage tier creates references or pointers to data blocks, allowing new shards to access data without physically duplicating it across the network, significantly reducing bandwidth consumption while maintaining data consistency through shared storage semantics.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Techniques for fast online scaling of a database via a split architecture including decoupled storage and compute tiers in a database service are described. A cluster of database (DB) nodes is scaled to add a new DB node. The scaling includes determining a split for data of a first volume managed by an existing DB node. A second DB node is obtained, and the first volume is cloned according to a lightweight copy technique to yield a second volume for use by the second DB node. After the cloning, a set of database modifications are applied to the second volume based on modifications caused by database traffic received by the first DB node, involving the volume, during the cloning of the first volume. Each DB node may drop the portion of the volume that it does not need according to the split.