Database Sharding Bucket Reassignment Without Data Migration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database sharding systems face challenges in efficiently redistributing data when the number of data stores is increased, as the even distribution of primary keys across buckets is disrupted, leading to inefficiencies in scalability and performance.
Innovation Solution
The solution involves generating a new number of buckets that is a multiple of the new number of data stores, allowing data items to be reassigned without migration, and redistributing data to ensure even distribution across physical data stores, using a modulo-based mapping and bucket assignment configuration to maintain data integrity and scalability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If the number of data stores is increased to improve scalability, then the system capacity increases, but the even distribution of primary keys across buckets is disrupted
Solution Approach 1:
The system segments the bucket assignment space into multiple independent hash functions. When data stores are added or removed, only the affected segment needs to be reassigned while other segments maintain their original assignments, preserving overall distribution stability while adapting to changes in system capacity.
Solution Approach 2:
The system changes the parameter of hash function selection dynamically. Different hash functions are applied to different ranges of primary keys, and this mapping can be adjusted when the number of data stores changes, allowing the system to maintain even distribution across varying numbers of data stores without complete remapping.
2Stability of the object's composition
If data migration is performed to redistribute data across increased data stores, then even distribution is restored, but system performance and operational complexity deteriorate
Solution Approach 1:
The redistribution operation is segmented to affect only the minimum necessary portion of the data. By dividing the bucket assignment into independent segments, the system can perform localized rebalancing without requiring global data migration, thus maintaining performance while restoring even distribution.
Solution Approach 2:
The system performs preliminary calculations to identify exactly which buckets need redistribution before executing the operation. This allows for targeted data movement rather than comprehensive migration, reducing the performance impact and operational complexity of the rebalancing process.
3Stability of the object's composition
If data migration is performed to redistribute data, then even distribution is achieved, but loss of time and operational overhead increase
Solution Approach 1:
The data redistribution task is divided into independent segments that can be processed concurrently. Only the segments affected by changes in data store configuration require migration, significantly reducing the total time required compared to migrating all data.
Solution Approach 2:
The system performs only the partial action necessary to achieve even distribution - migrating only the specific buckets that need rebalancing - rather than performing excessive action by migrating all data. This minimizes the time loss while still achieving the desired distribution state.
Data Source
AI summary
Disclosed are various embodiments for facilitating data redistribution in database sharding. A database is maintained that is horizontally partitioned into a set of first buckets using modulo-based assignments. A fixed number of the first buckets are stored in each of multiple physical data stores. The database is repartitioned into a set of second buckets using modulo-based assignments. The number of second buckets in the set is a multiple of the sum of a positive integer and the quantity of the physical data stores. The data in the database is unmoved between the physical data stores by repartitioning.