Distributed Database Data Rebalance via Partition Space Indexing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data rebalance methods in distributed databases require additional storage space and take a long time to execute, leading to service congestion.
Innovation Solution
A data processing method that involves obtaining data rebalance information, creating partition spaces, updating data ranges, and updating the correspondence between partition spaces and sets in the partition table, allowing for efficient data distribution without the need for additional storage or long execution times.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data rebalance is completed by copying a partition table and synchronizing data, then data distribution can be updated, but additional storage space is required and execution time increases
Solution Approach 1:
The patent extracts only the necessary metadata (data range information) from the full data copies, storing just the partition key value ranges and corresponding data locations in a separate index structure. This allows the system to track and manage data distribution without requiring storage of complete data copies, thus resolving the contradiction between updating data distribution and minimizing storage space usage.
Solution Approach 2:
The patent creates a lightweight copy of only the partition metadata (data range index) rather than copying the entire partition table data. This selective copying approach enables the system to maintain a backup reference for data range mapping without duplicating the actual data, thereby achieving distribution updates with minimal additional storage requirements.
2Adaptability or versatility
If data rebalance is completed by copying and synchronizing data, then data distribution can be updated, but execution time becomes very long
Solution Approach 1:
The patent extracts and stores only the essential data range metadata (partition key ranges and data locations) in an index structure, eliminating the need to process and synchronize entire data copies. This extraction approach reduces the amount of data that needs to be copied and processed during rebalance operations, significantly decreasing execution time while maintaining the ability to update data distribution.
Solution Approach 2:
The patent pre-establishes and maintains an index structure that maps data ranges to partition locations before rebalance operations are needed. This preliminary organization of metadata allows the system to quickly determine data migration paths during rebalance without performing full data synchronization, thus reducing execution time while enabling flexible data distribution updates.
3Adaptability or versatility
If data rebalance is completed by copying and synchronizing data, then data distribution can be updated, but service congestion occurs
Solution Approach 1:
The patent extracts only the necessary metadata (data range information) from full data copies, reducing the volume of data that needs to be processed during rebalance operations. This extraction approach minimizes the resource consumption and processing time, thereby preventing service congestion and maintaining high productivity during data distribution updates.
Solution Approach 2:
The patent creates a lightweight copy of only the partition metadata (data range index) rather than copying the entire partition table data. This selective copying enables faster rebalance operations with reduced processing overhead, preventing service congestion while maintaining the ability to perform data distribution updates without impacting overall system productivity.
Data Source
AI summary
A data processing method for an electronic device includes: obtaining data rebalance information representing a new partition plan for a partition table, the partition table being a table for data distribution based on a partition key; creating, for a set specified in the partition plan, a corresponding partition space, and recording, in the corresponding partition space, a current data range of a set corresponding to the partition space, the data range comprising one of a value range of the partition key and a value list of the partition key; updating the data range recorded in the corresponding partition space based on the partition plan, and updating correspondence between the corresponding partition space and sets of the partition table by using the updated data range in the corresponding partition space; and updating data distribution in the sets of the partition table based on the correspondence.