Distributed Database Data Redistribution via Incremental Splitting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data redistribution methods in distributed databases face challenges in implementing online redistribution without service interruption, especially when dealing with massive data, as they often require pre-designed data distribution rules and cannot efficiently handle incremental data redistribution.
Innovation Solution
A data processing method involving a cluster manager and resource managers that splits and imports original and incremental data dynamically, using Structured Query Language (SQL) statements to verify data integrity and adjust import processes based on check values and time thresholds, allowing for continuous operation during redistribution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data pre-distribution mode is used for online redistribution, then service interruption is avoided, but the system cannot handle databases with massive data due to unreasonable data splitting requirements
Solution Approach 1:
The patent segments the data redistribution process into multiple incremental phases. Instead of redistributing all data at once, it divides the data into chunks and redistributes them incrementally over multiple cycles, allowing the system to handle massive databases without requiring unreasonable initial splitting rules.
Solution Approach 2:
The patent implements dynamic data redistribution by allowing the redistribution strategy to adapt during the process. The system dynamically adjusts which data is redistributed in each incremental phase based on current system state and requirements, rather than following a fixed pre-designed splitting scheme.
2Stability of the object's composition
If incremental data is redistributed as a whole for a certain base, then data consistency is maintained, but redistribution cannot be implemented for databases with mass data due to excessive data volume
Solution Approach 1:
The patent segments incremental data into smaller manageable units for redistribution. Instead of treating all incremental data for a base as a single large unit, it divides and redistributes them in smaller batches across multiple operations, making the process feasible for databases with mass data.
Solution Approach 2:
The patent applies partial action by redistributing only a portion of incremental data in each phase rather than all incremental data at once. This allows the system to maintain data consistency for redistributed portions while progressively handling the entire dataset over multiple cycles.
3Ease of manufacture
If data is split in advance according to data characteristics with specified routing rules, then redistribution can be performed, but unreasonable design leads to huge databases that cannot be redistributed
Solution Approach 1:
The patent performs preliminary actions by establishing a flexible framework for incremental redistribution without requiring complete and perfect data splitting rules in advance. The system prepares the infrastructure for incremental redistribution and allows rules to be refined and adjusted during the actual redistribution process.
Solution Approach 2:
The patent implements dynamic adjustment of data distribution rules during redistribution. Rather than relying on static, pre-designed routing rules that must be perfect beforehand, the system allows rules to evolve and be optimized based on actual redistribution progress and system performance feedback.
4Reliability
If online redistribution is implemented without service interruption, then service availability is maintained, but the complexity of managing incremental data import and verification increases
Solution Approach 1:
The patent maintains continuity of useful action by enabling incremental data redistribution to proceed without interrupting service. The system continuously performs data export, splitting, import, and verification operations in the background while the database remains operational, ensuring service availability throughout the redistribution process.
Solution Approach 2:
The patent implements feedback mechanisms for verifying incremental data integrity during online redistribution. The system continuously monitors and verifies the correctness of redistributed incremental data, using feedback from verification results to guide subsequent redistribution operations and ensure data consistency without service interruption.
Data Source
Figure 1~2
Figure 3
Figure 4
AI summary
Provided are a data processing method, a cluster manager, a resource manager and a data processing system. The method includes: receiving a data redistribution request; instructing, according to the data redistribution request, a first resource manager to split original data stored in a first node and import each split sub-original-data into a corresponding second node, where the first resource manager is configured to manage data stored in the first node; and instructing the first resource manager to import incremental data which is stored in the first node during splitting and importing processes into a second node corresponding to the incremental data.