MPP Database Online Repartitioning via Staged Migration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional MPP databases face inefficiencies due to static partitioning algorithms that fail to adapt to changing data distributions, leading to unbalanced systems and requiring manual offline repartitioning, which disrupts runtime operations.
Innovation Solution
A method for dynamically re-allocating data entries between partitions while maintaining the MPP database online, using staged updates to the query catalog and preserving primary-key/foreign-key relationships, allowing for seamless migration of data without interrupting operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Stability of the object's composition
If manual repartitioning is performed to rebalance database partitions, then database balance is improved, but runtime operation is interrupted
Solution Approach 1:
The patent implements dynamic repartitioning where the database system automatically detects partition imbalances and performs repartitioning operations without manual intervention. The system continuously monitors data distribution and adapts the partitioning scheme in real-time, transforming the static manual process into a dynamic automated one that maintains both balance and operational continuity.
Solution Approach 2:
The database system performs self-diagnosis and self-repair by automatically detecting when partitions become unbalanced and initiating repartitioning operations without requiring database administrator intervention. The system manages its own optimization, eliminating the need for manual offline maintenance while preserving runtime operations.
2Ease of manufacture
If static partitioning algorithm is used during database creation, then initial database setup is simplified, but adaptability to changing data distributions is reduced
Solution Approach 1:
The patent transforms the static partitioning algorithm into a dynamic one that automatically adapts to changing data distributions. The system monitors data patterns over time and modifies partitioning strategies accordingly, allowing the database to evolve with changing requirements while maintaining the simplicity of initial setup.
Solution Approach 2:
The system implements feedback mechanisms that continuously monitor data distribution patterns and use this information to automatically adjust partitioning algorithms. The feedback loop enables the system to learn from changing data patterns and optimize partitioning without manual intervention, combining simplicity with adaptability.
3Manufacturing precision
If database is taken offline for repartitioning, then data consistency is ensured, but service availability is reduced
Solution Approach 1:
The patent performs preliminary actions by creating backup partitions and validation mechanisms before executing repartitioning operations. The system prepares recovery plans and ensures data integrity checks are in place beforehand, allowing safe online repartitioning that maintains both consistency and availability.
Solution Approach 2:
The system introduces intermediary mechanisms such as temporary storage areas and validation layers that mediate between the old and new partition structures. These intermediaries ensure data consistency during transitions while allowing the database to remain online and accessible throughout the repartitioning process.
Data Source
Figure 1~9
Figure 2
Figure 3
AI summary
A massively parallel processing (MPP) database can be re-partitioned/re-balanced while remaining on-line through a staged migration procedure. Staged migration may include a first stage and a second stage. During the first stage, entries in an existing partition are reallocated to the new partition, and the catalog is updated to associate the re-allocated entries with both the existing partition and the new partition such that queries for the re-allocated entries are directed toward the existing partition and the new partition. During the second stage, the re-allocated entries are migrated from the existing partition to the new partition, and after the migration is complete, the catalog is re-updated to associate the migrated entries with the new partition such that new queries are directed toward the new partition.