MPP Database Cluster Expansion via Split-Read Data Redistribution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Massively parallel processing (MPP) database systems face challenges in redistributing data evenly across servers during cluster expansion, leading to performance issues due to uneven data distribution and workload unbalancing, especially when new servers are added.
Innovation Solution
The method involves determining an updated bucket-server mapping for an MPP database cluster, redistributing data based on this mapping, and incrementally moving data from initial servers to new servers, while maintaining atomicity and isolation to ensure seamless expansion without impacting query performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Stability of the object's composition
If data is redistributed across all servers during cluster expansion, then even data distribution is achieved, but query performance degrades due to increased workload on existing servers
Solution Approach 1:
The patent segments the redistribution process by introducing a split-read mechanism that divides read operations into two parts: one part accesses data from the old location while the other part accesses data from the new location. This segmentation allows gradual migration of data without requiring all data to be redistributed simultaneously, thus maintaining query performance while achieving even data distribution over time.
Solution Approach 2:
The patent implements preliminary action by pre-calculating the new bucket configuration and determining which data blocks need to be migrated before actual redistribution begins. The system prepares migration plans in advance, identifies target servers for each data block, and sets up the necessary infrastructure before the redistribution process starts, minimizing disruption to ongoing query operations.
2Stability of the object's composition
If cluster expansion is performed offline, then data redistribution can be completed thoroughly, but system availability is reduced due to downtime
Solution Approach 1:
The patent applies dynamics by implementing a dynamic redistribution process that can adapt to ongoing query operations. The system continuously monitors data migration progress and adjusts the redistribution process in real-time, allowing servers to remain operational throughout the expansion process. The dynamic nature of the solution enables the system to maintain availability while progressively achieving even data distribution.
Solution Approach 2:
The patent introduces an intermediary mechanism in the form of a coordination layer that manages the transition between old and new data locations. This intermediary handles the complexity of data migration, routing queries appropriately during the transition period, and ensuring data consistency without requiring the entire system to be taken offline. The intermediary enables seamless cluster expansion while maintaining system availability.
3Quantity of substance
If all data is redistributed to new servers, then storage capacity is fully utilized, but workload becomes unbalanced across servers
Solution Approach 1:
The patent implements local quality by allowing different servers to have different roles and characteristics in the distributed system. Instead of forcing uniform data distribution across all servers, the system enables certain servers to specialize in specific types of data or operations based on their capabilities and current workload. This local optimization allows storage capacity to be fully utilized while maintaining workload balance through differentiated server roles.
Solution Approach 2:
The patent applies parameter changes by dynamically adjusting data distribution parameters based on server performance metrics and workload characteristics. The system monitors various parameters such as query response time, data access patterns, and server capacity utilization, and automatically adjusts the redistribution strategy to optimize both storage utilization and workload balance. This adaptive parameter adjustment ensures that no single server becomes a bottleneck while maximizing storage capacity usage.
Data Source
AI summary
In one embodiment, a method includes determining a number of initial servers in a massively parallel processing (MPP) database cluster and determining an initial bucket configuration of the MPP database cluster, where the initial bucket configuration has a number of initial buckets. The method also includes adding a number of additional servers to the MPP database cluster to produce a number of updated servers, where the updated servers include the initial servers and the additional servers and creating an updated bucket configuration in accordance with the number of initial servers, the initial bucket configuration, and the number of additional servers, where the updated bucket configuration has a number of updated buckets. Additionally, the method includes redistributing data of the MPP cluster in accordance with the updated bucket configuration.