Flexible Distributed MPP Partition Management
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional MPP database systems are inflexible, requiring the number of database partitions to match the number of fixed processors, making it cumbersome and time-consuming to add or remove partitions, as the entire setup process must be repeated.
Innovation Solution
A resource group manager initiates a management instance on an initial machine, determines the desired number of database partitions, and dynamically allocates them across available machines, allowing for flexible expansion and contraction of the resource group by adding or removing machines, and relocating management instances as needed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If the number of database partitions is fixed to match the number of processors, then the system setup is simple, but the system lacks flexibility and scalability when adding or removing partitions
Solution Approach 1:
The patent implements dynamic allocation by allowing the number of database partitions to be decoupled from the fixed number of processors. The system can dynamically add or remove partitions independent of processor count, enabling flexible adaptation to changing data storage needs without requiring system reconfiguration or additional hardware setup.
2Quantity of substance
If additional partitions are desired in a traditional MPP system, then data distribution capacity increases, but the entire setup process must be repeated which is time-consuming
Solution Approach 1:
The system performs preliminary actions by pre-configuring the dynamic allocation framework and partition management infrastructure in advance. This allows partitions to be added or removed on-demand without repeating the entire setup process, as the underlying system architecture is already prepared to handle dynamic partition changes.
3Quantity of substance
If the number of partitions exceeds the number of available machines, then storage capacity increases, but resource utilization becomes unbalanced
Solution Approach 1:
The patent applies merging by allowing multiple database partitions to be combined and hosted on a single machine when the number of partitions exceeds available machines. This dynamic consolidation enables the system to maintain balanced resource utilization while providing expanded storage capacity, as partitions can be allocated across machines based on current resource availability rather than requiring a one-to-one mapping.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
An embodiment method for massively parallel processing includes initiating a management instance on an initial machine, the management instance generating an initial partition corresponding to the initial machine, determining a total number of partitions desired for processing a database, the total number of partitions including the initial partition, determining a number of additional machines available to process the database, grouping the initial machine and the additional machines together in a pod, and launching the management instance on the additional machines in the pod to generate the total number of partitions desired for the database. Additional embodiment methods and an embodiment system operable to perform such methods are also disclosed.