MPP Database Resource Reservation via Global Manager
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Massively parallel processing (MPP) database systems face challenges in aligning resources with workload across data nodes, leading to performance slowdowns due to single-point resource insufficiencies, as existing resource management systems do not effectively consider individual node resource availability when dispatching queries.
Innovation Solution
A two-layer resource management mechanism with a global resource manager and local resource managers using a three-phase communication protocol to ensure sufficient resources are reserved before query execution, facilitating efficient workload distribution across data nodes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If queries are dispatched without resource reservation, then query execution speed is improved, but resource insufficiency occurs on data nodes
Solution Approach 1:
The system performs resource reservation before query execution by sending reserve resource messages to data nodes and receiving acknowledgments. This preliminary action ensures resources are secured in advance, preventing resource insufficiency during query execution while maintaining efficient throughput through batch processing of reservations.
2Reliability
If resources are reserved for all queries, then resource insufficiency is prevented, but system throughput decreases
Solution Approach 1:
The system reserves resources for only those queries that can be immediately executed based on current resource availability, rather than reserving for all queries in the queue. The global resource manager processes resource reservations in batches and allows queries to wait when resources are insufficient, thereby maintaining high throughput while ensuring resource availability for executable queries.
3Manufacturing precision
If resource reservation is implemented, then resource allocation accuracy is improved, but system complexity increases
Solution Approach 1:
The resource management system is segmented into a global resource manager that handles high-level coordination and local resource managers on each data node that handle local resource tracking and reservation. This segmentation allows precise resource allocation through distributed decision-making while reducing overall system complexity by localizing management functions.
Solution Approach 2:
The global resource manager acts as an intermediary between the query queue and data nodes, coordinating resource reservations and managing the protocol flow. This intermediary layer simplifies the interaction complexity by centralizing coordination logic while allowing data nodes to maintain simple local resource management.
4Productivity
If concurrent queries are executed in parallel, then system throughput is improved, but resource skewness increases
Solution Approach 1:
The system implements feedback through acknowledgment messages from data nodes to the global resource manager, reporting resource availability status. This feedback mechanism allows the global resource manager to monitor resource skewness and adjust query dispatch decisions, selecting queries that can be executed on nodes with sufficient resources, thereby maintaining throughput while reducing resource distribution imbalance.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
In one embodiment, a method for managing database resources includes selecting a first query from a queue of queries and transmitting, by a global resource manager to a portion of a plurality of data nodes, a plurality of reserve resource messages. The method also includes receiving, by the global resource manager from the portion of the plurality of data nodes, a plurality of acknowledgement messages and transmitting, by the global resource manager to a coordinator node, an execute query message when the plurality of acknowledgement messages are positive acknowledgements.