Hierarchical Scheduler Nodes for Distributed Database Query Execution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed database systems, establishing a large number of connections at runtime for query execution can slow down the process, and efforts to pool connections across different queries are often inefficient due to resource usage for unreused connections.
Innovation Solution
Implementing a hierarchical structure with global scheduler nodes, local scheduler nodes, and worker nodes, where the global scheduler generates an execution plan and distributes fragments to local scheduler nodes, which then coordinate the execution, using established connections to minimize the number of connections needed and avoid direct worker node connections.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a large number of connections are established at runtime for query execution, then query execution can be performed, but the execution speed slows down
Solution Approach 1:
The system establishes connections between scheduler nodes and worker nodes in advance, before query execution begins. These pre-established connections are maintained in a connection pool and reused across multiple queries, eliminating the need to establish new connections at runtime and thus reducing connection establishment time while maintaining execution speed
2Loss of energy
If connections are pooled across different queries, then resource usage is reduced, but efficiency decreases due to overhead from unreused connections
Solution Approach 1:
The system divides the connection pool into separate segments or pools for different scheduler nodes and worker nodes. Each scheduler node has dedicated connections to its assigned worker nodes, and these connections are organized in a hierarchical structure. This segmentation allows connections to be efficiently managed and reused within specific contexts while minimizing overhead from maintaining unreused connections across the entire system
3Device complexity
If direct connections are established between worker nodes, then message routing is simplified, but the number of connections increases significantly
Solution Approach 1:
The system introduces scheduler nodes as intermediary components between worker nodes. Instead of establishing direct connections between all worker nodes, messages are routed through the scheduler node that acts as a mediator. This hierarchical arrangement reduces the total number of connections required while maintaining the ability to coordinate execution across multiple worker nodes through the scheduler
Data Source
AI summary
In one respect, there is provided a distributed database system that includes a global scheduler node and a first local scheduler node. The global scheduler node can be configured to generate an execution plan for a query requiring data stored at and/or managed by a first and a second worker node assigned to the first local scheduler node. The execution plan can include a first fragment having operations to be performed by the first and second worker node. The first local scheduler node can be configured to coordinate the execution of a portion of the first fragment of the execution plan by distributing, to the first and second worker node, operations included in the first fragment of the execution plan, and routing, between the first and second worker node, messages associated with executing the first fragment of the execution plan. Related methods and computer program products are also provided.