Parallel Query Sub-plan Scheduling for Data Platform Node Utilization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional data platforms experience poor utilization of execution nodes due to lockstep scheduling, leading to resource stalls when processing small data sets in larger query plans, where only a subset of execution nodes are actively used while others remain idle.
Innovation Solution
Implementing parallel execution of sub-plans using multiple execution nodes by identifying candidate sub-plans that can be efficiently executed on a single node, determining a cost estimate for parallel execution, and distributing these sub-plans across available execution nodes to maximize resource utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If lockstep scheduling is used to process query plans, then execution simplicity is maintained, but execution node utilization deteriorates with idle nodes during small data set processing
Solution Approach 1:
The query plan is divided into multiple independent sub-plans that can be executed in parallel. Each sub-plan represents a portion of the overall query that can be processed independently by different execution nodes, enabling parallel execution while maintaining simplicity through structured decomposition of the query into manageable segments.
Solution Approach 2:
The system transitions from sequential execution (single dimension) to parallel execution across multiple execution nodes (adding spatial dimension). By distributing sub-plans across multiple nodes simultaneously, the system exploits the additional dimension of parallel processing to improve utilization without sacrificing execution clarity.
2Device complexity
If all execution nodes are reserved for each individual sub-plan, then execution control is simplified, but resource efficiency deteriorates due to resource stalls
Solution Approach 1:
Instead of allocating all execution nodes to a single sub-plan at a time, the system segments both the query into sub-plans and the execution nodes into multiple workers. This allows multiple sub-plans to be distributed across multiple nodes simultaneously, reducing resource stalls while maintaining manageable control through the segmented structure.
Solution Approach 2:
The system dynamically assigns sub-plans to execution nodes based on current workload and availability rather than statically reserving nodes for each sub-plan. This dynamic allocation allows execution nodes to be flexibly reused across different sub-plans, improving resource efficiency while maintaining execution control through adaptive scheduling.
3Device complexity
If sequential execution of sub-plans is used, then scheduling overhead is reduced, but processing time increases due to lack of parallelism
Solution Approach 1:
The query is segmented into independent sub-plans that can be executed in parallel by different execution nodes. This segmentation enables simultaneous processing of multiple sub-plans, significantly reducing total processing time while the scheduling overhead remains manageable due to the structured and modular nature of the segmented sub-plans.
Solution Approach 2:
By executing multiple sub-plans in parallel across multiple execution nodes, the system ensures continuous useful action throughout the processing period. Instead of nodes sitting idle during sequential execution, all nodes remain actively engaged in processing different sub-plans simultaneously, maximizing resource utilization and reducing overall processing time.
Data Source
AI summary
Sub-plans are executed in parallel using a plurality of execution nodes, which can be part of a data platform. In particular, various embodiments identify sub-plans (e.g., fragments or portions of one or more child operators) of a root operator in a query plan such that the identified sub-plans that are candidates for execution on a single execution node, determine a cost estimate for causing the candidate sub-plans to be executed in parallel using multiple execution nodes, and cause the candidate sub-plans to be executed in parallel based on the cost estimate.