Query Task Execution with Balanced Parallelism for Lower Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Adjusting the degree of parallelism of operators in query tasks involving large data tables can lead to increased resource consumption and prolonged query latency without necessarily improving overall efficiency.
Innovation Solution
Determine candidate degrees of parallelism based on the number of child tables obtained by pre-partitioning the target data table, select a target degree of parallelism within a preset range based on CPU cores, and evenly allocate child tables to computing units to avoid additional data exchange operators.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the degree of parallelism of operators is increased to achieve concurrent execution of query tasks, then query throughput is improved, but memory resource consumption increases and query latency is prolonged
Solution Approach 1:
The patent dynamically adjusts the degree of parallelism parameter based on the number of child tables obtained by pre-partitioning the target data table. By establishing a mapping relationship between the number of child tables and the optimal degree of parallelism, the system selects from candidate degrees of parallelism to achieve even allocation of child tables to computing units, thereby improving query throughput while avoiding excessive memory resource consumption.
2Speed
If the degree of parallelism is adjusted to increase concurrent execution capability, then processing speed is improved, but query latency increases due to additional data exchange operators
Solution Approach 1:
The patent performs pre-partitioning of the target data table into child tables before query execution. This preliminary action establishes a foundation for even allocation of data to computing units, reducing the need for additional data exchange operators during query execution and thereby decreasing query latency while maintaining processing speed.
Solution Approach 2:
The system selects the degree of parallelism parameter based on the number of child tables from pre-partitioning, ensuring that the allocation of child tables to computing units is even. This parameter selection avoids the introduction of additional data exchange operators that would otherwise increase query latency.
3Productivity
If operators are increased to achieve concurrent execution, then query task throughput is improved, but device complexity increases
Solution Approach 1:
The patent determines the degree of parallelism as a parameter based on the number of child tables obtained by pre-partitioning the target data table. By selecting from candidate degrees of parallelism and ensuring even allocation of child tables to computing units, the system achieves improved query task throughput while avoiding unnecessary operator complexity.
Data Source
AI summary
This disclosure provides a query task execution method, apparatus, computer device, storage medium. The method includes: receiving a query task for a target data table; determining at least one candidate degree of parallelism for executing the query task based on a first number of child tables obtained by pre-partitioning the target data table; selecting a target degree of parallelism within a preset range of degrees of parallelism from the at least one candidate degree of parallelism; and determining required computing units according to the target degree of parallelism, and evenly allocating the child tables to the computing units which concurrently execute the query task based on the allocated child tables.