Distributed Query Plan Generation for Parallel Database Execution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Database management systems face challenges in optimizing performance for complex queries and handling large volumes of transactions, leading to inefficient processing and response times due to the need for specialized execution engines for different database types and the burden on the database layer.
Innovation Solution
An execution engine is decoupled from the database layer, capable of generating and optimizing query plans across various database types, performing complex operations while reducing the processing load on the database layer by executing operations like joins and projections, and generating distributed query plans without fragmenting the physical plan into multiple parts.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a single database layer handles all query processing, then database control and consistency are maintained, but processing burden and response time worsen
Solution Approach 1:
The system segments the query processing workload by dividing the query plan into multiple fragments that can be executed in parallel across different threads and locations. The query optimizer generates a distributed execution plan where different operators are assigned to different execution contexts, enabling concurrent processing while maintaining centralized query optimization and control through the database layer.
Solution Approach 2:
The patent introduces an intermediary execution engine layer between the database layer and the query interface. This execution engine receives queries, optimizes them, generates distributed execution plans, and coordinates execution across multiple locations. It acts as a mediator that offloads processing burden from the database layer while maintaining control through centralized plan generation and coordination.
2Speed
If query execution is centralized in one location, then coordination is simplified, but processing speed and scalability worsen
Solution Approach 1:
The execution plan is segmented into multiple independent or loosely-coupled fragments that can be executed in parallel across different locations and threads. Each fragment represents a portion of the overall query processing that can be independently optimized and executed, enabling speedup through parallelism while the database system coordinates the fragments through the centralized query optimizer.
Solution Approach 2:
The patent transitions from single-location sequential execution to multi-location parallel execution by adding spatial distribution as a new dimension. Query operations are mapped across multiple physical locations and processing threads, transforming the execution model from one-dimensional sequential processing to multi-dimensional parallel processing, thereby increasing throughput and reducing response time.
3Adaptability or versatility
If specialized execution engines are created for different database types, then query optimization is improved, but system complexity and maintenance burden worsen
Solution Approach 1:
The patent implements a universal query execution engine that can handle multiple database types and query workloads through a unified architecture. Rather than creating specialized engines for different database types, the system uses a single execution engine that generates distributed execution plans adaptable to various database configurations, data types, and query patterns, thereby supporting multiple database types without increasing execution engine complexity.
Data Source
AI summary
A method, a system, and a computer program product for generating an executable query plan. A query requiring access to data stored in a database system is received. Based on the received query, a query execution plan having a plurality of query execution pipelines is generated. Each query execution pipeline in the plurality of query execution pipelines is configured to execute a plurality of operations in a predetermined order associated with each query execution pipeline. The generated query execution plan is fragmented into a plurality of fragments. Each fragment has one or more query execution pipelines in the plurality of query execution pipelines. Each fragment of the generated query execution plan is transmitted to one or more execution locations in a plurality of execution locations for execution. Each fragment of the generated query execution plan is executed at one or more execution locations. One or more results of executions are combined to generate a response to the received query.