Streaming Query Pipeline Scheduling for Higher Throughput
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database query execution for streaming data in computing systems suffers from inefficiencies such as low cluster utilization, waiting for I/O resources, and straggler tasks, leading to poor performance and low throughput.
Innovation Solution
Implementing pipelined execution of database queries by scheduling stages of the query in parallel for different batches of streaming data, with stateless stages overlapping and stateful stages executed sequentially to maintain logical correctness.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If sequential batch processing is used, then data processing correctness is maintained, but resource utilization is low and execution time is long
Solution Approach 1:
The patent segments the database query execution into multiple independent stages (e.g., scan, filter, aggregate, join) that can be executed in parallel across different batches. Each stage processes a specific portion of the query logic, allowing concurrent execution of multiple batches at different stages simultaneously, thereby increasing throughput while maintaining correctness through stage dependency management.
Solution Approach 2:
The patent performs preliminary actions by pre-compiling query plans and pre-allocation of processing resources before actual batch execution. Query plans are compiled in advance and cached, and processing stages are pre-configured with their required resources, eliminating compilation overhead during batch execution and enabling faster parallel processing.
2Productivity
If parallel batch execution is implemented, then resource utilization improves, but coordination overhead and complexity increase
Solution Approach 1:
The patent divides the query execution into discrete, manageable stages with well-defined inputs and outputs. Each stage can be independently scheduled and managed, reducing the complexity of coordinating parallel execution. The stage-based segmentation allows for modular scheduling where each stage's dependencies are clearly defined, making it easier to manage parallel batch execution.
Solution Approach 2:
The patent introduces an intermediary layer (the stage execution manager) that coordinates between the query plan and the parallel batch execution. This intermediary manages the scheduling, resource allocation, and result aggregation for multiple concurrent batches, abstracting away the complexity from the underlying execution engine while maintaining efficient resource utilization.
3Productivity
If I/O waiting is reduced through parallel execution, then throughput increases, but scheduling overhead increases
Solution Approach 1:
The patent performs query plan compilation and resource allocation in advance, before parallel batch execution begins. This preliminary preparation eliminates the need for repeated compilation and scheduling decisions during batch execution, reducing scheduling overhead while maintaining high throughput during the actual data processing phase.
Solution Approach 2:
The patent ensures continuous useful action by overlapping the execution of multiple batches across different stages. While one batch is being processed at the scan stage, another batch can simultaneously execute at the filter stage, and a third at the aggregate stage. This continuous pipeline execution minimizes idle time and reduces the relative impact of scheduling overhead.
Data Source
AI summary
A database system performs pipelined execution of queries that process batches of streaming data. The database system compiles a database query to generate an execution plan and determines a set of stages based on the execution plan. The database query processes streaming data comprising batches. A scheduler schedules pipelined execution stages of the database query. Accordingly, the database system performs execution of a particular stage processing a batch of the streaming data in parallel with subsequent stages of the database query processing previous batches of the streaming data. The system further maintains watermarks for different stages of the database query.