Projection Pipeline Task Sizing for Faster Query Result Streaming

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Database management systems face challenges in optimizing query performance due to complex queries and high transaction volumes, leading to inefficient resource utilization and prolonged wait times for clients.

Innovation Solution

Implementing a query execution engine that divides query execution into phases with result streaming, using pause and resume thresholds to manage buffer levels and enforce task size upper bounds, thereby optimizing resource allocation and reducing unnecessary computation.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Manufacturing precision

If the database execution engine processes complex queries with high transaction volumes, then query completeness and accuracy are improved, but query execution time and resource consumption increase

Engineering Contradiction:

Improvequery result accuracyVSAvoidquery execution time

Core Design Contradiction:

Manufacturing precisionVSLoss of time

Solution Approach 1:

The query execution process is divided into multiple phases (analysis phase, optimization phase, execution phase) with intermediate result streaming. The scheduling operator breaks down the query into smaller tasks that can be processed and streamed incrementally, allowing early result delivery while maintaining complete query processing for accuracy.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system performs preliminary query analysis and generates execution plans before full execution. Configuration parameters and pause thresholds are pre-configured to enable early result streaming, allowing the system to deliver partial results quickly while continuing processing to ensure complete accuracy.

Inventive Principle:

Principle #10Preliminary action

2Productivity

If the database system increases parallelization for high transaction volumes, then throughput is improved, but resource utilization efficiency deteriorates

Engineering Contradiction:

Improvetransaction throughputVSAvoidresource consumption

Core Design Contradiction:

ProductivityVSLoss of energy

Solution Approach 1:

The scheduling operator dynamically adjusts the degree of parallelization and task distribution based on system load, available resources, and query characteristics. Configuration parameters allow dynamic control of parallel execution threads and buffer management, optimizing resource utilization while maintaining high throughput during peak transaction volumes.

Inventive Principle:

Principle #15Dynamics

Data Source

PatentUS20260044506A1Smaller task sizes for the projection pipeline

Publication Date: 2026.02.12 SAP SE

AI summary

A database execution engine initiates generation of a query plan on a projection pipeline for a received query. In response to initiating generation of the query plan, the database execution engine determines a first estimate of total work that will be parallelized by a scheduling operator for the query plan. Also, the database execution engine determines a second estimate of a final result size at the projection pipeline. Next, the database execution engine calculates a target number of rows based on a pause threshold and one or more configuration parameters. Then, the database execution engine calculates a task size upper bound by multiplying the target number of rows by the first estimate divided by the second estimate. During execution of the query plan, the database execution engine applies the task size upper bound to task sizes for result streaming threads scheduled by the scheduling operator on the projection pipeline.