Query Pipeline Early Filtering for Top-K Join Pruning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database query plans are inefficient when a top K sort operation is performed subsequent to a join operation, as they waste processing cycles by processing elements that are not needed to fulfill the command due to the system's inability to know whether further rows have a join partner without further knowledge.
Innovation Solution
Implementing an early filter with intra-pipeline predicate back-propagation, where a dynamic variable is introduced to push a specialized filter down below the join operation, allowing for efficient pruning of elements before they are processed by the join operation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a top K sort operation is performed subsequent to a join operation in existing database query plans, then the query can be executed, but processing cycles are wasted by processing elements that are not needed to fulfill the command
Solution Approach 1:
The patent applies preliminary action by pushing a specialized filter down below the join operation to prune elements before they are processed by the join. This early filtering action occurs in advance of the join operation, preventing unnecessary elements from entering the processing pipeline and thus eliminating wasted processing cycles while maintaining correct query execution.
2Productivity
If elements are filtered early in the pipeline, then processing efficiency is improved, but the system must introduce additional filter mechanisms below the join operation
Solution Approach 1:
The patent introduces an intermediary specialized filter mechanism that acts as a mediator between the join operation and the data flow. This filter serves as an intermediate component that enables early pruning of unnecessary elements without fundamentally altering the core join operation, thus improving processing efficiency while adding controlled complexity through a targeted intermediate component.
3Reliability
If the system processes all elements through the join operation, then completeness is maintained, but unnecessary elements consume processing resources
Solution Approach 1:
The patent applies local quality by implementing a specialized filter with specific pruning criteria tailored to the top K sort operation's needs. This localized filtering approach applies quality control selectively at the appropriate position in the query plan, ensuring that only relevant elements proceed through the join operation while maintaining the completeness of necessary results and eliminating redundant processing.
Data Source
AI summary
In an example embodiment, an early filter is applied with a query plan using intra-pipeline predicate back-propagation. Specifically, the query plan may be thought of as a pipeline of operations. A runtime variable var may be introduced, and a specialized filter using var may be pushed down below the join operation. Var is a dynamic variable that is updated to track a value from the sort or similar operation (such as max (heap), reflecting the maximum value of a max-heap used by the sort or similar operation). The runtime variable gets initialized once the heap reaches a minimum number of elements (such as K in the case of a top K sort). Thus, before the heap reaches that minimum number of elements, the filter does not apply. Once the heap does reach that minimum number of elements, the filter does apply and acts to filter elements. Since the filter has been pushed down below the join operation, this saves processing cycles.