Optimized Exclusion Filter Expression for Query Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional query processing systems face inefficiencies in filtering data due to resource-intensive operations, especially when applying filters to large datasets, and often evaluate common filters multiple times, leading to increased costs and performance issues, particularly in parallel query scenarios.
Innovation Solution
The implementation of an optimized exclusion filter expression is created from a full filter expression, which reduces the overall cost by iteratively removing high-cost predicates and adjusting the expression tree, allowing for initial data exclusion with minimal resource usage and subsequent evaluation of the full filter expression only on non-excluded data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If each query processor applies filters independently on the complete data set, then filter application is simple and straightforward, but resource utilization increases and common filters are evaluated multiple times
Solution Approach 1:
The filter expression is segmented into two parts: an optimized exclusion filter expression that is evaluated once at the data source, and the full filter expression that is applied to the remaining data. This segmentation allows common filters to be evaluated only once instead of multiple times across parallel query processors, significantly reducing resource utilization while maintaining operational simplicity.
Solution Approach 2:
The optimized exclusion filter expression is created and applied as a preliminary action before the full filter expression is evaluated. By performing this preliminary filtering at the data source level, the system eliminates the need for multiple evaluations of common filters across different query processors, thereby reducing overall resource consumption while keeping the filter application process simple.
2Reliability
If multiple passes are made over the same data set for different queries, then each query can be answered accurately, but processing time and resource consumption increase
Solution Approach 1:
Multiple query filter expressions are merged into a single optimized exclusion filter expression that is evaluated once at the data source. This merging approach allows different queries to share common filtering logic, eliminating redundant passes over the data set while maintaining the accuracy required for each individual query.
Solution Approach 2:
A preliminary filtering action is performed using the optimized exclusion filter expression before the full filter expression is applied. This preliminary action reduces the data set size early in the process, thereby reducing the time and resources required for subsequent filtering operations while ensuring query accuracy is maintained.
3Measurement precision
If expensive predicates are included in the filter expression, then filtering precision is improved, but system resource consumption increases
Solution Approach 1:
The filter expression is segmented into an optimized exclusion filter expression that uses a subset of predicates for initial filtering, and the full filter expression that includes all predicates for final precision filtering. This segmentation allows the system to use less resource-intensive predicates for the initial pass while reserving expensive predicates for the final precision filtering step, thereby balancing filtering precision with resource consumption.
Solution Approach 2:
The optimized exclusion filter expression applies a partial set of filtering predicates that are sufficient to reduce the data set size significantly, without requiring all expensive predicates. This partial action approach allows the system to achieve substantial filtering effectiveness using fewer resources, reserving expensive predicates for only the necessary final precision checks.
Data Source
AI summary
A computer-implemented method for querying a data source using an optimized exclusion filter expression created from a full filter expression is described. The method includes receiving one or more queries defined by the full filter expression, wherein the one or more queries are for obtaining an output from the data source; performing a reduction on the full filter expression to determine the optimized exclusion filter expression; applying the optimized exclusion filter expression in the data source to exclude data; and applying the full filter expression to data not excluded by the optimized exclusion filter expression.