Disjunctive Join Optimization Using Bloom Filters
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing query processing systems face inefficiencies in handling disjunctive joins, particularly in distributed storage systems, where disjunctive joins are restricted to single-server parallelism and require extensive data processing and network resources due to the lack of effective filtering mechanisms.
Innovation Solution
The implementation of bloom filters and range bloom filters to reduce the number of rows in the probe-side input for disjunctive joins, combined with the use of hash tables for conjunctive and disjunctive joins, allowing for efficient data filtering and deduplication, thereby optimizing data flow and resource utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If disjunctive joins are performed without effective filtering mechanisms, then the join operation can be completed, but the amount of data processed and network resources consumed increase significantly
Solution Approach 1:
The patent applies preliminary filtering using bloom filters and range bloom filters before executing the disjunctive join operation. These filters are constructed in advance to identify and eliminate rows that cannot satisfy the join condition, thereby reducing the volume of data that needs to be processed in the subsequent join operation and minimizing network resource consumption.
2Device complexity
If disjunctive joins are restricted to single-server parallelism, then the system implementation is simplified, but the query execution efficiency is reduced
Solution Approach 1:
The patent segments the disjunctive join operation into multiple independent conjunctive join components, each of which can be executed in parallel across multiple servers. By decomposing the complex disjunctive join into simpler conjunctive join tasks, the system achieves higher parallelism and query execution efficiency while maintaining manageable system complexity through modular design.
3Measurement precision
If extensive data processing is performed for disjunctive joins, then the join results are accurate, but the resources needed increase
Solution Approach 1:
The patent employs bloom filters and range bloom filters as preliminary filtering mechanisms that use minimal computational resources to eliminate non-matching rows before the main join operation. This preliminary action ensures that only potentially matching rows undergo extensive processing, thereby maintaining join results accuracy while significantly reducing the overall resource consumption.
Solution Approach 2:
The patent introduces bloom filters as intermediary data structures that mediate between the input data and the join operation. These intermediaries provide a lightweight pre-filtering layer that preserves accuracy by not eliminating any true matches while filtering out non-matches, thus reducing the resource burden on the subsequent join operation.
Data Source
AI summary
Joining data using a disjunctive operator is described. An example computer-implemented method can include generating, with a processing device, a query plan for a query, the query comprising a join operator expression for a disjunctive predicate, wherein the join operator expression includes a conjunctive predicate and a disjunctive operator. The method may further include generating a bloom filter for the disjunctive operator. Additionally, the method may include generating a result set as a result of evaluating the join operator expression using the disjunctive operator and bloom filter for the disjunctive predicate.