Disjunctive Join Optimization Using Bloom Filters and Hash Tables
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems face inefficiencies in processing disjunctive joins due to limited parallelism and high data processing requirements, particularly when dealing with disjunctive predicates, which restricts the degree of parallelism and increases computational and network resources needed.
Innovation Solution
The implementation of bloom filters and range bloom filters to reduce the number of rows in the probe-side input for disjunctive joins, combined with the use of hash tables for conjunctive and disjunctive joins, allowing for separate hash tables to be built for each disjunct and conjunctive component, thereby optimizing data flow and reducing computational and network resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If bloom filters and range bloom filters are used to filter probe-side rows, then the amount of data processed and transmitted is reduced, but the device complexity increases due to additional filtering structures
Solution Approach 1:
Bloom filters and range bloom filters are constructed in advance from the build-side input data before the actual join operation. This preliminary filtering structure allows the system to quickly determine which probe-side rows might match build-side rows, eliminating the need to process all probe-side rows and thereby reducing the amount of data that needs to be processed and transmitted during the join operation.
2Productivity
If separate hash tables are built for each disjunct and conjunctive component, then query efficiency is enhanced and parallel processing is enabled, but the device complexity increases
Solution Approach 1:
The join operation is divided into multiple independent segments corresponding to each disjunct and conjunctive component in the join predicate. Separate hash tables are constructed for each segment, allowing the system to process each segment independently and in parallel. This segmentation enables the use of multiple servers for parallel processing, significantly improving query efficiency while the modular structure manages the complexity through organized separation of processing tasks.
3Device complexity
If disjunctive joins are processed without optimization, then the implementation is simpler, but the execution time increases due to restricted parallelism and high data processing requirements
Solution Approach 1:
The system performs preliminary filtering using bloom filters and range bloom filters before executing the main join operation. This preliminary action identifies and eliminates probe-side rows that cannot possibly match any build-side rows, thereby reducing the volume of data that requires full join processing. This optimization significantly reduces execution time by avoiding unnecessary processing of non-matching rows while maintaining implementation feasibility through the use of well-established filtering techniques.
Data Source
AI summary
Joining data using a disjunctive operator using a lookup table is described. An example computer-implemented method can include receiving a query with a set of conjunctive predicates and a set of disjunctive predicates. The method may also include generating a lookup table for each predicate in the sets of conjunctive predicates and disjunctive predicates. The method, for each row in a probe-side table, may also further include looking up a value associated with that row in each of the lookup tables and adding the row to a results set when there is a match. Additionally, the method may also include returning the results set.