Conditional Filters for Dynamic Database Join Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data processing methods, such as Bloom filters and cuckoo filters, are limited in allowing membership testing for a single set and require scanning the input table to build predicate-specific filters during query time, which is inefficient for database join processing where the relevant set is dynamic and determined by predicates.
Innovation Solution
The implementation of conditional filters that generate fact keys and attribute vectors for fact objects, allowing for efficient set membership testing by storing these in a data catalog, enabling rapid query processing without the need for scanning the input table, and using Bloom filters to determine attribute key matches.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If predicate-specific filters are built at query time using existing methods, then membership testing can be performed, but scanning the input table is required which reduces query performance
Solution Approach 1:
The patent pre-computes and stores filter data structures (Bloom filters, cuckoo filters) in the data catalog during data loading, rather than building them at query time. This preliminary action allows the query engine to directly use these pre-built filters for fast membership testing without scanning the input table, thus resolving the contradiction between accurate membership testing and query performance
2Measurement precision
If existing data sketches are used for membership testing, then single set membership can be tested, but they cannot handle dynamic sets determined by predicates in join processing
Solution Approach 1:
The patent extends traditional single-set data sketches to support predicate-specific filters that can handle dynamic sets. By storing multiple filter data structures in the data catalog (one for each predicate combination), the system achieves universality where a single infrastructure supports both single-set and multi-set membership testing, enabling adaptability to different join scenarios while maintaining membership testing precision
3Reliability
If predicate-specific filters are built during query time, then accurate membership testing is possible, but the process requires scanning the input table which increases processing time
Solution Approach 1:
The system pre-computes filter data structures during data loading and stores them in the data catalog, eliminating the need to scan the input table at query time. This preliminary action preserves filter accuracy while dramatically reducing query processing time by allowing direct access to pre-built filters
4Quantity of substance
If traditional Bloom filters are used, then memory efficient membership testing is achieved, but they are limited to fixed sets and cannot adapt to predicate-specific dynamic sets
Solution Approach 1:
The patent creates a universal filter infrastructure where Bloom filters and other data sketches are stored in the data catalog with predicate information. This allows the same memory-efficient filter structures to serve multiple predicates and dynamic sets, achieving both memory efficiency and predicate-specific adaptability simultaneously
Data Source
AI summary
Embodiments are directed to data processing. A plurality of fact objects and a plurality of attribute objects may be provided such that each of the attribute objects may be associated with one or more fact objects. A fact key may be generated for each of the plurality of fact objects based on information associated with each fact object. Attribute objects associated with each of the plurality of fact objects may be determined based on attribute information associated with each fact object. An attribute key for each of the one or more attribute objects may be generated based on the attribute information. The attribute keys and a plurality of fact keys may be stored at a plurality of storage locations in a data catalog such that each storage location corresponds to one of the plurality of fact keys.