Predicate Evaluation Algorithm for Database Join Operations
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current database management systems face inefficiencies in operations like joining, sorting, and searching due to the need to evaluate all elements of two data sets against a predicate, leading to high computational complexity and inefficiencies, especially with large data sets.
Innovation Solution
A method that partitions each data set into sub-groups, evaluates the probability of predicate satisfaction using boundary elements, and recursively refines sub-groups until base case sub-groups are reached, allowing for parallel processing across multiple computing cores.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If all elements of both data sets are evaluated against the predicate function, then complete accuracy is achieved, but computational complexity becomes prohibitive with large data sets
Solution Approach 1:
The patent divides data sets into sorted sub-groups and partitions them into blocks, evaluating only boundary elements first to identify promising sub-group pairings. This segmentation reduces the number of elements requiring full predicate evaluation while maintaining accuracy through systematic refinement of candidate pairs.
Solution Approach 2:
The patent performs preliminary evaluation of boundary elements before evaluating internal elements. By sorting data and assessing boundary conditions first, the system identifies and eliminates non-promising sub-group pairings early in the process, avoiding unnecessary computational work on elements that cannot satisfy the predicate.
2Reliability
If a nested loop join algorithm is used to ensure all element pairs are compared, then completeness of results is guaranteed, but processing time becomes excessive for large data sets
Solution Approach 1:
The patent segments the exhaustive comparison process into hierarchical levels: first comparing boundary elements of sorted sub-groups, then refining to internal elements only for promising pairs. This segmentation maintains result completeness while dramatically reducing processing time by eliminating unnecessary comparisons early in the evaluation hierarchy.
Solution Approach 2:
The patent performs preliminary sorting of data sets and preliminary evaluation of boundary elements before conducting full predicate evaluation. This preliminary action establishes an ordered structure that enables efficient identification of candidate pairs, ensuring no valid matches are missed while minimizing the number of expensive full evaluations required.
3Productivity
If data sets are sorted and processed using merge join, then processing efficiency is improved, but the algorithm becomes blocking and cannot be parallelized
Solution Approach 1:
The patent segments the data set into independently sortable sub-groups that can be processed in parallel. Each computing core can sort and evaluate its assigned sub-groups independently, eliminating the blocking nature of traditional merge join while maintaining the efficiency benefits of sorted processing through subsequent pairing of results.
Solution Approach 2:
The patent transitions from sequential merge join processing to a multi-dimensional parallel architecture where multiple computing cores independently process different sub-group pairings. This dimensional change from single-threaded to multi-threaded processing maintains sorting efficiency while enabling parallel execution across multiple cores.
4Speed
If hash join is used to achieve fast lookup, then processing speed is greatly improved, but the algorithm is constrained by available memory size
Solution Approach 1:
The patent segments the data into sorted sub-groups that can be processed with reduced memory requirements. Instead of loading entire data sets into hash tables, the system processes smaller sorted segments, evaluating boundary elements first and only loading necessary portions into memory, thereby reducing memory capacity requirements while maintaining processing speed.
Data Source
AI summary
The present invention relates to a method and apparatus for analysing a pair of data sets with respect to a predicate, the predicate having a predicate criterion, each data set comprising one or more elements sorted according to the predicate criterion. The invention comprises recursive partitioning of the data sets in a parallel manner across multiple computing cores until a base case is reached. The invention is particularly suited for carrying out join, filter and sort operations on large data sets within database management systems.