Inner Segment Join Operator for Wildcard Data Matching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for joining data segments with wildcards to large source datasets are inefficient, requiring multiple queries and being cost-prohibitive for large datasets, especially when segments have overlapping data and varying dimensions.
Innovation Solution
An inner segment join operator that builds a location map to efficiently index values and wildcards in join columns, allowing for a single scan of the source dataset and segments to perform an inner join, using a hash function and wildcard list to match values and wildcards.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the hash join algorithm is used to join segments with source dataset, then the join operation can be performed, but it requires an extra query per segment resulting in long and complicated flows that are unsuitable for large datasets
Solution Approach 1:
The patent applies preliminary action by pre-processing the segments to build location maps and wildcard maps before the actual join operation. This pre-computation of segment locations and wildcard positions allows the join to proceed efficiently without requiring separate queries for each segment, thus resolving the time loss issue while maintaining productivity
2Reliability
If a full query of the source dataset is performed for each segment, then all matching data points can be found, but the cost becomes prohibitive for large datasets
Solution Approach 1:
The patent applies segmentation by dividing the join operation into distinct phases: building location maps for each segment, constructing wildcard maps, and then performing the actual join. This segmentation allows the system to process segments independently and efficiently without requiring exhaustive queries of the entire source dataset for each segment, thus reducing energy consumption while maintaining join accuracy
Solution Approach 2:
The patent introduces location maps and wildcard maps as intermediary data structures that mediate between the segments and the source dataset. These intermediaries store pre-computed location information, allowing the join operation to quickly identify matching rows without performing full dataset queries, thereby reducing computational resource consumption while ensuring accurate joins
3Adaptability or versatility
If segments with wildcards and overlapping data are joined using traditional methods, then all matching segments can be identified, but the process becomes complex and inefficient for large scale datasets
Solution Approach 1:
The patent applies local quality by creating specialized data structures (location maps and wildcard maps) that are tailored to the specific needs of wildcard segment joining. Each map is constructed with local optimizations for handling wildcards and overlapping segments, allowing the system to efficiently handle diverse segment types without requiring a universally complex algorithm, thus reducing overall system complexity while maintaining adaptability
Data Source
AI summary
Embodiments relate to efficiently performing an inner join on segments from a data source, where the segments have wildcards in join columns. An algorithm for an inner segment join operator may build a location map that describes locations, by segment and column, of values in the join columns of rows in the segments. A map of wildcard locations, by segment and column, may also be constructed. The location map may be supplemented with information from the wildcard map. The data source is scanned and contents of rows in the data source are queried against the location map to join rows with segments that match due to either matching values or wildcard presence. A user interface may be provided to facilitate construction of joins using the inner segment join operator.