Two-Level Set Similarity Join Indexing for Stable Candidate Filtering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for set similarity join are inefficient and unstable when applied to real-world datasets that do not fit neatly into the SSJ or HSJ scenarios, leading to excessive computational and memory costs due to inappropriate pruning and verification strategies.
Innovation Solution
A two-level signature scheme is employed, where sets are indexed with a first signature, a subset is selected based on a cost function, reindexed with a second signature, and a hybrid index is generated to reduce the number of distance calculations by distributing sets across multiple records, using a hybrid index to filter and verify candidates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If a prefix filter method is applied to HSJ datasets, then the method assumes tokens are uncommon and creates inverted lists, but this produces long inverted lists requiring large numbers of candidate verifications and distance computations
Solution Approach 1:
The patent segments the indexing process into two levels: a first level using prefix filtering and a second level using partition-enumeration. This segmentation allows the system to adapt to different dataset characteristics (SSJ and HSJ scenarios) by applying appropriate filtering strategies at each level, thereby resolving the contradiction between method applicability and computational efficiency.
Solution Approach 2:
The patent implements a dynamic indexing strategy where the system can switch between different indexing methods (prefix filter vs. partition-enumeration) based on dataset characteristics. The cost function evaluates whether to use prefix filtering or partition-enumeration for each dataset, making the system adaptive to different scenarios and resolving the contradiction between ease of manufacture and productivity.
2Ease of manufacture
If partition-enumeration method is applied to SSJ datasets, then the method partitions sets and searches partitions individually, but query-side enumeration becomes computationally expensive due to large universe size
Solution Approach 1:
The patent segments the search process into two stages: first using prefix filtering to identify candidate sets, then using partition-enumeration only for the remaining candidates. This segmentation reduces the computational burden by applying the more expensive partition-enumeration only where necessary, resolving the contradiction between method applicability and computational power requirements.
Solution Approach 2:
The patent changes the indexing parameters dynamically based on dataset characteristics. For SSJ datasets with large universes, the system uses prefix filtering with optimized threshold parameters to reduce the number of candidates before applying partition-enumeration. This parameter adjustment reduces query-side enumeration costs while maintaining method applicability.
3Adaptability or versatility
If known approaches are applied to real-world datasets, then the methods operate partially outside their optimal scenarios, but this leads to unstable and inefficient performance
Solution Approach 1:
The patent creates a universal indexing system that can handle both SSJ and HSJ scenarios effectively. By implementing both prefix filtering and partition-enumeration with a cost function that selects the appropriate method, the system achieves multi-functionality across different dataset types, thereby improving both adaptability and performance stability.
Solution Approach 2:
The patent uses a cost function that evaluates dataset characteristics and provides feedback on which indexing method to use. This feedback mechanism allows the system to adapt to real-world datasets by selecting the most appropriate indexing strategy, resolving the contradiction between adaptability and performance stability.
Data Source
AI summary
Provided is a method for the set similarity join, wherein each set represents a process and each token represents a process step. The process comprises a series of process steps executed in at least one source computer system. Hence, similar sets represent similar processes within a collection of processes. The method is based on a two-level signature scheme. Having indexed the sets using a first signature into inverted lists, selected lists, in particular long lists, are reindexed using a second signature. As a result, the number of candidates and thus the number of required distance calculations can be effectively reduced. Its experimental evaluation has shown that the method consistently outperforms state-of-the-art algorithms on datasets with diverging characteristics, suggesting a stable solution for a wide range of applications.