PARTENUM Index for Exact Set-Similarity Lookups
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional set-similarity indexing techniques, such as locality-sensitive hashing (LSH), provide only probabilistically approximate lookups, missing some output sets and being inefficient for large data sets, whereas exact lookups are often required in data cleaning and database management systems.
Innovation Solution
The PARTENUM index structure supports exact set-similarity lookups by using a signature scheme that exploits the frequency of elements and partitions of multi-dimensional vectors, achieving sublinear query time and subquadratic space for uniformly distributed input sets, and can be adapted for disk-based implementations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If locality-sensitive hashing (LSH) based indexes are used for set-similarity lookups, then the lookup speed is improved, but the accuracy deteriorates because output sets are missed probabilistically
Solution Approach 1:
The index structure partitions the multi-dimensional vector space into multiple segments or regions. Each segment maintains its own index structure, allowing the system to divide the large search space into manageable parts. This segmentation enables faster local searches while maintaining global accuracy by ensuring that similar sets are likely to fall within the same segment.
Solution Approach 2:
The patent transforms the set-similarity problem into a multi-dimensional vector space representation. By mapping sets to points in a higher-dimensional space where similarity corresponds to spatial proximity, the system can leverage geometric properties and dimensionality-specific indexing techniques to achieve both speed and accuracy in set-similarity lookups.
2Quantity of substance
If conventional indexing methods are used for large data sets, then the space complexity is reduced, but the query time increases requiring full scans
Solution Approach 1:
The system performs preliminary actions during the index construction phase by pre-computing and storing multi-dimensional vector representations of all sets, along with their similarity relationships. This preprocessing work is done once during indexing, allowing subsequent queries to be answered quickly without requiring full scans, thus trading initial computation time for efficient query performance.
Solution Approach 2:
The patent changes the representation parameters of the data by transforming sets into multi-dimensional vectors with specific properties. This parameter transformation enables the use of efficient vector-based indexing and similarity computation, reducing both space complexity through compact representations and query time through optimized similarity searches.
3Measurement precision
If exact set-similarity lookups are implemented, then the lookup accuracy is improved, but the computational complexity increases
Solution Approach 1:
The patent introduces multi-dimensional vectors as an intermediary representation between the original set data and the similarity computation process. These vectors serve as mediators that capture the essential similarity properties of sets in a form that is computationally efficient to compare, thereby achieving exact lookups with reduced computational complexity.
Data Source
AI summary
Input set indexing for set-similarity lookups. The architecture provides input to an indexing process that enables more efficient lookups for large data sets (e.g., disk-based) without requiring a full scan of the input. A new index structure is provided, the output of which is exact, rather than approximate. The similarity of two sets is specified using a similarity function that maps two sets to a numeric value that represents similarity of the two sets. Threshold-based lookups are addressed where two sets are considered similar if the numeric similarity score is above a threshold. The structure efficiently identifies all input sets within a distance k (e.g., a hamming distance) of the query set. Additional information in the form of frequency of elements (the number of input sets in which an element occurs) is used to improve index performance.