Error-Tolerant Set Containment Indexing for Textual Record Matching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing set similarity lookup techniques, such as the Jaccard Coefficient, are inefficient for longer textual strings and require a large suite of indexing techniques, making them impractical for data cleaning platforms, and lack effective error tolerance in textual record matching.
Innovation Solution
The method involves obtaining a frequency threshold and query set, determining minimal infrequent tokens or token sets, and building an index using these tokens or token sets, with string transformations to enhance similarity measurement, allowing for error-tolerant set containment indexing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If Jaccard Coefficient is used for set similarity lookup, then set similarity measurement is enabled, but efficiency deteriorates for longer textual strings
Solution Approach 1:
The patent segments the textual string into multiple tokens and creates an inverted index structure where each token maps to a list of records containing that token. This segmentation allows the system to efficiently retrieve candidate records by intersecting token lists rather than comparing entire strings, thereby maintaining measurement precision while improving lookup efficiency for long textual strings.
2Measurement precision
If multiple indexing techniques are used to improve set similarity lookup, then measurement accuracy is improved, but device complexity increases
Solution Approach 1:
The patent creates a universal inverted index structure that can handle multiple set similarity measures (Jaccard Coefficient, Jaccard Containment, and other asymmetric measures) through a single indexing mechanism. The index stores token-frequency information that can be used to compute different similarity metrics, eliminating the need for multiple specialized indexing techniques and reducing overall system complexity.
3Reliability
If traditional set similarity measures are used, then symmetric similarity is achieved, but error tolerance in textual record matching is insufficient
Solution Approach 1:
The patent implements asymmetric set similarity measures by distinguishing between query tokens and record tokens in the inverted index. The Jaccard Containment measure computes similarity as the ratio of query tokens found in records to total query tokens, rather than using symmetric intersection-over-union. This asymmetric approach provides better error tolerance for textual record matching by allowing records to contain additional tokens beyond the query while still achieving high similarity scores.
Data Source
AI summary
The claimed subject matter provides a method and a system for the efficient indexing of error tolerant set containment. An exemplary method comprises obtaining a frequency threshold and a query set. All tokens or token sets within the query set are determined, and then all minimal infrequent tokens or all minimal infrequent tokens sets of data records are found and used to build an index. The minimal infrequent tokens or minimal infrequent tokensets are processed in a fixed order, and then a collection of signatures for each minimal infrequent token or token set is determined.