Token-Based Entity Resolution Reducing Comparisons
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional entity resolution methods face challenges in efficiently comparing records due to the N-Squared problem, particularly when dealing with large datasets, and struggle with free-form, unstructured data that requires intimate knowledge of data structure and location.
Innovation Solution
A token-based entity resolution system that transforms records into tokens, calculates token importance using tf-idf values, prunes unimportant tokens, and compares only records sharing common tokens, reducing the number of comparisons needed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If each record is compared to every other record to find similar records, then all possible matches are identified, but the computational complexity becomes N-Squared which is impractical for large datasets
Solution Approach 1:
The patent segments records into discrete tokens (words or phrases) and creates a token-record mapping structure. This segmentation allows the system to compare token sets rather than entire records, reducing computational complexity from O(N²) to approximately O(T×N) where T is the average number of tokens per record, making large-scale entity resolution practical while maintaining match identification completeness.
Solution Approach 2:
The patent performs preliminary actions by pre-computing token sets for all records and building a token-record mapping index before the actual matching process. This preliminary tokenization and indexing enables efficient retrieval of candidate matches during entity resolution without requiring full record comparisons, thus improving computational efficiency while preserving reliability.
2Productivity
If record blocking is used to reduce comparisons by requiring exact value matches in fixed locations, then computational efficiency improves, but the approach requires intimate knowledge of data structure and cannot handle free-form data
Solution Approach 1:
The patent employs dynamic tokenization that adapts to the content of free-form records rather than requiring fixed data structures. The system extracts meaningful tokens (words or phrases) from variable-length text fields, allowing it to handle unstructured data efficiently. This dynamic approach maintains computational efficiency by limiting comparisons to token-based matching while accommodating diverse data formats and structures.
Solution Approach 2:
The patent changes the matching parameter from fixed-position field values to extracted token sequences. By transforming the comparison criterion from structure-dependent (requiring knowledge of data layout) to content-dependent (based on token presence), the system achieves both computational efficiency and adaptability to free-form data without requiring intimate knowledge of underlying data structures.
3Productivity
If the nearest neighborhood/sliding window approach is used with a predefined key, then the number of comparisons is reduced, but it is difficult to define an appropriate key for free-form data
Solution Approach 1:
The patent implements self-service entity resolution where the system automatically generates token-based keys from the record content itself without requiring user-defined keys. The tokenization process extracts meaningful units from free-form text, and the token-record mapping automatically creates the indexing structure needed for efficient comparison. This eliminates the complexity of key definition while maintaining reduced comparison counts.
Solution Approach 2:
The patent replaces the mechanical key-definition process with an automated text processing mechanism. Instead of requiring users to manually identify and define comparison keys (a complex human task), the system uses algorithmic tokenization to automatically generate meaningful comparison units from the data content, substituting human expertise with a scalable computational process.
4Productivity
If token importance calculation and pruning is applied to reduce comparisons, then resource requirements are reduced, but there is a risk of missing important matches if unimportant tokens are incorrectly removed
Solution Approach 1:
The patent applies partial action by pruning only the least important tokens (those with lowest TF-IDF scores) while retaining tokens above a certain importance threshold. This selective pruning reduces the number of tokens that need to be compared across records, improving resource efficiency. The threshold-based approach ensures that tokens potentially important for accurate matching are preserved, maintaining reliability while reducing computational burden.
Solution Approach 2:
The patent introduces TF-IDF (term frequency-inverse document frequency) scoring as an intermediary mechanism to objectively evaluate token importance. This quantitative metric serves as a mediator between raw token data and the pruning decision, providing a systematic way to identify and remove genuinely unimportant tokens while preserving those that contribute meaningfully to record similarity, thus maintaining match identification accuracy.
Data Source
AI summary
A token-based database management system described herein may reduce an amount of comparisons during entity resolution of records. The system includes a token creator configured to create tokens from records, a token-record mapping creator configured to create a token-record mapping of tokens to records, a token importance calculator configured to calculate token importance values for the tokens, a token pruner configured to identify a token of the current record as unimportant based on token importance values of the tokens of the current record, and to remove the unimportant token from the token-record mapping, a record selector configured to select only records sharing at least one common token with the current record, and a record comparator configured to compare the current record with each of the selected records to determine whether the current record matches any of the selected records.