Semi-Structured Data Record Comparison Using Token Indexing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for comparing semi-structured data records, such as network management messages, online social network user profiles, and VoIP call records, face challenges due to the lack of well-defined formats, leading to inefficient and inaccurate matching processes, especially as data volumes grow, and often require manual intervention or significant computational resources.
Innovation Solution
A system and method that tokenizes and hashes data records to generate indexes and calculate similarity scores, allowing for efficient comparison and matching of alphanumeric tokens across multiple data sources, including the use of a middle-man entity for enhanced matching accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If general similarity measures such as edit distance and n-gram are used to compare data records, then the comparison process is simple and clear, but the computational cost becomes prohibitive when dealing with millions of profiles
Solution Approach 1:
The patent segments data records into discrete tokens and creates indexes for these tokens. This segmentation allows the system to compare only relevant token sets rather than performing exhaustive string comparisons across entire records, dramatically reducing computational complexity while maintaining comparison accuracy.
Solution Approach 2:
The patent performs preliminary tokenization and index creation on data records before comparison operations. By pre-processing the data into token sets and storing them in indexes, the system eliminates the need for repeated full-text processing during comparisons, significantly improving computational efficiency for large-scale profile matching.
2Extent of automation
If blind comparison methods are used to match data records, then the comparison can be performed automatically, but the matching accuracy deteriorates due to lack of contextual understanding
Solution Approach 1:
The patent changes the parameter representation from raw strings to token sets with associated metadata (such as token types, frequencies, and positional information). This parameter transformation enables automated comparison while preserving contextual information, allowing the system to distinguish between semantically different tokens that may appear similar in blind string comparison.
3Measurement precision
If manual methods are used to compare and analyze network management messages, then the analysis accuracy is maintained, but the processing time and labor resources increase significantly
Solution Approach 1:
The patent creates token-based copies of the original data records that preserve the essential information needed for comparison. These token representations serve as simplified proxies that can be processed automatically while maintaining the accuracy benefits of manual analysis, eliminating the need for time-consuming human review.
Data Source
AI summary
A method for comparing data records, including extracting, by a computer processor, alphanumeric tokens from a plurality of data records, wherein the plurality of data records are generated by a plurality of entities, generating, by the computer processor, a plurality of indexes each referencing an entity of the plurality of entities by at least one of the alphanumeric tokens that is associated with the entity, extracting target alphanumeric tokens from a target data record of a target entity, identifying a candidate entity from the plurality of entities based on the target alphanumeric tokens and a first index of the plurality of indexes, and calculating, by the computer processor, a first score representing a first similarity measure between a candidate data record selected from the plurality of data records that belongs to the candidate entity and the target data record of the target entity.