Dataset Metadata Generation for Record Linkage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Record linkage between structured and unstructured datasets is challenging due to limited attribute values in unstructured datasets, making it difficult to compare and match records effectively.
Innovation Solution
A method that selects attributes, determines distinct values, forms groups of words, identifies distinct word sequences, and calculates their presence levels within these groups, generating metadata to support record matching and completion of missing attributes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If record linkage is performed between structured and unstructured datasets, then data integration capability is improved, but matching accuracy deteriorates due to limited attribute values in unstructured datasets
Solution Approach 1:
The patent transforms the matching process from attribute-value comparison to word sequence analysis. By converting unstructured text into structured word sequences with presence levels, the system adds a new dimension of analysis that works across both structured and unstructured datasets, resolving the mismatch in data formats while maintaining matching accuracy
Solution Approach 2:
The patent changes the parameters used for comparison from traditional attribute values to word sequence presence levels. This parameter transformation allows unstructured data to be compared using the same metrics as structured data, improving matching accuracy without sacrificing data integration capability
2Measurement precision
If additional background data is collected to improve matching accuracy, then matching precision is improved, but system complexity and data processing requirements increase
Solution Approach 1:
The patent enables the system to generate its own metadata and word sequence information directly from the input datasets without requiring external background data. The unstructured dataset itself provides the necessary information through word sequence extraction, making the system self-sufficient and reducing overall complexity
Solution Approach 2:
The patent performs preliminary metadata generation and word sequence extraction during the data ingestion phase. By preparing the data in advance with embedded metadata, the system eliminates the need for complex real-time background data queries, reducing processing complexity while maintaining high matching precision
Data Source
AI summary
The present disclosure relates to a method for enabling a processing of a dataset of records having a set of attributes. The method comprises: selecting a first attribute of the set of attributes and a subset of one or more second attributes of the set of attributes. Distinct values of the subset of second attributes may be determined from the dataset. For each distinct value of the determined distinct values records of the dataset that have said each distinct value may be identified, and a group of words may be formed from values of the first attribute of the identified records. Distinct word sequences may be identified in the formed groups and a level of presence of each word sequence of the word sequences in each of the formed groups may be determined. At least part of the levels of presence may be provided as metadata.