Linkage Point Discovery for Heterogeneous Data Integration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The increasing complexity and heterogeneity of data sources, including unstructured and semi-structured data, pose challenges for manual schema matching and data integration, especially due to noise and errors, making it difficult to identify linkage points between data records from different sources.
Innovation Solution
A framework that transforms semi-structured data into record-attribute-value triples, uses indexing and search algorithms with similarity functions to identify linkage points, and filters false positives, enabling efficient data record linking across heterogeneous databases.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If manual schema matching is used for data integration, then simplicity and ease of operation are maintained, but productivity and ability to handle large heterogeneous data sources deteriorate
Solution Approach 1:
The system performs self-service by automatically discovering linkage points between data sources through algorithmic analysis of attribute values, eliminating the need for manual schema matching while handling large heterogeneous data sources efficiently
Solution Approach 2:
The patent replaces the mechanical manual process of schema matching with an automated computational system that uses indexing, search algorithms, and similarity functions to discover linkage points between data sources
2Ease of operation
If specific heuristics are used for schema matching, then ease of operation is maintained for certain data types, but adaptability to different data types and domains deteriorates
Solution Approach 1:
The system achieves universality by designing a general-purpose framework that can handle multiple data types and domains through a unified approach using indexing and similarity-based search algorithms, rather than requiring domain-specific heuristics
3Loss of information
If noise and errors from extracted data are present, then data completeness is maintained, but measurement precision and difficulty of detecting linkage points increase
Solution Approach 1:
The system uses feedback mechanisms by evaluating multiple candidate linkage points through similarity functions and selecting the best matches, allowing it to tolerate noise and errors while maintaining precision in identifying true linkage points
Data Source
AI summary
Data records are linked across a plurality of datasets. Each dataset contains at least one data record, and each data record is associated with an entity and includes one or more attributes of that entity and a value for each attribute. Values associated with attributes are compared across datasets, and matching attributes having values that satisfy a predetermined similarity threshold are identified. In addition, linkage points between pairs of datasets are identified. Each linkage point links one or more pairs of data records. Each data record in the pair of data records is contained in one of a given pair of datasets, and each pair of data records is associated with a common entity having matching attributes in the given pair of datasets. Data records associated with the common entities are linked across datasets using the identified linkage points.