Dimensionally Reduced Signatures for Automated Data Linking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data linking methods require user interaction and are not scalable for large numbers of data sources, as they rely on quadratic comparisons and are inefficient, limiting their ability to automatically link data elements across diverse data sources.
Innovation Solution
An automated method using similarity-based signatures, combining data value instances and semantic annotations, with Locality Sensitive Hashing to reduce the number of comparisons, allowing for efficient and automatic linking of data elements across multiple data sources without user intervention.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If exhaustive comparison of all data element pairs is performed to ensure complete link discovery, then linking accuracy is improved, but computational complexity increases quadratically
Solution Approach 1:
The patent segments the exhaustive comparison task by introducing signatures as intermediate representations. Instead of comparing all data element pairs directly, the system first computes signatures for each data element, then compares signatures to identify candidate pairs for detailed linking analysis. This segmentation reduces the quadratic complexity while maintaining linking accuracy through the use of similarity-preserving signatures.
Solution Approach 2:
The patent introduces signatures as an intermediary between raw data elements and the linking decision process. These signatures serve as compact representations that preserve similarity information, allowing the system to filter and prioritize candidate pairs before performing expensive detailed comparisons. This intermediary layer enables accurate link discovery without exhaustive pairwise analysis.
2Quantity of substance
If all data sources are linked to achieve complete data integration, then data coverage is improved, but processing time increases
Solution Approach 1:
The patent performs preliminary computation of signatures for all data elements before the actual linking process. These pre-computed signatures enable rapid similarity assessment and candidate pair identification, allowing the system to scale to large numbers of data sources without linearly increasing processing time. The preliminary signature computation phase separates the heavy lifting from the actual linking decisions.
Solution Approach 2:
The patent transforms the original data elements into a different parameter space through signature computation. This parameter transformation preserves the essential similarity information while reducing dimensionality and enabling more efficient comparisons. By changing from raw data representation to signature representation, the system achieves faster processing across multiple data sources.
3Measurement precision
If instance values are used for matching data elements to improve accuracy, then linking precision is improved, but computational cost increases
Solution Approach 1:
The patent extracts essential similarity information from instance values into compact signatures. Instead of using all instance value details in every comparison, the system extracts the most discriminative features into signature representations. This extraction maintains linking precision by preserving similarity information while dramatically reducing the computational cost of comparisons.
Solution Approach 2:
The patent creates simplified copies of data elements in the form of signatures. These signature copies retain the essential similarity characteristics needed for accurate linking but use significantly fewer computational resources to store and compare. The copying approach allows instance-based matching precision with reduced computational overhead.
Data Source
AI summary
Data elements from data sources and having a data value set are linked by using hash functions to determine a dimensionally reduced instance signature for each data element based on all data values associated with that data element to yield a plurality of dimensionally reduced instance signatures of equivalent fixed size such that similarities among the data values in the data value sets across all data elements is maintained among the plurality of instance signatures. Candidate pairs of data elements to link are identified using the plurality of instance signatures in locality sensitive hash functions, and a similarity index is generated for each candidate pair using a pre-determined measure of similarity. Candidate pairs of data elements having a similarity index above a given threshold are linked.