Data Record Integration via Probabilistic Graphical Clustering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for querying large datasets are inefficient due to the lack of obvious relationships between data records stored in different formats and locations, leading to high computing power consumption and difficulty in managing and organizing data across various platforms.
Innovation Solution
A system and method that preprocesses data by converting it into a common format, normalizes schema, blocks records, and uses graphical analysis to determine probabilistic relationships, allowing seemingly unrelated datasets to be queried as a unified set with foreign key references, thereby reducing computing power requirements and improving performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional brute force methods are used to query large datasets, then all datasets can be analyzed, but computing power consumption becomes excessively high
Solution Approach 1:
The patent segments the large-scale data matching problem into multiple processing stages: data preprocessing, blocking to create subsets, pairwise comparison within blocks, and graph-based relationship determination. This segmentation allows the system to process data in manageable chunks rather than attempting to compare all records simultaneously, significantly reducing computing power requirements while maintaining analysis completeness.
Solution Approach 2:
The patent performs preliminary actions by pre-processing data to normalize formats, creating blocking keys before pairwise comparison, and pre-establishing graph structures. These preliminary steps organize the data in advance to enable more efficient processing during the actual querying phase, reducing the computational burden of the main analysis operation.
2Adaptability or versatility
If data records from different sources are stored in different formats and locations, then data diversity is maintained, but relationship detection between records becomes difficult
Solution Approach 1:
The patent applies local quality by performing format normalization and schema alignment at the local level of individual data blocks and records, rather than requiring global standardization of all data sources. Each block can be processed with its own normalization rules, allowing the system to handle diverse formats while detecting relationships within and across blocks.
Solution Approach 2:
The patent introduces intermediary structures including blocking keys that bridge different data sources, and graph structures that serve as intermediaries to represent relationships between records from disparate formats. These intermediaries translate between different data representations, enabling relationship detection without requiring direct compatibility between all source formats.
3Measurement precision
If all pairwise comparisons of records are performed to determine relationships, then comprehensive relationship detection is achieved, but the number of comparisons becomes computationally prohibitive
Solution Approach 1:
The patent segments the pairwise comparison operation by first dividing records into blocks based on blocking keys, then performing comparisons only within each block rather than across all records. This segmentation reduces the number of comparisons from O(n²) to a much smaller number while using graph-based transitive closure to infer relationships that span multiple blocks, maintaining accuracy while improving productivity.
Solution Approach 2:
The patent performs partial pairwise comparisons only within blocked subsets rather than all possible pairs. By combining this partial action with graph-based relationship inference, the system achieves sufficient relationship detection accuracy without the excessive computational cost of complete pairwise comparison, balancing precision and productivity.
4Ease of operation
If data is normalized to a common format, then unified querying is enabled, but data processing time increases
Solution Approach 1:
The patent performs format normalization as a preliminary action during the data loading and blocking phases, before the actual querying operation. By normalizing data in advance and organizing it into blocked structures with unified schemas, the system enables fast unified querying without paying the normalization cost during each query operation, thus improving ease of operation while minimizing processing time loss.
Data Source
AI summary
Methods and systems for determining relationships between two or more nominally unrelated data sources utilizing a combination of probabilistic modeling and graphical clustering are described. The systems and methods for utilizing probabilistic model functions as a way of determining and judging the likelihood that two records from different systems are related to the same entity.