Data Segment Inheritance Clustering for Large-Scale Record Linkage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Identifying relevant data in large-scale databases with billions of records is challenging due to the computational infeasibility of comparing datasets without a concrete strategy, and datasets associated with different individuals or events may not be connected without proper determination of their relationships.
Innovation Solution
A computer-implemented method using an iterative clustering algorithm to group data string ranges based on a similarity metric, attributing sets of data string ranges to data inheritances, and applying filtering to remove mismatched segments.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If datasets are compared without a concrete strategy, then comprehensive data comparison is achieved, but computational feasibility deteriorates
Solution Approach 1:
The patent segments data comparison into two stages: first comparing data instances at the record level to identify potential matches, then comparing only the matched segments in detail. This segmentation reduces the computational burden from comparing all datasets against each other to comparing only relevant portions, resolving the contradiction between comprehensive comparison and computational feasibility.
Solution Approach 2:
The patent performs preliminary comparison of data instances to identify matches before conducting detailed segment comparison. This preliminary action filters out non-matching datasets early in the process, preventing unnecessary computational resources from being wasted on clearly unrelated data, thus maintaining both completeness and feasibility.
2Measurement precision
If data inheritance is determined through iterative clustering, then data linkage accuracy is improved, but processing time increases
Solution Approach 1:
The patent applies partial action by performing clustering iterations only on matched data segments rather than all data instances. The iterative clustering process continues until convergence or after a predetermined number of iterations, balancing the need for accurate data linkage with time constraints. This partial approach ensures sufficient accuracy without excessive processing time.
Solution Approach 2:
The iterative clustering algorithm incorporates feedback mechanisms where the similarity metric is recalculated based on previous clustering results, and the process adjusts assignments accordingly. This feedback allows the algorithm to converge on accurate data inheritances efficiently, improving linkage accuracy while limiting processing time through convergence criteria.
3Loss of information
If data string ranges are grouped into data inheritances, then data relationships are clarified, but system complexity increases
Solution Approach 1:
The patent segments data string ranges into distinct data inheritances based on clustering results, creating organized groups that represent clear data relationships. This segmentation transforms the complexity of raw data into structured inheritance categories, making relationships explicit while managing system complexity through systematic organization.
Solution Approach 2:
The patent introduces data inheritances as intermediary structures between raw data instances and final data relationships. These inheritances serve as mediating categories that simplify the representation of complex data connections, making relationships clearer without requiring the system to directly manage all possible data interaction possibilities.
Data Source
AI summary
A computing device may receive a target data instance. The computing device may identify a plurality of matched segments that match to the target data instance for at least a threshold length. The computing device may define, based on overlapping of the matched segments, the target data instance as a plurality of data string ranges, wherein each divided data string is matched to a set of overlapping matched segments. The computing device may apply an iterative clustering algorithm to group the plurality of data string ranges based on values of a similarity metric among data string ranges that are assigned to a given group. The computing device may attribute a first set of data string ranges that are assigned to a first group to a first inheritance.