Data Segment Inheritance Clustering for Large-Scale Record Linking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Identifying relevant data in large-scale databases with billions of records is challenging due to the computational infeasibility of comparing numerous datasets without a concrete strategy, and datasets associated with different individuals or events may not be connected without determining their relationship.
Innovation Solution
A computer-implemented method using an iterative clustering algorithm to group data string ranges based on a similarity metric, attributing sets of data string ranges to data inheritances, and applying filtering to remove mismatched segments.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If datasets are compared without a concrete strategy, then comprehensive data comparison is achieved, but computational feasibility deteriorates
Solution Approach 1:
The patent segments data comparison into two stages: first comparing data instances at the record level to identify potential matches, then comparing only the matched segments in detail. This segmentation reduces the computational burden from comparing all datasets completely to comparing only relevant portions, resolving the contradiction between comprehensive comparison and computational feasibility.
Solution Approach 2:
The patent performs preliminary actions by first identifying and filtering data instances that share common attributes or patterns before conducting detailed comparison. This preliminary sorting and filtering creates a reduced dataset for thorough analysis, making comprehensive comparison computationally feasible by eliminating irrelevant data early in the process.
2Measurement precision
If all datasets are compared to identify relevant data, then data relevance is improved, but processing time increases
Solution Approach 1:
The patent divides the data comparison process into sequential segments: initial filtering by common attributes, identification of matched data instances, and detailed comparison of matched segments. This segmentation allows the system to quickly identify relevant data through the first segments while reserving detailed analysis for only the necessary portions, significantly reducing overall processing time.
Solution Approach 2:
The patent applies partial action by comparing only the portions of datasets that are likely to be relevant based on preliminary attributes, rather than exhaustively comparing all data. This partial comparison approach identifies sufficient relevant data without the time cost of complete comparison, achieving acceptable data relevance with reduced processing time.
3Measurement precision
If data instances are analyzed for inheritance, then data relationship identification is improved, but algorithm complexity increases
Solution Approach 1:
The patent segments the inheritance analysis into discrete steps: extracting data instances, comparing them against reference data, identifying matches based on similarity metrics, and attributing matches to specific data inheritances. This segmentation breaks down the complex inheritance detection problem into manageable operations, reducing overall algorithm complexity while maintaining detection precision.
Solution Approach 2:
The patent introduces an intermediary reference dataset that mediates the comparison between target data instances and potential inheritance sources. Instead of directly comparing all possible data instances (which would be complex), the reference data serves as an intermediary that simplifies the inheritance attribution process by providing a standardized comparison basis.
Data Source
AI summary
A computing device may receive a target data instance. The computing device may identify a plurality of matched segments that match to the target data instance for at least a threshold length. The computing device may define, based on overlapping of the matched segments, the target data instance as a plurality of data string ranges, wherein each divided data string is matched to a set of overlapping matched segments. The computing device may apply an iterative clustering algorithm to group the plurality of data string ranges based on values of a similarity metric among data string ranges that are assigned to a given group. The computing device may attribute a first set of data string ranges that are assigned to a first group to a first inheritance.