Database Record Clustering for Entity Linking Without Hierarchy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The challenge in data processing environments is accurately matching, linking, and clustering records from multiple data sources, especially in hierarchical structures where relationships and connections are complex and often incomplete, making it difficult to understand the structure and determine how entities fit within it.
Innovation Solution
The implementation of a method that clusters database records into sets based on similarity, associates matching records, and re-clusters them to determine relationships and output relevant information, even in the absence of a clear hierarchy, using co-convergence and clustering techniques.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If hierarchical structures are used to organize data, then relationships and connections between entities can be understood, but the complexity of matching and linking records increases when data grows large
Solution Approach 1:
The patent segments the data processing task into multiple stages: initial clustering of database records into groups based on field value similarity, followed by iterative matching within clusters, and then re-clustering based on matching results. This segmentation breaks down the complex problem of matching all records against each other into manageable phases, reducing overall system complexity while maintaining matching accuracy
Solution Approach 2:
The patent performs preliminary clustering of database records into clusters based on similarity of corresponding field values before conducting the actual matching process. This preliminary organization groups potentially matching records together, reducing the search space and making the subsequent matching process more efficient and scalable to large datasets
2Measurement precision
If complete hierarchical data is available, then accurate entity relationships can be determined, but data integration becomes more difficult when data is incomplete or from multiple sources
Solution Approach 1:
The patent implements a feedback mechanism through iterative matching processes. Records are initially clustered, then matching is performed within clusters, and based on the matching results, records are re-clustered. This iterative feedback loop allows the system to progressively refine cluster assignments and improve relationship determination accuracy even when starting with incomplete data from multiple sources
Solution Approach 2:
The patent performs matching iterations on subsets of records within clusters rather than requiring complete data processing. By focusing computational effort on partial sets of records that are most likely to match based on initial clustering, the system can determine relationships with acceptable precision even when complete hierarchical data is not available
3Adaptability or versatility
If traditional clustering methods are used, then data processing is straightforward, but the ability to handle complex relationships without hierarchy is limited
Solution Approach 1:
The patent employs dynamic re-clustering where cluster assignments are not fixed but are updated iteratively based on matching results. Records can move between clusters as the algorithm progresses, allowing the system to adapt to complex relationships in the data without requiring a predefined hierarchical structure, while maintaining operational simplicity through automated iterative processes
Data Source
AI summary
Certain implementations of the disclosed technology include systems and methods for linking entities in an internal database by utilizing co-convergence and clustering. The method may include clustering database records into a first set of clusters having corresponding first cluster identifications (IDs). The clustering may be based at least in part on determining similarity among corresponding field values. The method may include associating mutually matching database records, by performing at least one matching iteration for each of the database records. The method may include determining similarity among corresponding field values of the database records, re-clustering at least a portion of the database records into a second set of clusters, the re-clustering based at least in part on the associating mutually matching database records and on the determining similarity among corresponding field values of the database records.