Hierarchical Cluster IDs for Database Record Matching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Merging database tables to cluster common records across multiple databases is a time-consuming and costly process, often due to the presence of redundant records with overlapping but not identical information, which existing methods fail to address effectively.
Innovation Solution
The implementation of hierarchical clustering using an ordinal classifier to evaluate the degree of match between records, allowing for different confidence levels in clustering, and performing conflict resolution to detect hard conflicts, thereby generating hierarchical cluster identifiers that reflect the confidence tiers of record matches.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional merging methods are used to cluster records across database tables, then records can be combined, but the process is time-consuming and costly
Solution Approach 1:
The patent replaces traditional mechanical merging operations with a hashing-based computational system. Records are transformed into hash values through deterministic functions, enabling automatic clustering without manual or sequential processing. This substitution of mechanical table-merging with algorithmic hash-based grouping dramatically reduces processing time and computational resources.
2Measurement precision
If records are clustered with high confidence thresholds, then accuracy of matching is improved, but fewer records are clustered together
Solution Approach 1:
The patent segments the clustering process into multiple hierarchical tiers with progressively relaxed confidence thresholds. Tier 1 applies strict matching criteria for high-confidence clusters, while Tier 2 and Tier 3 apply progressively more lenient criteria to capture additional records. This segmentation allows the system to maintain high accuracy for core clusters while still achieving comprehensive coverage across all tiers.
Solution Approach 2:
The patent introduces a hierarchical dimension to the clustering results, organizing clusters into multiple levels (Tier 1, Tier 2, Tier 3) based on confidence thresholds. This dimensional organization allows users to navigate between strict and lenient matching criteria, effectively adding a confidence-level dimension to the traditional binary clustered/not-clustered classification.
3Quantity of substance
If records are clustered with lenient confidence thresholds, then more records are clustered together, but the risk of incorrect matching increases
Solution Approach 1:
The patent segments clusters into hierarchical tiers where Tier 3 contains the largest number of records with the most lenient matching criteria, while Tier 1 contains the most reliable clusters with strict criteria. This segmentation allows the system to capture comprehensive record coverage across all tiers while maintaining high reliability in the upper tiers for critical operations.
Solution Approach 2:
The system provides feedback through the hierarchical structure, allowing users to observe cluster confidence levels and adjust their selection accordingly. By presenting Tier 1, Tier 2, and Tier 3 clusters with their respective confidence characteristics, the system enables users to feedback-driven selection of appropriate cluster tiers based on their specific reliability requirements.
4Loss of information
If multiple database tables are merged to eliminate redundant records, then data completeness is improved, but the complexity of the merging process increases
Solution Approach 1:
The patent replaces complex mechanical merging operations with simple hash-based grouping operations. Instead of comparing and reconciling records across multiple tables through complex algorithms, the system transforms records into hash values and groups them automatically. This substitution dramatically simplifies the process while achieving complete data integration across all source tables.
Solution Approach 2:
The patent creates hash value copies of original records as the basis for clustering. These hash representations serve as simplified proxies that capture the essential grouping characteristics without requiring manipulation of the full complex record structures. This copying approach reduces the complexity of merging operations while preserving data completeness.
Data Source
AI summary
The present disclosure relates clustering similar data records together in a hierarchical clustering scheme. Each tier in a cluster corresponds to a minimal match score, which reflects a degree of confidence. A hierarchical cluster ID is generated for respective data records. The hierarchical cluster ID may be made up of a series of values, wherein each value reflects a tier within the hierarchical clustering scheme. A user may enter a partial hierarchical cluster ID to select clusters associated with a lower confidence. Thus, in some embodiments, the hierarchical cluster ID is variable in length in a manner that corresponds to the tiers in the hierarchical clustering scheme.