Record Merging via Transitive Matching and Partitioning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing entity resolution systems lack transparency in decision-making processes for deduplicating records, fail to identify problematic entities, and lack auditability for comparative metrics, making it difficult to manage large databases efficiently.
Innovation Solution
A system and method that perform record-pairing based on transitive deterministic matching criteria, apply non-transitive matching rules, and partition record-pairs using independent grouping operations to generate maximal connected components, which are then merged to produce a reduced set of records, while providing audit trails and comparative metrics.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If automated entity resolution systems are used to deduplicate records, then productivity is improved, but device complexity increases and auditability is lost
Solution Approach 1:
The patent segments the entity resolution process into distinct modular components: record-pairing operation (generating candidate pairs), partitioning operation (grouping records into connected components), and merging operation (consolidating duplicates). This segmentation allows each module to be independently optimized and maintained, reducing overall system complexity while maintaining high deduplication efficiency through automated processing.
2Manufacturing precision
If complex matching rules are applied to accurately identify duplicates, then manufacturing precision is improved, but device complexity increases
Solution Approach 1:
The patent implements dynamic matching rules that can be configured and adjusted without changing the underlying system architecture. The system allows administrators to define custom matching criteria (exact match, fuzzy match, probabilistic thresholds) that can be modified based on data characteristics, maintaining high matching accuracy while keeping the core system simple and manageable.
3Productivity
If traditional entity resolution systems are used, then productivity is improved, but loss of information occurs due to lack of decision transparency
Solution Approach 1:
The patent incorporates comprehensive audit trail functionality that automatically records and stores all matching decisions, rule applications, and operational parameters throughout the deduplication process. This feedback mechanism provides complete transparency into system decisions while running at full automated speed, allowing stakeholders to review, validate, and understand the reasoning behind each merge operation without slowing down productivity.
4Reliability
If manual review processes are implemented to identify problematic entities, then reliability is improved, but productivity decreases
Solution Approach 1:
The patent implements preliminary automated filtering and classification that identifies and flags potentially problematic entities (such as records with ambiguous matching scores or conflicting attributes) before they require human review. This preliminary action handles the majority of cases automatically, maintaining high productivity, while only presenting a small subset of genuinely problematic cases for manual validation, thus improving reliability without significantly impacting overall processing speed.
Data Source
AI summary
A non-transitory computer-readable storage medium is provided to store computer-readable program code to receive an unmerged record set, comprising a first plurality of data records, to generate record-pairs from the first plurality of data records, based upon a set of transitive deterministic matching criteria, apply a set of non-transitive matching rules to the record-pairs, perform a partitioning operation on the record-pairs, using a plurality of independent grouping operations, wherein a plurality of matched record groups are generated. The computer-readable program code may determine a set of maximal connected components from the plurality of matched record groups, perform a merge operation on the set of maximal connected components to generate a set of merged records, the set of merged records comprising a second plurality of data records, less than the first plurality of data records, and send the merged records for storage in a non-transitory computer readable storage medium.