Record Linking with Dynamic Entity Thresholds for Data Purity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data management systems face issues with transitive linking of records using a static, global auto-link-threshold value, leading to the creation of large, heterogeneous entities with dissimilar records, which can result in inefficient data management and decision-making.
Innovation Solution
Implementing a dynamic, entity-based local auto-link-threshold value that adjusts based on the similarity and impurity within each entity, using equations to determine whether to link records, thereby preventing the formation of large-sized heterogeneous entities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If a static, global auto-link-threshold value is used for record linking, then the linking process is simple and consistent, but large heterogeneous entities with dissimilar records are formed
Solution Approach 1:
The patent applies local quality by transitioning from a single global auto-link-threshold value to entity-specific local threshold values. Each entity maintains its own threshold based on the similarity characteristics of its member records, allowing the system to adapt the linking criteria to the specific nature of each entity while maintaining operational simplicity through automated threshold calculation.
Solution Approach 2:
The patent implements dynamics by making the auto-link-threshold value dynamic rather than static. The threshold automatically adjusts based on the impurity level and similarity characteristics of each entity, enabling the system to respond to changing data conditions and prevent the formation of heterogeneous entities without requiring manual intervention.
2Manufacturing precision
If a dynamic, entity-based local auto-link-threshold value is used, then entity purity is maintained, but the system complexity increases
Solution Approach 1:
The patent applies self-service by enabling entities to automatically determine their own threshold values based on their internal characteristics. The system calculates local threshold values autonomously using the similarity and impurity metrics of each entity's member records, eliminating the need for complex external configuration or manual threshold setting while maintaining high entity purity.
3Productivity
If transitive linking is performed with a global threshold, then record linking is comprehensive, but dissimilar records are included in the same entity
Solution Approach 1:
The patent resolves this contradiction by applying local quality through entity-specific threshold values that reflect the similarity characteristics of each entity's member records. This allows comprehensive linking within entities while maintaining reliability by preventing dissimilar records from being grouped together, as each entity's threshold is tailored to its specific data characteristics.
Data Source
AI summary
Records linking is provided. Two records are selected from a plurality of records corresponding to a customer for pair-wise record comparison. It is determined whether the two records are included in different entities. A local auto-link-threshold value of the different entities is identified in response to determining that the two records are included in different entities. An attribute comparison is performed between the two records. A comparison score is generated based on the attribute comparison between the two records. It is determined whether the comparison score is greater than the local auto-link-threshold value of the different entities. The two records are linked in response to determining that the comparison score is greater than the local auto-link-threshold value of the different entities.