Entity Normalization via Name Signatures for Duplicate Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for identifying duplicate objects in collections are ineffective when objects have different types or include incomplete/inaccurate information, as they assume homogeneity and are not computationally efficient for large datasets.
Innovation Solution
A system and method that normalizes object names to generate signatures, grouping objects based on these signatures to identify duplicates within buckets, using a matcher to compare objects within the same bucket, optimizing the process by reducing pair-wise matching across all buckets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If conventional duplicate identification methods are used, then they work well for homogeneous object types, but they become ineffective when objects have different types or incomplete information
Solution Approach 1:
The patent transforms object names into normalized forms by changing their parameter representation. The normalization process standardizes various name formats (abbreviations, synonyms, different capitalizations) into a common parameter form, enabling effective comparison across different object types and handling of incomplete information.
Solution Approach 2:
The patent introduces an intermediary normalization step between the original object names and the comparison process. This intermediary transformation layer converts diverse name representations into a standardized intermediate form that facilitates reliable duplicate detection across heterogeneous object types.
2Reliability
If pair-wise matching is performed across all objects, then all duplicates can be identified, but the processing time increases significantly for large datasets
Solution Approach 1:
The patent segments the large-scale pair-wise matching problem into smaller sub-problems by first grouping objects into candidate sets based on normalized name similarity. This segmentation reduces the computational complexity from O(n²) pair-wise comparisons to comparing only objects within the same candidate set, significantly reducing processing time while maintaining completeness.
Solution Approach 2:
The patent performs preliminary grouping of objects into candidate duplicate sets before conducting detailed pair-wise matching. This preliminary action filters out objects that are unlikely to be duplicates, so that the computationally intensive pair-wise comparison is only applied to a small subset of candidate objects, thereby reducing overall processing time.
3Productivity
If objects are grouped into buckets based on signatures, then processing efficiency improves, but additional computational steps are required for normalization and signature generation
Solution Approach 1:
The patent performs preliminary normalization and signature generation for all objects before the duplicate identification process. This preliminary action transforms raw object names into standardized forms and compact signatures in advance, so that the subsequent bucketing and matching operations can proceed efficiently without repeating these computationally intensive transformations.
Solution Approach 2:
The patent creates simplified copies of object names in the form of signatures that capture the essential identifying characteristics. These signature copies are much more compact and easier to compare than the original names, enabling efficient bucketing and reducing the complexity of subsequent matching operations while preserving duplicate detection accuracy.
Data Source
AI summary
A system and method identifying duplicate objects from a plurality of objects. The system and method normalizes name values of objects, groups objects into buckets based at least in part on the normalized name values, matches objects within the same bucket based on a selected matcher, and identifies the matching objects as duplicate objects.