Entropy-Based Duplicate Object Detection in Data Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for determining duplicate objects in databases are ineffective when dealing with incomplete or imperfect information, as they either rely on specific facts that may be missing or inaccurate, or treat all facts equally, leading to inaccurate results.
Innovation Solution
The method identifies common facts with matching attributes and values, calculates the entropy of these matching facts, and compares the sum of entropies to an entropy threshold to determine if two objects are duplicates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If specific facts (e.g., SSN, ISBN, UPC) are used for comparison, then duplicate detection is effective when all objects contain these facts, but the approach fails when objects have incomplete or inaccurate information
Solution Approach 1:
The patent changes the parameter used for comparison from specific fact values (SSN, ISBN) to entropy values calculated from multiple facts. Instead of relying on a single primary key, the system calculates entropy for each fact and combines them, allowing the approach to work with incomplete or inaccurate information while maintaining reliability in duplicate detection
2Adaptability or versatility
If all common facts are compared with equal weight, then the method can handle incomplete information, but it produces inaccurate results by treating all facts equally
Solution Approach 1:
The patent applies local quality by assigning different weights to different facts based on their entropy values. Instead of treating all facts equally, the system calculates entropy for each fact and gives higher weight to facts with higher entropy (more distinguishing power). This allows the method to handle incomplete information while maintaining accuracy by focusing on the most informative facts
3Reliability
If multiple facts are compared to improve accuracy, then the system can identify duplicates with incomplete primary keys, but the complexity of determining fact importance increases
Solution Approach 1:
The patent applies self-service by using the facts themselves to determine their own importance through entropy calculation. Each fact's entropy is calculated based on its distribution across objects, and this entropy value automatically serves as its weight in the comparison. This self-determined weighting mechanism reduces the need for external complexity in determining fact importance
Data Source
AI summary
A system and method determines whether two objects are duplicate objects. The system and method matches common facts of the two objects based on a match measure, combines the entropies of the matching common facts, and determines whether the two objects are duplicate objects based on the sum of entropies.