Deduplication Algorithm Using Aggregate Constraints
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data deduplication methods are inefficient in handling duplicated information from multiple sources due to errors and variations in data representation, leading to inaccurate results and increased costs in revenue recovery and data mining applications.
Innovation Solution
A deduplication algorithm that leverages aggregate and groupwise constraints, using textual similarity to restrict the search space and maximize constraint satisfaction, allowing for approximate satisfaction of constraints rather than rigid enforcement, thereby improving accuracy and efficiency in data deduplication.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional graph partitioning algorithms are used for deduplication, then deduplication can be performed, but the process becomes inefficient and time consuming
Solution Approach 1:
The patent segments the deduplication problem into two distinct phases: (1) generating candidate groups using approximate nearest neighbor search to identify potentially duplicate records, and (2) verifying these candidates using graph partitioning algorithms. This segmentation allows the computationally intensive verification step to operate only on a small subset of candidates rather than all record pairs, dramatically improving efficiency while maintaining accuracy
Solution Approach 2:
The patent performs preliminary filtering by generating candidate groups before applying the full graph partitioning algorithm. By pre-identifying likely duplicate groups using approximate nearest neighbor search, the system prepares a reduced search space that makes subsequent verification computationally feasible, thus resolving the time efficiency contradiction
2Ease of operation
If simple duplicate elimination by using 'select distinct' queries is used, then the process is simple, but it is inadequate for handling different representations of the same entity
Solution Approach 1:
The patent introduces textual similarity as an intermediary measure between simple equality comparison and complex graph partitioning. By computing similarity scores for candidate groups and using these scores to guide the verification process, the system achieves accurate deduplication of semantically equivalent records while maintaining operational simplicity through automated similarity-based candidate generation
3Reliability
If constraints are imposed inflexibly as hard constraints, then constraint satisfaction is maximized, but the system cannot handle errors in the data
Solution Approach 1:
The patent applies constraints partially rather than as hard requirements. By computing a satisfaction score that measures the degree to which constraints are met and using this score to rank and select candidate groups, the system can handle data errors gracefully while still maximizing constraint satisfaction for the subsets of data that can be reliably constrained
Data Source
AI summary
A deduplication algorithm that provides improved accuracy in data deduplication by using aggregate and/or groupwise constraints. Deduplication is accomplished using only as many of these constraints that are satisfied rather than be imposed inflexibly as hard constraints. Additionally, textual similarity between tuples is leveraged to restrict the search space. The algorithm begins with a coarse initial partition of data records and continues by raising the similarity threshold until the threshold splits a given partition. This sequence of splits defines a rich space of alternatives. Over this space, an algorithm finds a partition of the input that maximizes constraint satisfaction. In the context of groupwise aggregation constraints for deduplication all SQL (structured query language) aggregates are allowed, including summation.