Database Deduplication via Vector Clustering and Confidence Scoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current deduplication engines in database systems are not scalable to handle millions of records and require user confirmation for duplicate entry removal, making them inefficient for large datasets.
Innovation Solution
A method and system for deduplicating data entries using a multi-layer entity graph and machine learning algorithms to compute attribute similarities, cluster rows based on similarities, and provide an output with clustered data entries and confidence scores, enabling automated deduplication without user intervention.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional deduplication engines are used to identify duplicate entries, then duplicate detection can be achieved, but the system cannot scale to handle millions of records and requires user confirmation
Solution Approach 1:
The patent segments the deduplication process into multiple independent components: (1) data indexing and representation as feature vectors, (2) similarity computation using cosine similarity metric, (3) clustering using DBSCAN algorithm, and (4) duplicate group formation. This segmentation allows each component to be optimized independently and processed in parallel, enabling the system to scale to millions of records while maintaining deduplication accuracy.
2Extent of automation
If traditional deduplication engines are used, then duplicate entries can be identified, but user confirmation is required which reduces automation
Solution Approach 1:
The system implements self-service deduplication by automatically computing similarity metrics between data entries, performing clustering analysis using DBSCAN, and generating duplicate groups without requiring user intervention. The algorithm autonomously determines duplicate relationships based on configurable similarity thresholds, enabling fully automated deduplication processes that eliminate time-consuming manual confirmation steps.
3Measurement precision
If exact matching is used for deduplication, then processing is simple, but partial duplicates with similar but not identical attributes are not detected
Solution Approach 1:
The patent transforms the deduplication problem from exact string matching to continuous similarity measurement by converting data attributes into numerical feature vectors. The cosine similarity metric computes the angular distance between vectors, providing a continuous similarity score between 0 and 1. This parameter transformation enables detection of partial duplicates with similar attributes while maintaining computationally efficient operations through vectorized calculations.
Data Source
AI summary
A system and method for data entries deduplication are provided. The method includes indexing an input data set, wherein the input data set is in a tabular formant and the indexing includes providing a unique Row identifier (RowID), wherein rows are the data entries; computing attribute similarity for each column across each pair of rows; computing, for each pair of rows, row-to-row similarity as a weighted sum of attribute similarities; clustering pairs of rows based on their row-to-row similarities; and providing an output data set including at least the clustered pairs of rows.