Duplicate Detection via Clustering Centroids
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems face challenges in real-time detection of duplicate invoices or data records, particularly in large datasets, leading to impractical computational complexity and delayed payment processing.
Innovation Solution
A real-time duplicate detection system that groups invoices over a time period into clusters based on similarity, allowing for reduced comparisons between new invoices and cluster centroids, enabling efficient real-time detection by minimizing the number of required comparisons.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional duplicate detection compares every new invoice with all previous invoices, then detection accuracy is improved, but computational complexity and processing time increase exponentially
Solution Approach 1:
The patent segments the large set of historical invoices into multiple clusters based on similarity criteria (vendor, amount, date, line items). Each cluster represents a group of similar invoices. When a new invoice arrives, the system only compares it against relevant clusters rather than all historical invoices, dramatically reducing computational complexity while maintaining detection accuracy through targeted comparisons.
2Device complexity
If batch processing is used to perform duplicate detection comparisons, then computational load is reduced, but payment processing is delayed more than twenty-four hours
Solution Approach 1:
The patent performs preliminary actions by pre-processing and clustering historical invoices before they are needed for duplicate detection. The clustering is performed in advance, organizing invoices into similarity groups. When a new invoice arrives, the system can quickly determine which clusters to search, enabling real-time or near-real-time duplicate detection without the delays associated with batch processing all comparisons at once.
3Measurement precision
If manual review of every invoice is performed for duplication, then detection accuracy is improved, but invoice payment processing becomes unacceptably slow
Solution Approach 1:
The patent implements self-service duplicate detection through automated clustering and comparison algorithms. The system automatically groups historical invoices by similarity and performs targeted comparisons with new invoices without human intervention. This automated approach maintains high detection accuracy by systematically analyzing relevant invoices while processing volumes at machine speed, eliminating the need for slow manual review of every invoice.
Data Source
AI summary
Disclosed are various embodiments for real-time detection of duplicate data records. A duplicate detection application generates a set of clusters from a set of data records by grouping each data record in the set of data records according to similarity to respective centroid data records of the set of clusters. The duplicate detection application determines whether a particular data record has a potential duplicate in the set of data records by first comparing the particular data record to the respective centroid data records to identify a most similar cluster in the set of clusters. The duplicate detection application then compares the particular data record to each data record in the most similar cluster.