Transaction Record Deduplication With Targeted Positional Fuzzy Matching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing record deduplication processes in financial transaction systems fail to effectively utilize structural layout and relative positional differences within documents for accurate matching and deduplication, leading to inefficiencies and errors.
Innovation Solution
Implement a targeted fuzzy matching process that computes relative positional differences between documents, using a similarity function to generate a difference similarity vector and aggregate it into a final score for document-level similarity, enhancing the accuracy of deduplication by focusing on positional differences.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional fuzzy matching is used for deduplication, then the process is simple to implement, but accuracy deteriorates due to extraction errors and lack of positional information
Solution Approach 1:
The patent segments the document comparison process into distinct components: extracting structural elements (headers, line items, totals), computing positional differences for each element, and aggregating these into a composite similarity score. This segmentation allows the system to handle complexity in a structured way while maintaining high accuracy by considering multiple document features separately and combining them systematically.
Solution Approach 2:
The patent introduces a new dimension to traditional fuzzy matching by incorporating positional information. Instead of only comparing text content, the system computes positional differences between corresponding elements in duplicate documents and integrates this spatial dimension into the similarity calculation, thereby significantly improving deduplication accuracy.
2Measurement precision
If structural layout information is utilized for matching, then deduplication accuracy improves, but computational complexity increases
Solution Approach 1:
The patent applies local quality by focusing computational effort on specific critical regions of documents rather than processing the entire document uniformly. It identifies and prioritizes structural elements such as headers, line items, and totals that are most indicative of document identity, computing positional differences only for these key areas. This selective approach maintains high accuracy while reducing overall computational complexity.
3Measurement precision
If relative positional differences are computed for all elements, then deduplication precision improves, but processing time increases
Solution Approach 1:
The patent extracts and computes positional differences only for specific critical structural elements (headers, line items, totals) rather than all elements in the document. By taking out only the most diagnostically valuable positional information, the system achieves high deduplication precision while avoiding the excessive processing time that would result from analyzing every element.
Data Source
AI summary
A computer-implemented method is disclosed. The method includes obtaining a candidate pair of a plurality of digitally stored documents from a document database. The method automatically computes relative positional differences between each digitally stored document of the candidate pair. The method compares the relative positional differences with a similarity function to form a difference similarity vector for the candidate pair. The difference similarity vector includes components corresponding to each relative positional difference. The method aggregates the components of the difference similarity vector to determine a final score for the candidate pair. The method determines a document-level similarity metric from the final score. The method determines whether the document-level similarity metric is above a threshold value and then classifies the candidate pair based on determining that the document-level similarity metric is above the threshold value to de-duplicate the plurality of digitally stored documents in the candidate pair.