Near-Duplicate Document Grouping With Structured Similarity Output
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing computer systems struggle to accurately and efficiently identify near-duplicate documents in large libraries of text documents, consuming storage space and processing power while negatively affecting downstream processes.
Innovation Solution
A computer system that transforms text documents using transliteration and Unicode encoding, clusters them based on unique character frequencies, filters using TF-IDF cosine similarity, and generates similarity scores from neighbor-character frequency matrices to identify duplicates and near-duplicates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional duplicate detection methods are used on large libraries of text documents, then storage space and processing power are consumed, but accuracy in identifying near-duplicate documents deteriorates
Solution Approach 1:
The patent segments the document comparison process into multiple stages: first computing lightweight similarity scores using character n-gram overlap, then applying more computationally intensive methods only to pairs that exceed a threshold. This hierarchical segmentation allows accurate identification of near-duplicates while minimizing overall processing power consumption by avoiding exhaustive comparison of all document pairs.
Solution Approach 2:
The patent applies partial action by computing similarity scores for only those document pairs that are likely to be duplicates based on initial filtering criteria, rather than computing scores for all possible pairs. This selective approach maintains detection accuracy for relevant cases while significantly reducing processing power consumption by skipping unnecessary computations.
2Measurement precision
If comprehensive similarity scoring is performed for all document pairs, then duplicate detection accuracy improves, but processing time increases
Solution Approach 1:
The patent performs preliminary actions by pre-computing character n-gram frequencies and similarity scores for all documents before conducting full duplicate detection. This preprocessing step creates reusable representations that enable rapid filtering and comparison, reducing the time required for comprehensive similarity scoring while maintaining detection accuracy.
Solution Approach 2:
The patent segments the similarity scoring process into a two-stage approach: first computing rapid character n-gram overlap scores to identify candidate duplicate pairs, then performing more accurate but time-consuming similarity assessments only on those candidates. This segmentation reduces overall processing time while preserving detection accuracy for true duplicates.
3Productivity
If no duplicate detection is performed, then processing resources are saved, but downstream processes are negatively affected by duplicate documents
Solution Approach 1:
The patent implements self-service by enabling the document processing system to automatically identify and flag near-duplicate documents without requiring manual intervention or extensive external processing. The system uses efficient algorithms that consume minimal resources to perform duplicate detection, thereby improving downstream process efficiency while avoiding the resource expenditure of more comprehensive analysis or manual review.
Data Source
AI summary
Techniques described herein provide for generation of structured output for documents identified using automated near-duplicate detection. In one example, a system can receive a set of documents including at least one pair of similar documents determined to be similar to one another based on similarity scores generated using a predefined similarity scoring technique. The system can generate document groups by merging together pairs of documents that share at least one document. The system can, for each of the document groups, identify a representative document for the document group. The system can generate an output for display including a section for each document group, in which each section includes the representative document for the document group and, for each document in the document group, the similarity score relative to the representative document for the document group.