Near-Duplicate Text Detection via Clustering and Matrix Scoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large libraries of text documents often contain duplicate or near-duplicate documents, which consume storage space, require processing power, and can negatively affect downstream processes, making it challenging for computer systems to accurately identify near-duplicate documents.
Innovation Solution
A process involving transformation, clustering, filtering, feature extraction, and matrix similarity scoring, where text documents are transformed using transliteration and Unicode encoding, clustered based on character frequencies, filtered using TF-IDF values, and converted into neighbor-character frequency matrices to generate similarity scores, identifying documents as duplicates or near-duplicates if scores meet predefined thresholds.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional duplicate detection methods are used on large libraries of text documents, then storage space is consumed and processing power is required, but the accuracy of identifying near-duplicate documents deteriorates
Solution Approach 1:
The patent segments the duplicate detection process into multiple stages: initial clustering using character frequencies to group potentially similar documents, then pairwise comparison only within clusters using TF-IDF vectors, and finally matrix representation for near-duplicate identification. This segmentation reduces the overall computational complexity from O(n²) comparisons to O(k₁ + k₂ + ... + kₘ) where each kᵢ is the smaller number of comparisons within cluster i.
Solution Approach 2:
The patent performs preliminary clustering of documents based on character frequencies before conducting detailed pairwise similarity comparisons. This preliminary action groups documents that are likely to be similar, so that subsequent expensive TF-IDF computations are only performed on documents within the same cluster rather than on all document pairs, significantly reducing processing requirements.
2Measurement precision
If comprehensive similarity scoring is performed on all document pairs, then identification accuracy improves, but processing time and computational resources increase
Solution Approach 1:
The patent divides the document set into clusters based on character frequencies, then performs comprehensive similarity scoring only within each cluster rather than across all documents. This segmentation maintains high identification accuracy for near-duplicates while reducing processing time by limiting pairwise comparisons to smaller subsets.
Solution Approach 2:
The patent applies different processing strategies to different groups of documents: character frequency analysis for initial clustering, TF-IDF vectors for pairwise comparison within clusters, and matrix representation for final near-duplicate identification. Each document pair receives the appropriate level of analysis based on its cluster membership and similarity score, optimizing both accuracy and efficiency.
3Productivity
If character frequency-based clustering is used to group documents, then processing efficiency improves by reducing comparison scope, but measurement precision of similarity detection may deteriorate
Solution Approach 1:
The patent uses character frequency-based clustering as a first-stage segmentation that efficiently groups documents with similar character distributions. This initial clustering reduces the search space for subsequent similarity detection without sacrificing accuracy, because documents with different character frequencies are unlikely to be near-duplicates regardless of semantic similarity.
Solution Approach 2:
The patent introduces TF-IDF vectors as an intermediary representation between character frequency clustering and final similarity scoring. The TF-IDF vectors capture term-level information that complements the character-level information from clustering, providing a more nuanced basis for detecting near-duplicates while maintaining the efficiency gains from clustering.
Data Source
AI summary
Techniques described herein provide for automated detection of near-duplicate documents. In one example, a system can cluster documents into a set of clusters based on character frequencies associated with the documents. For a given cluster, the system can generate first similarity scores associated with every pair of documents in the cluster. The system can then select a filtered group of documents associated with first similarity scores that meet or exceed a first predefined similarity threshold. Next, the system can convert the filtered group of documents into matrix representations. The system can generate second similarity scores for every pair of matrix representations. The system can then identify documents, from among the filtered group of documents, associated with second similarity scores that meet or exceed a second predefined similarity threshold. The identified documents can be duplicate or near-duplicate text documents.