Automated Near-Duplicate Detection Through Character-Frequency Clustering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing computer systems struggle to accurately and efficiently identify near-duplicate documents in large libraries of text documents, consuming storage space and processing power while negatively affecting downstream processes.
Innovation Solution
A computer system processes text documents by transforming them using transliteration and Unicode encoding, clustering based on unique character frequencies, filtering with TF-IDF cosine similarity, and generating neighbor-character frequency matrices to identify duplicate and near-duplicate documents.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If existing computer systems process text documents to identify duplicates using traditional methods, then they can detect duplicate documents, but they consume excessive storage space and processing power while lacking accuracy and efficiency
Solution Approach 1:
The patent segments the document comparison process into multiple stages: first clustering documents using character frequencies (a lightweight feature), then applying more computationally intensive TF-IDF cosine similarity only within clusters. This hierarchical segmentation reduces the overall computational burden while maintaining detection accuracy.
Solution Approach 2:
The patent performs preliminary clustering of documents based on character frequencies before conducting detailed similarity analysis. This preliminary action groups potential duplicates together, so that subsequent expensive TF-IDF computations are only performed on small subsets of documents rather than all document pairs, significantly improving processing efficiency.
2Reliability
If existing computer systems process text documents to identify duplicates using traditional methods, then they can detect duplicate documents, but they consume excessive storage space and processing power
Solution Approach 1:
The patent segments the document comparison process into multiple stages: first clustering documents using character frequencies (a lightweight feature), then applying more computationally intensive TF-IDF cosine similarity only within clusters. This hierarchical segmentation reduces the overall computational burden while maintaining detection accuracy.
Solution Approach 2:
The patent performs preliminary clustering of documents based on character frequencies before conducting detailed similarity analysis. This preliminary action groups potential duplicates together, so that subsequent expensive TF-IDF computations are only performed on small subsets of documents rather than all document pairs, significantly improving processing efficiency.
3Productivity
If existing computer systems process text documents to identify duplicates using traditional methods, then they can detect duplicate documents, but it negatively affects downstream processes
Solution Approach 1:
The patent segments the document comparison process into multiple stages: first clustering documents using character frequencies (a lightweight feature), then applying more computationally intensive TF-IDF cosine similarity only within clusters. This hierarchical segmentation reduces the overall computational burden while maintaining detection accuracy.
Solution Approach 2:
The patent performs preliminary clustering of documents based on character frequencies before conducting detailed similarity analysis. This preliminary action groups potential duplicates together, so that subsequent expensive TF-IDF computations are only performed on small subsets of documents rather than all document pairs, significantly improving processing efficiency.
Data Source
AI summary
Techniques described herein provide for automated near-duplicate detection for new text documents given text documents that were previously processed using automated near-duplicate detection for text documents. In one example, a system can receive new documents and documents that were previously processed using a predefined processing technique for automated near-duplicate detection. The system can process the new documents and cluster the new documents into multiple predefined clusters previously identified using the predefined processing technique. For each predefined cluster including at least one new document, the system can generate document groups by determining similarity scores using the predefined processing technique as applied to the documents in the predefined clusters. The system can identify a representative document for each document group and generate an output data structure including the document groups and the representative document for each group.