Near-Duplicate Text Comparison Using Boilerplate Filtering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing computer systems struggle to accurately and efficiently identify near-duplicate documents in large libraries of unstructured text, consuming storage space and processing power while negatively affecting downstream processes.
Innovation Solution
A computer system employs a process involving text transformation, clustering based on unique character frequencies, filtering using TF-IDF cosine similarity, and neighbor-character frequency matrix analysis to identify duplicate and near-duplicate documents, utilizing transliteration and Unicode encoding to handle diverse languages.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional text comparison methods are used to identify duplicate documents in large libraries, then comprehensive document analysis can be performed, but the system consumes excessive storage space and processing power while operating slowly
Solution Approach 1:
The patent segments text documents into character n-grams (substrings of specified lengths) to create compact representations. Instead of comparing entire documents, the system divides them into overlapping character sequences (e.g., 5-grams, 10-grams) that serve as fingerprints for rapid comparison, enabling efficient duplicate detection in large document libraries
Solution Approach 2:
The patent transforms text documents into different parameter representations including character frequency vectors and n-gram frequency distributions. By changing the representation parameters from raw text to frequency-based numerical vectors, the system enables rapid mathematical comparisons using cosine similarity while maintaining detection accuracy
2Measurement precision
If traditional text comparison methods are used to identify duplicate documents, then comprehensive analysis can be performed, but the system consumes excessive processing power and resources
Solution Approach 1:
The patent segments text documents into character n-grams to create compact representations. Instead of comparing entire documents, the system divides them into overlapping character sequences (e.g., 5-grams, 10-grams) that serve as fingerprints for rapid comparison, enabling efficient duplicate detection in large document libraries
Solution Approach 2:
The patent replaces traditional mechanical text comparison (character-by-character or word-by-word analysis) with mathematical vector operations. By representing documents as frequency vectors and using cosine similarity calculations, the system substitutes computationally intensive string matching with efficient linear algebra operations
3Quantity of substance
If duplicate documents are not identified and removed, then all documents can be processed, but storage space is wasted and downstream processes are negatively affected
Solution Approach 1:
The patent performs preliminary duplicate detection and filtering before documents enter downstream processing pipelines. By identifying and removing duplicate documents in advance using character n-gram comparison and frequency vector analysis, the system prevents wasted storage space and eliminates redundant processing in subsequent analytical workflows
Data Source
AI summary
Techniques described herein provide for text string comparison for documents identified using automated near-duplicate detection. In one example, a system can receive a pair of documents. The system can extract text strings from the documents. The system can normalize the extracted text strings using a predefined normalization scheme. The system can identify boilerplate text segments in the normalized text strings. The system can remove the boilerplate text segments from the normalized text strings to generate filtered text strings. The system can divide the filtered text strings by identifying section indicators. The system can, for each section, generate groupings of text strings and determine a similarity score between each pair of corresponding groupings to identify matching groupings of text strings. The system can generate an output for display showing the visual indications of the matched groupings of text strings.