Proxy Pad Score for Duplicate Document Clustering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing web-based information retrieval systems face challenges in handling duplicate content, which wastes resources and degrades user experience by presenting multiple documents with the same content, and struggle to distinguish between legitimate and spam content.
Innovation Solution
An automated method and system that identifies clusters of duplicate documents, determines quality scores, and generates a proxy pad score to identify and avoid indexing spam content, ensuring that only high-quality, unique documents are indexed and served as search results.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If the indexer indexes all crawled documents, then the index contains comprehensive information, but duplicate documents waste space and degrade user experience
Solution Approach 1:
The patent extracts and identifies duplicate documents from the crawled set using clustering algorithms, then excludes them from indexing. The system compares each crawled document against identified clusters, determines if it is a duplicate, and selectively indexes only unique documents, thereby removing the harmful duplicate content while preserving comprehensive unique information.
2Loss of energy
If the indexer selects one document from duplicate clusters to index, then resource waste is reduced, but it is difficult to distinguish between legitimate and spam content
Solution Approach 1:
The patent introduces proxy pad sites as intermediary entities that aggregate spam content from multiple sources. By identifying these intermediary proxy pad sites through clustering analysis and quality score comparison, the system can detect and exclude spam content effectively. The proxy pad sites serve as mediators that reveal the presence of coordinated spam operations, enabling the indexer to make informed decisions about which documents to exclude.
3Productivity
If the system identifies and avoids indexing duplicate documents, then resource efficiency improves, but the complexity of detecting duplicates increases
Solution Approach 1:
The patent segments the duplicate detection process into distinct phases: first identifying clusters of potentially duplicate documents using efficient clustering algorithms, then comparing each crawled document against these pre-formed clusters to determine duplicity. This segmentation transforms the complex O(N^2) comparison problem into a more manageable process, improving indexing efficiency while maintaining accurate duplicate detection.
Data Source
AI summary
A system may identify a set of first documents associated with an organization, and identify clusters to which the first documents belong. Each of a number of the identified clusters may include a group of documents that includes one of the first documents and one or more second documents associated with one or more different organizations. The system may determine a quality score for each of the documents in each of the identified clusters, and determine, for each of the number of the identified clusters, whether the quality score of the one of the first documents in the identified cluster is higher than the quality score of the one or more second documents in the identified cluster. The system may generate a proxy pad score based on the determinations, and store the proxy pad score.