Predictive Clustering for Duplicate Document Redirection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Crawler-based search engines face challenges in handling duplicate content on the web, leading to resource wastage and suboptimal search results, as they struggle to identify and manage multiple versions of the same documents.
Innovation Solution
An automated method and system that clusters documents, selects representative documents, determines final redirect targets, and re-clusters if the targets are duplicates, using predictive-based clustering techniques to identify and merge duplicate documents into a single canonical index.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the indexer indexes duplicate documents, then the search results may contain redundant information, but the resource consumption increases and the quality of search results deteriorates
Solution Approach 1:
The patent merges duplicate documents into clusters and selects a single representative document from each cluster to index. This consolidates multiple duplicate documents into one canonical document, eliminating redundant indexing while preserving the information content of all duplicates through the representative selection process.
Solution Approach 2:
The patent extracts and selects only the most relevant representative document from each cluster of duplicate documents for indexing. By taking out just the representative document rather than indexing all duplicates, the system reduces resource consumption while maintaining search result quality.
2Loss of energy
If the indexer selects a single representative document from duplicate clusters, then resource consumption is reduced, but the complexity of determining which document is representative increases
Solution Approach 1:
The patent employs a feedback mechanism where the system iteratively refines its clustering and representative selection. The algorithm adjusts its clustering based on feedback from quality measurements and redirect target analysis, progressively improving the selection of representative documents while managing computational complexity through iterative refinement.
Solution Approach 2:
The patent performs preliminary actions by pre-processing documents into initial clusters before final representative selection. This preliminary clustering organizes documents into manageable groups, making the subsequent representative selection process more efficient and less complex than analyzing all documents simultaneously.
3Measurement precision
If the system uses predictive-based clustering to identify duplicate documents, then the accuracy of duplicate detection improves, but the processing time and computational resources increase
Solution Approach 1:
The patent performs preliminary clustering actions to organize documents before applying predictive-based duplicate detection. This preliminary organization groups similar documents together, allowing the more computationally intensive predictive analysis to focus only on potential duplicates within clusters rather than all documents, thereby reducing overall processing time.
Solution Approach 2:
The patent segments the document processing into distinct phases: initial clustering, representative selection, and predictive duplicate detection. By segmenting the process, the system can apply less computationally intensive methods initially and reserve predictive analysis for specific critical comparisons, balancing accuracy with processing time.
Data Source
AI summary
An automated method may include clustering documents into a plurality of clusters, selecting a representative document, respectively, for two of the plurality of clusters, determining a final redirect target document for each of the representative documents in the two of the plurality of clusters, determining whether the final redirect target documents are duplicate documents of each other; and re-clustering the two of the plurality of clusters together if it is determined that the final redirect target documents are duplicate documents.