Document Identifier Clustering for Duplicate Content Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Search engines waste resources by fetching duplicate copies of dynamically-generated web pages that share the same content, as they lack a system to identify and manage document identifiers referencing the same content.
Innovation Solution
A system partitions document identifiers into clusters, generates equivalence rules to classify content-relevant and content-irrelevant parameters, and groups them into equivalence classes, allowing for the identification and management of duplicate documents to prevent unnecessary fetching.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If web crawlers fetch all document identifiers without identification, then complete document coverage is achieved, but resource waste increases significantly
Solution Approach 1:
The system performs preliminary clustering and equivalence rule generation on document identifiers before the actual crawling process. By pre-organizing document identifiers into clusters and establishing equivalence rules that identify content-irrelevant parameters, the system prepares the framework for efficient duplicate detection in advance, avoiding resource waste during the crawling phase while ensuring complete document coverage.
2Measurement precision
If document identifiers are clustered and equivalence rules are generated, then duplicate detection accuracy improves, but system complexity increases
Solution Approach 1:
The system segments the document identifier management process into distinct components: clustering document identifiers by structural similarity, generating equivalence rules for each cluster, and applying these rules during crawling. This segmentation transforms a complex monolithic system into manageable modular components, where each cluster-processing unit operates independently with its own equivalence rules, reducing overall system complexity while maintaining high duplicate detection accuracy.
3Quantity of substance
If equivalence classes are created with representative document identifiers, then storage efficiency improves, but processing overhead increases
Solution Approach 1:
The system merges multiple document identifiers that reference the same content into a single equivalence class represented by one representative document identifier. During crawling, when a new document identifier is encountered, the system checks if it belongs to an existing equivalence class and merges it accordingly, rather than treating it as a separate entity. This merging dramatically reduces the quantity of document identifiers that need to be stored and processed, while the merging operation itself is efficient due to the pre-established equivalence rules.
Data Source
AI summary
A system of reducing the possibility of crawling duplicate document identifiers partitions a plurality of document identifiers into multiple clusters, each cluster having a cluster name and a set of document parameters. The system generates an equivalence rule for each cluster of document identifiers, the rule specifying which document parameters associated with the cluster are content-relevant. Next, the system groups each cluster of document identifiers into one or more equivalence classes in accordance with its associated equivalence rule, each equivalence class including one or more document identifiers that correspond to a document content and having a representative document identifier identifying the document content.