Web Crawler Near-Duplicate Detection via Hash Comparison
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Web crawlers face inefficiencies due to the presence of near-duplicate documents on the web, which consume resources and require handling of various forms and versions of documents, leading to unnecessary resource utilization.
Innovation Solution
A system that generates hash values for documents and compares them with stored values to identify near-duplicates by determining bit positions that match, allowing for the identification and discarding of near-duplicate documents to conserve resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If web crawlers process all fetched documents, then comprehensive information collection is achieved, but resource consumption increases due to near-duplicate documents
Solution Approach 1:
The system performs preliminary action by generating hash values and comparing them against a database of previously seen document hashes before fully processing a document. This preliminary check identifies near-duplicate documents early in the crawling process, allowing the system to skip resource-intensive processing of redundant content while maintaining comprehensive information collection of unique documents.
Solution Approach 2:
The invention extracts only the essential identifying feature (hash value) from each document for comparison purposes, rather than processing the entire document content. By taking out just the hash value for duplicate detection, the system efficiently identifies near-duplicates without consuming resources on processing complete duplicate documents, thus resolving the contradiction between information completeness and resource consumption.
2Reliability
If web crawlers store complete document content, then detailed analysis is enabled, but storage requirements increase due to near-duplicate documents
Solution Approach 1:
The system uses hash values as compact representations (copies) of the complete document content for storage and comparison purposes. Instead of storing full duplicate documents, only their hash values are retained in the database, enabling efficient duplicate detection while maintaining the ability to verify document identity. This copying approach preserves analysis reliability for unique documents while dramatically reducing storage requirements.
3Loss of information
If web crawlers process every fetched document, then thorough indexing is achieved, but processing time increases due to near-duplicate documents
Solution Approach 1:
The system performs preliminary hash-based duplicate detection before initiating the time-consuming indexing process. By checking if a document's hash value matches existing entries in the database, the system identifies near-duplicates early and skips the lengthy indexing operation for those documents, thereby reducing overall processing time while maintaining thorough indexing of unique documents.
Solution Approach 2:
The invention applies partial action by performing only the essential duplicate detection step (hash comparison) for all fetched documents, and only performing the full indexing process for documents confirmed to be unique. This selective approach avoids the excessive processing time that would result from indexing every fetched document, including near-duplicates, while still achieving complete indexing of novel content.
Data Source
AI summary
A system generates a hash value for a fetched document and compares the hash value with a set of stored hash values to identify ones of the stored hash values with a sequence of bit positions, less than all of the bit positions, that match a corresponding sequence of bit positions of the hash value. The system also determines whether any of the identified hash values are substantially similar to the hash value and identify the fetched document as a near-duplicate of another document when one of the identified hash values is substantially similar to the hash value.