URL Normalization Rules for Web Crawler Duplicate Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Web crawlers face inefficiencies due to duplicative downloads of substantially identical web pages accessed through different URLs, wasting bandwidth and computational resources.
Innovation Solution
A method to identify and normalize URLs by comparing web resources, determining essential and irrelevant components, and applying rule-learning steps to generalize equivalence-class rules for reducing duplicative downloads.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If web crawlers download all pages referenced by URLs of a web site, then complete web site coverage is achieved, but duplicative downloads waste bandwidth and computational resources
Solution Approach 1:
The system performs preliminary analysis of URL patterns and document content before crawling to identify and normalize equivalent URLs. By pre-establishing normalization rules based on historical data and pattern recognition, the crawler can predict and avoid duplicative downloads in advance, rather than detecting duplicates after downloading
Solution Approach 2:
The system implements feedback mechanisms where downloaded documents are analyzed for content equivalence, and this information is used to refine URL normalization rules. The crawler continuously learns from past crawling experiences, adjusting its URL normalization strategy based on feedback about which URLs actually point to equivalent content
2Loss of energy
If web crawlers download only one instance of identical documents, then resource usage is optimized, but identifying URL equivalence patterns requires complex analysis
Solution Approach 1:
The URL analysis process is segmented into distinct components: pattern recognition, content hashing, equivalence class formation, and rule generalization. Each segment handles a specific aspect of the analysis, making the overall complex task manageable through modular processing stages
Solution Approach 2:
The system introduces intermediary structures such as equivalence classes and normalization rules that mediate between raw URLs and final download decisions. These intermediaries simplify the comparison process by grouping equivalent URLs together before the actual download decision is made
Data Source
AI summary
Different URLs that actually reference the same web page or other web resource are detected and that information is used to only download one instance of a web page or web resource from a web site. All web pages or web resources downloaded from a web server are compared to identify which are substantially identical. Once identical web pages or web resources with different URLs are found, the different URLs are then analyzed to identify what portions of the URL are essential for identifying a particular web page or web resource, and what portions are irrelevant. Once this has been done for each set of substantially identical web pages or web resources (also referred to as an “equivalence class” herein), these per-equivalence-class rules are generalized to trans-equivalence-class rules. There are two rule-learning steps: step (1), where it is learned for each equivalence class what portions of the URLs in that class are relevant for selecting the page and what portions are not; and step (2), where the per-equivalence-class rules constructed during step (1) are generalized to rules that cover many equivalence classes. Once a rule is determined, it is applied to the class of web pages or web resources to identify errors. If there are no errors, the rule is activated and is then used by the web crawler for future crawling to avoid the download of duplicative web pages or web resources.