URL Canonicalization for Search Crawler Efficiency

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Search engines face inefficiencies and increased costs due to redundant indexing caused by different URLs referring to the same content, which affects crawling quality and feasibility.

Innovation Solution

A method is introduced to receive a URL with parameters, identify parameter combinations that do not affect content, and generate URL rewrite rules to convert URLs into a canonical form, reducing redundancy by removing unnecessary parameters.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If the spider program crawls all URLs with different parameter combinations, then it ensures comprehensive content coverage, but it causes redundant indexing and significantly increases crawling cost and time

Engineering Contradiction:

Improvecontent coverage completenessVSAvoidcrawling efficiency

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The system changes the parameter representation of URLs by identifying and removing redundant parameters. It analyzes parameter combinations across multiple URLs, determines which parameters do not affect content, and transforms URLs into a canonical form with only essential parameters, thereby reducing redundancy while preserving content coverage

Inventive Principle:

Principle #35Parameter changes

Solution Approach 2:

The system creates a canonical copy of URLs by generating a standardized representation that eliminates redundant parameters. Instead of processing every variant URL, it creates and processes a single canonical version that represents all equivalent URLs, significantly reducing the number of URLs to be crawled and indexed

Inventive Principle:

Principle #26Copying

2Measurement precision

If the spider program indexes all different URLs referring to the same content, then it maintains accurate URL-to-content mapping, but it leads to decreased search engine quality and increased storage requirements

Engineering Contradiction:

ImproveURL-content mapping accuracyVSAvoidindexed data volume

Core Design Contradiction:

Measurement precisionVSQuantity of substance

Solution Approach 1:

The system merges multiple URLs that refer to the same content into a single canonical URL representation. By identifying URLs with different parameter combinations that resolve to the same content, it consolidates them into one standardized form, reducing the total number of indexed entries while preserving the URL-content relationship accuracy

Inventive Principle:

Principle #5Merging (Combining)

3Adaptability or versatility

If the system processes URLs with redundant parameters, then it maintains compatibility with dynamic web site structures, but it significantly increases the time and cost involved in crawling

Engineering Contradiction:

Improvedynamic site compatibilityVSAvoidcrawling time

Core Design Contradiction:

Adaptability or versatilityVSLoss of time

Solution Approach 1:

The system performs preliminary analysis of URL parameter patterns before full crawling begins. It pre-identifies redundant parameters by analyzing URL structures and content mappings in advance, then applies this knowledge to efficiently process subsequent URLs, reducing the time required for crawling dynamic sites while maintaining adaptability

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS7827254B1Automatic generation of rewrite rules for URLs

Publication Date: 2010.11.02 GOOGLE LLC

AI summary

A rewrite component automatically generates rewrite rules that describe how uniform resource locators (URLs) can be rewritten to reduce or eliminate different URLs that redundantly refer to the same or substantially the same content. The rewrite rules can be applied to URLs received when crawling a network to increase the efficiency of the crawl and the corresponding document index generated from the crawl.