Web Crawler Session ID Detection via URL Comparison

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Web crawlers inefficiently identify and handle web sites that use session identifiers, leading to repeated crawling of the same pages and duplication in search engine indexes due to changing session identifiers in URLs.

Innovation Solution

A method and device that compare URLs from multiple copies of web documents to determine if a site uses session identifiers, allowing for the extraction of session identifier rules and enhancement of web crawling by identifying and managing session identifiers.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If a web spider crawls URLs from a web site that uses session identifiers, then the spider can access and index web pages, but the spider repeatedly crawls the same pages because session identifiers cause URL variations, wasting time and bandwidth and filling the index with duplicate pages

Engineering Contradiction:

Improveweb crawling efficiencyVSAvoidtime spent on duplicate crawling

Core Design Contradiction:

ProductivityVSLoss of time

Solution Approach 1:

The system performs preliminary detection of session identifier usage by comparing URLs from multiple copies of web documents before actual crawling begins. By identifying web sites that use session identifiers in advance, the spider can prepare appropriate handling strategies (such as normalizing URLs or adjusting crawl logic) to prevent repeated crawling of the same pages.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system uses feedback from comparing URL variations across multiple document copies to detect session identifier usage. This feedback mechanism allows the spider to identify patterns in URL changes and adapt its crawling behavior accordingly, avoiding redundant processing of pages that would result from treating session-identifier-based URL variations as distinct pages.

Inventive Principle:

Principle #23Feedback

2Reliability

If session identifiers are embedded in URLs to track user behavior, then user tracking is enabled, but web crawlers cannot properly identify duplicate pages and repeatedly crawl the same content

Engineering Contradiction:

Improveuser tracking accuracyVSAvoidweb crawling efficiency

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The system segments the URL into distinct components, separating the session identifier portion from the base URL. By detecting which portion varies between copies of the same web document, the system can identify session identifiers as a separate element and handle them appropriately, allowing accurate user tracking while preventing duplicate crawling of the same base pages.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system introduces an intermediary detection mechanism that compares URL variations across multiple document copies to identify session identifiers. This intermediary process acts as a mediator between the session tracking requirement and the crawling efficiency requirement, enabling the system to recognize when URL variations are due to session identifiers rather than actual different pages.

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS7886217B1Identification of web sites that contain session identifiers

Publication Date: 2011.02.08 GOOGLE LLC

AI summary

Web sites are analyzed to determine whether the web sites are embedding session identifiers in web documents. The analysis is based on a comparison of in-host links of multiple copies of a document from a web site. Rules governing the insertion of session identifiers for the web site may be determined and used to assist in crawling the web site.