Web Crawler URL Selection Using Human Interaction Data

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

The existing web crawling methods are inefficient due to the presence of 'link farms' that distort web page rankings and waste resources by crawling interlinked pages not useful to humans, as they rely on link popularity algorithms rather than human interaction data.

Innovation Solution

A system and method that utilizes a web crawler with a URL selection module that prioritizes URLs with associated interaction data, such as human interaction records, to download and index web pages, focusing on pages with high human interaction and relevance, thereby filtering out irrelevant content.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If link popularity algorithms are used to rank web pages, then web page rankings can be determined, but resource waste increases and crawling efficiency decreases due to link farms

Engineering Contradiction:

Improvecrawling efficiencyVSAvoidresource waste

Core Design Contradiction:

ProductivityVSLoss of energy

Solution Approach 1:

The patent changes the ranking parameter from link popularity (outbound links) to human interaction data (inbound interactions such as clicks, views, or other user engagement metrics). This parameter transformation allows the system to prioritize pages that actually engage users over pages that merely have many outbound links, thereby reducing resource waste on link farms while maintaining crawling productivity.

Inventive Principle:

Principle #35Parameter changes

Solution Approach 2:

The system incorporates feedback from human interaction data into the crawling and ranking process. By monitoring and utilizing feedback signals from user interactions (such as which pages are clicked, viewed, or otherwise engaged with), the system dynamically adjusts its crawling strategy to focus on pages with genuine human interest, eliminating the need to waste resources on artificially linked pages.

Inventive Principle:

Principle #23Feedback

2Measurement precision

If link popularity algorithms are used, then web pages can be ranked, but the accuracy of rankings deteriorates due to manipulation by link farms

Engineering Contradiction:

Improveranking accuracyVSAvoidlink farm manipulation

Core Design Contradiction:

Measurement precisionVSObject-generated harmful factors

Solution Approach 1:

The patent converts the harmful effect of link farms (artificial link structures) into a beneficial filtering mechanism. By switching from link-outbound-based ranking to link-inbound human interaction-based ranking, the system naturally filters out link farm pages that lack genuine human engagement, while still utilizing link data to discover and crawl pages. This transforms the harmful manipulation into a useful signal for identifying authentic content.

Inventive Principle:

Principle #22Blessing in disguise (Convert harm into benefit)

Solution Approach 2:

The system introduces human interaction data as an intermediary factor between link structure and ranking. Instead of directly using link popularity as the ranking signal, the system uses human interaction data as a mediator that validates whether a page is genuinely useful. This intermediary layer filters out artificial link structures while maintaining the ability to discover pages through linking.

Inventive Principle:

Principle #24Intermediary (Mediator)

3Quantity of substance

If all URLs are crawled to ensure comprehensive coverage, then complete web indexing is achieved, but crawling time and resources increase significantly

Engineering Contradiction:

Improveweb pages indexedVSAvoidcrawling time

Core Design Contradiction:

Quantity of substanceVSLoss of time

Solution Approach 1:

The patent applies partial action by selectively crawling only the subset of URLs that have demonstrated human interaction data, rather than attempting to crawl every URL on the web. This partial crawling strategy focuses resources on pages that are likely to be useful to users, achieving sufficient coverage for search purposes without the time and resource expenditure required for complete web indexing.

Inventive Principle:

Principle #16Partial or excessive action

Solution Approach 2:

The system performs preliminary action by pre-identifying and prioritizing URLs based on existing human interaction data before initiating the crawling process. By using interaction data as a pre-filter or pre-rank, the system determines which pages to crawl first and which are less important, thereby reducing overall crawling time while maintaining comprehensive coverage of relevant content.

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS9940391B2System, method and computer readable medium for web crawling

Publication Date: 2018.04.10 ORACLE AMERICAN INC

AI summary

In a web crawler, a URL selection module selects URLs for pages to be downloaded. The URL selection module accesses an interaction data store that stores interaction data for web pages, including interaction data that indicates human interactions with the pages. To reduce the effects of link farms, the URL selection module filters the URLs to select only those URLs that have human interaction histories and provides the selected URLs to a download module for web page downloading.