Web Crawler Robots.txt Analysis for Illegitimate Site Detection

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current web crawlers face challenges in identifying illegitimate websites that engage in link spam, phishing, or infringe intellectual property rights, as they lack effective methods to differentiate between legitimate and illegitimate web pages, particularly when illegitimate sites attempt to manipulate search engine rankings or obscure their content.

Innovation Solution

A system comprising a requester module, analysis module, and record module that requests and analyzes 'robots.txt' rules from host servers to determine if a website is illegitimate by identifying discrepancies in access instructions for different types of robot computer programs, and storing indications of illegitimate websites for further action.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Loss of information

If web crawlers access all web pages to gather information, then information coverage is improved, but resource consumption increases and illegitimate pages cannot be effectively filtered

Engineering Contradiction:

Improveinformation coverageVSAvoidresource consumption

Core Design Contradiction:

Loss of informationVSLoss of energy

Solution Approach 1:

The system performs preliminary analysis of robots.txt rules before crawling web pages. By requesting and analyzing robots.txt files in advance, the crawler identifies illegitimate websites and excludes them from further crawling, preventing waste of computational resources on malicious sites while ensuring legitimate pages are processed

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The robots.txt file serves as an intermediary mechanism between the crawler and the target website. The analysis module examines this intermediary file to detect discrepancies that indicate illegitimate websites, enabling filtering decisions without directly accessing or processing the actual web page content

Inventive Principle:

Principle #24Intermediary (Mediator)

2Measurement precision

If web crawlers analyze robots.txt rules to identify illegitimate websites, then detection accuracy is improved, but system complexity increases

Engineering Contradiction:

Improvedetection accuracyVSAvoidsystem complexity

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The crawling system is segmented into distinct functional modules: a requester module that retrieves robots.txt files, an analysis module that examines the rules for discrepancies, and a crawler module that executes crawling based on analysis results. This segmentation allows complex detection logic to be isolated and managed separately from the core crawling functionality

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The robots.txt file acts as an intermediary that contains the detection logic externally. Rather than embedding complex detection algorithms within the crawler itself, the system leverages the existing robots.txt infrastructure and analyzes its contents for suspicious patterns, reducing the complexity of the crawler's internal logic

Inventive Principle:

Principle #24Intermediary (Mediator)

3Loss of information

If illegitimate websites are not identified, then search result completeness is improved, but user safety deteriorates due to phishing and malicious content

Engineering Contradiction:

Improvesearch result completenessVSAvoiduser safety

Core Design Contradiction:

Loss of informationVSObject-affected harmful factors

Solution Approach 1:

The system converts the harmful behavior of illegitimate websites attempting to manipulate search engines into a beneficial detection mechanism. By analyzing robots.txt rules that these sites use to control crawler access, the system identifies discrepancies that reveal malicious intent, thereby turning the sites' own attempts at deception into evidence of their illegitimacy

Inventive Principle:

Principle #22Blessing in disguise (Convert harm into benefit)

Solution Approach 2:

The system performs preliminary identification of illegitimate websites before they can harm users. By analyzing robots.txt rules in advance and flagging suspicious patterns, the system prevents malicious sites from being included in search results, protecting users before they can be exposed to phishing or malicious content

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS10104116B2System for detecting link spam, a method, and an associated computer readable medium

Publication Date: 2018.10.16 MAJESTIC 12

AI summary

A system for determining whether a website is an illegitimate website, the system comprising: a requester module configured to request one or more rules from a host server for a website and to receive a response from the host server in response to a request; an analysis module configured to determine whether a response or lack of a response received by the requester module indicates that the website is an illegitimate website; and a record module configured to store an indication that the website is an illegitimate website, wherein the one or more rules provide one or more instructions to a robot computer program regarding access of the website by the robot computer program.