Web Crawler Trap Detection via Date Pattern Regular Expressions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional web crawlers face inefficiencies and increased storage and load issues due to the collection of unnecessary web pages caused by infinite hyperlink loops in web-based calendar pages, leading to poor performance and resource consumption.
Innovation Solution
A method and system that cluster URLs based on predetermined standards, generate regular expressions from date patterns, and identify web traps by analyzing clustering results to detect and delete unnecessary web pages, thereby preventing infinite linking and optimizing web crawler operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If web crawlers collect all hyperlinks from web-based calendar pages, then comprehensive web page collection is achieved, but storage space is consumed and performance deteriorates due to infinite hyperlink loops
Solution Approach 1:
The patent applies preliminary action by generating regular expressions from date patterns before the web crawling process. These pre-generated patterns are used to identify and filter out calendar-related hyperlinks in advance, preventing the crawler from entering infinite loops and collecting unnecessary pages, thus improving performance while maintaining comprehensive collection of legitimate web pages
Solution Approach 2:
The patent extracts and removes the harmful component (calendar-related hyperlinks) from the web page collection process. By using regular expressions to identify and exclude URLs matching calendar date patterns, the system separates useful web pages from harmful infinite loop links, resolving the contradiction between comprehensive collection and performance
2Adaptability or versatility
If web-based calendar pages include hyperlinks to previous and subsequent months and years, then navigation functionality is improved, but infinite hyperlink loops are generated causing web traps
Solution Approach 1:
The patent converts the harmful infinite hyperlink loops into a detectable pattern. By analyzing the structure of calendar hyperlinks and generating regular expressions from their date patterns, the system transforms the previously harmful navigation links into identifiable targets that can be filtered out, while preserving the legitimate navigation functionality for users
3Manufacturing precision
If conventional web crawlers process all collected web pages, then complete indexing is achieved, but load on search engine increases due to unnecessary pages
Solution Approach 1:
The patent applies preliminary filtering by integrating regular expression-based calendar trap detection into the web crawling process itself. This preliminary action identifies and excludes calendar-related URLs before they are processed by the search engine, ensuring complete indexing of legitimate pages while reducing search engine load by preventing processing of unnecessary calendar pages
Data Source
AI summary
The present disclosure relates to a method, system and software executable by a processor associated with non-transitory computer-readable storage medium for detecting a trap of web-based calendar pages and building a retrieval database. According to an aspect of the disclosure, detecting a trap of web-based calendar pages includes clustering, by a clustering module, URLs corresponding to web pages stored in a database according to a predetermined standard, generating a regular expression by analyzing a date pattern included in a clustering result, and detecting, a cluster suspected of being a trap of web-based perpetual calendar pages using the generated regular expression.