Web Crawler Scheduling via Dynamic Priority Buckets
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing web crawler systems face inaccuracies in crawl frequency settings due to the need to pre-determine keyword popularity levels and fail to distinguish between different types of link addresses in the initial request process, leading to incomplete or repeated crawls on frequently updating websites.
Innovation Solution
A method and system for scheduling web crawlers using virtual buckets with different priority levels, where link addresses are stored based on their popularity levels, and the quantities of scheduled tasks are adjusted dynamically based on actual page quantities, ensuring that popular keywords are crawled frequently and unpopular keywords are not crawled repeatedly.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If keyword popularity levels are pre-determined to set crawl frequency, then crawl scheduling can be established, but crawl frequency accuracy deteriorates due to inability to reflect real-time information update rates
Solution Approach 1:
The patent implements dynamic crawl frequency adjustment by continuously monitoring information update rates and automatically modifying crawl schedules. The system transitions from static pre-determined popularity levels to dynamic real-time adjustment based on actual information update rates, ensuring crawl frequency accurately reflects current website activity levels
Solution Approach 2:
The system establishes a feedback mechanism where crawl results are analyzed to determine information update rates, which then feed back into adjusting crawl frequency. This closed-loop control ensures that crawl scheduling continuously adapts to actual website update patterns, improving both accuracy and responsiveness
2Device complexity
If uniform crawl frequency is applied to all keywords, then scheduling simplicity is maintained, but data completeness deteriorates for popular keywords and resource waste occurs for unpopular keywords
Solution Approach 1:
The patent applies different crawl frequencies to different keywords based on their specific information update rates. Instead of uniform treatment, the system tailors crawl schedules to local characteristics of each keyword's target website, ensuring popular keywords are crawled frequently enough to capture updates while unpopular keywords are crawled only when necessary
3Ease of operation
If secondary download link addresses are not distinguished in initial request processing, then request handling is simplified, but crawl completeness deteriorates due to missed important links
Solution Approach 1:
The patent segments link addresses into different categories, specifically distinguishing secondary download link addresses from other links. This segmentation allows the system to prioritize and properly process secondary download links separately, ensuring they are not missed while maintaining organized request handling through categorical differentiation
Data Source
AI summary
A method and a system for scheduling web crawlers according to keyword search. The method comprises: a scheduling end receiving a task request command sent by a crawling node; the scheduling end acquiring a secondary download link address from a priority bucket, generating tasks, adding the generated tasks into a task list, acquiring keyword link addresses from a dynamic bucket, deriving derivative link addresses of the quantities of pages corresponding to the keyword link addresses, generating tasks of the quantities of the pages according to the derivative link addresses of the quantities of the pages, adding the tasks of the quantities of the pages into the task list, acquiring a keyword link address from a basic bucket, generating tasks, adding the generated tasks into the task list, and the scheduling end returning the task list to the crawling node. By adjusting the quantities of the tasks allowed to be added from a virtual bucket, the quantities of scheduled link addresses of different types are flexibly adjusted. In addition, by crawling popular keywords more frequently, data miss is prevented, and repeated crawls of unpopular keywords is reduced.