Dynamic Crawl Scheduler for Web Page Prioritization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Search appliances often prioritize less important or unchanged pages for recrawling due to their initial sequence, leading to less important or frequently changed pages being overlooked in finite crawling periods.
Innovation Solution
Implementing a crawl scheduler that estimates the change period of web pages and adjusts the recrawl schedule based on importance ranks and hints, ensuring more important pages are recrawled more frequently and less important pages are not over-recrawled.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If pages are crawled in the initial sequence from the URL list, then the crawling process is simple to implement, but less important pages are crawled before more important pages
Solution Approach 1:
The patent implements a dynamic crawl schedule that adjusts recrawl timing based on page importance ranks and estimated change periods. The system transitions from a static first-come-first-served approach to a dynamic priority-based scheduling system where crawl intervals are calculated based on multiple factors including importance, change frequency, and overdue status.
Solution Approach 2:
The system changes the parameter of crawl timing from fixed sequence order to variable intervals based on importance rank and change period estimates. By introducing parameters such as importance weight, estimated change period, and overdue penalty, the system optimizes crawl scheduling to prioritize important and frequently changing pages.
2Device complexity
If the crawler focuses on pages appearing earlier in the URL list, then the crawling schedule is easy to manage, but more important or changed pages may be overlooked
Solution Approach 1:
The system incorporates feedback mechanisms by monitoring page changes and importance metrics to adjust crawl schedules. The estimated change period is updated based on historical crawl data, and the schedule is dynamically adjusted to ensure important pages are recrawled at appropriate intervals, preventing information loss.
Solution Approach 2:
The system performs preliminary estimation of page change periods and importance ranks before finalizing the crawl schedule. By pre-calculating these parameters and setting initial crawl intervals, the system prepares an optimized schedule that prevents important pages from being overlooked, reducing the need for reactive adjustments.
3Ease of manufacture
If recrawls are performed in sequence, then the implementation is straightforward, but less important pages are crawled to the exclusion of more important pages
Solution Approach 1:
The patent transforms the static sequential recrawl process into a dynamic priority-based system. The system calculates priority scores based on importance ranks, estimated change periods, and overdue status, then orders recrawls dynamically based on these scores, improving crawl efficiency by focusing resources on high-priority pages.
Solution Approach 2:
The system introduces multiple parameters including importance weight, estimated change period, and overdue penalty to determine crawl priority. By changing from a single parameter (sequence order) to multiple weighted parameters, the system achieves better crawl efficiency while maintaining reasonable implementation complexity.
4Quantity of substance
If the crawler allocates time to all pages equally, then the distribution is uniform, but important pages are not crawled frequently enough
Solution Approach 1:
The patent applies local quality by differentiating crawl frequency based on local page characteristics such as importance rank and change period. Instead of uniform treatment, the system assigns different crawl intervals to different pages based on their specific needs, ensuring important pages receive more frequent attention while maintaining overall coverage.
Solution Approach 2:
The system changes the crawl interval parameter from a fixed uniform value to variable values based on page-specific parameters. By introducing importance weight and estimated change period as modifying parameters, the system achieves both broad coverage and focused attention on important pages, balancing quantity and quality of crawl operations.
Data Source
AI summary
Determining a schedule for recrawling pages is disclosed. A crawling schedule that specifies a due date at which each page is to be crawled is determined according to a first scheme. A set of pages that includes one or more pages each of which has a due date that has passed is determined. The set of pages is ordered according to a second scheme.