URL Crawl Backlog Management via Priority-Based Early Rejection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In scenarios where URL crawl capacity is limited, existing systems fail to prioritize URL crawls effectively, leading to high latency for important crawls as less important ones are not rejected early enough, resulting in incomplete fulfillment of crawling demands.
Innovation Solution
Implementing a backlog management system that uses a priority queue to sort and manage URL crawl requests, where requests with higher priorities are processed first, and those failing to meet a priority threshold are rejected early to maintain low average latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all pending URL crawls are kept in the backlog queue, then completeness of crawling demands is improved, but average latency increases for important URL crawls
Solution Approach 1:
The patent extracts low-priority URL crawls from the backlog queue through early rejection based on priority thresholds. This removes unnecessary elements (low-priority crawls) from the system before they can consume resources and increase latency, while maintaining high-priority crawls that need to be processed promptly.
Solution Approach 2:
The patent applies different treatment to different parts of the backlog based on priority levels. High-priority URL crawls are maintained and processed promptly, while low-priority ones are rejected early. This differential treatment optimizes the overall system performance by ensuring critical crawls get low latency while non-critical ones don't waste resources.
2Loss of time
If early rejection of low-priority URL crawls is implemented, then average latency is reduced, but completeness of crawling demands deteriorates
Solution Approach 1:
The patent changes the parameter of priority threshold dynamically. By adjusting the priority threshold, the system can control which URL crawls are rejected early and which are maintained in the backlog. This allows optimization of latency for high-priority crawls while preserving the ability to fulfill important crawling demands by setting appropriate threshold levels.
3Productivity
If limited crawl capacity is shared among multiple applications, then resource utilization is improved, but latency for important URL crawls increases due to competition
Solution Approach 1:
The patent applies priority-based differential treatment to URL crawls from different applications. High-priority crawls receive preferential treatment by being maintained in the backlog and processed promptly, while low-priority crawls are rejected early. This ensures that shared resources are allocated to important tasks first, reducing latency for critical crawls while still utilizing capacity for less important ones.
Solution Approach 2:
The patent extracts low-priority competing requests from the system through early rejection. This removes competing elements that would otherwise consume crawl capacity and increase latency for high-priority requests, allowing important URL crawls to be processed more quickly while still maintaining overall resource utilization.
Data Source
AI summary
The technology described relates to reducing a backlog of pending URL crawls in view of a limited URL crawl capacity. This technology is useful for crawling URLs with low latency. Because of the limited crawl capacity, uncrawled URLs from crawl requests are entered into a backlog data structure of pending crawl requests. Various criteria are applied to the URL's that are requested to be crawled, so that less important URL crawls are rejected early from the backlog data structure. This early rejection tends to limit the backlog data structure to the more important pending URL crawls, and tends to keep the average latency low by quickly failing the less important requested URL crawls.