Web Crawler Cache Pre-loading for Initial Request Speed
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current web server caching techniques reduce response times for subsequent access requests but not for initial requests, as the cache is populated by client requests, leading to slower retrieval from disk for initial document access.
Innovation Solution
A webcrawler automatically populates the cache by sending requests to web pages based on their hierarchical structure, ensuring that documents are cached before initial client requests, thereby reducing response times by pre-loading the cache with web pages.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If the cache is populated by client requests, then the cache stores frequently accessed documents, but initial document retrieval is slow because the document must be retrieved from disk first
Solution Approach 1:
The patent applies preliminary action by having the web crawler proactively access and cache documents before actual client requests arrive. The crawler systematically traverses the web site hierarchy, pre-fetching documents and storing them in the cache during off-peak times or continuously in the background, so that when clients request these documents, they are already available in the cache for immediate retrieval.
2Speed
If a webcrawler pre-loads the cache, then initial client requests are served faster, but the system requires additional automated infrastructure
Solution Approach 1:
The patent applies self-service by designing the web crawler to autonomously navigate the web site hierarchy, automatically determine which documents to fetch, and independently manage the caching process without requiring complex external control systems. The crawler uses the existing site structure to guide its actions and serves itself by generating its own workload based on the document links it discovers.
3Loss of energy
If the cache is populated only when documents are requested, then the system uses minimal resources, but subsequent requests for the same document suffer from disk retrieval delays
Solution Approach 1:
The patent applies periodic action by having the web crawler operate in continuous background cycles, periodically traversing the web site hierarchy and refreshing the cache with newly added or updated documents. This periodic background operation ensures that the cache is proactively maintained without requiring continuous high resource consumption, balancing energy efficiency with the need for fast subsequent retrieval.
Data Source
AI summary
One embodiment of the invention is directed to automatically populating the cache of a web server. A webcrawler may send a request for a web page to the web server and receive the web page in response. The webcrawler may locate a link to a second web page in the received web page and determine if the link includes a predetermined pattern of characters. If the link includes the predetermined pattern of characters, the webcrawler may send a request to the webserver for the second web page. In another embodiment, the webcrawler may be made aware of changes to the content of a web site and revisit the web site after changes are made to repopulate the cache with the updated content.