Web Corpus Building via Reference Crawl Agent
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Building a web corpus efficiently is hindered by slow and non-selective web crawling processes, which result in high delays and resource inefficiencies due to limited crawl frequency, non-parallelizable resource retrieval, and the need to crawl the entire web despite only a subset being relevant.
Innovation Solution
The introduction of a reference web crawl agent that interfaces with a web crawler to reuse previously downloaded resources from a reference web crawl, reducing the need for repeated web queries and allowing parallel querying to accelerate the corpus building process without requiring additional hardware resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If web crawlers retrieve resources directly from the web, then the corpus contains up-to-date information, but the time required to build the corpus is excessive and delays are unpredictable
Solution Approach 1:
The patent applies preliminary action by having a reference web crawler pre-fetch and cache web resources in advance before they are actually needed. The reference crawler continuously builds a reference web crawl by downloading resources from the web and storing them in a cache, so when the main crawler needs resources, they are already available locally, eliminating the need for real-time web access and significantly reducing corpus building time.
Solution Approach 2:
The patent uses an intermediary approach by introducing a reference web crawl cache as a mediator between the main web crawler and the actual web resources. Instead of directly accessing the web, the crawler first queries the reference cache, which acts as an intermediary layer. This cache contains pre-downloaded resources that can be quickly retrieved, reducing the time delay while maintaining data availability.
2Loss of information
If web crawlers visit all referenced resources iteratively to build a complete view of the web, then the corpus comprehensiveness is improved, but the accumulated delays make the process unreasonably slow
Solution Approach 1:
The reference web crawler performs preliminary actions by pre-downloading and caching resources that are likely to be needed, including resources from multiple levels of references. This advance preparation allows the main crawler to retrieve resources without performing time-consuming iterative web access, while still maintaining a comprehensive view of the web through the pre-fetched reference crawl.
Solution Approach 2:
The patent applies partial action by having the reference crawler download not only the exact resources needed but also additional referenced resources in advance. The reference web crawl includes resources beyond what is strictly necessary, creating an excessive cache that ensures all possible referenced resources are available locally, eliminating iterative delays while maintaining completeness.
3Reliability
If web crawlers are restricted to low crawl frequencies to avoid overloading websites, then website health is preserved, but the productivity of corpus building is significantly reduced
Solution Approach 1:
The reference web crawler performs preliminary resource downloads during periods when the main crawler is not actively querying, building up a cache in advance. This allows the main crawler to operate at higher effective speeds by retrieving resources from the local cache rather than from the web, thus improving productivity without increasing the actual load on external websites.
Solution Approach 2:
The patent uses copying by creating a local copy of web resources through the reference web crawl cache. Instead of repeatedly accessing the original web sources, the system copies resources into the reference cache, which can then be accessed multiple times without imposing additional load on the original websites. This copying mechanism decouples corpus building speed from website crawl frequency constraints.
4Reliability
If the same resources are downloaded multiple times during corpus building, then resource availability is ensured, but resource inefficiency and time waste increase significantly
Solution Approach 1:
The reference web crawl cache creates a local copy of web resources that can be reused multiple times. When the main crawler needs resources, it first checks the reference cache for available copies. If resources are found in the cache, they are retrieved locally without re-downloading from the web, ensuring resource availability while eliminating repeated download time and network overhead.
Solution Approach 2:
The system recovers previously downloaded resources by storing them in the reference web crawl cache. Instead of discarding downloaded resources after a single use, the cache retains them for future queries, allowing the crawler to recover and reuse the same resources multiple times without re-accessing the web, thus improving efficiency while maintaining availability.
Data Source
Figure 1~3
Figure 4
Figure 5
AI summary
Computer-implemented method for building a web corpus (WCD) comprising the steps of: - sending by a web crawler (WC) a query to a reference web crawl agent (RWCA), this query containing a least one identifier of a resource, - receiving by the web crawler (WC) a response from the reference web crawl agent (RWCA); - if this response does not contain the resource identified by the identifier, downloading by the web crawler (WC) the resource from the website (WS) corresponding to the identifier and adding the resource to the web corpus (WCD; and - if this response contains the resource identified by the identifier, adding the resource to the web corpus (WCD).