Crawler Scheduler Reusing Stable Documents
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Search engines face challenges in efficiently managing their crawling capacity due to the vast number of web pages, necessitating automated mechanisms to determine which pages to crawl and avoid, given their limited capacity compared to the vast number of distinct documents on the World Wide Web.
Innovation Solution
A scheduler for a search engine crawler that determines which documents to download by using a reuse flag in scheduler records, deciding between retrieving documents from the World Wide Web or a document repository based on prior crawl data, and employing URL schedulers to allocate and manage URLs for crawling, with a reuse server identifying stable documents for retrieval from a repository.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the crawler retrieves all documents from the World Wide Web, then the search engine can access the most current document information, but the crawling capacity is exceeded due to the vast number of distinct documents
Solution Approach 1:
The patent creates a copy of previously crawled documents in a document repository. When a document is scheduled for retrieval, the system first checks the repository for an existing copy. If found and deemed stable, the copy is reused instead of retrieving the original from the World Wide Web. This copying approach allows the search engine to maintain current information without repeatedly accessing the same documents from the web, thereby managing crawling capacity effectively.
Solution Approach 2:
The patent implements preliminary retrieval and storage of documents in the document repository during initial crawls. This preliminary action prepares documents for future reuse, eliminating the need to re-download them from the World Wide Web. By performing this action in advance, the system optimizes subsequent crawling operations and maintains information freshness without exceeding crawling capacity.
2Reliability
If the crawler downloads all scheduled documents from the World Wide Web, then complete document updates are ensured, but network bandwidth and server load increase significantly
Solution Approach 1:
The system maintains copies of documents in a local document repository instead of continuously downloading them from the World Wide Web. When a document is scheduled for retrieval, the system checks the repository for an existing copy. If the document is deemed stable and a copy exists, the system reuses the copy rather than downloading the original, significantly reducing network bandwidth consumption while ensuring document update completeness through the copy mechanism.
Solution Approach 2:
The document repository serves the crawler's own needs by providing locally stored copies of previously crawled documents. The system uses its own previously retrieved resources to satisfy future retrieval requests, eliminating the need to continuously consume network bandwidth for the same documents. This self-service approach optimizes network resource utilization while maintaining reliable document updates.
3Measurement precision
If the scheduler retrieves documents from the World Wide Web for each scheduled URL, then the most current version is obtained, but the crawling process becomes less efficient due to repeated retrievals of stable documents
Solution Approach 1:
The patent stores copies of previously crawled documents in a document repository. The scheduler checks this repository before retrieving documents from the World Wide Web. When a stable document is found in the repository, the system reuses the copy instead of performing a new retrieval. This approach maintains document version accuracy by ensuring the copied document reflects the current stable version, while significantly improving crawling operation efficiency by eliminating redundant network requests.
Solution Approach 2:
The system uses feedback from previous crawl operations to determine whether to retrieve documents from the World Wide Web or reuse copies from the repository. The scheduler receives feedback about document stability and retrieval history, allowing it to make informed decisions about the optimal retrieval method. This feedback mechanism ensures that stable documents are reused while maintaining version accuracy, thereby improving overall crawling efficiency.
4Quantity of substance
If the search engine crawls all scheduled documents, then comprehensive document coverage is achieved, but the crawler's capacity limits are exceeded
Solution Approach 1:
The patent implements a document repository that stores copies of previously crawled documents. When the scheduler needs to retrieve a document, it first checks the repository for an existing copy. This copying mechanism allows the crawler to maintain comprehensive document coverage by reusing copies of stable documents rather than attempting to download every document from the World Wide Web, thereby managing crawling capacity effectively while achieving extensive document coverage.
Solution Approach 2:
The patent segments the document retrieval process into two paths: retrieving new documents from the World Wide Web and reusing existing copies from the document repository. This segmentation allows the crawler to manage its capacity by directing retrieval requests to the appropriate source, achieving comprehensive document coverage without exceeding capacity limits through intelligent routing of retrieval operations.
Data Source
AI summary
A search engine crawler includes a scheduler for determining which documents to download from their respective host servers. Some documents, known to be stable based on one or more record from prior crawls, are reused from a document repository. A reuse flag is set in a scheduler record that also contains a document identifier, the reuse flag indicating whether the document should be retrieved from a first database, such as the World Wide Web, or a second database, such as a document repository. A set of such scheduler records are used during a crawl by the search engine crawler to determine which database to use when retrieving the documents identified in the scheduler records.