Shared Cookie Database for Multi-Crawler Coordination
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional network crawlers lack the ability to obtain and handle cookies, and coordinate multiple crawlers for accessing web sites that require cookies, leading to incomplete data retrieval and inefficiencies in search engine indexing.
Innovation Solution
A system of multiple network crawlers with a shared cookie database that retrieves and manages cookies, detects cookie errors, and updates cookie information, enabling access to cookie-regulated web sites and improving data retrieval efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If multiple network crawlers are used to scan documents frequently, then the freshness of the search engine index is improved, but the complexity of coordinating cookie management across crawlers increases
Solution Approach 1:
The patent merges cookie management functionality into a shared database that is accessible by all network crawlers. Instead of each crawler independently managing cookies, the system combines cookie storage and retrieval operations into a centralized repository, allowing multiple crawlers to share cookies and coordinate their access to cookie-protected resources efficiently.
Solution Approach 2:
The cookie database serves multiple functions: it stores cookies, validates cookies, coordinates crawler access, and manages cookie expiration. This multi-functional approach reduces the overall system complexity by consolidating what would otherwise require separate mechanisms for each function across multiple crawlers.
2Device complexity
If conventional network crawlers are used without cookie facilities, then the crawler system is simpler, but the ability to access cookie-protected web sites is lost
Solution Approach 1:
The patent introduces a cookie database as an intermediary between the network crawlers and cookie-protected web sites. The database acts as a mediator that handles all cookie-related operations, allowing crawlers to access protected content without each crawler needing to implement complex cookie management logic independently.
Solution Approach 2:
The cookie database automatically handles cookie validation, expiration checking, and retrieval operations. When a crawler needs to access a cookie-protected resource, the database self-manages the cookie provisioning process, reducing the complexity burden on individual crawlers while maintaining access capability.
Data Source
AI summary
A web-crawler system includes a plurality of network crawlers configured to fetch documents from hosts on a network and a cookie database shared by the plurality of network crawlers. The cookie database stores cookies and associated information for use by the plurality of network crawlers. Each network crawler is configured to retrieve one or more cookies from the cookie database so as to enable access to documents on at least one of the hosts on the network. In some embodiments, each of the network crawlers may be configured to detect any of a plurality of predefined cookie errors associated with fetching a document. In some embodiments, each of the network crawlers may also be configured to detect when a cookie in the cookie database has expired and to obtain a replacement cookie.