Server-Side Client Script Evaluation for Dynamic Link Crawling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional Web crawlers are unable to locate and index dynamically generated links due to their inability to execute client-side scripts outside the context of a Web browser, limiting their ability to crawl and index documents containing such links.
Innovation Solution
A facility is provided that enables server-side evaluation of client-side scripts by executing them using a server-side scripting engine, which provides equivalent browser objects to simulate the execution environment, allowing dynamically generated links to be detected and added to the crawl queue.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional Web crawlers are used to crawl and index Web pages, then the crawling process is simple and efficient, but dynamically generated links cannot be located or indexed
Solution Approach 1:
The patent introduces a client-side script execution engine as an intermediary component between the Web crawler and the dynamically generated links. This engine simulates a Web browser environment to execute client-side scripts and generate the dynamic links, allowing the crawler to access and index content that would otherwise be inaccessible. The intermediary resolves the contradiction by adding the necessary complexity only where needed (in the execution engine) while keeping the rest of the crawling system relatively simple.
Solution Approach 2:
The crawling system is segmented into distinct functional components: the Web crawler itself, the client-side script execution engine, and the link generation module. This segmentation allows each component to be optimized independently - the crawler focuses on efficient page discovery while the execution engine handles the complex task of script interpretation and dynamic link generation, resolving the contradiction between simplicity and capability.
2Loss of information
If client-side scripts are executed outside a Web browser context, then dynamically generated links can be discovered, but execution compatibility issues arise
Solution Approach 1:
The patent creates a simplified copy of the Web browser execution environment that runs on the server side. Instead of attempting to run scripts in their original client-side context, the system copies the essential browser objects and execution context to a server environment where they can be executed reliably. This copying approach resolves the contradiction by preserving the execution semantics needed for dynamic link generation while eliminating the compatibility issues of cross-context execution.
3Loss of information
If all Web pages including those with dynamic links are crawled, then complete indexing is achieved, but processing time increases significantly
Solution Approach 1:
The patent applies partial action by selectively executing client-side scripts only on Web pages that contain dynamic link generation, rather than processing every page uniformly. The system identifies pages with script tags and focuses computational resources on those specific pages, achieving complete indexing of dynamic content while minimizing unnecessary processing time on static pages.
Solution Approach 2:
The system performs preliminary analysis of Web pages to identify those containing client-side scripts before executing them. By pre-processing pages to detect script presence and organizing them for targeted execution, the system avoids the time cost of executing scripts on every page while ensuring no dynamic content is missed, thus resolving the contradiction between completeness and efficiency.
Data Source
AI summary
A facility is described for server-side evaluation of client-side script, such as to crawl and index dynamically generated links. In various embodiments, the facility provides an object that emulates a browser object. The provided object exposes a function for performing steps comprising receiving a first link and storing the received first link in a list of links to be processed. The facility receives a second link to be processed, loads a second document identified by the received second link, and assigns the provided object to a global variable so that a script associated with the loaded second document invokes the function exposed by the provided object instead of a function with a similar name that is exposed by the emulated browser object. The function with the similar name exposed by the emulated browser object handles the received first link without storing it in the list of links to be processed.