Rules Engine for Dynamic Web Script Link Discovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional web crawlers fail to index dynamically generated content from scripts, resulting in incomplete search results for users as they cannot interpret or evaluate scripts, leading to missed data in the indexing process.
Innovation Solution
A rules engine connected to the crawler parses webpages, extracts and evaluates script segments to identify and validate links generated by the scripts, storing these links and associated content in an index, using rules from a database to optimize the indexing process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If a conventional crawler is used to parse webpages, then the crawler can access and store webpage data in an index, but the crawler cannot interpret or evaluate scripts, resulting in incomplete indexing of dynamically generated content
Solution Approach 1:
A rules engine is introduced as an intermediary component between the crawler and the indexing system. The rules engine receives webpages from the crawler, extracts script segments, evaluates them against stored rules, and generates links that are then added to the index. This intermediary enables script interpretation without requiring the crawler itself to have script evaluation capabilities.
Solution Approach 2:
The webpage processing is divided into distinct segments: the crawler handles HTML parsing and basic data extraction, while the rules engine specifically handles script extraction and evaluation. This segmentation allows each component to specialize in its strength, with the rules engine focusing on interpreting scripting code to generate dynamic links.
2Productivity
If the crawler ignores scripts to maintain simple processing, then the crawler operates efficiently, but the index does not include data corresponding to dynamically generated content
Solution Approach 1:
The rules engine serves as a specialized intermediary that handles the complex task of script evaluation. By separating this function from the main crawler, the system achieves complete indexing of dynamic content while keeping the crawler itself relatively simple and efficient.
Solution Approach 2:
The rules engine uses rules stored in a rules database that can be copied and applied to multiple script segments. This allows the system to handle diverse scripting code without requiring complex custom evaluation logic for each case, maintaining processing efficiency while achieving comprehensive indexing.
Data Source
AI summary
A computer-implemented method, a computer system, and computer media for discovering links in scripts are provided. The computer system includes a crawler, a rules engine, and an index that are utilized to store links generated by scripts located in webpages in the index. The crawler traverses a network to locate webpages having scripts. The rules engine parses the located webpages and extracts the scripts based on rules that are satisfied by segments of the extracted scripts. The rules engine evaluates the segments of the extracted scripts to generate links. After the rules engine validates the links, the rules engine transmits the links to the index for storage.