Web Crawler Access to Dynamic Pages via Portal Mediator
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Web crawlers are unable to access and index dynamic web pages, which are created on-the-fly or through script execution, as they lack the capability to parse and extract content from pages with non-HTML elements like JavaScript, Flex, or Silverlight, making it difficult to search and index such content.
Innovation Solution
A method and apparatus that enables a web crawler to access dynamic web pages by requesting resource identifiers from components associated with a primary application, which renders and parses the content into a Document Object Model (DOM) for text or HTML extraction, allowing for indexing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a web crawler uses traditional hyperlink-based navigation to access web pages, then it can efficiently crawl static web pages, but it cannot access dynamic web pages that are created on-the-fly or through script execution
Solution Approach 1:
The patent introduces a portal application as an intermediary between the web crawler and dynamic web content. The portal application receives crawler requests, executes internal logic to generate dynamic pages, and returns both the rendered content and extracted text/HTML. This mediator enables the crawler to access dynamic pages without needing to execute scripts or understand complex dynamic rendering processes.
Solution Approach 2:
The portal application performs preliminary actions by pre-rendering dynamic web pages and pre-extracting text content before the crawler arrives. When a crawler requests a dynamic page, the portal application has already prepared the rendered HTML and extracted text, allowing immediate delivery without requiring the crawler to execute scripts or perform complex parsing operations.
2Loss of information
If a web crawler attempts to parse and extract content from pages containing non-HTML content like JavaScript, Flex, or Silverlight code, then it may access dynamic content, but it fails to correctly parse and index the content
Solution Approach 1:
The patent extracts the content rendering and text extraction functionality from the crawler itself and places it within the portal application. The portal application renders the dynamic content using its internal scripting engine and extraction logic, then delivers only the cleaned text and HTML to the crawler. This removes the burden of complex parsing from the crawler while preserving complete content extraction.
3Adaptability or versatility
If dynamic web pages are created through script execution or portal application logic, then the pages can be dynamically generated, but the links and contents become unreachable for typical web crawlers
Solution Approach 1:
The portal application serves as a mediator that bridges the gap between dynamic page generation and crawler accessibility. It receives crawler requests, executes the necessary application logic to generate dynamic pages, and returns both the rendered content and structured text data. This intermediary approach maintains dynamic page generation while ensuring reliable crawler access.
Data Source
AI summary
A method and apparatus for enabling an external application such as a web crawler access to dynamic web pages associated with a primary application such as a portal page. The primary application addresses each component associated with it and requests a list of resource identifiers. Each component implements an interface and provides such list of resource identifiers. The list is returned to the external application, which then optionally requests the contents of the page associated with each resource identifier. The component provides the content of the page, which is then parsed by a parsing module associated with the primary application. The parsing module transforms the content into a data structure such as a Document Object Model, and then extracts text or Hypertext Markup Language code from the data structure. The text is then returned to the external application fro searching, indexing or other purposes.