Web Page Indexing via Runtime Attribute Extraction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional web crawling methods for indexing internet content are inefficient due to frequent page changes, leading to outdated indexes and mismatches between search requests and results, while indexing entire resource structures fail to support user-viewable data and HTML previews.
Innovation Solution
The proposed system generates an attribute for rendered web pages, including a uniform resource locator, and adds it to a complex resource structure, ensuring that the search engine indexes the content as it is displayed to the user, with incremental updates triggered by page modifications, using a knowledge management system and display component.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If web crawling is used to index web content, then the index can be updated, but frequent page changes cause the index to become outdated and lead to performance issues
Solution Approach 1:
The system performs preliminary actions by generating attributes and indexing URLs during the page design phase (design time) rather than waiting for runtime crawling. When a web page is created or modified in the composition tool, the system proactively generates the URL attribute and updates the search index immediately, ensuring the index is current before the page is even published.
Solution Approach 2:
The web page composition system serves its own indexing needs by automatically generating URL attributes and triggering index updates when pages are created or modified. This self-service mechanism eliminates the need for external crawlers to discover and index pages, making the system self-sufficient for indexing purposes.
2Quantity of substance
If the complete resource structure is indexed, then more data is available for searching, but the indexed data does not match user-viewable content leading to search mismatches
Solution Approach 1:
The system extracts only the essential URL attribute from the complex resource structure for indexing purposes, rather than indexing the entire resource structure. This selective extraction ensures that only the actual user-viewable page URLs are indexed, matching what users will see and eliminating mismatches between indexed data and displayed content.
Solution Approach 2:
Different parts of the resource structure serve different functions: the complex resource structure maintains design-time content organization, while the extracted URL attribute provides runtime accessibility information. This local differentiation ensures that indexing focuses on the specific attribute (URL) that matters for user access, rather than indiscriminately indexing all resource structure data.
3Device complexity
If HTML preview is not supported in search results, then the search system is simpler, but user experience is degraded
Solution Approach 1:
The system performs preliminary rendering of HTML previews during the indexing phase. When generating URL attributes for indexing, the system also creates and stores HTML preview content that can be directly used in search results, eliminating the need for complex runtime preview generation and providing immediate visual feedback to users.
Data Source
AI summary
A system, article and method of indexing web content is presented. An attribute is generated for a rendered web page rendered by a display component. The attribute includes a uniform resource locator to the rendered web page, and the rendered web page is associated with a complex resource structure that defines content of a design time version of the rendered web page. The attribute is added to the complex resource structure, and the uniform resource locator from the attribute is indexed in a search engine index.