Web Crawler Link Equivalency Detection via Structural Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current web crawlers face challenges in efficiently identifying equivalent universal resource locators (URLs) on websites, as they often require examining page content and structure, leading to inaccurate results and the need for expert configuration, especially when websites change their layout on subsequent visits.
Innovation Solution
A computer-implemented process that determines whether all required URLs have been visited, locates the next URL to be crawled, identifies equivalent URLs, and adds them to a list if necessary, allowing for efficient crawling by recognizing structurally similar pages and avoiding duplicate processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a web crawler examines page content to determine link equivalency, then accuracy of equivalency detection improves, but the complexity of the crawling process increases and requires expert configuration
Solution Approach 1:
The patent extracts and compares only the structural framework of web pages (HTML tags, document object model structure) while excluding variable content elements. This selective extraction enables accurate equivalency detection without requiring analysis of entire page contents, thereby reducing complexity while maintaining precision.
Solution Approach 2:
The patent transforms the approach from content-based comparison to structure-based comparison by changing the parameters used for equivalency determination. Instead of comparing textual content and semantic meaning, the system compares structural attributes such as HTML tag hierarchy, element positions, and document object model relationships.
2Reliability
If a web crawler explores all equivalent links on a website, then completeness of website exploration improves, but the time required for crawling increases significantly
Solution Approach 1:
The patent merges multiple equivalent links into a single representative link for crawling purposes. By identifying links with equivalent structural patterns, the system combines their exploration value into one crawl action, ensuring completeness of information gathering while dramatically reducing the number of actual crawl operations required.
Solution Approach 2:
The patent creates a universal structural pattern that can identify multiple equivalent links across different pages. This universal approach allows the crawler to recognize and skip equivalent links systematically, making the crawling process more efficient while maintaining comprehensive coverage of unique content.
3Productivity
If a web crawler uses heuristics to omit page portions for comparison, then crawling speed improves, but the accuracy of equivalency determination deteriorates
Solution Approach 1:
The patent segments the web page into structural components (HTML elements, tags, and their hierarchical relationships) rather than treating it as a continuous content block. This segmentation allows precise identification of which structural elements to compare for equivalency determination, maintaining accuracy while enabling efficient processing through targeted comparison of segmented structural features.
Data Source
AI summary
A computer-implemented process for identifying equivalent links on a page responsive to a determination that the crawler has not visited all required universal resource locators, locates a next URL to be crawled to form a current URL and processes the current URL to identify equivalent URLs. Responsive to a determination that the crawler has not visited the current URL, determine whether necessary to crawl all identified equivalent URLs and responsive to a determination that it is necessary to crawl all identified equivalent URLs, adding all equivalent URLs to a list of URLs to be crawled.