Focused Web Crawler Link Scoring and Priority Queue
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing web crawlers face inefficiencies in collecting relevant information from web pages due to their linear or haphazard search methods, requiring significant processing power and memory, and struggle to identify target information in complex hyperlink networks without central listings or databases.
Innovation Solution
A method and system for focused web crawling using a processor and database that assigns link scores to web page links based on their probability of leading to a target page type, sorting them in a priority queue, and utilizing classifiers and parsers to efficiently navigate and extract relevant information from web pages, particularly for personnel directories like faculty listings.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If existing web crawlers use linear or haphazard search methods to collect information from web pages, then they can eventually find target information, but they require large amounts of processing power and memory, and take significant time
Solution Approach 1:
The patent changes the parameter of link evaluation from uniform treatment to scored prioritization. By assigning different link scores based on relevance criteria and processing links in descending order of their scores, the system transforms the crawling process from a blind linear search to a targeted prioritized search, significantly improving computational efficiency while maintaining the ability to find target information
Solution Approach 2:
The patent performs preliminary scoring and ranking of links before actual crawling begins. By pre-calculating link scores and organizing links in a priority queue in descending order, the system prepares the crawling path in advance, avoiding the need to process irrelevant pages during the actual information collection phase
2Measurement precision
If existing web crawlers search through pages and subpages in a linear fashion through complicated hyperlink networks, then they can find target webpages, but they require large amounts of processing power and memory storage
Solution Approach 1:
The patent introduces link scoring as a new parameter to evaluate and prioritize links. By assigning numerical scores to links based on relevance criteria and processing them in descending order, the system transforms the crawling approach from exhaustive linear search to targeted prioritized search, reducing processing power and memory requirements
Solution Approach 2:
The patent extracts and prioritizes only the most relevant links for crawling by assigning scores and processing links in descending order. This extraction of high-value links from the complete link set allows the system to ignore low-scoring irrelevant pages, significantly reducing the processing power and memory needed while maintaining the ability to find target information
3Adaptability or versatility
If existing web crawlers explore links in each accessed page in a linear fashion, then they can comprehensively search the web, but they analyze hundreds or thousands of irrelevant webpages before reaching the target
Solution Approach 1:
The patent changes the processing order parameter from linear to prioritized based on link scores. By sorting links in descending order of their scores and processing high-scoring links first, the system dramatically reduces the time to reach target webpages while maintaining comprehensive search capability through the scoring mechanism that ensures relevant pages are prioritized
Solution Approach 2:
The patent performs preliminary scoring and sorting of all links before crawling begins. By organizing links in a priority queue in descending order of relevance scores, the system prepares an optimized crawling path in advance, eliminating the time waste of analyzing hundreds or thousands of irrelevant pages while preserving comprehensive search coverage
Data Source
AI summary
Embodiments herein provide for a more efficient web crawler (102) that accesses a plurality of links from a web page (106). In certain embodiments, the web crawler (102) determines link scores for each of the plurality of links, with each link score corresponding to a probability that the associated link leads to a web page (106) of a target page type. The plurality of links and the associated link scores are added to a priority queue (114). The web crawler (102) accesses a second web page (106) using the highest-ranked link in the priority queue (114), obtains links from that web page (106), identifies a link score for each of those links, and adds those links to the priority queue (114) based on their link scores. This process can be repeated for several pages, as needed, using the highest-rank link in each instance, until the crawler (102) identifies a web page (106) of a target page type.