Hash Table URL Prioritization for Web Crawling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for organizing uniform resource locators (URLs) for web crawling fail to prioritize unexplored URLs, leading to redundant crawling of already visited URLs while neglecting un-crawled ones.
Innovation Solution
A system and method that utilize a hash table to select portions of stored URLs, identify them, and determine processing based on associated data, including page scores and global information, to efficiently prioritize and crawl URLs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If current methods for organizing URLs are used, then URLs can be stored and processed, but unexplored URLs are not prioritized and already explored URLs are crawled redundantly
Solution Approach 1:
The patent applies preliminary action by pre-organizing URLs in a hash table structure with dedicated slots for unexplored and explored URLs before crawling begins. This pre-arranged organization enables immediate prioritization of unexplored URLs during processing, eliminating the need to determine priority during crawling and preventing redundant crawls of already explored URLs.
Solution Approach 2:
The patent segments the URL collection into distinct categories using a hash table with separate slots for unexplored URLs and explored URLs. This segmentation allows the crawling system to efficiently distinguish between URLs that need to be crawled and those that have already been processed, thereby improving crawling efficiency and preventing redundant operations.
2Reliability
If current URL organization methods are used, then all URLs can be stored, but the same URL is crawled twice while other URLs remain un-crawled
Solution Approach 1:
The patent implements feedback by maintaining a recorded list of explored URLs and using this information to guide subsequent crawling decisions. The system continuously updates the explored URL list based on crawling results and uses this feedback to prevent re-crawling of the same URLs, ensuring complete and efficient processing of all unique URLs.
Solution Approach 2:
The patent prepares the URL processing system in advance by creating a hash table with dedicated slots for unexplored and explored URLs before crawling begins. This preliminary organization ensures that the system is ready to track and prioritize URLs effectively from the start, preventing both redundant crawls and missed URLs throughout the crawling process.
3Ease of operation
If URLs are not prioritized, then simple organization is maintained, but unexplored URLs are not processed before already explored URLs
Solution Approach 1:
The patent segments URLs into distinct categories (unexplored and explored) using a hash table structure with separate slots. This segmentation provides a simple yet effective prioritization mechanism where unexplored URLs are naturally prioritized for crawling, maintaining ease of operation while dramatically improving crawling efficiency by preventing redundant processing.
Data Source
AI summary
In one general embodiment, a computer program product for determining whether to process a uniform resource locator (URL) comprises a computer readable storage medium having program instructions embodied therewith, wherein the computer readable storage medium is not a transitory signal per se, and where the program instructions are executable by a processor to cause the processor to perform a method comprising selecting, by the processor, a portion of a hash table, identifying, by the processor, a plurality of uniform resource locators (URLs) stored within the selected portion of the hash table, and for each of the plurality of identified URLs, individually determining, by the processor, whether to process the identified URL, based on data associated with the identified URL.