Web Crawler Using Locality Sensitive Hash for Unvisited Content
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing web crawlers face inefficiencies due to redundant processing of similar web pages, as current solutions lack scalability and efficiency in identifying structurally similar pages, leading to increased time and resource consumption.
Innovation Solution
A computer-implemented process using Locality Sensitive Hash (LSH) values to identify unvisited portions of web pages by computing an LSH value for received information, determining its similarity to previously visited content, and crawling only unvisited portions, thereby reducing redundant analysis.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If web crawlers process all web pages completely, then analysis thoroughness is improved, but processing time and resource consumption increase
Solution Approach 1:
The patent segments web pages into hierarchical structures (pages, sections, elements) and processes them at different levels. By identifying similar pages at the page level using LSH and skipping redundant element-level processing, the system maintains analysis thoroughness for unique content while reducing overall processing time through selective segmentation.
Solution Approach 2:
The patent performs preliminary actions by computing LSH values for page structures before detailed element analysis. This preliminary identification of similar pages allows the crawler to skip redundant processing of identical elements across multiple pages, reducing processing time while maintaining thoroughness for genuinely different content.
2Reliability
If web crawlers use MD5 hash values to identify duplicate pages, then duplicate detection capability is improved, but system complexity and search requirements increase
Solution Approach 1:
The patent changes the parameter used for duplicate detection from MD5 hash values to Locality Sensitive Hash (LSH) values. LSH provides a more efficient parameter that can quickly identify similar pages without requiring complex search through large record repositories, thus maintaining duplicate detection capability while reducing system complexity.
Solution Approach 2:
The patent creates a simplified representation (copy) of page structure using LSH values that captures essential similarity information without requiring full page content storage or complex indexing. This copying approach allows efficient duplicate detection without the complexity of maintaining comprehensive record repositories.
3Loss of information
If web crawlers process redundant HTML elements, then completeness of data collection is improved, but processing efficiency deteriorates
Solution Approach 1:
The patent applies local quality by treating different parts of web pages differently based on their uniqueness. Elements that are identical across multiple pages are marked as redundant and skipped, while unique elements are processed in detail. This selective local processing maintains data collection completeness for unique information while significantly improving processing efficiency by avoiding redundant work.
Data Source
AI summary
Identifying unvisited portions of visited information to visit includes receiving information to crawl, wherein the information is representative of one of web based information and non-web based information, computing a locality sensitive hash (LSH) value for the received information, and identifying a most similar information visited thus far. Identifying unvisited portions of visited information further includes determining whether the LSH of the received information is equivalent to most similar information visited thus far and, responsive to a determination that the LSH of the received information is not equivalent to most similar information visited thus far, identifying a visited portion of the received information using information for most similar information visited thus far and crawling only unvisited portions of the received information.