Adaptive Web Crawler for Price Data Prioritization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional search engines are inefficient in prioritizing webpages for crawling and indexing price and product information, as they focus on overall content changes rather than specific price or product updates, leading to inadequate prioritization of relevant data.
Innovation Solution
A system and method that utilize Crawl Agents and Indix Servers to identify and prioritize URIs containing price and product attributes, adjusting crawl queues based on observed changes and user interest, employing Parse Maps to extract and validate price and product information from webpages.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional crawlers prioritize webpages based on overall content changes, then general webpage updates are captured, but price and product information updates are not prioritized adequately
Solution Approach 1:
The patent segments the webpage content into distinct elements, specifically identifying price and product information as separate crawlable components. By dividing the webpage into structured elements (price tags, product descriptions, specifications), the crawler can selectively monitor and prioritize these specific segments rather than treating the entire page as a single unit, thereby improving detection precision for price changes while maintaining crawling efficiency.
Solution Approach 2:
The patent applies local quality by assigning different priority levels and crawl frequencies to different portions of the webpage. Price and product information elements are designated as high-priority local regions that require frequent monitoring, while other static content receives lower priority. This localized differentiation enables the system to focus computational resources on extracting price and product data efficiently without being bogged down by processing entire pages uniformly.
2Loss of information
If crawlers monitor all webpage content changes, then comprehensive data is collected, but computational resources are wasted on non-price content
Solution Approach 1:
The patent extracts and isolates price and product information from the broader webpage content. By using structured data extraction techniques that specifically target price elements (such as price tags, cost indicators) and product attributes (descriptions, specifications, images), the system separates the essential commercial data from unrelated content. This extraction process ensures complete capture of price and product information while eliminating the computational overhead of processing and analyzing irrelevant webpage elements.
Solution Approach 2:
The patent implements partial action by monitoring only the specific portions of webpages that contain price and product information, rather than analyzing entire pages. The crawler selectively applies parsing and data extraction operations to identified price-related elements, leaving the rest of the webpage content unprocessed. This partial monitoring approach maintains information completeness for commercial data while significantly reducing computational resource consumption compared to full-page analysis.
3Reliability
If traditional search engines crawl webpages uniformly, then all content is indexed, but price information is not updated timely
Solution Approach 1:
The patent introduces dynamic crawling behavior that adapts to the specific characteristics of different webpage elements. Price information elements are assigned dynamic priority levels and refresh intervals based on their importance and volatility. The system dynamically adjusts crawl frequency for pages containing price changes, increasing monitoring intensity when price updates are detected and reducing intensity for stable content. This dynamic approach ensures timely price information updates while managing system complexity through adaptive rather than static crawling rules.
Data Source
AI summary
Content is obtained from a webpage accessed via a URI, which URI is obtained from a URI queue. The content is parsed for price and product information according to a parse map, with the resulting parse result being stored. The priority of URIs in the URI queue is adjusted based on analysis of the parse result for changes in price and product attributes and according to other criteria. The parse map may be one associated with the URI or a general purpose parse maps. The parse result may be validated by human- and machine-based systems, including by graphically labeling price and product information in the content for human confirmation or correction.