Virtual-Browser Retail Crawling for Faster Product Data Extraction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing web crawlers take a long time to discover all webpages on large retailer websites and generate significant performance impacts on the website due to numerous requests.
Innovation Solution
A retail crawling engine that utilizes a virtual browser to process webpages efficiently by suppressing irrelevant elements, processing engine control statements, and implementing self-healing mechanisms to adapt to webpage changes, while combining data from multiple sources for accurate product data extraction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a web crawler requests all webpages on a large retailer website, then complete product data extraction is achieved, but the processing time increases significantly and website performance is impacted
Solution Approach 1:
The patent segments the webpage processing into distinct phases: rendering only critical elements (product data), suppressing non-critical elements (ads, navigation), and selectively extracting information. This segmentation allows the crawler to focus computational resources on extracting product data while ignoring irrelevant content, thereby reducing processing time without compromising extraction accuracy
Solution Approach 2:
The patent extracts and processes only the essential product data elements from webpages, removing unnecessary processing of non-critical elements such as advertisements, navigation menus, and footer content. By taking out only the relevant information needed for product catalog maintenance, the system achieves accurate data extraction while minimizing processing time and website performance impact
2Measurement precision
If a web crawler generates numerous requests to discover all webpages, then complete product data is collected, but website performance is significantly impacted
Solution Approach 1:
The patent implements preliminary action by using an engine control statement framework that pre-defines extraction rules, element selectors, and data mapping configurations before crawling begins. This preliminary setup allows the crawler to efficiently process webpages without needing to generate excessive requests for discovery, as the extraction logic is already optimized and ready to execute
Solution Approach 2:
The patent introduces an intermediary layer (the engine control statement system) that mediates between the crawler and the target website. This intermediary optimizes request generation by using structured directives to guide the crawler's behavior, reducing unnecessary requests while ensuring complete product data collection through systematic navigation and extraction rules
3Measurement precision
If traditional web crawlers process all webpage elements, then complete data is extracted, but processing efficiency decreases
Solution Approach 1:
The patent applies local quality by treating different elements of the webpage differently based on their importance. Critical product data elements (prices, titles, images) are processed with high priority and detailed extraction, while non-critical elements (ads, social media links, navigation) are suppressed or processed minimally. This differentiated approach maintains extraction completeness for product data while significantly improving processing efficiency
Solution Approach 2:
The patent employs partial action by selectively processing only the necessary portions of each webpage required for product data extraction. Instead of processing all elements equally, the system applies extraction logic only to relevant product information elements, achieving complete product data collection without the overhead of processing unnecessary webpage content
Data Source
AI summary
A computer system extracts product data from a website and correlates product records from multiple sources to one another as corresponding to the same product. A website is crawled efficiently by rendering webpages using a virtual browser that ignores blacklisted elements, extracts data from objects without rendering, and suppressing retrieval of remote resources. Data is extracted according to engine control statements including a selector and extractor. A website may be crawled repeatedly and changes in extracted data may be detected and flagged. Engine control statements may be automatically changed in response to detecting a change in the configuration of the website. Images of product records may be correlated with one another by first comparing text of the product records and selecting images for comparison based on composition. Images are compared using a machine learning model. Images determined to be similar may be presented to a human for a correlation decision.