Browserless Web Data Extraction for Product Variant Retrieval
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data extraction techniques from web pages require cooperation from the website and involve computationally expensive rendering and bandwidth consumption, especially when dealing with multiple product variants, leading to inefficiencies.
Innovation Solution
Automated data extraction is performed without using a browser and without rendering the entire web page, by parsing structured data from visible and hidden elements, simulating browser operations to obtain information such as product variants like color, size, and availability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional scraping techniques are used to obtain information from websites, then data can be extracted from rendered web pages, but the process becomes computationally expensive and consumes excessive bandwidth
Solution Approach 1:
The patent extracts only the essential structured data elements (product title, price, availability, variants) from the web page HTML without requiring full rendering. The system identifies and extracts data directly from the page source code, separating the necessary information from the complete visual presentation, thereby reducing computational overhead while maintaining extraction accuracy.
Solution Approach 2:
The patent segments the web page data extraction process into specific target elements (product information, pricing, availability, variants) rather than processing the entire rendered page. By focusing extraction on discrete data points and their corresponding HTML structures, the system minimizes computational resources while ensuring complete data capture.
2Measurement precision
If traditional scraping techniques are used to obtain information from websites, then data can be extracted from rendered web pages, but the process requires significant bandwidth consumption
Solution Approach 1:
The system extracts only the minimal necessary HTML structures containing product information without downloading or rendering the complete web page. By targeting specific data elements and their associated HTML code, the patent reduces bandwidth consumption while maintaining complete data extraction accuracy for all product variants.
3Loss of information
If JavaScript evaluation is performed to reveal information about product variants, then complete product information can be obtained, but the process leads to a combinatorial problem when dealing with multiple variants
Solution Approach 1:
The patent performs preliminary extraction of product variant information directly from the HTML source code before any JavaScript execution or combinatorial processing is needed. By identifying and extracting all variant data (size, color, price, availability) in advance from the page structure, the system avoids the combinatorial complexity that would arise from evaluating JavaScript for each variant combination.
Solution Approach 2:
The system segments the extraction of product variant information into discrete, independently extractable elements from the HTML structure. Each variant attribute (size, color, price, availability) is extracted as a separate data point, eliminating the need for complex JavaScript evaluation and combinatorial processing while ensuring complete information capture.
4Loss of information
If multiple requests or scrapes are performed to obtain information about multiple product configurations, then complete product data can be retrieved, but the process becomes inefficient
Solution Approach 1:
The patent merges the extraction of all product configuration information (different sizes, colors, prices, availability) into a single scraping operation. By identifying the structured HTML elements that contain all variant information and extracting them simultaneously, the system retrieves complete product configuration data in one efficient pass, eliminating the need for multiple separate requests or scrapes.
Data Source
AI summary
Various embodiments provide techniques for automatically extracting data from web pages. Such extraction can take place without the use of a browser, and without necessarily rendering the entire web page. Thus, data extraction can be performed more efficiently and more quickly, while reducing the computing resources needed to perform such operations. In at least one embodiment, data extraction and translation are performed by automatically parsing structured data from visible and hidden elements of a web page.