Web Inventory Page Extraction Using Adaptive HTML Pattern Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for extracting product information from web pages of different merchants are inefficient, inaccurate, and require substantial human labor, failing to adapt to varying merchant configurations and periodic changes in product offerings.
Innovation Solution
A machine-automated, merchant-agnostic process using computer scripts and machine learning to identify inventory pages and extract product information, such as names and prices, without prior knowledge of web page setups, utilizing HTML structure detection and machine learning to enhance accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If machine-based methods are used to extract product information from web pages, then automation is improved, but accuracy deteriorates due to inability to adapt to different merchant configurations
Solution Approach 1:
The system dynamically adapts its extraction approach by detecting recurring HTML structures across different merchant websites and adjusting its parsing logic accordingly. The machine learning component continuously learns from new merchant configurations, enabling the system to maintain high accuracy while automating extraction across diverse web page layouts and structures.
Solution Approach 2:
The system changes its extraction parameters based on the detected HTML structure patterns of different merchants. By identifying recurring structural elements and adapting extraction rules dynamically, the system maintains high accuracy across varying merchant configurations without requiring manual reprogramming for each new merchant.
2Measurement precision
If human labor is used to extract product information, then accuracy is improved, but productivity deteriorates due to manual effort requirements
Solution Approach 1:
The system performs self-learning by automatically detecting recurring HTML structures from newly encountered merchant websites and updating its extraction logic without human intervention. This self-service capability enables the system to maintain human-level accuracy while achieving automated extraction speeds, eliminating the need for manual configuration for each new merchant.
Solution Approach 2:
The system incorporates feedback mechanisms where extraction results are continuously analyzed to improve future extractions. By learning from successful extractions and adjusting its patterns accordingly, the system achieves both high accuracy and automated efficiency, combining the best of human expertise with machine speed.
3Extent of automation
If existing machine-based methods are used, then automation is improved, but adaptability deteriorates when dealing with different merchants and online platforms
Solution Approach 1:
The system achieves universal adaptability by detecting recurring HTML structure patterns that are common across different merchants and platforms. A single automated system can extract data from multiple online platforms and merchant configurations by identifying and adapting to these universal structural patterns, eliminating the need for merchant-specific customization.
Solution Approach 2:
The system dynamically adapts to different merchants and platforms by continuously learning their specific HTML structure patterns while maintaining a universal extraction framework. This dynamic adaptability allows the automated system to handle diverse online platforms and merchant configurations effectively.
Data Source
AI summary
The present disclosure provides a method of automatically extracting data from web pages and analyzing the extracted data to generate an output. A plurality of web pages of a plurality of merchants is accessed. Based on the accessing of the web pages, a subset of the plurality of web pages is identified as inventory pages that contain information about products or services offered for sale. The inventory pages are electronically scanned to extract a price for each of the products or services. An output is generated that includes a listing of the products or services and prices associated with the products or services, respectively.