Automated Site Navigation via Key Request Extraction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing techniques for obtaining information from e-commerce websites, such as APIs and web scraping, require cooperation from the website and can be computationally expensive, especially when dealing with multiple product configurations, and are inefficient.
Innovation Solution
Automated sequential site navigation is performed without a browser, recording interactions with a website, analyzing requests and responses to determine data expectations, and simulating browser operations to extract information efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If web scraping techniques are used to obtain information from e-commerce websites, then data extraction capability is improved, but computational cost and resource consumption increase significantly
Solution Approach 1:
The patent extracts only the essential data elements needed from web pages without rendering the entire page. It identifies and extracts specific data points (product information, pricing, availability) directly from the HTML structure, bypassing the need for full page rendering and associated computational overhead.
Solution Approach 2:
The patent segments the web page processing into discrete, targeted extraction operations. Instead of processing the entire page, it divides the task into extracting specific data elements from specific locations in the HTML, reducing overall computational requirements while maintaining extraction effectiveness.
2Loss of information
If traditional web scraping with full page rendering is performed, then complete data extraction is achieved, but processing time increases
Solution Approach 1:
The patent performs preliminary analysis of the HTML structure to identify where required data elements are located before actual extraction begins. It pre-processes the HTML to locate specific data points, enabling direct extraction without the need to render or process the entire page content, thus reducing processing time while maintaining data completeness.
3Adaptability or versatility
If browser-based web scraping is used, then JavaScript-rendered content can be extracted, but resource consumption and complexity increase
Solution Approach 1:
The patent creates a simplified copy of the web page's data structure from the HTML source, extracting only the essential data elements needed. It copies relevant information directly from the HTML without requiring a full browser environment, reducing system complexity while maintaining the ability to handle various content types through direct HTML parsing.
Data Source
AI summary
A website is navigated automatically in a sequential manner, for example by performing a series of specific network requests in a specified order, without necessarily using a browser, and without necessarily rendering each web page in the sequence. Interactions with a website are automatically recorded, stored, and analyzed. Requests and responses can be associated with particular input/output operations. Based on the analysis, the system can determine what data is expected by web servers, so that it can effectively simulate browser operations in connection with those web servers, and extract useful information from the data received in response to those operations. Thus, data extraction and/or other operations can be performed more efficiently and more quickly, while reducing the computing resources needed.