Web Data Extraction Wrapper Generation via HTTP Interaction Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current web data extraction methods, such as visual-clue based wrappers, suffer from significant runtime overhead due to browser rendering and are inefficient, while HTTP wrappers require manual expertise and are less robust, necessitating a method that combines the advantages of both without their disadvantages.
Innovation Solution
A method to automatically generate a second wrapper that can extract web data without using a web browser engine by analyzing the interactions and code of a first wrapper, transforming it into a more efficient HTTP wrapper that suppresses unnecessary requests and maintains robustness.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If visual-clue based wrappers are used for data extraction, then wrapper creation is simplified and robustness is improved, but runtime performance deteriorates significantly due to browser rendering overhead
Solution Approach 1:
The patent creates a copy of the visual wrapper's extraction logic and transforms it into an HTTP-level wrapper that replicates the same data extraction functionality without browser rendering. This copy allows the wrapper to maintain robustness while achieving significant speed improvements by operating at a lower computational layer.
Solution Approach 2:
The patent replaces the mechanical browser rendering system with a direct HTTP request system. Instead of using a web browser engine to parse HTML, render CSS, and execute JavaScript, the system directly sends HTTP requests to the web server and extracts data from the raw responses, eliminating the heavy computational overhead of the rendering mechanism.
2Productivity
If HTTP wrappers are used for data extraction, then extraction speed is improved by avoiding browser rendering, but wrapper creation complexity increases and robustness decreases
Solution Approach 1:
The system automatically generates HTTP wrappers by analyzing the interactions between a visual wrapper and the web server. The wrapper generator captures HTTP requests, responses, and parameter transformations, then automatically constructs the HTTP wrapper without requiring manual programming. This self-service approach eliminates the need for expert manual creation while maintaining robustness through automated analysis.
Solution Approach 2:
The patent implements a feedback mechanism where the visual wrapper executes against the target website, capturing HTTP interactions and parameter transformations. This feedback information is then used to automatically construct the HTTP wrapper, ensuring it correctly handles the server's response format and maintains robust data extraction capabilities.
3Ease of operation
If visual-clue based wrappers are used, then data extraction is performed through browser rendering, but Internet traffic increases due to unnecessary requests for images, fonts, and ads
Solution Approach 1:
The patent extracts only the necessary HTTP requests that contain the required data from the full set of browser-generated requests. By analyzing the visual wrapper's interactions, the system identifies and sends only the essential HTTP requests to the server, eliminating unnecessary requests for images, fonts, CSS, and JavaScript that would otherwise be loaded during browser rendering.
Data Source
AI summary
Methods for automatically generating a wrapper for extracting web data and corresponding computer systems are disclosed. In one arrangement, a first wrapper is used to generate a second wrapper. The first wrapper extracts target data from one or more target web pages hosted by one or more target web servers. The second wrapper is capable of extracting the same target data from the same one or more target web pages without using a web browser engine to perform a) sending requests to the one or more target web servers, and/or b) processing replies from the one or more target web servers. The generation of the second wrapper comprises analysing one or both of the following: (i) code defining the first wrapper, (ii) interactions between the first wrapper and the one or more target web servers that occur during execution of the first wrapper.