Web Crawler Configuration Tool for Automated Extraction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current web page information extraction methods, particularly those based on regular expressions, require manual configuration and have low automation levels, making them inefficient for high-volume extraction and requiring technical expertise, which poses challenges for non-professionals and increases development costs due to the need for multiple targeted extractors for varying website structures.
Innovation Solution
A web crawler configuration tool using a graphical user interface (GUI) that allows users to configure web crawling actions through a hierarchical structure, such as a DOM tree, enabling selection of nodes and configuration of extraction options like text extraction, image capture, and form processing, with options applicable to multiple nodes and providing a preview of actions, thereby simplifying the extraction process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If multiple targeted extractors are used to handle different website structures, then extraction accuracy for specific websites is improved, but device complexity and maintenance difficulty increase significantly
Solution Approach 1:
The patent creates a universal extractor that can handle multiple website structures through automatic classification. The system analyzes webpage characteristics and automatically routes requests to appropriate extraction templates, eliminating the need for manually maintaining multiple targeted extractors while preserving extraction accuracy across different website types.
Solution Approach 2:
The system performs automatic classification of website structures without manual intervention. The extractor automatically identifies the type of website being accessed and selects the appropriate extraction template, enabling the system to serve itself rather than requiring manual configuration for each new website structure.
2Measurement precision
If manual configuration of regular expressions is required, then extraction precision can be improved, but ease of operation deteriorates due to high technical threshold
Solution Approach 1:
The patent introduces an automatic classification system as an intermediary between the user and the extraction process. Instead of requiring users to directly configure regular expressions, the system automatically classifies the website structure and selects appropriate extraction templates, thereby maintaining extraction precision while eliminating the need for users to master regular expression syntax.
Solution Approach 2:
The system replaces the manual mechanical process of configuring regular expressions with an automated classification and template selection mechanism. This substitution eliminates the need for users to manually write and configure complex regular expressions while maintaining or improving extraction precision through systematic template matching.
3Reliability
If targeted extractors are developed for each website type, then extraction reliability is improved, but productivity decreases due to high development costs
Solution Approach 1:
The patent segments the extraction system into modular components: a classification module that identifies website types and template modules that handle specific extraction tasks. This segmentation allows the system to maintain reliability for different website types while improving productivity, as new website types can be handled by adding new classification rules rather than developing complete extractors from scratch.
Solution Approach 2:
The system performs preliminary classification of website structures before extraction. By pre-analyzing and categorizing website types, the system prepares the appropriate extraction template in advance, ensuring reliable extraction while reducing development time. The classification framework is established beforehand, allowing rapid deployment for new website types.
Data Source
AI summary
Web crawling configuration includes: obtaining a webpage comprising a plurality of receiving a user selection of a node in the webpage; presenting a set of web crawling configuration options pertaining to a web crawling action to be performed with respect to the node, the set of web crawling configuration options depending at least in part on a type of an element included in the node and comprising: a first option to perform a first web crawling action in the event that the node include a first type of the element; and a second option to perform a second web crawling action in the event that the node includes a second type of the element; receiving a user input specifying the web crawling configuration option; and storing user specified web crawling configuration option, performing the web crawling action on the node according to the user input, or both.