Web Page Structure Determination for Automated Scraping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for scraping web pages are inefficient due to the need for manual determination of web page structures, which is unreliable and time-consuming, especially when dealing with multiple domains and changing data structures, leading to network congestion and reluctance from content providers.
Innovation Solution
A system and method for scraping web pages based on a predetermined web page template, which automatically determines the web page structure and stores scraped data in a local cache, allowing for efficient data retrieval and generation of web pages across multiple domains.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If manual determination of web page structures is used, then scraping can be performed, but the process is unreliable and time-consuming
Solution Approach 1:
The system performs self-service by automatically determining web page structures through AI/ML models without requiring manual intervention. The scraper autonomously analyzes HTML documents, identifies data fields, and maps them to templates, eliminating the need for human operators to manually define scraping structures for each website.
Solution Approach 2:
The patent replaces manual mechanical processes with automated AI/ML-based systems. Instead of human operators manually analyzing and defining web page structures, the system uses trained machine learning models to automatically extract, analyze, and map data fields, substituting human cognitive work with computational intelligence.
2Productivity
If scrapers are used to obtain content from multiple domains, then content dissemination is achieved, but network congestion occurs and content providers are reluctant to provide bandwidth
Solution Approach 1:
The system performs preliminary actions by pre-processing and analyzing web page structures before actual scraping operations. By using AI/ML models to pre-identify data fields and map them to templates, the system optimizes scraping queries in advance, reducing the volume of data transferred and minimizing network congestion during actual content retrieval.
Solution Approach 2:
The patent extracts only the specific data fields needed from web pages using AI/ML-based field identification, rather than retrieving entire web pages. The system analyzes HTML structures, identifies relevant data elements, and extracts only those specific fields, significantly reducing network bandwidth consumption compared to traditional full-page scraping methods.
Data Source
AI summary
A system and method for scraping web pages based on a predetermined web page template is disclosed. The method includes: requesting a first plurality of web pages from a web server, each web page including a markup language document having a first plurality of data fields; determining a web page structure for the first plurality of web pages, wherein a first data field of the first plurality of data fields is matched to a second data field of a second plurality of data fields of a web page template; scraping a second plurality of web pages from the web server based on the determined web page structure; and storing scraped data from the second plurality of web pages in a local cache.