Data Acquisition System for Dynamic Web Event Scraping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional systems for acquiring data from webpages face challenges due to the varied formatting, multiplicity, and temporal changes of events, which they are unable to effectively address.
Innovation Solution
A computer system and method that utilize a selection module, mapping module, navigation module, and storing module to acquire and update structured data from linked pages through screen-scraping, employing headless browser instances to navigate and load pages, and store data in a uniform database format, with an updating module that only updates changed data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If conventional systems are used to acquire data from webpages, then data acquisition can be performed, but the systems cannot effectively address varied formatting, multiplicity, and temporal changes of events
Solution Approach 1:
The system dynamically adapts to changes in webpage formatting and event data structures by implementing flexible data acquisition rules that can be modified based on observed patterns. The system evolves its acquisition strategies over time to handle temporal changes in event data, making the data acquisition process adaptable rather than static.
Solution Approach 2:
The system changes acquisition parameters such as selectors, path data, and classification rules based on the varied formatting of different webpages and events. By adjusting these parameters dynamically, the system maintains reliability across diverse and changing data sources without requiring complete system redesign.
2Productivity
If screen-scraping is performed to acquire structured data from linked pages, then data acquisition capability is improved, but system complexity increases due to multiple modules and headless browser instances
Solution Approach 1:
The system divides the data acquisition process into distinct functional modules: acquisition module for scraping data, selection module for identifying relevant elements, mapping module for classifying data types, navigation module for traversing page structures, and updating module for maintaining data. This segmentation allows each module to specialize in one aspect, improving overall productivity while managing complexity through modular design.
Solution Approach 2:
The headless browser instances serve multiple functions: they navigate to linked pages, load and render pages for scraping, execute selection and mapping operations, and support updating operations. By making the browser instances multi-functional, the system reduces the need for separate specialized components, thereby managing complexity while maintaining high productivity.
3Measurement precision
If all structured data is updated frequently, then data accuracy is maintained, but computational resources and time are wasted on unchanged data
Solution Approach 1:
The updating module implements a feedback mechanism that compares newly acquired data with previously stored data to determine whether updates are necessary. By providing feedback on data changes, the system only updates when accuracy improvements are needed, avoiding wasted computational resources on unchanged data while maintaining data accuracy.
Solution Approach 2:
Instead of updating all structured data uniformly, the system applies partial updates only to specific data elements that have changed. This selective approach performs less than a complete update would, but it is sufficient to maintain accuracy while significantly reducing time and resource consumption.
Data Source
AI summary
Provided are systems and methods for acquiring data hosted on a server and updating acquired data. The system includes an acquisition module for acquiring the data from one or more linked pages each linked to from a main page by an event link by following each event link, the acquisition module including a selection module for selecting one or more event data hosted on the linked page by selecting an element on the linked page and storing path data corresponding to the selected element, a mapping module for classifying a type for each event datum, a navigation module for navigating a navigation structure of each linked page to vary a selection of each element, and a storing module for storing the event links and the event data. The system further includes an updating module for updating the event data according to the stored path data.