Web Scraping Validation via Transaction Record Comparison
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing web scraping scripts fail to detect changes in websites, leading to functional issues when website structures are updated, as they cannot autonomously identify the need for updates.
Innovation Solution
A method and system that establish a network connection with web applications, execute scraping instructions, receive electronic messages, generate records from scraped data and messages, validate the scraping instructions by identifying missing transactions, and generate alerts to update the scripts when necessary.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If web scraping scripts are used to extract data from websites, then data can be obtained and processed, but the scripts fail to detect when website changes occur, leading to functional issues
Solution Approach 1:
The patent implements a feedback mechanism where transaction data obtained from web scraping is compared against expected transaction patterns. When discrepancies are detected (indicating website changes), the system generates alerts to notify users that script updates are needed. This closed-loop feedback system continuously monitors script effectiveness and triggers updates when website structures change.
Solution Approach 2:
The system performs preliminary validation by comparing scraped transaction data against known good transaction patterns before full processing occurs. This preliminary check allows the system to proactively identify when website changes have occurred, enabling preventive maintenance of scraping scripts before complete functionality breaks down.
2Reliability
If scraping scripts are manually updated to adapt to website changes, then functionality can be maintained, but the process requires continuous human intervention and time
Solution Approach 1:
The patent implements a self-service mechanism where the system automatically detects website changes by comparing transaction data patterns and autonomously generates alerts for script updates. This eliminates the need for continuous manual monitoring and intervention, allowing the system to self-diagnose functionality issues and notify users only when updates are actually needed, significantly reducing time loss.
Solution Approach 2:
The automated feedback loop continuously monitors scraping effectiveness by comparing obtained transaction data against expected patterns. When changes are detected, the system automatically triggers update notifications, replacing the manual update process with an automated system that reduces both human intervention and time loss.
3Reliability
If validation mechanisms are added to detect script issues, then reliability improves, but system complexity increases
Solution Approach 1:
The patent uses a copying approach by creating simplified reference copies of expected transaction patterns and comparing them against actual scraped data. Rather than implementing complex validation algorithms, the system uses straightforward pattern matching against copied ideal transaction structures, achieving reliable validation with minimal added complexity.
Solution Approach 2:
The validation mechanism focuses on specific critical transaction attributes rather than attempting to validate entire website structures. By applying validation only to key local elements (transaction data patterns), the system achieves effective script validation without requiring comprehensive complex validation of all website components.
Data Source
AI summary
A method that involves establishing a network connection with a plurality of servers executing at least one web application; executing, via the network connection, a set of computer executable scraping instructions on the at least one web application to obtain a set of scraped data; receiving a plurality of electronic messages transmitted via a network; generating a first set of records from the set of scraped data; generating a second set of records from the plurality of electronic messages; validating the set of computer executable scraping instructions by locating a record in the second set of records that identifies a transaction missing in the first set of records to obtain a validation result; and generating, based on the validation result, an alert to update the set of computer executable scraping instructions.