Automated Software Robot Generator for Web Data Extraction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current technologies lack the ability to automatically generate software robots that can manipulate existing web sites and web applications for machine-to-machine communication, leading to a bottleneck in implementing machine-to-machine interactions due to the need for manual coding by skilled engineers, and there is a lack of systems for monitoring and analyzing network transactions to facilitate automated operations.
Innovation Solution
A system and method that generates computer programs to automate data retrieval and manipulation of web content, using tracing and filtering mechanisms to collect and analyze data from web sites, and generate source code that mimics user interactions, enabling robotic manipulation and automated transactions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If manual coding by skilled engineers is used to create software robots for machine-to-machine communication, then the software robots can be customized and optimized, but the implementation time and cost increase significantly
Solution Approach 1:
The system captures actual browser interactions and network transactions as templates, then automatically generates software robot code that replicates these human-performed tasks. This copying approach eliminates manual coding while preserving the functional reliability of the software robots.
Solution Approach 2:
The system enables self-service by allowing the software to automatically generate its own control code through automated analysis of captured transactions. The generated code can be directly executed or lightly modified, eliminating the need for skilled engineers to manually program each software robot.
2Ease of operation
If HTML is used as the data exchange format for web communication, then human users can easily interact with web pages, but machine-to-machine data exchange becomes problematic and inefficient
Solution Approach 1:
The system makes HTML serve dual purposes: it remains the presentation format for human users while simultaneously serving as the data exchange format for machines. The automated analysis and code generation tools enable machines to efficiently process and extract structured data from HTML without requiring a separate data format.
3Adaptability or versatility
If existing web technologies are used without automated generation tools, then web pages can be accessed by human users, but the ability to automatically manipulate web sites for machine operations is lacking
Solution Approach 1:
The system introduces an intermediary layer between existing web technologies and automated operations. This intermediary consists of the automated analysis and code generation tools that translate human-oriented web interactions into machine-executable software robot code, enabling automated manipulation without changing the underlying web technologies.
Data Source
AI summary
A system and a method for using a computer program to automate the process of searching and or retrieving data from a particular data pool for introduction to a user is provided. More specifically, a program is used that may robotically manipulate existing web sites using computer source codes that are generated all or in part by the computer program and/or software. The system may trace, monitor and analyze a browser or network application to produce transactions and manipulate the data retrieved to automatically access previously un-accessible data from a web site.