Autonomous GUI Interaction System Using Semantic Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional automation systems lack the intelligence and intrinsic understanding of computer systems, leading to increased specificity and maintenance requirements as tasks become more complex, limiting their ability to handle unexpected deviations and resulting in potential failure and increased costs.
Innovation Solution
A computer-implemented system that uses machine learning and artificial intelligence to autonomously interact with external devices by interpreting GUI data into semantic objects, developing a system model, and executing tasks while accounting for environmental perturbations and system randomness, employing engines like Workflow, Rules, and Event Engines.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional automation procedures are used to interact with computer systems, then explicit control and precision are achieved, but the system cannot account for unexpected deviations and requires high maintenance costs
Solution Approach 1:
The system employs machine learning models that enable the automation system to learn and adapt autonomously from observed interactions and deviations, eliminating the need for manual procedure updates and human intervention to maintain robustness against unexpected system behaviors
Solution Approach 2:
The automation system transitions from static, pre-defined procedures to dynamic, adaptive workflows that can modify their behavior in real-time based on observed system states and deviations, allowing the system to handle unexpected situations without increasing procedural complexity
2Adaptability or versatility
If traditional automation procedures are used to interact with computer systems, then specific interactions are automated, but the system lacks adaptability to handle unexpected deviations
Solution Approach 1:
The system implements continuous feedback loops where machine learning models observe system interactions, learn from deviations and unexpected behaviors, and automatically adjust automation procedures to improve adaptability while maintaining high levels of autonomous operation
Solution Approach 2:
Machine learning models serve as intermediaries between the automation system and the computer system being interacted with, enabling the system to handle unexpected deviations through learned patterns without requiring direct human intervention or reducing automation extent
3Ease of manufacture
If traditional automation procedures are used, then initial development is straightforward, but maintenance costs increase over time to account for system limitations
Solution Approach 1:
The machine learning-based automation system automatically learns and adapts to changing system behaviors and requirements, eliminating the need for manual maintenance and procedure updates, thereby reducing maintenance costs while preserving ease of initial development
Data Source
AI summary
According to an embodiment of the present invention, a computer implemented system that automates development, maintenance and execution of procedures to autonomously interact with one or more external devices, comprises: an input configured to receive interaction data and to detect state data from an external computer system, the user interaction data comprising GUI data; a memory component configured to store the interaction data, the state data and relationship data between objects, events and resultant states where an event represents an interaction with the external computer system and where a resultant state represents a state resulting from an interaction; a semantic processor configured to interpret the interaction data into semantic objects and develop a system model, using a learning algorithm, based on the semantic objects, the state data and the relationship data; and an execution processor configured to execute tasks and roles accounting for environmental perturbations and system randomness.