On-Demand RPA via Screen Capture and Confusion Indices
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional process automation systems lack cognitive capabilities to learn and adapt continuously, follow user actions, determine user intent, and predict goals, leading to inefficiencies in repetitive tasks and inability to provide self-learning agents or share automation methods effectively.
Innovation Solution
A system utilizing a hardware processor to execute applications with a monitoring program for screen capture and image processing, generating metadata and confusion indices to create a decision table and process automation model, enabling adaptive process automation by identifying successful user interactions and validating models with adaptive threshold values.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If conventional process automation systems are used to perform repetitive tasks, then task execution can be automated, but the systems lack cognitive capabilities to learn and adapt continuously
Solution Approach 1:
The system employs machine learning models that enable the automation system to self-learn from user interactions and system behaviors. The cognitive engine continuously trains models using captured screen images, user actions, and task outcomes, allowing the system to improve its automation capabilities autonomously without requiring manual reprogramming for each new task scenario.
Solution Approach 2:
The system implements continuous feedback loops where user actions, system responses, and task outcomes are captured and fed back to the machine learning models. This feedback mechanism enables the system to learn from successful and unsuccessful task completions, adjusting its behavior to improve future automation performance and adapt to changing requirements.
2Adaptability or versatility
If current systems attempt to follow user actions and determine user intent, then goal-oriented automation can be achieved, but the systems are technologically incapable of doing so
Solution Approach 1:
The system replaces traditional rule-based mechanical automation approaches with cognitive systems using machine learning and image processing. Instead of pre-programmed sequences, the system uses AI models to interpret screen images, recognize user intent, and dynamically determine optimal automation paths, enabling goal-oriented automation without explicit programming.
Solution Approach 2:
The system introduces a cognitive engine as an intermediary between user actions and automation execution. This engine captures and analyzes user interactions, interprets intent through machine learning models, and translates them into automated actions, bridging the gap between human behavior and machine execution.
3Measurement precision
If the system captures and analyzes user interactions to create automation models, then accurate process automation can be generated, but significant processing and computational resources are required
Solution Approach 1:
The system performs preliminary actions by capturing and storing screen images, user actions, and task outcomes during the learning phase. This pre-captured data is then used to train machine learning models offline, reducing the computational burden during actual automation execution while maintaining high accuracy in process automation generation.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
Enables continuous learning and efficient task accomplishment by understanding user goals and optimal paths, guiding users, and suggesting optimal processes, thus improving automation capabilities across multiple applications and devices.
Implementation Method 1
identifying, via the hardware processor, one or more graphical user interface elements included in the one or more graphical user interfaces, using one or more computerized image processing techniques
Data Source
AI summary
System and methods for creating on-demand robotic process automation are described. In one example hardware processor-implemented method embodiment, an application providing graphical user interfaces is executed, and a monitoring program executing independently of the application. Using the monitoring program, a screen capture image is obtained of the graphical user interfaces. Graphical user interface elements are identified using a computerized image processing technique. Metadata on presence of confirmatory predictors for the elements, and confirmatory predictor vectors for the interfaces, are generated. Confusion indices for the confirmatory predictors and the confirmatory predictor vectors are calculated, Threshold values are generated based on the calculated confusion indices. A decision table is generated storing sequences of user interactions with the graphical user interface elements. A subset of the stored sequences is identified as yielding a successful outcome. A process automation model is generated and validated based on the identified subset, using the threshold values.