Generative AI RPA Workflow Extraction From User Interaction Sequences
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing UI automation technologies face challenges in providing effective automation without system-level kernel hook information and require extensive driver and application-level functionality, while user interactions are often unknown.
Innovation Solution
Utilizing generative AI/ML models to analyze recorded user interactions, determine sequences of user interactions, and generate RPA robots by training models to recognize related sequences through n-grams over a sliding window, with techniques to optimize n-gram values and importance metrics.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If driver level and application level interactions are used for UI automation, then automation functionality is achieved, but system complexity increases and kernel hook information becomes unavailable
Solution Approach 1:
The patent introduces an intermediary AI/ML model that acts as a mediator between user interactions and automation execution. Instead of directly implementing complex driver-level automation, the system uses the AI model to analyze recorded user interactions and generate automation workflows, thereby reducing the need for extensive kernel hook information and simplifying the automation architecture
Solution Approach 2:
The system records and analyzes copies of user interaction sequences rather than directly intercepting kernel-level events. By creating and analyzing copies of user interaction data through recording mechanisms, the system can train AI models to recognize patterns and generate automations without requiring deep system-level access or complex driver functionality
2Measurement precision
If user interactions are recorded and analyzed using n-grams over sliding windows, then sequence recognition accuracy improves, but computational processing time increases
Solution Approach 1:
The system uses n-grams with varying values of n (from 1 to maximum) over sliding windows to analyze user interaction sequences. By applying partial analysis with different n-gram depths, the system can achieve good recognition accuracy for common patterns without always performing the most computationally intensive full-sequence analysis, thus balancing accuracy with processing time
Solution Approach 2:
User interactions are recorded and pre-processed into n-gram sequences in advance before automation execution. This preliminary action of recording and structuring interaction data allows the AI model to be trained offline on historical patterns, reducing the computational burden during runtime when automations need to be executed
Data Source
AI summary
Use of generative artificial intelligence (AI)/machine learning (ML) models is disclosed to determine sequences of user interactions with computing systems, extract common processes, and generate robotic process automation (RPA) robots. The generative AI/ML model may be trained to recognize matching n-grams of user interactions and/or a beneficial end state. Recorded real user interactions may be analyzed, and matching sequences may be implemented as corresponding activities in an RPA workflow.