AI-Driven RPA Workflow Generation from User Interaction Sequences
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current UI automation technologies face challenges in determining sequences of user interactions and generating robotic process automation (RPA) robots, especially when key presses and mouse clicks are not available at the system level, and in identifying efficient common processes without extensive driver and application-level functionality.
Innovation Solution
The use of AI/ML models to analyze time-ordered user interactions, recognize related sequences, and generate RPA workflows by comparing n-grams of user interactions over a sliding window, associating interactions with RPA activities, and determining the most efficient sequences for automation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If driver level and application level functionality are used to implement UI automation, then interaction capability is improved, but device complexity increases
Solution Approach 1:
The patent replaces traditional driver-level and application-level interaction mechanisms with an AI-based visual recognition system. Instead of using low-level keyboard hooks and mouse click simulators, the system uses computer vision to detect UI elements and perform actions based on visual patterns, thereby reducing dependency on complex driver functionality while maintaining interaction capabilities
Solution Approach 2:
The patent introduces an AI-based visual recognition layer as an intermediary between the user interface and the automation system. This intermediary captures visual information from the screen, processes it through neural networks to identify UI elements and their functions, and translates visual patterns into automation actions, eliminating the need for direct driver-level access
2Measurement precision
If n-grams with two or more values of n are compared over a sliding window, then sequence recognition accuracy is improved, but computational time increases
Solution Approach 1:
The patent pre-processes user interaction data into n-gram sequences and stores them in an optimized data structure before training the AI model. By preparing the data in advance with multiple n-gram values already calculated and organized, the system reduces computational overhead during the actual sequence recognition phase, allowing accurate multi-value n-gram comparison without excessive computational time
Solution Approach 2:
The patent implements a dynamic sliding window mechanism that adapts the window size and n-gram values based on the characteristics of the data being analyzed. The system can adjust the granularity of n-gram comparison dynamically, using smaller n-values for quick initial matching and larger n-values for refined verification, thereby balancing accuracy requirements with computational efficiency
Data Source
AI summary
Use of artificial intelligence (AI)/machine learning (ML) models is disclosed to determine sequences of user interactions with computing systems, extract common processes, and generate robotic process automation (RPA) robots. The AI/ML model may be trained to recognize matching n-grams of user interactions and/or a beneficial end state. Recorded real user interactions may be analyzed, and matching sequences may be implemented as corresponding activities in an RPA workflow.