RPA Activity Naming via Auto-Detected UI Labels
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current robotic process automation (RPA) systems require advanced programming skills to identify and interact with user interface elements, making them inaccessible to developers without formal training, and existing methods for automating repetitive tasks are not intuitive enough for non-technical users.
Innovation Solution
The development of a method and system that uses graphical user interface (GUI) tools and computer vision techniques to automatically identify user-facing labels of target elements within a user interface, allowing for the creation of RPA scripts that mimic human interactions, such as mouse clicks and text inputs, through a user-friendly interface.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If traditional RPA programming methods are used, then automation functionality is achieved, but programming complexity and skill requirements increase
Solution Approach 1:
The system enables self-service automation by allowing users to interact with UI elements through natural GUI operations (clicks, hovers) rather than requiring programming knowledge. The robot automatically identifies and interacts with target elements based on user actions, eliminating the need for users to write or understand code for automation tasks.
Solution Approach 2:
An intermediary layer is introduced between the user and the automation execution. This layer captures user interactions with the GUI, translates them into automation scripts, and manages the robot's execution. The intermediary handles the complexity of script generation and element identification, presenting a simplified interface to users.
2Extent of automation
If traditional RPA programming methods are used, then automation functionality is achieved, but ease of operation decreases
Solution Approach 1:
Users perform automation setup through natural GUI interactions rather than programming. The system captures these interactions and automatically generates the corresponding automation logic, making the process as easy as manually performing the task once.
Solution Approach 2:
The system copies user interactions with the GUI and uses them as the basis for automation scripts. By recording and replicating manual operations, the system creates automated versions of tasks without requiring users to understand programming concepts.
3Ease of operation
If computer vision techniques are used to identify UI elements, then accessibility to non-technical users improves, but processing time increases
Solution Approach 1:
The system performs preliminary actions by capturing screenshots of the UI and pre-processing them during the design phase. UI elements are identified and stored with their visual characteristics in advance, so that during execution, the robot can quickly match elements without performing complex image processing from scratch.
Solution Approach 2:
Instead of performing real-time computer vision analysis during automation execution, the system creates copies of UI element characteristics (visual features, positions, hierarchies) during setup. These copied characteristics are stored and used for rapid element identification during robot execution, significantly reducing processing time.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Some embodiments describe improvements to an interface for designing robotic process automation (RPA) software. In response to receiving user input that selects a target UI element for an RPA activity (e.g., mouse click, text input), some embodiments automatically identify a label element co-displayed with the respective target element. An activity configuration interface displays an activity name and/or description using the label element as a descriptor and/or substitute for the target element, making the interface more intuitive and accessible to developers without a programming background.