Mobile RPA Workflow Authoring Through a Mirrored Device GUI
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Robotic process automation (RPA) on mobile computing platforms faces challenges due to hardware and software heterogeneity, requiring substantial knowledge of hardware and software engineering, and differing user interaction methods such as gestures, making conventional RPA systems from desktop computers not easily translatable to mobile devices.
Innovation Solution
A method employing a hardware processor to execute an RPA design application that displays a model GUI mimicking a mobile device, allowing user input for gestures and target element selection, resulting in an RPA script that automates interactions on mobile devices, including touchscreen gestures and operations, thereby simplifying RPA development for mobile platforms.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If conventional RPA systems designed for desktop computers are used, then automation of repetitive computing tasks can be achieved, but the systems cannot easily be translated to mobile platforms due to hardware and software heterogeneity
Solution Approach 1:
The patent creates a model GUI that copies the visual appearance and structure of the target mobile device interface. This model serves as a template that the RPA system can work with without needing to understand the underlying hardware and software differences of various mobile devices. The model GUI captures the essential UI elements and their relationships, allowing the same automation script to work across different mobile platforms.
Solution Approach 2:
The model GUI acts as an intermediary layer between the RPA automation system and the diverse mobile device interfaces. Instead of the RPA system directly interacting with various mobile device UIs (which would require handling hardware and software heterogeneity), all interactions go through the standardized model GUI, which translates them into device-specific actions.
2Productivity
If RPA automation is implemented on mobile devices, then productivity can be improved by automating mobile tasks, but substantial knowledge of hardware and software engineering is required
Solution Approach 1:
By copying the mobile device interface into a model GUI that can be displayed and manipulated on a desktop computer, the system allows users to create and test automation scripts in a familiar desktop environment. Users can interact with the model GUI using standard desktop input devices (mouse, keyboard) rather than needing to physically handle mobile devices or understand mobile-specific programming constraints.
Solution Approach 2:
The patent replaces the need for direct physical interaction with mobile devices (touchscreen gestures, physical button presses) with desktop-based interaction methods. The model GUI translates desktop mouse clicks and keyboard inputs into the corresponding mobile device actions, eliminating the need for users to have specialized knowledge of mobile device operation and programming.
3Measurement precision
If user gestures such as tapping, swiping, and pinching are supported, then mobile device interaction accuracy can be improved, but the complexity of gesture recognition and implementation increases
Solution Approach 1:
The model GUI copies not only the visual appearance of the mobile interface but also the gesture interaction patterns. When a user performs a gesture on the model GUI (such as swiping across the touchscreen display), the system recognizes this gesture and translates it into the corresponding action on the target mobile device. This allows accurate gesture recognition to be implemented once in the model GUI rather than needing to be implemented separately for each mobile device.
Data Source
AI summary
In some embodiments, a robotic process automation (RPA) design application provides a user-friendly graphical user interface that unifies the design of automation activities performed on desktop computers with the design of automation activities performed on mobile computing devices such as smartphones and wearable computers. Some embodiments connect to a model device acting as a substitute for an actual automation target device (e.g. smartphone of specific make and model) and display a model GUI mirroring the output of the respective model device. Some embodiments further enable the user to design an automation workflow by directly interacting with the model GUI.