RPA UI Control Detection for Resilient Playback Across Screen Variations
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing robotic process automation (RPA) systems face challenges in interacting reliably with user interfaces due to variations in screen size, resolution, and layout, leading to inconsistent playback of software automation processes.
Innovation Solution
Enhanced RPA systems utilize programmatic examination to detect and manipulate user interface controls, enabling resilient playback by altering the visual appearance of displayed interfaces to ensure necessary controls are visible, and employing programmatic interaction to match and execute recorded actions across varying interface conditions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If RPA systems interact with user interfaces using traditional methods, then the system can perform basic automation tasks, but the reliability deteriorates when screen size or resolution varies
Solution Approach 1:
The system changes the parameter of control identification from fixed pixel coordinates to dynamic parameters including control type, text content, and relative position. This allows the bot to locate controls regardless of screen size or resolution variations by matching control characteristics rather than absolute positions.
Solution Approach 2:
The system transitions from static control location methods to dynamic adaptation by detecting control parameters during playback and adjusting control locations based on actual interface conditions. The bot can dynamically scroll or navigate to find controls when their expected positions are not visible.
2Adaptability or versatility
If RPA systems use fixed control location methods, then the playback process is simple to implement, but the system cannot adapt to different screen configurations
Solution Approach 1:
The system introduces an intermediary layer between the bot instructions and the actual interface interaction. This intermediary component detects control parameters, matches them against expected controls, and resolves discrepancies by scrolling or navigating, thereby enabling adaptation without requiring complex modifications to the core bot logic.
Solution Approach 2:
The control detection process is segmented into distinct steps: capturing the interface, detecting control parameters, matching controls, and executing actions. This segmentation allows each step to be optimized independently and simplifies the overall complexity by breaking down the adaptive process into manageable components.
3Ease of operation
If the system scrolls to make controls visible during playback, then controls become accessible regardless of initial visibility, but the playback time increases
Solution Approach 1:
The system performs preliminary detection of control parameters and their expected positions before executing the main playback sequence. By pre-identifying which controls need to be scrolled to or navigated to, the system can optimize the playback path and minimize unnecessary scrolling operations, thereby reducing overall playback time while ensuring all required controls are accessible.
Data Source
AI summary
Robotic process automation (RPA) systems with improved playback capabilities. Certain embodiments can provide resilient playback of software automation processes by providing the capability to record, compute and store parameters for user interface controls detected from a screen image of a user interface. These parameters can be used to assist in locating correct corresponding user interface controls within a screen image presented at playback of a software automation process. Other embodiments can, additionally or alternatively, provide resilient playback of software automation processes by providing enhanced capability to locate user interface controls within a screen image of a user interface. In some embodiments, one or more of the user interface controls located within the screen image of the user interface can be used to manipulate the user interface so that other user interface controls become visible within the screen image. Advantageously, embodiments disclosed herein allow software automation processes to operate with greater reliability and flexibility.