RPA Document Scrolling for Hidden UI Element Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Robotic process automation (RPA) systems face challenges in identifying target UI elements within dynamic user interfaces, particularly when the interface changes or when documents are too large to fit within the UI window, leading to failed automation tasks.
Innovation Solution
The implementation of an automatic scrolling mechanism in RPA robots, which determines if a UI container is scrollable and activates scroll controls to reveal subsequent parts of the document, allowing the robot to continue searching for the target element, thereby enhancing the robot's ability to interact with large documents like spreadsheets and webpages.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the RPA robot attempts to identify target UI elements in large documents without automatic scrolling, then the identification process is simple, but the robot fails to locate elements when they are outside the current UI view
Solution Approach 1:
The system performs preliminary actions by automatically scrolling through the document before attempting to identify the target element. The robot detects whether the element is visible in the current UI view, and if not, it activates scroll controls to bring the element into view before proceeding with identification. This preliminary scrolling action ensures that the target element is always visible when identification is attempted.
Solution Approach 2:
The RPA robot serves itself by autonomously determining when scrolling is needed and executing the scroll action without human intervention. The system monitors the visibility of target elements and automatically activates scroll controls when elements are outside the current view, enabling the robot to independently handle large documents of varying sizes.
2Reliability
If the RPA robot manually scrolls through large documents to find target elements, then element identification becomes possible, but the automation process becomes less reliable due to interface changes
Solution Approach 1:
The system dynamically adapts to interface changes by continuously monitoring the UI environment and adjusting its scrolling behavior accordingly. The robot detects changes in interface layout, element positioning, and scroll control locations, and modifies its identification and scrolling strategy to maintain reliable automation despite interface variations.
Solution Approach 2:
The RPA robot employs feedback mechanisms by continuously checking whether the target element is visible in the current UI view and whether scroll controls are present and functional. Based on this feedback, the system adjusts its behavior - scrolling when elements are hidden, and stopping when elements are visible or no further scrolling is possible, thereby maintaining robustness against interface changes.
3Adaptability or versatility
If the RPA system uses basic element identification without scrolling capability, then the system is easier to implement, but it cannot handle documents larger than the UI window
Solution Approach 1:
The RPA robot achieves universality by implementing a scrolling mechanism that can handle documents of any size within the UI window. The system not only identifies visible elements but also autonomously scrolls to make hidden elements visible, thereby becoming capable of processing both small and large documents with a single unified approach.
Solution Approach 2:
The system transitions from a two-dimensional view (only currently visible UI elements) to a three-dimensional approach by adding the scrolling dimension. The robot can navigate through multiple UI views vertically and horizontally, effectively accessing elements anywhere in the document regardless of their initial position, thereby handling documents of any size.
Data Source
AI summary
In some embodiments, a robotic process automation (RPA) robot is configured to search for a target element within a first part of a document currently exposed within a user interface. When the search fails, the robot may automatically actuate a scroll control of the respective UI to cause it to bring another part of the respective document into view. The robot may then continue searching for the RPA target within the newly revealed part of the document. In some embodiments, the robot automatically determines whether the respective document is scrollable, and identifies the scroll control according to a type of target application (e.g., spreadsheet vs. web browser).