RPA Document Scrolling for Hidden UI Element Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Robotic process automation (RPA) systems face challenges in identifying target UI elements within dynamic user interfaces, particularly when the interface changes or when documents are too large to fit within the UI window, leading to failures in automating repetitive tasks.
Innovation Solution
The method involves using a hardware processor to attempt to identify a target element within a user interface, and if the initial attempt fails, determining if the UI container is scrollable and activating the scroll control to reveal subsequent parts of the document, allowing another attempt to identify the target element.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the RPA system attempts to identify target UI elements in large documents, then the automation capability is improved, but the identification fails when elements are outside the visible UI area
Solution Approach 1:
The system performs preliminary actions by detecting whether the target UI element is within the visible area before attempting identification. If the element is outside the visible area, the system pre-activates scroll controls to bring the element into view, ensuring successful identification subsequent to this preliminary positioning action.
Solution Approach 2:
The scroll control activation serves as an intermediary action between the system and the target UI element. By activating the scroll control, the system mediates the positioning of the UI container to make the target element visible, enabling subsequent successful identification and interaction.
2Adaptability or versatility
If the RPA system handles dynamic UI changes, then the adaptability is improved, but the robustness against interface changes deteriorates without automatic scrolling
Solution Approach 1:
The system performs preliminary detection to determine whether the target UI element is within the visible area before attempting identification. This preliminary positioning action ensures that subsequent identification operations occur on visible elements, improving reliability against interface changes.
Solution Approach 2:
The system autonomously determines when scrolling is needed and activates scroll controls without human intervention. This self-service capability allows the RPA system to automatically adapt to large documents and dynamic UI changes, maintaining robustness across varying interface conditions.
3Adaptability or versatility
If the RPA system automatically scrolls to find target elements, then the scalability to handle large documents is improved, but the complexity of the automation process increases
Solution Approach 1:
The system performs preliminary detection to determine whether the target UI element is within the visible area before attempting identification. This conditional approach ensures scrolling is only activated when necessary, avoiding unnecessary complexity in the automation process while maintaining scalability.
Solution Approach 2:
The system autonomously manages the scrolling process by detecting visibility conditions and automatically activating scroll controls when needed. This self-service mechanism handles large documents and dynamic UI changes without requiring complex manual scripting, improving scalability while keeping the automation process relatively simple.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
In some embodiments, a robotic process automation (RPA) robot is configured to search for a target element within a first part of a document currently exposed within a user interface. When the search fails, the robot may automatically actuate a scroll control of the respective UI to cause it to bring another part of the respective document into view. The robot may then continue searching for the RPA target within the newly revealed part of the document. In some embodiments, the robot automatically determines whether the respective document is scrollable, and identifies the scroll control according to a type of target application (e.g., spreadsheet vs. web browser).