RPA Target Element Identification via Visual Fallback
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Robotic process automation (RPA) systems face challenges in unambiguously identifying target user interface elements due to changes in the appearance of interfaces, such as positioning, color schemes, and fonts, between design and runtime, leading to potential failures in automation tasks.
Innovation Solution
The method involves using a script that encodes both a target ID and visual appearance data for a user interface element, allowing the system to search for matching elements at runtime and automatically identify the target element based on these criteria, even if the target ID does not match, ensuring robustness and scalability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the system uses target ID to identify user interface elements, then the identification process is simple and fast, but the system fails when the interface appearance changes between design and runtime
Solution Approach 1:
The system performs preliminary actions by capturing screenshots of the user interface at design time and extracting visual features (color, shape, position, size) before runtime. These pre-extracted features are stored and used during runtime to identify target elements, eliminating the need for complex real-time analysis and ensuring reliable identification even when interface appearances change.
Solution Approach 2:
The patent introduces visual feature extraction as an intermediary mechanism between the target ID and the actual element identification. Instead of directly relying on target IDs or complex real-time image processing, the system uses pre-extracted visual features (color histograms, shape descriptors, position coordinates) as intermediaries to match and identify target elements at runtime, simplifying the overall identification process while improving reliability.
2Measurement precision
If the system performs comprehensive visual analysis to identify target elements, then identification accuracy improves, but processing time increases
Solution Approach 1:
The system performs comprehensive visual analysis in advance by capturing screenshots and extracting detailed visual features (color histograms, shape descriptors, position, size) during the design phase. This preliminary processing allows the runtime system to quickly match pre-extracted features with current interface elements, achieving high identification accuracy without time-consuming real-time analysis.
Solution Approach 2:
The patent extracts specific visual features (color, shape, position, size) from the user interface elements during design time and stores them for later use. By extracting and storing only the essential identifying features rather than performing complete image analysis at runtime, the system achieves accurate element identification with minimal processing time during execution.
3Adaptability or versatility
If the system uses multiple identification criteria (target ID and visual appearance), then robustness against interface changes improves, but the complexity of the identification process increases
Solution Approach 1:
The system performs preliminary extraction of multiple visual features (color histograms, shape descriptors, position coordinates, element size) during the design phase and stores them for runtime use. This advance preparation enables the system to robustly identify target elements even when interface appearances change, while keeping the runtime process simple by using pre-computed features for direct comparison.
Solution Approach 2:
The patent merges multiple identification criteria (target ID matching and visual appearance matching) into a unified identification process. By combining these approaches and using pre-extracted visual features as a fallback or complementary mechanism to target ID matching, the system achieves adaptability to interface changes without significantly increasing process complexity, as both methods work together in a coordinated manner.
Data Source
AI summary
A software robot is designed to carry out an activity (e.g., a mouse click, a text input, etc.) on a target element (e.g., a button, an input field, etc.) of a user interface. The robot's code specification is configured to include an on-screen image of the target element and a text displayed by the target element. The robot is configured to automatically identify the target element at runtime according to an element ID specified in the source code of the user interface, and when such identification fails, to identify the target element according to the text and image stored in the robot's code.