Hierarchical UI Element Detection for Robust RPA Target Matching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
RPA robots face challenges in accurately identifying and interacting with graphical elements of user interfaces due to changes in appearance and location, which current technologies have not adequately addressed.
Innovation Solution
Implementing a computer-implemented method that uses graphical element search techniques, fuzzy logic selection for anchors and targets, and hierarchical graphical element detection to enhance the accuracy of UI element identification, including selector searches, fuzzy selector searches, image and text matching, and geometric validation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional selector search is used for UI element identification, then the system is simple and fast, but it fails when UI elements change appearance or location
Solution Approach 1:
The system dynamically selects and switches between different search algorithms (selector search, image matching, text matching, CV matching) based on the specific UI element and context. This dynamic adaptation allows the system to maintain high identification accuracy while managing complexity by only using advanced algorithms when necessary.
Solution Approach 2:
The system changes the parameters and approaches of element identification by using multiple different search algorithms with varying levels of complexity. Each algorithm has different parameters (e.g., image similarity thresholds, text matching criteria) that can be adjusted to balance accuracy and computational cost.
2Reliability
If multiple search algorithms are implemented to handle UI changes, then identification accuracy improves, but processing time increases
Solution Approach 1:
The element identification process is segmented into multiple stages with increasing complexity. The system first attempts simple selector search, and only if that fails does it proceed to more complex image matching, text matching, or CV matching. This segmentation reduces average processing time while maintaining high accuracy for challenging cases.
Solution Approach 2:
The system performs partial searches using multiple algorithms simultaneously or in sequence, stopping as soon as a match is found with sufficient confidence. This avoids the excessive action of always running all algorithms to completion, thereby reducing processing time while maintaining identification accuracy.
3Adaptability or versatility
If fuzzy logic and hierarchical detection are used, then flexibility to accommodate UI changes improves, but system complexity increases
Solution Approach 1:
The system implements a universal detection framework that can handle multiple types of UI elements and changes through a single hierarchical architecture. This framework provides adaptability to various UI design changes while managing complexity by reusing the same detection mechanisms across different element types.
Solution Approach 2:
The detection system uses a nested hierarchical structure where simple detection methods are embedded within more complex ones. For example, selector search is nested within image matching, which is nested within CV matching. This nesting provides adaptability to handle UI changes at multiple levels of complexity while organizing the system in a manageable hierarchical manner.
Data Source
AI summary
Graphical element search technique selection, fuzzy logic selection for anchors and targets, and hierarchical graphical element identification for robotic process automation (RPA) are disclosed. The fuzzy logic selection of anchors and targets may be part of a larger, tiered, or hierarchical process for identifying graphical elements in the UI. When a selector for a UI element is not found with at least a confidence threshold, similar elements potentially corresponding to the selector for a UI element target may be searched based on fuzzy matching of the target and corresponding anchor(s). Geometric matching may also be employed between the target UI element and its respective anchor(s). The combination of fuzzy matching and geometric matching may allow for more flexible and accurate identification of the exact selector with which an RPA robot is attempting to interact.