UI Anchor Determination for Duplicate Element Identification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current UI automation technologies require user intervention and manual input for anchor detection, leading to variability in anchor selection across users, which complicates the identification of target graphical elements.
Innovation Solution
Implement a context-based mechanism that automatically determines additional anchors for target UI elements, using positional and geometric associations, UI descriptors, and multi-anchor matching to ensure unique identification, even when a single anchor is insufficient.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If manual anchor detection is used, then user flexibility is maintained, but anchor selection variability increases and reliability decreases
Solution Approach 1:
The system automatically determines anchors by analyzing the target graphical element's properties and context without requiring user intervention. The computer automatically selects anchors based on geometric relationships, positional associations, and UI descriptors, making the system self-sufficient in anchor determination while ensuring consistent and reliable anchor selection across different users and sessions.
2Reliability
If multiple anchors are used to ensure unique identification, then target identification reliability improves, but system complexity increases
Solution Approach 1:
The system performs preliminary analysis of the target graphical element to pre-determine suitable anchors before the actual identification process. By analyzing UI descriptors, geometric properties, and contextual relationships in advance, the system prepares a set of candidate anchors that can be efficiently evaluated during runtime, reducing the complexity of the matching process while ensuring reliable target identification.
Solution Approach 2:
The anchor determination process is divided into distinct segments: analyzing target properties, identifying candidate anchors based on geometric and positional criteria, evaluating anchor suitability using UI descriptors, and selecting the final anchor set. This segmentation of the complex identification process into manageable stages reduces overall system complexity while maintaining high identification reliability.
3Productivity
If automatic anchor determination is implemented, then productivity increases, but handling of ambiguous targets becomes more difficult
Solution Approach 1:
The system incorporates feedback mechanisms that analyze the results of anchor determination and target identification to continuously improve its performance. When the system encounters ambiguous targets or duplicate anchors, it uses feedback from the identification outcome to adjust its anchor selection criteria, refine its geometric and positional analysis, and enhance its ability to handle ambiguous situations in future operations, thereby maintaining high productivity while improving difficulty handling.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Automatic anchor determination for target graphical element identification in user interface (UI) automation is disclosed. A context-based mechanism assists in discriminating between duplicate target UI element candidates. More specifically, additional anchors may be determined and automatically added for a target UI element that provide context and are visible in an area surrounding the target. During design time, a target UI element may be indicated by a user of a designer application and a corresponding anchor may be determined. When a pair of UI elements is found having the same or similar characteristics and/or relationships to the target- anchor pair, an additional anchor is automatically identified without requesting user input. The additional anchor may be selected from the UI elements within a radius of the target UI element.