Fallback UI Element Locators for Resilient Automation Playback

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Automation programs fail to locate target user interface (UI) elements due to changes in software application user interfaces, such as updates or dynamic elements, leading to unsuccessful automation processes.

Innovation Solution

Implementing fallback element locators, including candidate XPaths generated by machine learning models, to identify UI elements during playback, and utilizing native systems to create and prioritize fallback XPaths based on UI element interactions during design time.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If automation programs use fixed UI element locators recorded during design time, then automation execution is simple and fast, but automation fails when UI changes occur

Engineering Contradiction:

Improveautomation execution success rateVSAvoidelement locator mechanism complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The system performs preliminary actions by generating multiple candidate fallback element locators during design time before automation playback. These candidate locators are stored and ready to be used if the primary locator fails, preventing automation failure due to UI changes without adding complex runtime decision-making

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system changes the parameter of element locator from a single fixed value to multiple candidate values with different priorities. When UI changes occur, the system tries alternative locator parameters in priority order until a successful match is found, thereby adapting to UI variations while maintaining a structured approach

Inventive Principle:

Principle #35Parameter changes

2Reliability

If multiple fallback element locators are generated and stored, then automation resilience to UI changes improves, but system complexity and resource usage increase

Engineering Contradiction:

Improveautomation resilience to UI changesVSAvoidnumber of element locators stored

Core Design Contradiction:

ReliabilityVSQuantity of substance

Solution Approach 1:

The system applies local quality by creating candidate fallback element locators specifically for UI elements that are critical to automation success. Not all UI elements generate fallback locators - only those identified as important during design time, thereby reducing the total number of locators stored while maintaining resilience for critical operations

Inventive Principle:

Principle #3Local quality

Solution Approach 2:

The system uses partial action by generating a limited number of candidate fallback locators (e.g., top 3-5 candidates) rather than exhaustively generating all possible locators. This provides sufficient resilience for most UI changes while avoiding the overhead of storing and processing an excessive number of candidates

Inventive Principle:

Principle #16Partial or excessive action

3Adaptability or versatility

If candidate fallback locators are generated using machine learning models, then automation can adapt to UI changes, but processing time and computational resources increase

Engineering Contradiction:

Improveautomation adaptability to UI changesVSAvoidfallback locator generation time

Core Design Contradiction:

Adaptability or versatilityVSLoss of time

Solution Approach 1:

The system performs the computationally intensive machine learning-based fallback locator generation during design time rather than during playback. This preliminary action ensures that all adaptive capabilities are prepared in advance, so during actual automation execution, the system only needs to evaluate pre-generated candidates, minimizing runtime overhead

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system uses cost-effective and efficient techniques for generating fallback locators, such as template-based approaches or simple pattern matching for common UI elements, rather than always using complex machine learning models. This balances adaptability with computational efficiency, especially for frequently updated or simple UI elements

Inventive Principle:

Principle #27Cheap short-living objects (Disposable)

Data Source

PatentUS20250315279A1Fallback user interface identification techniques for automation processes

Publication Date: 2025.10.09 AUTOMATION ANYWHERE INC

AI summary

Systems and methods for locating target user interface (UI) elements that automation programs intend to interact when performing automation programs. The automation programs can, for example, operate to perform actions as part of a workflow process to complete tasks. Advantageously, when UI elements referenced in automation programs cannot be located based on information recorded about the UI elements during the phase of designing the automation programs. One approach can involve creating candidate fallback element locators (e.g., XPaths) for UI elements during the automation program design phase. If an automation program fails during playback, one or more of these candidate fallback element locators can be used to locate a target UI element. In another approach, when an automation program playback fails, portions of a software application's user interface code can be identified and used as inputs to machine learning model(s), which can generate candidate fallback element locators that can used to locate target UI elements.