UI Element Detection via Dynamic Recovery Strategies
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Automated testing applications face challenges in correctly identifying and locating user interface elements due to changes in marker attributes, requiring substantial manual updates and maintenance, especially in dynamic user interfaces, leading to inefficiencies and increased resource consumption.
Innovation Solution
An AI-driven test execution framework that uses scenario documents written in human-readable intermediate language to automatically locate and interact with user interface elements, even when identifiers change, by capturing and comparing initial and subsequent DOM data, and employing recovery strategies like tree localization, multiple-attribute, and visual positioning approaches.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If automated testing applications use marker attributes to locate user interface elements, then element identification is straightforward, but the system fails when marker attributes change requiring manual updates
Solution Approach 1:
The system dynamically adapts its element location strategy by switching from static marker attribute matching to dynamic visual and structural analysis when changes are detected. The automated testing application monitors for attribute changes and automatically adjusts its localization methodology, transforming from a rigid marker-dependent system to a flexible multi-strategy system that maintains reliability despite UI changes.
Solution Approach 2:
The system changes its operational parameters by transitioning from relying on specific marker attributes to using visual characteristics and DOM structural parameters. When marker attributes change, the system alters its identification parameters to use alternative attributes, visual patterns, or positional relationships, thereby maintaining element location accuracy without manual intervention.
2Measurement precision
If manual updates are performed to maintain test scripts, then element location accuracy is preserved, but maintenance time and resource consumption increase
Solution Approach 1:
The automated testing system performs self-maintenance by automatically detecting when user interface elements change and autonomously updating its element location strategies. The system monitors its own test execution results, identifies failed element locations, and self-corrects by alternative location methods without requiring human intervention, thereby eliminating manual maintenance time while preserving location accuracy.
Solution Approach 2:
The system implements a feedback loop where test execution results are analyzed to detect element location failures. This feedback triggers automatic recovery processes that attempt alternative location strategies, and successful recoveries are fed back into the system for future use. This continuous feedback mechanism maintains element location accuracy while eliminating the need for manual maintenance updates.
3Ease of manufacture
If traditional automated testing is used, then initial setup is straightforward, but substantial manual maintenance is required for dynamic user interfaces
Solution Approach 1:
The system segments the element location process into multiple independent strategies (marker attribute matching, visual pattern recognition, DOM structural analysis). Each segment can operate independently or in combination, allowing the system to maintain simple initial setup while automatically selecting or combining segments based on the specific UI context, thereby reducing manual maintenance requirements.
Solution Approach 2:
The automated testing system implements multi-functional element location capabilities that can handle both static and dynamic user interfaces using the same core framework. The system universally applies multiple location strategies across different UI types and change scenarios, eliminating the need for separate manual maintenance processes for different interface types and thereby increasing automation extent.
Data Source
AI summary
The present disclosure involves systems, software, and computer implemented methods for automatically detecting user interface elements. One example method includes accessing master frame information for a user interface of an application. The master frame information includes first captured user interface information captured during a first execution of a scenario for the application. A subsequent execution of the scenario is performed, including capturing second captured user interface information. A determination is made that the subsequent execution of the scenario has not passed successfully, due to a non-functional error. A determination is made that the non-functional error is based on a failure to locate a user interface element specified in the master frame. A recovery strategy is performed, using the first captured user interface information and the second captured user interface information, to automatically locate the user interface element.