UI Element Identification via Visual Signatures
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Software testing tools face challenges in reliably identifying user interface elements after application updates, as changes in element attributes render existing locators ineffective, leading to failed tests and the need for rewritten test scripts.
Innovation Solution
Generating a signature for user interface elements that includes descriptive attributes, allowing for comparison with candidate elements in updated applications to determine matching elements, enabling continued testing operations without updating test scripts.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional locators specifying element attributes are used to identify user interface elements, then test scripts can locate elements in the initial application, but the locators become ineffective after application updates that change element attributes
Solution Approach 1:
The patent transforms the element identification approach by changing from attribute-based locators to visual feature-based signatures. Instead of relying on mutable attributes like IDs or classes that change during updates, the system extracts stable visual parameters (colors, shapes, positions, text content) that remain consistent across application updates, thereby maintaining identification reliability while adapting to changes.
Solution Approach 2:
The patent creates a visual copy or representation of the user interface element through its signature - a set of visual characteristics that replicate the element's appearance and properties. This visual copy serves as a stable identifier that can be matched against updated elements without requiring the original attribute-based locator, enabling the test script to find the corresponding element even after updates.
2Productivity
If application updates are performed to improve functionality, then new features and fixes are deployed, but existing test scripts fail due to changed element attributes
Solution Approach 1:
The patent performs preliminary extraction of visual signatures for all test elements before application updates occur. These signatures are stored and used as the basis for element identification after updates. By preparing the visual fingerprint in advance, the system ensures that test scripts can automatically locate elements post-update without requiring manual script rewriting, thus maintaining test validity while enabling frequent updates.
Solution Approach 2:
The patent implements a feedback mechanism where the visual signature extraction and matching process provides information about element correspondence between updated and original applications. This feedback allows the testing system to automatically adapt to updates by identifying matched elements through signature comparison, rather than relying on brittle attribute-based locators that break with updates.
3Ease of manufacture
If attribute-based locators are used for element identification, then test scripts are simple to write initially, but extensive reconfiguration is needed after updates
Solution Approach 1:
The patent replaces the mechanical attribute-matching system with a visual signature recognition system. Instead of manually configuring locators based on element attributes and maintaining them through updates, the system automatically extracts visual signatures and performs matching. This substitution maintains the simplicity of initial test script creation while dramatically easing maintenance, as the visual matching process automatically adapts to updates without manual reconfiguration.
Data Source
AI summary
Systems and methods for performing automated software testing on user interface elements are disclosed. For instance, a first element of an application can be identified. A signature can be generated for the first element. The signature for the first element can include one or more attributes descriptive of the first element. After an update of the application, a candidate element signature can be generated for one or more candidate elements in the updated application. The signature for the first element can be compared to each candidate element signature to determine whether the first element matches each candidate element, thereby enabling the first element to be located after the update of the application.