UI Element Reidentification via Local and Global Description Mining
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing automation methods for user interface (UI) elements rely on unique identifiers that are prone to changes in the UI, making them unreliable for consistent automation.
Innovation Solution
The proposed solution involves performing description mining for concepts and reidentification checks for UI elements by generating and storing local and global descriptions associated with concept identifiers, using pattern mining techniques and machine learning models to ensure robust and consistent identification.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If unique identifiers (XPath expressions, CSS selectors) are used to identify UI elements, then automation can be implemented, but the identifiers become unreliable when the UI changes
Solution Approach 1:
The system performs description mining in advance to create multiple descriptive representations (XPath expressions, CSS selectors, text descriptions, image hashes) for each UI element concept. These pre-generated descriptions are stored and can be used interchangeably, so when the UI changes, alternative descriptions can be selected without requiring re-engineering of the automation workflow.
Solution Approach 2:
The system changes the identification approach from relying on a single unique identifier to using multiple descriptive parameters. By generating and storing multiple descriptions (structural, textual, visual) for each UI element, the system can switch between different parameter combinations based on which ones remain valid after UI changes, thereby maintaining reliability.
2Ease of manufacture
If manual definition of unique identifiers is performed during application modelling, then automation workflows can be created, but the process requires significant re-engineering when software changes
Solution Approach 1:
The system performs description mining in advance to create multiple descriptive representations (XPath expressions, CSS selectors, text descriptions, image hashes) for each UI element concept. These pre-generated descriptions are stored and can be used interchangeably, so when the UI changes, alternative descriptions can be selected without requiring re-engineering of the automation workflow.
Solution Approach 2:
The system creates multiple copies of element descriptions in different formats and representations. Instead of relying on a single identifier, it generates redundant descriptive copies (structural descriptions, text content, visual characteristics) that can serve as backups, eliminating the need to re-engineer workflows when original identifiers become invalid.
3Reliability
If domain experts use trial-and-error with domain knowledge to identify robust UI elements, then more reliable identification is achieved, but the process becomes time-consuming and complex
Solution Approach 1:
The system performs automatic description mining and generates multiple robust descriptions for UI elements without requiring manual trial-and-error by domain experts. The automated process extracts structural, textual, and visual characteristics, creating a comprehensive set of identifiers that are as reliable as expert-generated ones but without the time cost of manual intervention.
Solution Approach 2:
The system replaces the manual mechanical process of expert trial-and-error identification with an automated computational approach. Machine learning models and pattern recognition algorithms automatically generate and evaluate multiple descriptions, substituting human expert effort with automated processing that achieves similar or better reliability more efficiently.
Data Source
Figure 1~2
Figure 3~4
Figure 5
AI summary
There is provided a method (100) for performing description mining for a concept. The method comprises: acquiring (S110) a plurality of reference descriptions, wherein each of the plurality of reference descriptions is associated with a user interface (UI) element corresponding to a type of UI element represented by the concept; generating (S120), based on the plurality of reference descriptions, at least one of a local description for the concept and a global description for the concept, wherein the local description for the concept comprises a description component that is specific to a respective UI, and wherein the global description for the concept comprises a description component that is not specific to a UI; and storing (S130) the at least one of generated local description and global description such that it is associated with a concept identifier corresponding to the concept.