Robust GUI Element Identification Using Alternative Representations
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing GUI element acquisition methods are not robust due to variations in GUIs across platforms, over time, and due to changes in design or functionality, leading to inconsistent and unreliable identification of elements.
Innovation Solution
A method involving a representation set of alternative representations is applied to identify GUI elements, with modifications made to the set if initial identification is indefinite, using attributes, paths, and user input to ensure unique identification.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a single representation method is used to identify GUI elements, then the identification process is simple, but the reliability is poor due to GUI variations across platforms and over time
Solution Approach 1:
The patent divides the identification approach into multiple independent representation methods (e.g., attribute-based, path-based, image-based representations). Each representation independently attempts to identify the GUI element, and their results are aggregated to determine the final identification, thereby improving reliability through diversity of approaches
Solution Approach 2:
The patent creates a universal identification system that uses multiple alternative representations applicable across different platforms and GUI versions. The system can adaptively select and combine different representation types depending on the specific GUI element and context, making the identification process robust to platform and version variations
2Reliability
If multiple alternative representations are applied to identify GUI elements, then the reliability improves, but the complexity of the identification process increases
Solution Approach 1:
The patent pre-generates and stores multiple alternative representations for each GUI element during the element creation or capture phase. When identification is needed, the system retrieves these pre-prepared representations and applies them, avoiding the time-consuming process of generating representations on-demand and reducing identification latency
Solution Approach 2:
The patent implements a staged identification process where representations are applied in order of expected effectiveness or complexity. The system may stop after finding a confident match using simpler representations, or proceed to more complex representations only if needed, thus avoiding unnecessary processing time while maintaining high reliability
3Measurement precision
If the representation set is modified based on indefinite results, then the identification accuracy improves, but the system complexity increases
Solution Approach 1:
The patent implements a feedback mechanism where the results of applying representations are monitored and analyzed. When indefinite or conflicting results are detected, the system automatically adjusts the representation set by adding, removing, or weighting representations based on their performance, creating a self-optimizing identification system
Solution Approach 2:
The patent makes the representation set dynamic and adaptive rather than static. The system can modify which representations are active, their weights, or their parameters based on runtime conditions, GUI element types, and historical performance data, allowing the identification system to adapt to changing GUI structures and platforms
Data Source
AI summary
A method, apparatus and product for GUI element acquisition using a plurality of alternative representations of the GUI element. The method comprising: obtaining a selection of a Graphical User Interface (GUI) element; obtaining a representation set of the GUI element, wherein the representation set comprises a plurality of alternative representations, each of which provides a manner of identifying the GUI element in GUIs; applying the representation set to identify the GUI element in a GUI of a program, wherein said applying the representation set comprises, for each representation of the alternative representations, independently applying the each representation to identify one or more GUI elements corresponding the each representation; and in response to determining that said applying the representation set provides an indefinite result, obtaining a modified representation set.