GUI Element Identification Using Alternative Representations
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing GUI element acquisition methods are not robust due to variations in platform, user roles, and changes over time, leading to inconsistent and unreliable identification of GUI elements.
Innovation Solution
A method involving a representation set of alternative representations is applied to identify GUI elements, with modifications made to the set when indefinite results occur, using client devices and servers to refine the representation set based on user input and machine learning.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a single representation method is used to identify GUI elements, then the identification process is simple, but the reliability is low due to platform variations and GUI changes
Solution Approach 1:
The patent segments the identification process into multiple independent representation methods (e.g., text-based, structure-based, image-based). Each representation independently attempts to identify the GUI element, and their results are aggregated to improve reliability. This segmentation allows the system to handle platform variations and GUI changes more robustly.
Solution Approach 2:
The patent creates a universal identification system that works across multiple platforms (iOS, Android, Windows, web) by implementing multiple representation methods that can adapt to different platform-specific GUI characteristics. The system is designed to be platform-agnostic while maintaining compatibility with various GUI frameworks.
2Reliability
If multiple alternative representations are applied to identify GUI elements, then the reliability improves, but the complexity of the identification process increases
Solution Approach 1:
The patent implements feedback mechanisms where the results from each representation method are evaluated and used to adjust the identification process. If some representations fail or produce inconsistent results, the system learns from this feedback and adjusts its strategy, such as weighting certain representations more heavily or attempting alternative identification approaches.
Solution Approach 2:
The patent dynamically changes parameters of the identification process based on the performance of different representations. For example, if text-based representations are reliable for a particular GUI element, the system increases their weight; if image-based representations perform better in certain contexts, it adjusts accordingly. This adaptive parameter adjustment maintains ease of operation while improving reliability.
3Measurement precision
If the representation set is modified based on indefinite results, then the accuracy improves, but the time required for acquisition increases
Solution Approach 1:
The patent performs preliminary actions by pre-processing and analyzing GUI elements to create a initial representation set before actual identification is needed. This preliminary analysis includes pre-computing text features, structural properties, and image characteristics, so that when identification is required, the system can quickly apply these pre-computed representations without extensive real-time processing.
Solution Approach 2:
The patent implements dynamic adjustment of the representation set based on the specific context and requirements. The system can dynamically add or remove representations from the active set based on the complexity of the identification task, the available time budget, and the performance of previous attempts. This dynamic approach balances accuracy and acquisition time adaptively.
Data Source
AI summary
A method, apparatus and product for GUI element acquisition using a plurality of alternative representations of the GUI element. The method comprising: obtaining a selection of a Graphical User Interface (GUI) element; obtaining a representation set of the GUI element, wherein the representation set comprises a plurality of alternative representations, each of which provides a manner of identifying the GUI element in GUIs; applying the representation set to identify the GUI element in a GUI of a program, wherein said applying the representation set comprises, for each representation of the alternative representations, independently applying the each representation to identify one or more GUI elements corresponding the each representation; and in response to determining that said applying the representation set provides an indefinite result, obtaining a modified representation set.