GUI Annotation via Visual Candidate Selection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Manual testing of graphical user interface (GUI) objects is inefficient, especially when a large number of input possibilities are involved, and automated testing requires knowledge of and access to the underlying source code, making it complicated for non-technical users.
Innovation Solution
Creating annotations for unannotated GUI objects based on the properties and positions of surrounding objects, allowing non-technical users to automate testing and documentation without needing access to the source code by identifying nearby candidate GUI objects and selecting the best candidate for annotation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If automated testing is implemented using test scripts that reference GUI objects by source code names, then testing efficiency and automation capability are improved, but device complexity and difficulty of operation increase due to requirement of source code knowledge
Solution Approach 1:
The patent introduces an intermediary layer between the test script and the GUI object. Instead of directly referencing source code names, the system uses visual identification (bounding boxes, positions, sizes) as an intermediary to map test commands to GUI objects. This mediator translates non-technical visual references into actionable test script commands, eliminating the need for source code knowledge while maintaining automation capability.
Solution Approach 2:
The patent creates a virtual copy or representation of the GUI object's visual characteristics (bounding box, position, size) that can be referenced in test scripts without accessing the actual source code. This copy allows test automation to operate on visual properties rather than requiring knowledge of the underlying object names or identifiers in the source code.
2Ease of operation
If manual interaction with GUI objects is used for testing, then ease of operation is maintained, but productivity decreases due to inefficiency when dealing with large numbers of input possibilities
Solution Approach 1:
The system enables automated testing that performs itself without requiring manual intervention for each test case. The test script automatically identifies GUI objects using visual characteristics, executes test commands, and cycles through multiple input possibilities without human involvement. This self-service automation maintains the simplicity of manual operation while achieving the efficiency of automated testing.
3Extent of automation
If test scripts reference GUI objects using source code names, then automation capability is improved, but device complexity increases due to requirement of source code access and knowledge
Solution Approach 1:
The patent extracts the essential identification information from the source code and represents it through visual properties (bounding boxes, positions, sizes). By taking out the dependency on source code names and replacing it with visual characteristics, the system maintains automation capability while reducing device complexity. Test scripts no longer need to access or understand the source code structure.
Solution Approach 2:
Visual identification properties serve as an intermediary layer that simplifies the relationship between test scripts and GUI objects. Instead of directly complex relationships with source code names, the intermediary visual properties provide a simple, intuitive reference system that reduces overall system complexity while enabling full automation.
Data Source
AI summary
A system and method of creating annotations for graphical user interface (GUI) objects is provided. The annotations may be based on best candidate GUI objects that are in the vicinity of GUI objects that need annotation. The best candidate GUI object may be selected based on the position of an unannotated GUI object, a position of candidate GUI objects, a role of the unannotated GUI object, and a precedence criteria of the unannotated GUI object. The precedence criteria may include a position preference for candidate GUI objects in relation to unannotated GUI objects. The annotations may also be based on information within a self-described GUI object. Options can be specified to define how the annotations are extracted from candidate GUI objects and self-described GUI objects. The annotation of GUI objects can assist non-technical users in the automated testing and documentation of GUIs.