Image-Based Automation GUI State Matching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Image-based application automation systems fail to match images related to input events with graphical user interface elements that have multiple states, leading to playback failures during the replay phase due to mismatched visual appearances.
Innovation Solution
The system determines whether the graphical user interface includes a portion satisfied by one image associated with an input event and automatically selects another image from a group if the first image does not match, repeating the process until a matching image is found or all images have been selected.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the system uses a single image to represent a GUI element for automation, then the automation script is simple and fast to execute, but the system fails when the GUI element appears in different states
Solution Approach 1:
The system performs preliminary actions by capturing multiple images of a GUI element in different states during the record phase. These images are stored and associated with the element before playback occurs. During playback, the system selects the appropriate pre-captured image that matches the current state, avoiding the need to capture images in real-time and ensuring reliable matching regardless of the element's state.
Solution Approach 2:
The system changes the parameter of image selection by comparing visual characteristics of the current GUI element state with multiple pre-captured images. Instead of using a single fixed image, the system dynamically selects which image to use based on matching visual parameters such as color, shape, and text content, allowing the automation to adapt to different states of the same GUI element.
2Reliability
If the system captures multiple images of GUI elements in different states, then the automation becomes more reliable across different states, but the complexity of image management and matching increases
Solution Approach 1:
The system implements self-service by automatically comparing the current GUI element with multiple pre-captured images and selecting the matching one without requiring manual intervention. The automation system itself performs the image matching and selection process, using visual comparison algorithms to identify the correct image based on the current state of the GUI element, thereby managing the complexity internally rather than requiring external management.
Solution Approach 2:
The system applies universality by creating a single automation script that can handle multiple states of a GUI element through multiple associated images. Instead of requiring separate scripts for each state, the universal script automatically selects the appropriate image from the set, making the automation system multi-functional and capable of handling various states with a single unified approach.
3Measurement precision
If the system uses exact image matching for GUI elements, then the matching is precise and accurate, but the system fails when the GUI element's visual appearance changes due to state changes
Solution Approach 1:
The system applies dynamics by making the image selection process adaptive rather than static. Instead of using a fixed single image for matching, the system dynamically selects from multiple pre-captured images based on the current state of the GUI element. This dynamic selection process maintains high matching accuracy by choosing the image that best corresponds to the current visual state, while adapting to different states through the multiple-image approach.
Solution Approach 2:
The system performs preliminary capture of multiple images representing different states of a GUI element before automation playback. These pre-captured images serve as templates for matching during playback. By having multiple pre-prepared images covering different states, the system maintains precise matching accuracy for each state without needing to adapt in real-time, as the adaptation work was done in advance during the record phase.
Data Source
AI summary
In one implementation, an image-based application automation system determines that a first image from a plurality of images associated with an input event does not satisfy any portion of a graphical user interface at a first time. The image-based application automation system then selects, independent of user input, a second image from the plurality of images after the first time, and identifies a portion of the graphical user interface satisfied by the second image.