GUI Element Detection Using Property and Image Matching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Automated functional testing of applications faces challenges in accurately detecting graphical user interface (GUI) elements due to high false positives and false negatives, particularly when using property value matching or image recognition alone, which can lead to premature termination of testing and incorrect failure of applications.
Innovation Solution
Combining GUI element detection by property values with image detection, where the faster property value method is used by default, and the image method is employed only if property value detection fails, with further limited property value matching if image detection is successful, to improve accuracy and reduce false positives.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If property value matching is used for GUI element detection, then detection speed is improved, but false positives and false negatives increase
Solution Approach 1:
The patent combines property value matching and image recognition methods into a unified detection system. The system first attempts property value matching for speed, then uses image recognition to verify results, merging the advantages of both approaches to achieve both fast detection and high accuracy.
Solution Approach 2:
The system performs property value matching as a preliminary filtering step before conducting image recognition. This preliminary action quickly eliminates obvious mismatches and reduces the computational burden of the more intensive image recognition process, thereby maintaining speed while improving accuracy.
2Reliability
If image recognition is used for GUI element detection, then detection accuracy is improved, but detection speed decreases
Solution Approach 1:
The system applies image recognition selectively rather than to all GUI elements. It uses property value matching first to identify candidate elements, then applies image recognition only when needed to verify or resolve ambiguities, performing partial action to balance accuracy and speed.
Solution Approach 2:
Property value matching serves as an intermediary step between the detection request and image recognition. It pre-processes the detection task by identifying likely candidates based on properties, which then narrows the scope for image recognition and reduces its computational overhead.
3Productivity
If property value matching is used, then detection efficiency is improved, but false positives occur
Solution Approach 1:
The system implements feedback by using image recognition results to verify and correct property value matching outcomes. When property value matching produces potential false positives, the image recognition component provides feedback to confirm or reject the detection, thereby maintaining efficiency while improving precision.
4Measurement precision
If image detection is used, then false positives are reduced, but processing time increases
Solution Approach 1:
The detection process is segmented into two phases: a fast property value matching phase that handles most cases efficiently, and a slower image recognition phase that is invoked only when needed. This segmentation reduces the average processing time while maintaining high precision by applying the intensive image analysis selectively.
Data Source
AI summary
A script directing functional testing of an application specifies values for properties of a specific graphical user interface (GUI) and specifies an image of the specific GUI element. A computing device detects whether the application has displayed the specific GUI element by evaluating the values for the properties of the specific GUI element against GUI elements that the application has displayed. If unsuccessful, the computing device detects whether the application has displayed the specific GUI element by evaluating the image of the specific GUI element against a screen image including the GUI elements that the application has displayed.