GUI Actionable Object Detection via Cursor Scanning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current AI/ML algorithms face difficulties in accurately identifying actionable graphical objects in graphical user interfaces, leading to incomplete software application testing, as they sometimes misclassify clickable objects as text objects.
Innovation Solution
The system identifies actionable graphical objects by generating cursor movements using a scanning process within a grid, detecting changes in cursor types, and integrating these objects into the testing process, utilizing machine learning algorithms to validate the GUI and learn patterns over time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If AI/ML algorithms are used to detect graphical objects in GUI, then automation of testing is improved, but measurement precision of graphical object identification deteriorates
Solution Approach 1:
The patent introduces an intermediary verification process between the AI/ML detection and final object identification. This involves generating cursor movements that simulate user interaction and observing cursor type changes as the cursor hovers over different areas. The cursor type change acts as a mediator that confirms whether a detected object is truly actionable, thereby improving identification accuracy while maintaining automation.
Solution Approach 2:
The system implements feedback by using the cursor type change information to verify and correct the initial AI/ML detection results. When the cursor hovers over an area and its type changes (e.g., from arrow to hand pointer), this feedback confirms the presence of an actionable object. The system uses this feedback to refine its identification, ensuring higher precision in distinguishing actionable objects from non-actionable elements.
2Measurement precision
If cursor movement scanning is added to detect actionable objects, then measurement precision of object identification is improved, but device complexity increases
Solution Approach 1:
The system uses the existing cursor mechanism and its natural behavior (changing type when hovering over actionable objects) to perform the detection function. Instead of adding complex new detection hardware or software, the system leverages the self-service capability of the cursor to indicate actionable objects through its type changes, thereby improving detection accuracy without significantly increasing system complexity.
Solution Approach 2:
The cursor type change serves as a simple intermediary signal that bridges the gap between the testing system and the actionable objects. Rather than implementing complex detection algorithms, the system uses the cursor's inherent behavior as a mediator to identify actionable objects, keeping the added complexity minimal while achieving improved measurement precision.
3Reliability
If comprehensive testing of all graphical objects is performed, then reliability of software testing is improved, but loss of time in testing process increases
Solution Approach 1:
The system performs preliminary action by using AI/ML algorithms to quickly detect and classify graphical objects before conducting detailed verification. This initial classification allows the system to prioritize which objects need comprehensive testing (actionable objects) and which can be processed more quickly, thereby improving testing reliability without proportionally increasing testing duration.
Solution Approach 2:
The system applies partial action by focusing comprehensive verification only on actionable objects identified through cursor type changes, rather than performing exhaustive testing on all graphical objects. This selective approach ensures reliable testing of critical elements while reducing overall testing time by not over-testing non-actionable elements.
Data Source
AI summary
An area of a graphical user interface that potentially comprises an actionable graphical object is identified. An actional graphical object is an object that generates an event when clicked on. For example, an actional graphical object may be a button, a menu, a menu item, a check box, a text field, a text area, a tab, and/or the like. A cursor movement is generated in the area of the graphical user interface (e.g., using a grid). The cursor movement uses a scanning process to a detect a change in a cursor type (e.g., from an arrow cursor to a link cursor). In response to detecting the change in the cursor type, the actionable graphical object is identified in the area of the graphical interface. The actionable graphical object may then be integrated into a testing process to validate the graphical user interface.