GUI Event Recording with Vicinity Label Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current software solutions for recording user-driven events in GUI applications are limited, particularly when dealing with nameless interactive controls, as they fail to provide meaningful action logs due to lack of readable and identifiable properties, leading to difficulties in reproducing bugs and correcting issues.
Innovation Solution
A method that automatically searches for descriptive information near GUI interactive controls using preferred UI guidelines to generate rich action logs, which includes identifying labels or contents in images, thereby providing clearer user intentions and improving reproducibility of user interactions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If previous software solutions record user-driven events, then action logs are generated, but the logs lack meaningful information when interactive controls are nameless
Solution Approach 1:
The system performs preliminary action by automatically searching for and assigning descriptive labels to interactive controls before recording user actions. This ensures that even nameless controls have meaningful identifiers in the action logs, preventing information loss without requiring developers to manually add properties to each control
Solution Approach 2:
The system introduces an intermediary mechanism that bridges the gap between nameless controls and meaningful log entries. This intermediary automatically searches for nearby UI elements with descriptive text and uses that text to identify controls in action logs, thereby preserving information without adding complexity to the control structure
2Measurement precision
If users manually convey bug reproduction steps, then debugging information is transmitted, but accuracy is lost due to human memory limitations
Solution Approach 1:
The system implements self-service by automatically recording user-driven events and generating accurate reproduction steps without requiring user intervention. The system captures UI events, searches for nearby descriptive text, and creates detailed action logs that accurately represent user actions, eliminating the need for users to manually recall and convey bug reproduction steps
3Measurement precision
If automatic vicinity searching is performed for descriptive information, then action log accuracy improves, but processing time increases
Solution Approach 1:
The system applies partial action by performing vicinity searching only when needed - specifically when a control lacks an accessible name or when the current search fails to find suitable descriptive text. The system searches in progressively expanding radii and stops when sufficient descriptive information is found, avoiding unnecessary full searches and reducing overall processing time
Data Source
AI summary
A method of recording user-driven events within a computing system includes receiving an indication that a user interacted with a first graphical user interface (GUI) user interactive control. The method includes automatically searching at least one vicinity near the first GUI user interactive control for descriptive information associated with the first GUI user interactive control, and automatically recording a user-driven event that represents the user interaction with the first GUI user interactive control and includes the descriptive information.