Display-independent GUI Control Recognition via Screenshot Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Complex graphical user interfaces (GUIs) in applications make it difficult for users to identify and operate controls, especially when display properties change, leading to difficulties in providing effective guidance independent of display settings.
Innovation Solution
A method and system for display-independent recognition of GUI controls, involving screenshot capture, analysis, and optical character recognition to identify and store images of controls, allowing for guidance script execution independent of display properties, including activation of scroll bars to reveal invisible regions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If users rely on pre-defined topics or search functions to find guidance, then they can retrieve information, but it becomes difficult to find out which sequence of steps is required to execute a particular task
Solution Approach 1:
The system captures screenshots of the GUI and stores images of controls along with their coordinates. This creates visual copies of the control states that can be referenced during guidance, allowing users to see exactly what controls exist and their positions without relying on text-based help documentation.
Solution Approach 2:
The system introduces an intermediary layer between the user and the complex GUI by automatically generating and storing control images with coordinates. This intermediary database allows the guidance system to reference specific control states without requiring users to navigate through complex help indices or search functions.
2Adaptability or versatility
If applications become more complex with many different controls, then functionality increases, but it becomes difficult for users to identify and operate controls
Solution Approach 1:
The system creates visual copies of all GUI controls by capturing screenshots and extracting control images. This allows users to reference the actual visual appearance of controls rather than trying to remember or search for them, significantly reducing identification difficulty in complex applications.
Solution Approach 2:
The system segments the GUI into individual controllable elements by identifying and storing separate images of each control with its coordinates. This segmentation allows the guidance system to focus on and provide instructions for specific controls without overwhelming users with the entire complex interface at once.
3Ease of operation
If guidance is provided based on display properties, then it can be tailored to specific views, but it becomes dependent on window size, resolution, or theme changes
Solution Approach 1:
The system stores actual images of controls captured from the display, rather than relying on coordinate-based or property-based references. These stored images serve as universal references that remain valid regardless of window size, resolution, or theme changes, ensuring guidance consistency across different display settings.
Solution Approach 2:
The system captures and stores control images at their actual display properties, creating a reference database that adapts to different display configurations. When providing guidance, the system references these stored images rather than calculating positions based on display parameters, making the guidance independent of window size, resolution, or theme changes.
Data Source
AI summary
A method for display-independent recognition of a graphical user interface (GUI) control, the method comprising capturing a screenshot of a GUI of a computerized application, determining a cursor position in the screenshot, analyzing an area surrounding the cursor position, to identify a GUI control, and storing an image of the GUI control.