Display-Independent Computerized Guidance Script Recording
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Complex applications with many controls can be difficult for users to navigate, as existing guidance methods rely on specific display properties, making them ineffective when display settings change, and users often require assistance to perform tasks.
Innovation Solution
A method and engine for recording and executing display-independent computerized guidance scripts, which capture screenshots, analyze GUI controls, and record operation sequences, allowing for independent operation of GUI controls regardless of display properties.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If guidance scripts are created based on specific display properties, then the guidance can be precise for a particular display configuration, but the guidance becomes ineffective when display settings change
Solution Approach 1:
The patent captures screenshots of the GUI and stores them as reference templates. When executing guidance, the system compares current GUI elements against these captured images rather than relying on coordinate-based or property-based matching. This visual copying approach allows the guidance to adapt to different display settings while maintaining precision in identifying controls.
Solution Approach 2:
The system changes the parameter for control identification from display-dependent parameters (coordinates, colors, sizes) to display-independent visual templates. By capturing the actual visual appearance of controls at runtime and using image matching, the guidance system can accommodate parameter changes in display settings without failing.
2Ease of operation
If users manually create guidance scripts, then they can provide customized assistance, but it requires significant time and effort from users
Solution Approach 1:
The system performs self-service by automatically capturing screenshots and generating guidance scripts without requiring manual input from users. The automated recording engine captures the user's interactions and generates the guidance script autonomously, eliminating the time-consuming manual creation process while preserving customization based on actual usage patterns.
Solution Approach 2:
The system performs preliminary actions by capturing screenshots and recording operations before generating the final guidance script. This preliminary capture of visual and operational data allows the system to automatically generate comprehensive guidance without requiring users to manually create scripts step-by-step.
3Measurement precision
If the system analyzes the entire screenshot to identify GUI controls, then it can accurately identify controls, but it increases processing time and complexity
Solution Approach 1:
The patent segments the screenshot analysis by focusing on specific regions around the cursor position rather than analyzing the entire screenshot uniformly. The system captures and analyzes only the relevant area where the user is interacting, which maintains identification accuracy while significantly reducing processing time and computational complexity.
Data Source
AI summary
A method for recording a display-independent computerized guidance script, the method comprising capturing a screenshot of a GUI of a computerized application, analyzing an area, in the screenshot, surrounding a determined cursor position, to identify a GUI control, recording a manual operation sequence of the computerized application, and assigning the GUI control to a step of the operation sequence.