GUI Interaction Detection Through Adaptive Spying Mode Selection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing GUI spying modes for intelligent process capture and automation platforms are inconsistent in accuracy and efficiency across different applications, leading to errors or inaccurate results, necessitating human intervention to select the optimal mode, which hampers usability and efficiency.
Innovation Solution
A method involving analyzing device state information using multiple GUI spying modes, classifying the estimated interaction information with a reference model, and outputting accurate GUI interaction information based on classification results, utilizing APIs, computer vision tools, and deep learning models to determine the optimal spying mode.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If multiple GUI spying modes are used to improve accuracy, then measurement precision improves, but device complexity increases
Solution Approach 1:
The system segments the GUI spying functionality into multiple independent modes (UIA, Win32, computer vision, OCR) that can be selected and executed separately. Each spying mode operates as an independent module that analyzes device state information through different technical approaches, allowing the system to choose the most appropriate mode for each specific GUI element type without requiring all modes to run simultaneously.
Solution Approach 2:
The system changes the parameter of spying mode selection dynamically based on the type of GUI element being analyzed. Different spying modes are activated depending on the application context and element characteristics, transforming a static single-mode approach into a dynamic multi-mode system that adapts parameters (spying mode selection) to optimize accuracy for each specific case.
2Measurement precision
If human operators select the optimal spying mode, then measurement precision improves, but loss of time increases
Solution Approach 1:
The system implements self-service by automatically selecting the optimal spying mode without requiring human operator intervention. The intelligent process capture and automation platform autonomously evaluates the current GUI state and selects the most appropriate spying mode from the available options, eliminating the need for manual mode selection during process capture while maintaining high accuracy.
Solution Approach 2:
The system uses feedback mechanisms where the outcome of each spying mode attempt is evaluated and used to determine subsequent actions. If a spying mode fails to accurately capture GUI interaction information, the system receives feedback about the failure and automatically switches to an alternative mode, creating a closed-loop system that continuously optimizes selection based on real-time performance.
3Device complexity
If a single GUI spying mode is used, then device complexity decreases, but measurement precision deteriorates
Solution Approach 1:
The system achieves universality by creating a multi-functional GUI spying framework that can handle diverse application types and GUI element characteristics through a single unified architecture. The platform incorporates multiple spying modes (UIA for Windows applications, Win32 for legacy applications, computer vision for proprietary interfaces, OCR for text extraction) within one system, allowing it to universally adapt to various application scenarios without requiring separate specialized systems.
Data Source
AI summary
A computer implemented method for determining graphical user interface, GUI, interaction information for an end user device is described. The method comprising analysing device state information using one or more GUI spying modes to estimate GUI interaction information for the one or more GUI spying modes; classifying the estimated GUI interaction information for the one or more GUI spying modes based on a reference model; and outputting GUI interaction information based on the classification results.