Computer Vision GUI Automation for Cross-Platform Adaptability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional GUI automation tools are limited by their reliance on fixed user interface element locations, which fail if these locations change, and they struggle to operate effectively across different execution environments and platforms.
Innovation Solution
A computer-vision based system that uses machine learning to recognize and identify user interface components within a graphical user interface, generating operating system-level events to perform actions independently of specific platform or execution environment, allowing for autonomous and flexible GUI operation across various platforms.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If conventional GUI automation tools use fixed user interface element locations, then the automation process is simple to implement, but the system fails when UI element locations change and lacks adaptability
Solution Approach 1:
The patent replaces the mechanical coordinate-based input system with a computer-vision-based recognition system. Instead of relying on fixed numerical coordinates to locate UI elements, the system uses image recognition algorithms to automatically identify and locate UI components based on their visual characteristics, enabling adaptation to UI layout changes without manual reconfiguration
Solution Approach 2:
The system dynamically changes the parameter used for UI element identification from fixed coordinates to visual feature-based recognition. By transitioning from static coordinate parameters to dynamic image recognition parameters, the system can adapt to different UI layouts and design changes while maintaining automation capability
2Adaptability or versatility
If conventional automation tools rely on platform-specific coordinates, then the implementation is straightforward for a single platform, but the system cannot operate effectively across different execution environments
Solution Approach 1:
The patent implements a universal computer-vision-based GUI automation system that can operate across multiple platforms and execution environments. The image recognition approach serves multiple functions: it identifies UI elements, determines their locations, and enables interactions without being tied to platform-specific coordinate systems, thereby achieving cross-platform compatibility while maintaining reliable operation
3Adaptability or versatility
If the system uses computer-vision based recognition to identify UI components, then the system becomes adaptable to UI changes and cross-platform, but the processing complexity and computational requirements increase
Solution Approach 1:
The system creates a visual model or representation of the GUI interface through image recognition. By capturing and processing visual information about UI components, the system builds a digital copy or model of the interface structure, enabling automated interaction based on visual characteristics rather than requiring complex platform-specific coordinate mappings
Data Source
AI summary
Using computer-vision based training information, a user interface (UI) component of an application-level user interface of an application and rendering coordinates of the UI component within the application-level user interface are recognized. A functional class that is mapped within the computer-vision based training information to the UI component and that is used to instantiate the UI component as part of the application-level user interface is identified in accordance with the computer-vision based training information. A replica object of the identified functional class is instantiated within a user interface container separately from the application. An operating system-level event that specifies a functional operation of the UI component and the recognized rendering coordinates of the UI component is generated from the instantiated replica object on an operating system event queue that provides inputs to the application.