AI Model Recognizing UI Elements via Computer Vision
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current UI automation technologies face challenges in recognizing applications, screens, and UI elements without system-level or application-level information, limiting their ability to perform effective interactions.
Innovation Solution
Training an AI/ML model using computer vision (CV) to recognize applications, screens, and UI elements, supplemented by optical character recognition (OCR) and user interaction tracking, allows for the identification of graphical elements and user interactions without prior knowledge, enabling robust UI automation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If driver-level and application-level interactions are used for UI automation, then automation functionality can be implemented, but system-level information and kernel hook information are not available
Solution Approach 1:
The patent introduces an AI/ML model as an intermediary that bridges the gap between visual UI elements and automation actions. The model processes screenshots and identifies UI elements without requiring direct system-level hooks, enabling automation functionality while working within information constraints.
Solution Approach 2:
The patent replaces traditional mechanical approaches (driver-level hooks, kernel-level information access) with a computer vision-based AI/ML system. This substitution allows the system to obtain UI element information through visual processing rather than requiring privileged system-level access.
2Adaptability or versatility
If extensive driver level and application level functionality is required, then UI automation platform can be built, but system complexity increases
Solution Approach 1:
The patent extracts the essential automation capability from complex driver and application-level functionality by using a standalone AI/ML model. This extraction allows the system to achieve UI automation without incorporating extensive driver-level components, thereby reducing overall platform complexity.
Solution Approach 2:
The AI/ML model serves as a universal component that can identify various UI elements across different applications without requiring application-specific drivers. This multi-functional approach reduces the need for extensive specialized functionality, simplifying the platform architecture.
3Measurement precision
If AI/ML model is trained to recognize UI elements without a priori knowledge, then recognition accuracy improves, but training data requirements increase
Solution Approach 1:
The patent performs preliminary actions by recording user interactions and generating training data automatically during normal system usage. This preliminary data collection enables the AI/ML model to be trained on relevant UI elements and interaction patterns without requiring manual annotation of large datasets, thus achieving high accuracy with manageable data volumes.
Data Source
AI summary
Techniques for training an artificial intelligence (AI)/machine learning (ML) model to recognize applications, screens, and UI elements using computer vision (CV) and to recognize user interactions with the applications, screens, and UI elements. Optical character recognition (OCR) may also be used to assist in training the AI/ML model. Training of the AI/ML model may be performed without other system inputs such as system-level information (e.g., key presses, mouse clicks, locations, operating system operations, etc.) or application-level information (e.g., information from an application programming interface (API) from a software application executing on a computing system), or the training of the AI/ML model may be supplemented by other information, such as browser history, heat maps, file information, currently running applications and locations, system level and/or application-level information, etc.