Computer Vision UI Recognition for Automation Without System Hooks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current UI automation technologies face challenges in providing effective automation without system-level and application-level interactions, particularly when key presses, mouse clicks, and kernel hook information are not available.
Innovation Solution
Training a generative AI/ML model using computer vision to recognize applications, screens, and UI elements, supplemented by optical character recognition and user interaction tracking, to perform UI automation without relying on system-level or application-level information.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If driver level and application level interactions are used for UI automation, then automation functionality is achieved, but system complexity and dependency on system-level information increase
Solution Approach 1:
The patent replaces traditional mechanical interaction methods (driver-level hooks, kernel-level information capture) with a vision-based system that uses computer vision to observe and interpret UI elements visually, eliminating the need for system-level access and complex driver integration
Solution Approach 2:
The patent introduces an intermediary AI/ML model that acts as a bridge between the automation system and the UI, translating visual observations into automation actions without requiring direct system-level interactions, thus simplifying the overall system architecture
2Measurement precision
If key presses, mouse clicks, and kernel hook information are used, then user interaction recognition is achieved, but availability and portability across different systems decrease
Solution Approach 1:
The patent creates visual copies (screenshots, video frames) of the UI interface and uses these copies for analysis and recognition, allowing the system to understand user interactions without accessing actual system-level event data, thereby enabling cross-platform portability
Solution Approach 2:
The patent substitutes system-level event capture mechanisms with vision-based observation, replacing the need for platform-specific kernel hooks and driver-level access with a universal visual interface that can be analyzed across different operating systems and applications
Data Source
AI summary
Techniques for training a generative artificial intelligence (AI)/machine learning (ML) model to recognize applications, screens, and UI elements using computer vision (CV) and to recognize user interactions with the applications, screens, and UI elements are disclosed. Optical character recognition (OCR) may also be used to assist in training the generative AI/ML model. Training of the generative AI/ML model may be performed without other system inputs such as system-level information (e.g., key presses, mouse clicks, locations, operating system operations, etc.) or application-level information (e.g., information from an application programming interface (API) from a software application executing on a computing system), or the training of the generative AI/ML model may be supplemented by other information, such as browser history, heat maps, file information, currently running applications and locations, system level and/or application-level information, etc.