UI Automation Engine Using Visual Element Recognition
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing UI automation techniques struggle with complex UIs, particularly those with dynamic content, requiring manual scripting and are prone to errors due to resolution mismatches and performance issues, making them inefficient and error-prone.
Innovation Solution
A system utilizing an automation engine with a region-based convolutional neural network (RCNN) model captures and classifies UI elements, enabling dynamic response to UI changes without manual scripting, and performs pre- and post-validation to ensure accurate execution of interactions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If existing OCR-based UI automation techniques are used, then automation of simple UIs is possible, but they fail with complex UIs containing dynamic content and require manual scripting
Solution Approach 1:
The patent replaces traditional OCR-based mechanical text recognition with a deep learning-based visual recognition system. The system uses neural networks to directly analyze UI screenshots and identify interactive elements, their types, and positions, eliminating the need for manual scripting and XPath-based approaches that fail with dynamic content.
Solution Approach 2:
The patent changes the fundamental parameters of UI automation by transitioning from coordinate-based interactions (x,y positions) to element-based interactions (identified through visual recognition). This allows the system to adapt to dynamic content where elements may move or change position, as the deep learning model continuously identifies elements based on their visual characteristics rather than fixed coordinates.
2Extent of automation
If manual scripting with tools like Selenium is used, then specific use cases can be automated, but the automation must be re-done when the target application changes
Solution Approach 1:
The patent implements self-service automation where the deep learning model automatically adapts to UI changes without requiring manual intervention. The system continuously learns from UI screenshots, automatically updates its understanding of the interface structure, and maintains automation functionality even when the target application undergoes changes, eliminating the need to re-write scripts.
Solution Approach 2:
The patent incorporates feedback mechanisms where the system validates automated interactions by analyzing UI state changes after executing actions. This feedback loop allows the system to detect when UI elements have changed and automatically adjust its recognition models, ensuring continued reliability without manual re-scripting.
3Extent of automation
If screenshots are captured at specific resolution for automation recording, then interactions can be recorded, but replaying on different resolution UIs results in errors
Solution Approach 1:
The patent transitions from two-dimensional coordinate-based automation (x,y positions on screenshots) to a scale-invariant element-based approach. The deep learning model identifies UI elements through their visual characteristics and relative positions, allowing the same automation logic to work across different resolutions by recognizing elements rather than targeting fixed coordinates.
4Ease of operation
If existing automation tools are used, then basic UI interactions can be automated, but they lack capabilities for complex UIs without manual scripting
Solution Approach 1:
The patent creates a universal automation system based on deep learning that can handle diverse UI types and complexities through a single platform. The system performs multiple functions including element identification, classification, interaction execution, and validation, replacing the need for multiple specialized tools and manual scripting approaches.
Data Source
Figure 1
Figure 2A~2B
Figure 3A
AI summary
Aspects of the present disclosure provide systems, methods, and computer-readable storage media that support creation and execution of automated sequences of user interface (UI) interactions. To facilitate creation and execution of automated sequences of UI interactions, an automation engine is provided and includes a model configured to capture image data when creating the sequence of UI interactions. The model may also be used during replay of the sequence of UI interactions. For example, the model may be used during replay of the sequence of UI interactions to locate UI elements corresponding to the UI interactions, or to perform pre-and/or post-validation of action execution. The automation engine may also provide processes to enable location of dynamic content, such as UI elements that may be presented in different or unexpected locations, and processes to address complex UI elements, such as data grids, tree views, and drawings (e.g., CAD drawings).