Surface Automation for Black Box UI via Image Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Robotic Process Automation (RPA) faces challenges in black box environments, such as virtualization and legacy applications, where user interface elements are inaccessible due to secure VPN layers and limited interface usage, preventing effective automation.
Innovation Solution
Surface automation techniques, including image processing algorithms like object detection and optical character recognition, are used to capture and understand user interface information from screenshots, enabling the identification and manipulation of application elements, even in remote sessions or restricted environments.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional RPA methods using application technology specific APIs are used, then automation can be implemented in standard environments, but it cannot be applied in black box environments where technical understanding is unavailable
Solution Approach 1:
The patent creates a visual copy of the application interface by capturing screenshots and uses image processing algorithms to analyze this copy instead of directly accessing the application's technical handles or APIs. This allows the bot to interact with the visual representation of the interface, enabling automation in black box environments where traditional API-based methods fail.
Solution Approach 2:
The patent introduces an intermediary layer between the RPA bot and the application interface. This intermediary uses image processing, object detection, and optical character recognition to translate visual interface elements into actionable information, bridging the gap between the bot and the protected application elements without requiring direct technical access.
2Reliability
If virtualization is used to provide enhanced security and compliance, then high availability at low cost is achieved, but user interface elements become inaccessible due to secure VPN layers
Solution Approach 1:
The patent captures visual copies of the application interface through screenshots and processes these copies using image analysis algorithms. This allows the RPA system to interact with the application's visual elements without needing to penetrate the secure VPN layer or access the virtualized interface directly, maintaining security while enabling automation.
Solution Approach 2:
The patent replaces traditional mechanical interface interaction (direct programmatic access to UI elements) with optical/electromagnetic interaction (image capture and processing). By using computer vision and optical character recognition, the system can 'read' and interpret visual interface elements without requiring direct connection to the protected virtual environment.
3Adaptability or versatility
If image processing algorithms are used to capture and understand UI information from screenshots, then automation in black box environments is enabled, but processing time and computational resources increase
Solution Approach 1:
The patent divides the image processing task into distinct segments: capturing screenshots, detecting objects, recognizing text, and identifying interface elements. By segmenting these complex tasks into manageable steps, the system can process images more efficiently and optimize each step individually to reduce overall processing time.
Solution Approach 2:
The patent applies image processing algorithms selectively rather than universally - only to the portions of the interface that require analysis. By using object detection to identify relevant elements first, then applying more intensive processing only to those areas, the system reduces unnecessary computational overhead while maintaining comprehensive analysis capability.
Data Source
AI summary
Disclosed herein are system, method, and computer program product embodiments for surface automation in black box environments. An embodiment operates by determining scenarios of an application for automation; detecting the scenario during an execution of an application; capturing and storing one or more user interface screenshots of the scenario; identifying and storing user interface information from the user interface screenshot; implementing a sequential set of instructions comprising at least one textual element detection technique and at least one non-textual element detection technique; and executing the sequential set of instructions.