RPA UI Mapping With Computer Vision for Reusable UI Objects
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current robotic process automation (RPA) technologies face challenges in efficiently building and maintaining user interface (UI) descriptors for automating applications, as growing a repository of UI objects organically is slow and requires manual effort, especially with dynamic UI elements.
Innovation Solution
A UI mapper tool that facilitates the indication and inclusion of UI elements in a UI tree, allowing for their reuse across applications, and automatically captures UI elements using computer vision to speed up the process of creating UI object libraries, which can be used for subsequent automations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If UI elements are added manually item-by-item to the repository, then the repository grows organically with accurate descriptors, but the process is slow and time-consuming
Solution Approach 1:
The system uses computer vision to capture screenshots of the application UI and automatically generates UI element descriptors by analyzing the visual content. This copying approach allows rapid creation of UI object repositories without manual item-by-item entry, while maintaining accuracy through automated image-to-descriptor conversion
Solution Approach 2:
The manual mechanical process of adding UI elements one by one is replaced with an automated computer vision system that captures screenshots and automatically extracts UI element information. This substitution dramatically increases repository growth speed while maintaining descriptor quality through algorithmic analysis
2Reliability
If developers build descriptors manually for each UI action, then the automation is reliable and accurate, but the development time and effort increase significantly
Solution Approach 1:
The system performs preliminary actions by automatically capturing UI screenshots and generating descriptors before the actual automation workflow is needed. This advance preparation creates a ready-to-use UI object repository that can be directly utilized in RPA workflows, eliminating the need for manual descriptor creation during workflow development
Solution Approach 2:
The automated system copies UI element information from screenshots to create descriptors, enabling rapid workflow creation. This copying mechanism preserves automation reliability by accurately capturing UI element properties while dramatically reducing the time required to build automation workflows
3Ease of manufacture
If the UI repository is built organically item-by-item, then each UI object is well-documented with proper descriptors, but the overall process lacks efficiency
Solution Approach 1:
The manual mechanical process of documenting each UI object is replaced with automated computer vision technology that captures screenshots and extracts UI element information. This substitution maintains documentation quality through systematic analysis while dramatically improving repository building efficiency
Solution Approach 2:
The system enables self-service by allowing the UI repository to be automatically populated through screenshot capture and analysis. The computer vision system independently identifies, extracts, and documents UI elements without requiring manual intervention, thereby maintaining documentation quality while improving efficiency
Data Source
AI summary
A user interface (UI) mapper for robotic process automation (RPA) is disclosed. The UI mapper may initially capture UI elements to fetch UI elements faster for later use and allow an RPA developer to “map” the UI elements for automating an application. This may enable subsequent developers who potentially do not have programming knowledge to build RPA workflows using these predefined “target” UI elements.