Multi-anchor UI Element Detection for RPA Error Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Robotic process automation (RPA) systems face errors due to single anchoring methods, leading to duplication, false positives, and missed elements, particularly in virtual or remote machine configurations, where factors like DPI, scaling, and button shapes cause issues in user interface automation.
Innovation Solution
Implementing multi-anchoring methods for user interface analysis during RPA development and runtime to reduce errors and increase performance by using multiple anchors for element detection and training on extracted images, applicable across desktop, web, and virtual machine configurations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If a single anchor or reference point is utilized with computer vision to identify UI elements, then the system complexity is reduced, but the reliability of element identification deteriorates due to duplication, errors, false positives, and missed elements
Solution Approach 1:
The patent divides the single anchor identification task into multiple anchor points. Instead of relying on one reference point, the system uses multiple anchors to identify UI elements, which segments the identification process into multiple verification points, thereby improving reliability without significantly increasing overall system complexity
Solution Approach 2:
The system implements feedback mechanisms where multiple anchor points provide cross-validation for element identification. The results from multiple anchors are aggregated and used to confirm or reject element identification, creating a feedback loop that enhances reliability while maintaining manageable system complexity
2Ease of operation
If single anchoring is used for UI automation, then the ease of operation is improved, but the productivity deteriorates due to errors requiring correction and rework
Solution Approach 1:
The system performs preliminary actions by establishing multiple anchor points before the actual UI element identification process. This preliminary setup of multiple references prevents errors during execution, eliminating the need for correction and rework, thereby improving productivity while maintaining ease of operation
Solution Approach 2:
The patent applies beforehand cushioning by using multiple anchor points as a buffer against identification errors. The redundant anchors provide a safety margin that cushions against false positives and missed elements, preventing productivity loss from errors while keeping the system easy to operate
3Measurement precision
If multiple anchors are used for element detection, then the measurement precision of UI elements is improved, but the device complexity increases due to additional processing requirements
Solution Approach 1:
The patent merges the results from multiple anchor points into a single unified identification outcome. By combining the information from multiple anchors through aggregation and consensus mechanisms, the system achieves high measurement precision while avoiding the complexity of managing completely separate identification systems
Data Source
AI summary
Multiple anchors may be utilized for robotic process automation (RPA) of a user interface (UI). The multiple anchors may be utilized to determine relationships between elements in the captured image of the UI for RPA. The results of the anchoring may be utilized for training or retraining of a machine learning (ML) component.