RPA User Interface Recognition for Layout-Resilient Task Execution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Robotic process automation systems lack robustness and resilience, failing when encountering deviations from pre-programmed conditions or user interface changes, and are deterministic, ignoring data errors without specific programming.
Innovation Solution
Implementing machine learning to analyze and recognize user interface components, determining relevant areas and data for tasks, allowing flexible execution of tasks regardless of interface changes through modules like area determination, recognition, and data processing, using techniques like OCR and neural networks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If robotic agents are programmed to retrieve specific data from user interfaces at specific locations, then task execution is accurate and efficient, but the system lacks robustness when user interfaces change layout or component positions
Solution Approach 1:
The patent applies dynamics by transitioning from static, location-based task execution to dynamic, function-based task execution. The robotic agent continuously analyzes user interface components to identify their current locations and functions, allowing the system to adapt to layout changes while maintaining accurate task execution. The agent dynamically updates its understanding of component positions and relationships rather than relying on pre-programmed fixed locations.
Solution Approach 2:
The patent changes the fundamental parameters of task execution from location-based coordinates to function-based identification. Instead of programming specific pixel coordinates or fixed positions, the system identifies components by their functional characteristics and relationships, allowing the same task to be executed accurately regardless of where components appear on the interface.
2Reliability
If robotic agents are programmed with specific contingencies for every possible deviation, then robustness improves, but device complexity and programming effort increase significantly
Solution Approach 1:
The patent implements self-service by enabling the robotic agent to autonomously analyze and understand user interface changes without requiring pre-programmed contingencies for every possible scenario. The agent independently identifies component functions, determines appropriate actions, and adapts to new situations through continuous analysis, eliminating the need for exhaustive programming of edge cases.
Solution Approach 2:
The system uses dynamic analysis to continuously assess the current state of user interfaces and adapt behavior in real-time. Rather than relying on static programming for every contingency, the agent dynamically determines appropriate responses based on current component functions and relationships, reducing programming complexity while maintaining robustness.
3Productivity
If robotic agents execute pre-programmed tasks deterministically, then task execution is efficient and fast, but the system cannot handle data errors or unexpected conditions without specific programming
Solution Approach 1:
The patent applies preliminary action by having the robotic agent continuously analyze and understand user interface components before executing tasks. This preliminary analysis establishes a foundation for efficient execution while maintaining adaptability, as the agent already understands component functions and relationships upfront, enabling both fast execution and flexible error handling.
Solution Approach 2:
The system implements feedback mechanisms where the robotic agent continuously monitors user interface states and adjusts its behavior based on analyzed component functions. This feedback loop maintains efficient task execution while enabling the system to handle errors and unexpected conditions, as the agent can detect anomalies and adapt its approach based on real-time analysis rather than relying solely on pre-programmed error handling.
Data Source
AI summary
Systems and methods relating to enhancing capabilities of robotic process automation systems. A system and method includes recognizing and analyzing the components of a user interface on which at least one task is to be executed. The task can be executed regardless of changes to the user interface as the components of the task are based on the presence and function of areas of the user interface and not on the location of the components necessary to execute the task.