RPA Playback Using Screen Image Control Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Robotic process automation (RPA) systems face challenges in maintaining reliable and flexible interactions with software applications due to changes in playback engines and user interface variations, leading to errors when executing legacy software robots or interacting with updated software programs.
Innovation Solution
The implementation of resilient playback capabilities that detect and correct errors by identifying user interface controls within captured screen images, allowing RPA systems to adapt and continue operating despite changes in screen size, resolution, or playback engine updates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If RPA systems interact with software applications through user interfaces as a human would do, then integration difficulties are eliminated and application level tasks can be automated, but playback engines may evolve over time and may not continue to support legacy software robots, and user interfaces may appear differently due to variations in screen size or resolution
Solution Approach 1:
The patent transitions from traditional coordinate-based control object identification to image-based identification. By capturing screen images and using image processing techniques, the system can locate control objects based on their visual appearance rather than fixed coordinates, making the RPA system adaptable to different screen sizes, resolutions, and UI variations while maintaining reliable operation of legacy software robots
Solution Approach 2:
The system changes the parameters used to identify control objects from fixed screen coordinates to visual characteristics extracted from captured screen images. This allows the playback engine to locate control objects even when the UI appearance changes due to different screen configurations or playback engine versions, resolving the contradiction between adaptability and reliability
Data Source
AI summary
Robotic process automation (RPA) systems with improved playback capabilities are disclosed. Certain embodiments can provide resilient playback of software automation processes by providing enhanced capabilities to locate user interface controls within a screen image of a user interface of an application program being utilized by the software automation processes. Using such capabilities, embodiments can provide resilient playback of software automation processes by providing the capability to resolve playback errors by detecting user interface controls within at least one screen image of a user interface that was captured when the software automation process was created. Advantageously, embodiments disclosed herein allow software automation processes to continue to operate over time and thus with greater resiliency and flexibility.