Screen Response Validation for RPA Robots
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current robotic process automation (RPA) technologies lack effective methods to validate screen responses, leading to potential misinterpretation of interactions with computing systems, as RPA robots cannot visually verify the outcomes of their actions.
Innovation Solution
Implementing a screen response validation system that continuously compares current and previous screenshots to identify changes, using optical character recognition (OCR) and computer vision to determine the accuracy of robot interactions, allowing for corrective actions if discrepancies are found.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If RPA robots interact with computing systems without screen response validation, then automation speed and productivity are improved, but reliability and accuracy of interactions deteriorate
Solution Approach 1:
The patent implements screen response validation that continuously monitors and validates the computing system's screen responses after robot interactions. The validation system captures screenshots, compares them against expected outcomes, and provides feedback to determine whether the interaction was successful. This feedback mechanism ensures reliability while maintaining automation speed by only intervening when validation fails.
Solution Approach 2:
The patent performs preliminary validation by capturing and analyzing screenshots before the robot proceeds to the next interaction step. By validating screen responses in advance, the system ensures accuracy is maintained without significantly impacting the overall automation speed, as the validation occurs as part of the normal workflow rather than as a separate post-processing step.
2Reliability
If screen response validation is implemented, then reliability and accuracy of robot interactions are improved, but device complexity increases
Solution Approach 1:
The patent introduces a screen response validation system as an intermediary layer between the RPA robot and the computing system. This intermediary captures screenshots, performs image processing and text recognition, compares results against expected outcomes, and determines interaction success. By isolating the complexity within this dedicated validation module, the overall system architecture remains manageable while achieving high reliability.
Solution Approach 2:
The patent creates visual copies (screenshots) of the computing system's display to analyze and validate responses. By working with image copies rather than directly manipulating the actual display, the system can perform complex validation operations without interfering with the normal operation of the computing system, thus managing complexity effectively.
3Reliability
If screen response validation is implemented, then reliability is improved, but processing time and productivity are reduced
Solution Approach 1:
The patent performs partial validation by focusing on specific regions of the screen or key elements that are most critical to validate, rather than analyzing the entire screen. This selective approach maintains high reliability for critical interactions while reducing the overall processing time required for validation, thus balancing accuracy with productivity.
Solution Approach 2:
The patent implements periodic validation where screenshots are captured and analyzed at specific intervals or at key decision points in the automation workflow, rather than continuously. This periodic approach ensures reliability is maintained at critical moments while minimizing the time overhead by avoiding constant validation at every single interaction step.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Screen response validation of robot execution for robotic process automation (RPA) is disclosed. Whether text, screen changes, images, and/or other expected visual actions occur in an application executing on a computing system that an RPA robot is interacting with may be recognized. Where the robot has been typing may be determined and the physical position on the screen based on the current resolution of where one or more characters, images, windows, etc. appeared may be provided. The physical position of these elements, or the lack thereof, may allow determination of which field(s) the robot is typing in and what the associated application is for the purpose of validation that the application and computing system are responding as intended. When the expected screen changes do not occur, the robot can stop and throw an exception, go back and attempt the intended interaction again, restart the workflow, or take another suitable action.