Remote RPA Script Generation via Screen Stream Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional robotic processing automation (RPA) technologies face challenges in automating tasks on remote apparatuses where a designer cannot be installed, and require image-based automation scenarios, making it difficult to handle changes in GUI object locations and requiring OCR for text processing.
Innovation Solution
A method and system for robotic processing automation that transmits screen streams from remote apparatuses, monitors user events, and generates automation scripts based on object-related information, enabling automation of tasks on remote apparatuses without the need for local designer installations, by converting coordinates and using object-related data for script generation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If conventional RPA uses GUI object capture and coordinate information to develop automation scenarios, then automation can be performed on local applications, but automation cannot be performed on remote applications where a designer cannot be installed
Solution Approach 1:
The patent introduces a remote terminal screen as an intermediary layer between the local RPA system and the remote application. This intermediary enables the RPA system to access and control remote applications without requiring direct installation of the designer on the remote apparatus, thus resolving the contradiction between automation capability and adaptability to remote environments.
Solution Approach 2:
The patent creates a virtual copy of the remote application's GUI interface through the remote terminal screen. This copy allows the RPA system to interact with the remote application as if it were locally installed, enabling automation scenarios to be developed and executed on remote applications without physical access or designer installation on the remote side.
2Extent of automation
If automation scenarios are developed using captured image and coordinate information of remote apparatus, then automation can be implemented, but automation fails when the location of the GUI object to be automated is changed
Solution Approach 1:
The patent implements a feedback mechanism where the system continuously monitors the remote terminal screen for user events and object information. When GUI objects change location or appearance, the system detects these changes through ongoing monitoring and updates the automation scenarios accordingly, ensuring continued functionality despite GUI variations.
Solution Approach 2:
The patent enables dynamic adjustment of automation parameters based on detected changes in the remote GUI environment. When object locations or characteristics change, the system modifies the corresponding parameters in the automation scenario, allowing the automation to adapt to the new state without manual intervention.
3Loss of information
If optical character recognition (OCR) technology is used for reading text by processing remote apparatus information, then text extraction is possible, but the process becomes complex and less accurate
Solution Approach 1:
The patent extracts text information directly from the remote terminal screen's GUI object data rather than using image-based OCR. By accessing the actual text properties of GUI elements through the remote terminal interface, the system obtains text information without undergoing complex image processing, thus reducing complexity while maintaining accuracy.
Data Source
AI summary
Disclosed are a method for robotic processing automation. The method for robotic processing automation includes transmitting to a remote apparatus, a screen stream request for a target program executed in the remote apparatus, receiving from the remote apparatus, a screen stream corresponding to the screen stream request and displaying the screen stream corresponding to the screen stream request, monitoring whether a user event occurs in the displayed screen stream, transmitting to the remote apparatus, when the user event occurs, an information inquiry request for an object related to the user event, receiving from the remote apparatus, object-related information corresponding to the information inquiry request, and generating an automation script for automatically executing the target program based on at least one of the screen stream or the object-related information.