RPA Change Detection for GUI Object Shift and Shape Variation

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing RPA systems face challenges when the working environment changes, leading to errors and requiring users to recreate or modify bots, which is resource-intensive and inefficient, especially in dynamic environments.

Innovation Solution

An RPA apparatus with a change detection method that automatically detects changes in the working environment, updates object information, and corrects errors without user intervention, using a computer device with a processor and memory that includes an RPA performing module and a change detection module to identify and adapt to changes in the GUI or object positions and shapes.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If a bot is created based on a working environment at the time of creation, then the bot can operate normally in that environment, but when the working environment changes (GUI resolution, website version, program version), the bot malfunctions or stops working

Engineering Contradiction:

Improvebot operation reliabilityVSAvoidenvironmental adaptability

Core Design Contradiction:

ReliabilityVSAdaptability or versatility

Solution Approach 1:

The system performs preliminary actions by capturing screenshots of the working environment at different times and storing them for comparison. This allows the bot to proactively detect environmental changes before they cause malfunction, enabling preventive adaptation rather than reactive troubleshooting.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system implements feedback by continuously comparing current screenshots with previously stored screenshots, detecting changes in the working environment, and automatically notifying users or adjusting bot behavior based on detected changes. This closed-loop feedback mechanism maintains bot reliability in changing environments.

Inventive Principle:

Principle #23Feedback

2Adaptability or versatility

If a new bot is recreated or existing bot is modified to adapt to changed working environment, then the bot can operate normally in the new environment, but it requires a lot of time and resources even for experienced users

Engineering Contradiction:

Improveenvironmental adaptabilityVSAvoidbot recreation or modification time

Core Design Contradiction:

Adaptability or versatilityVSLoss of time

Solution Approach 1:

The system enables self-service by automatically detecting environmental changes through screenshot comparison and generating notifications or adaptation suggestions without requiring user intervention. This eliminates the time-consuming manual bot recreation or modification process while maintaining adaptability to environment changes.

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The system replaces the mechanical process of manual bot modification with an automated computer vision-based screenshot comparison mechanism. This substitution eliminates the need for users to manually recreate or modify bots, significantly reducing the time and resources required for adaptation.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

3Adaptability or versatility

If the position or shape of an object referred to by RPA in a screen is changed, then the working environment is updated, but the bot cannot accurately find the object to manipulate

Engineering Contradiction:

Improveobject position and shape variabilityVSAvoidobject detection accuracy

Core Design Contradiction:

Adaptability or versatilityVSMeasurement precision

Solution Approach 1:

The system creates visual copies by capturing screenshots of the working environment and storing them as reference images. When objects change position or shape, the system compares current screenshots with stored copies to locate and identify objects, maintaining detection accuracy without requiring precise coordinate or shape specifications.

Inventive Principle:

Principle #26Copying

Solution Approach 2:

The system replaces traditional coordinate-based or shape-based object identification mechanisms with screenshot-based visual comparison. This substitution allows the bot to accurately find objects even when their position or shape changes, as the system visually matches objects based on their appearance in screenshots rather than relying on fixed spatial parameters.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Data Source

PatentUS11279040B2Robot process automation apparatus and method for detecting changes thereof

Publication Date: 2022.03.22 SAMSUNG SDS CO LTD

AI summary

According to aspects of the present disclosure, a change detection method for ROBOT process automation (RPA) performed by a computer device includes obtaining object information of a target object from first working environment information at a first time, and determining a comparison object detected in second working environment information at a second time as an object corresponding to the target object based on the obtained object information of the target information. Thus, even if an error occurs due to a change in a working environment, an RPA apparatus can operate normally by correcting the error by itself without the need for a user to recreate a new bot or modify the existing bot. Also, even if the position or shape of an object referred to by RPA in a screen is changed, the RPA apparatus can accurately find an object to manipulate based on surrounding information and perform RPA normally.