RPA Task Recording with UI Object Type Correction

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current Robotic Process Automation (RPA) task recording methods lack accuracy and versatility, failing to effectively capture user interactions with standardized user interface components, leading to inaccuracies in automated task execution.

Innovation Solution

A method for improving RPA recording accuracy by post-processing recorded tasks, which includes correcting target object types, removing noise events, and replacing partial event sequences with predefined activities, using event sequences that include information about user manipulations, target object positions, and holding times, and applying object type classification models to enhance accuracy and versatility.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Ease of operation

If basic user manipulation recording is used in RPA, then accessibility is improved for non-programmers, but accuracy of task recording deteriorates

Engineering Contradiction:

ImproveaccessibilityVSAvoidaccuracy of task recording

Core Design Contradiction:

Ease of operationVSMeasurement precision

Solution Approach 1:

The patent introduces an intermediary processing layer that automatically analyzes recorded user manipulations and translates them into standardized UI component interactions. This intermediary process bridges the gap between simple recording (easy for users) and accurate representation (needed for reliable automation), without requiring users to understand the complex translation process.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The patent replaces manual configuration and mechanical recording of each UI interaction with an automated intelligent system that uses algorithms to identify and classify UI components. This substitution transforms the mechanical process of recording every mouse movement and keyboard stroke into an intelligent system that understands UI semantics automatically.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Loss of information

If detailed event sequences are recorded for all user manipulations, then completeness of task information is improved, but complexity of processing deteriorates

Engineering Contradiction:

Improvecompleteness of task informationVSAvoidcomplexity of processing

Core Design Contradiction:

Loss of informationVSDevice complexity

Solution Approach 1:

The patent extracts and removes redundant or irrelevant information from the recorded event sequences, keeping only the essential UI component interactions needed for task automation. This extraction process reduces processing complexity while maintaining the completeness of critical task information by filtering out noise and focusing on meaningful actions.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The patent segments the continuous stream of user manipulations into discrete, meaningful UI component interactions. By dividing the complex event sequence into identifiable segments corresponding to specific UI components and their actions, the system simplifies processing while preserving complete task information in a structured, manageable format.

Inventive Principle:

Principle #1Segmentation

3Ease of manufacture

If RPA recording captures generic user manipulations, then ease of implementation is improved, but versatility of RPA scenario deteriorates

Engineering Contradiction:

Improveease of implementationVSAvoidversatility of RPA scenario

Core Design Contradiction:

Ease of manufactureVSAdaptability or versatility

Solution Approach 1:

The patent changes the parameters of recorded manipulations from generic coordinates and timestamps to semantic UI component parameters such as component type, properties, and hierarchical relationships. This parameter transformation enables the same recording process to work across different applications and UI frameworks, significantly improving versatility while maintaining ease of implementation through automated parameter mapping.

Inventive Principle:

Principle #35Parameter changes

Solution Approach 2:

The patent creates a universal recording system that can capture and represent UI interactions across multiple applications and interfaces using standardized component models. This universal approach allows a single RPA scenario to be adapted to various contexts and applications, enhancing versatility without requiring separate implementation for each specific case.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS20240142943A1Method and system for task recording using robotic process automation technology

Publication Date: 2024.05.02 SAMSUNG SDS CO LTD

AI summary

Provided is a method for recording a task including: recording a task using robotic process automation (RPA) solution; sequentially obtaining each event included in an event sequence generated as a result of the recording, wherein the event sequence includes information about a type of user manipulation, information about a target object position, and information about a target object type; correcting the target object type of the obtained event using the target object position of the obtained event; reproducing the obtained event using the target object position of the obtained event and a corrected target object type of the obtained event; and repeating the obtaining, the correcting, and the reproducing until an event included in the event sequence is exhausted.