Robotic Process Automation Control Identification via Image Differencing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Robotic process automation faces challenges in accurately identifying and selecting user interface controls due to variations in background content and the reliance on low-quality image representations, which can lead to incorrect icon selection and reduced automation robustness.
Innovation Solution
A robotic training system uses computer vision and user interface component analysis to detect control selection events by analyzing differences between images, determining the location of selected controls through color changes, and extracting a cropped representation of the control for automation scripts, thereby improving accuracy and reducing memory and resource requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the system uses image representations of user interface controls for automation, then the automation process can be implemented, but the accuracy of control identification decreases due to variations in background content and low-quality images
Solution Approach 1:
The patent segments the user interface control from its background by detecting color changes in difference images. Instead of using the entire screenshot containing both control and variable background content, the system extracts only the control portion by identifying pixels that changed between frames, thereby isolating the control element for accurate identification and matching during automation execution.
Solution Approach 2:
The patent extracts the essential identifying features of the control by creating a difference image that removes background content. By comparing consecutive frames and extracting only the pixels that changed (representing the control), the system creates a clean representation of the control without surrounding background elements, improving both accuracy and robustness of control identification.
2Measurement precision
If the system captures and processes multiple images to identify controls, then the accuracy of control selection improves, but the memory and computational resource usage increases
Solution Approach 1:
The patent converts the potentially harmful effect of processing multiple images (which increases resource usage) into a benefit by using frame differencing. Instead of storing and processing all image data, the system computes the difference between consecutive frames, which dramatically reduces the amount of data that needs to be retained in memory while still enabling accurate control identification through the extracted color change information.
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for creating user interface control images for robotic automation. One of the methods includes detecting, during a training process, a control selection event; retrieving a first image and a second image; determining a difference image that represents a difference between the first image and the second image; determining, using the difference image, a plurality of colors for pixels included in the difference image that are likely located outside of a control location for the control; determining one or more shapes depicted in the difference image that each have a color other than all of the colors in the plurality of colors; selecting, from the one or more shapes and using an event location, a particular shape as likely representing the control; and storing a cropped image of the control extracted from one of the two or more images.