On-Screen Application Object Detection for Reliable Process Discovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing process discovery techniques for robotic process automation face challenges in efficiently collecting and analyzing metadata from graphical user interface elements, including reliability issues with API-based and DOM-based methods, resource intensity, staleness, and privacy concerns when transmitting screenshots to remote servers.
Innovation Solution
A metadata extraction system using machine learning models, such as object detection and text recognition, processes application UI screenshots on the user's device in real-time, employing optimization techniques like caching and selective text recognition to reduce computational and resource usage, while maintaining privacy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If API-based or DOM-based methods are used to collect metadata from GUI elements, then the process discovery can be performed, but reliability issues and resource intensity problems occur
Solution Approach 1:
The patent replaces traditional API-based and DOM-based metadata collection methods with machine learning-based image processing. Instead of relying on programmatic interfaces that are unreliable and resource-intensive, the system captures screenshots and uses trained ML models to detect and extract metadata from visual GUI elements, fundamentally substituting the collection mechanism
Solution Approach 2:
The system creates visual copies (screenshots) of the GUI interface and processes these images to extract metadata. This copying approach allows the ML models to analyze the visual representation of the interface without requiring direct interaction with the application's internal structures, improving reliability while reducing resource demands
2Measurement precision
If screenshots are transmitted to remote servers for processing, then metadata extraction can be performed, but privacy concerns arise
Solution Approach 1:
The system enables the user's device to perform metadata extraction locally using trained machine learning models. Instead of sending screenshots to remote servers, the device itself processes the images and extracts metadata, making the system self-sufficient and eliminating privacy concerns associated with data transmission
3Measurement precision
If multiple trained ML models are used to process screenshots, then metadata extraction accuracy improves, but computational resources increase
Solution Approach 1:
The system applies multiple ML models selectively based on the specific metadata extraction needs. Rather than always deploying all available models, the system chooses the appropriate subset of models for each screenshot processing task, reducing unnecessary computational overhead while maintaining extraction accuracy
Data Source
AI summary
A method of gathering information about a process being performed by a user of a computing device having application programs and separate monitoring software installed thereon is described. The user performs the process by performing actions via a sequence of application user interface (UI) screens. The method comprises capturing screenshots of at least some application UI screens in the sequence to obtain a sequence of application screenshots; processing the sequence of application screenshots using multiple different trained machine learning (ML) models to extract a corresponding sequence of application UI screen metadata, the multiple different trained ML models including an object detection model and a text recognition model; using the sequence of application UI screen metadata to generate a representation of the process; and storing the representation of the process on the computing device and/or transmitting the representation of the process to another device different from the computing device.