UI Attribute Extraction for Accurate Process Discovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing process discovery techniques in robotic automation rely on low-level data collection, such as click and keystroke data, which limits the accuracy of identifying computerized processes and fails to differentiate between similar processes based on non-interactive user interface elements and contextual information.
Innovation Solution
Collect and analyze contextual information associated with user interface elements, including both interactive and non-interactive elements, to identify attributes like names, values, and locations, enhancing the accuracy of process discovery by differentiating between similar processes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If low-level data collection (click and keystroke data) is used for process discovery, then device complexity is reduced, but measurement precision and reliability of process identification deteriorate
Solution Approach 1:
The patent segments the data collection process into multiple layers: low-level interaction data (clicks, keystrokes) and high-level contextual data (UI element properties, screen information). This segmentation allows the system to maintain simplicity in data collection mechanisms while achieving high measurement precision through multi-layered attribute gathering from both interactive and non-interactive UI elements
Solution Approach 2:
The patent transitions from single-dimension data collection (only interaction events) to multi-dimensional data collection by incorporating contextual attributes of UI elements such as names, values, types, and locations. This dimensional expansion enables more accurate process differentiation without significantly increasing system complexity
2Ease of operation
If only interactive UI elements are monitored, then ease of operation is improved, but loss of information occurs regarding non-interactive elements that differentiate similar processes
Solution Approach 1:
The patent implements a universal monitoring mechanism that serves multiple functions: it captures both interactive user actions and contextual information from non-interactive UI elements through a single data collection framework. This multi-functional approach prevents information loss while maintaining ease of operation by using a unified monitoring system rather than separate mechanisms
3Measurement precision
If detailed contextual information from all UI elements is collected, then process discovery accuracy is improved, but quantity of data and processing requirements increase
Solution Approach 1:
The patent extracts only the most relevant attributes from UI elements (names, values, types, locations) rather than collecting complete screenshots or all possible element properties. This extraction approach maintains high process differentiation accuracy while significantly reducing the volume of collected data and associated processing requirements
Data Source
AI summary
System and methods for gathering information about a process being performed by a user of a computing device, the computing device having computer software programs and separate monitoring software installed thereon. Action information associated with zero, one or more actions performed by the user via a particular UI screen and contextual information associated with one or more UI elements visible in the particular UI screen is collected by the monitoring software. The contextual information is analyzed to identify attributes for the particular UI screen, each attribute corresponding to at least one UI element visible in the particular UI screen. Analyzing the contextual information comprises identifying, for each attribute, a respective attribute name, a respective attribute value, and/or a respective location in the particular UI screen. The attributes and information indicating their names, values, and/or locations is stored.