Screen Image Feature Extraction for Task Mining
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional task mining methods relying on pixel-wise comparison of screen images are limited in determining similarity between user interfaces unless they are visually near-identical or pre-defined categories, failing to effectively identify repetitive tasks for automation.
Innovation Solution
The approach involves preprocessing screen images using machine learning-based networks to extract image, text, and control element features, which are then encoded to generate representative features, enabling tasks such as similarity determination, clustering, and classification of user interfaces, with networks trained using masked language modeling and contrastive learning losses.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If pixel-wise comparison is used to determine similarity between screen images, then the method is simple to implement, but it can only determine similarity when user interfaces are visually near-identical or pre-defined categories are assigned
Solution Approach 1:
The screen image is divided into multiple regions of interest (ROIs) based on detected control elements and text boxes. Each ROI is processed independently to extract features, allowing the system to focus on functional elements rather than the entire visual interface. This segmentation enables effective comparison of interfaces with different visual layouts but similar functional structures.
Solution Approach 2:
The patent replaces the mechanical pixel-wise comparison approach with a machine learning-based feature extraction and encoding system. Instead of directly comparing pixel values, the system uses trained neural networks to extract semantic features from ROIs and encode them into representative vectors, enabling similarity determination based on functional content rather than visual appearance.
2Measurement precision
If multiple machine learning based networks are used to extract and encode features, then the accuracy of task mining is improved, but the system complexity increases
Solution Approach 1:
The patent combines multiple feature extraction processes (image features from ROIs, text features from text boxes, control element features from detected controls) into a unified encoding framework. The third machine learning network encodes all these different feature types into a single representative feature vector, merging multiple information sources into a cohesive representation that improves accuracy while managing system complexity through integration.
3Stability of the object's composition
If screen images are preprocessed with resizing and normalization, then the input consistency for machine learning networks is improved, but the processing time increases
Solution Approach 1:
The patent performs preprocessing operations (resizing to standardized dimensions and normalization of pixel values) on screen images before they are fed into the machine learning networks. This preliminary action ensures that all input images have consistent dimensions and value ranges, which improves the stability and performance of the neural networks. The preprocessing is done once per image before the main feature extraction process, minimizing its impact on overall processing time.
Data Source
AI summary
Systems and methods for extracting features from screen images for performing a task mining task are provided. A screen image depicting a user interface of a computing system is received. The screen image is preprocessed to generate a preprocessed screen image and processing results. Image features are extracted from the preprocessed screen image using a first machine learning based network. Text features and control element features are extracted from the processing results using a second machine learning based network. The text features and the control element features are encoded using a third machine learning based network to generate representative features of the screen image. A task is performed on the screen image based on one or more of the image features, the text features, the control element features, or the representative features. Results of the task are output.