GUI Operation Log Generation via Image Clustering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods face challenges in acquiring operation logs for GUI applications, particularly due to the complexity of specifying conditions in advance and the impracticality of creating mechanisms for multiple applications across various execution environments, especially in thin client environments where attribute values of GUI components cannot be acquired.
Innovation Solution
An operation log generation device that captures images of the operation screen, extracts candidate GUI components using image features, specifies operable components from event occurrence positions, and classifies images into clusters to generate operation logs, allowing for the reproduction of operation flows without requiring specific conditions or teacher data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a mechanism is developed to acquire attribute values of GUI components for each application and execution environment, then operation logs can be acquired with high precision, but the device complexity and cost become prohibitively high
Solution Approach 1:
The patent creates a universal operation log acquisition mechanism that works across multiple applications and execution environments (including thin client environments) without requiring separate mechanisms for each. The system uses a common approach of capturing screen images and detecting operation events that can identify GUI components, making the mechanism applicable universally rather than requiring application-specific implementations.
Solution Approach 2:
Instead of directly accessing and acquiring attribute values from GUI components (which requires deep integration with each application), the patent captures screen images as visual copies and detects operations on these images. This indirect copying approach allows operation log acquisition without requiring direct access to application internals, thereby reducing complexity while maintaining measurement precision.
2Loss of information
If conditions are specified in advance to filter operation events, then only meaningful operation events are recorded, but the ease of operation decreases due to complex configuration requirements
Solution Approach 1:
The system automatically performs operation log acquisition and filtering without requiring users to manually specify conditions in advance. The mechanism captures all operation events and uses automatic identification methods (such as image recognition and clustering) to distinguish meaningful operations from irrelevant ones, eliminating the need for complex pre-configuration while still achieving effective filtering.
3Adaptability or versatility
If image processing and clustering are used to identify GUI components, then operation logs can be acquired in thin client environments, but the processing time increases
Solution Approach 1:
The patent applies image processing and clustering selectively rather than to all possible operations. The system focuses processing on identifying GUI components that are actually interacted with during operations, rather than processing the entire screen continuously. This partial action approach maintains adaptability to thin client environments while reducing unnecessary processing time.
Data Source
AI summary
An acquisition unit detects an operation event of a user and acquires an occurrence position of the operation event in a captured image of an operation screen. A specifying unit specifies an image of the occurrence position of the operation event from among images to be candidates for a GUI component extracted from the captured image and records the image and the operation event in association with each other. A classifying unit classifies a set of recorded images into clusters according to the similarity between the images. A generation unit generates an operation log by using an image corresponding to the operation event of each classified cluster.