GUI Component Identification via Image Segmentation

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional GUI segmentation techniques struggle to accurately partition GUIs when hierarchy information is unavailable, such as in FLASH applications or Web 2.0 development toolkits that lack semantic information, leading to difficulties in determining GUI component boundaries and relationships.

Innovation Solution

The method involves using image segmentation techniques to identify GUI components from screen images, employing edge detection, geometric shape recognition, and template matching, which can function without DOM information, and providing data to facilitate GUI component identification and hierarchy inference.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If conventional GUI segmentation techniques rely on run time object hierarchy (DOM) information, then GUI component differentiation is possible when DOM is available, but the technique fails when DOM information is unavailable or insufficient (e.g., FLASH applications, Web 2.0 toolkits)

Engineering Contradiction:

ImproveGUI segmentation reliabilityVSAvoidAdaptability to GUIs without DOM information

Core Design Contradiction:

ReliabilityVSAdaptability or versatility

Solution Approach 1:

The patent introduces image processing techniques as an intermediary method to bridge the gap when DOM information is unavailable. By capturing screen images and applying image segmentation, edge detection, and template matching, the system can identify GUI components in applications like FLASH and Web 2.0 toolkits that lack proper DOM semantics, thus maintaining segmentation reliability across diverse GUI types.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The patent changes the fundamental parameter used for GUI segmentation from relying on DOM hierarchy information to using visual image data. By transforming the input from structured DOM data to unstructured screen images and applying computer vision algorithms, the system adapts to handle cases where DOM information is insufficient while maintaining segmentation accuracy.

Inventive Principle:

Principle #35Parameter changes

2Adaptability or versatility

If image segmentation techniques are used to identify GUI components from screen images, then GUI component identification can proceed without DOM information, but the complexity of the segmentation process increases

Engineering Contradiction:

ImproveAbility to process GUIs without DOM informationVSAvoidComplexity of segmentation process

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The patent applies segmentation by dividing the GUI identification process into distinct stages: screen image capture, image preprocessing, edge detection, template matching, and component identification. This multi-stage segmentation approach makes the complex image processing task more manageable and systematic, allowing the system to handle diverse GUI components through a structured pipeline.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent employs universal image processing techniques (edge detection, template matching, geometric shape recognition) that can identify various types of GUI components across different applications without requiring application-specific logic. These multi-functional algorithms handle buttons, text fields, icons, and other elements uniformly, reducing overall system complexity despite the sophisticated processing required.

Inventive Principle:

Principle #6Universality (Multi-functionality)

3Measurement precision

If multiple image segmentation techniques are employed (edge detection, geometric shape recognition, template matching), then GUI component identification accuracy improves, but the processing time and computational resources increase

Engineering Contradiction:

ImproveGUI component identification accuracyVSAvoidProcessing time

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The patent applies preliminary action by performing image preprocessing steps (grayscale conversion, noise filtering, edge enhancement) before the main segmentation tasks. By preparing the image data in advance with these preprocessing operations, the subsequent edge detection and template matching processes run more efficiently, reducing overall processing time while maintaining high identification accuracy.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent applies partial action by selectively applying different image segmentation techniques based on the specific GUI component being identified. Not all components require all three techniques (edge detection, geometric shape recognition, template matching); the system applies only the necessary techniques for each component type, optimizing processing efficiency while maintaining accuracy.

Inventive Principle:

Principle #16Partial or excessive action

Data Source

PatentUS8762873B2Graphical user interface component identification

Publication Date: 2014.06.24 MICRO FOCUS LLC

AI summary

Systems, methods, and other embodiments associated with graphical user interface (GUI) component identification are described. One example method includes identifying a set of GUI components. The identifying may be based on a screen image of a GUI. The screen image may comprise data describing a visual representation of the GUI. The example method may also include providing data identifying the set of GUI components.