GUI Object Detection via Edge Analysis and ML

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing computer testing automation techniques require prior metadata about GUI objects and cannot dynamically learn the behavior of a system under test without pre-existing knowledge, limiting their ability to detect objects of any shape or operate in fully automated or minimally supervised learning scenarios.

Innovation Solution

A learning engine that employs supervised and unsupervised machine learning to detect and learn the behavior of GUI objects from screenshots, using edge detection algorithms, deduplication, and object-type determination, allowing for fully automated testing without prior metadata and enabling detection of objects of any shape.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Difficulty of detecting and measuring

If prior metadata about GUI objects is required for testing automation, then testing can be performed with structured information, but the system cannot dynamically learn behavior or detect objects of any shape

Engineering Contradiction:

Improveobject detection capabilityVSAvoiddynamic learning capability

Core Design Contradiction:

Difficulty of detecting and measuringVSAdaptability or versatility

Solution Approach 1:

The system performs self-service by automatically generating metadata about GUI objects through machine learning and edge detection algorithms. Instead of requiring pre-existing metadata, the system autonomously learns object properties, relationships, and behaviors by analyzing screenshots and interacting with the GUI, thereby enabling both detection of any shape and dynamic learning without external metadata input

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The patent replaces the mechanical approach of manual metadata creation and structured object recognition with intelligent systems including supervised and unsupervised machine learning algorithms. These systems automatically extract features from images, generate metadata dynamically, and learn object behaviors without requiring pre-defined object models or structured information about the GUI

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Extent of automation

If fully automated testing is implemented without prior metadata, then automation extent increases, but the complexity of learning and detecting objects increases

Engineering Contradiction:

Improvetesting automation levelVSAvoidlearning engine complexity

Core Design Contradiction:

Extent of automationVSDevice complexity

Solution Approach 1:

The learning engine is segmented into distinct functional modules: screenshot capture, edge detection, object identification, metadata generation, and behavior learning. Each module performs a specific task in the automation pipeline, reducing overall system complexity by breaking down the complex process of fully automated testing into manageable, specialized components that can be developed and maintained independently

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces metadata as an intermediary data structure that bridges the gap between raw screenshot data and high-level testing decisions. The machine learning algorithms generate this intermediate metadata representation that captures essential object properties and relationships, making the complex visual information more manageable and easier to process for automated testing logic

Inventive Principle:

Principle #24Intermediary (Mediator)

3Adaptability or versatility

If metadata is generated dynamically instead of using pre-existing knowledge, then adaptability to new systems improves, but the time required to learn behavior increases

Engineering Contradiction:

Improvesystem adaptabilityVSAvoidlearning time

Core Design Contradiction:

Adaptability or versatilityVSLoss of time

Solution Approach 1:

The system performs preliminary actions by pre-processing screenshots through edge detection and feature extraction algorithms before actual testing begins. This preliminary metadata generation and object detection work is done in advance, creating a structured representation of the GUI that accelerates subsequent learning and testing operations, reducing the overall time required for dynamic adaptation

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS11461689B2Techniques for automatically testing/learning the behavior of a system under test (SUT)

Publication Date: 2022.10.04 PETURSSON SIGURDUR RUNAR

AI summary

Techniques are disclosed for systems and methods for learning the behavior of and/or for performing automated testing of a system under test (SUT). The learning/testing is accomplished solely via the graphical user interface (GUI) of the SUT and requires no a priori metadata/knowledge about the GUI objects. The learning engine operates by performing actions on the GUI and by observing the results of these actions. If the actions result in a change in the screen/page of the GUI then a screenshot is taken for further processing. Objects are detected from the screenshot, new actions that need to be performed on the objects are guessed, those actions are performed, the results are observed and the process repeats. Text labels on the screens are also read and are used for generating contextualized inputs for the screens. The learning process continues until any predetermined learning/testing criteria are satisfied.