Non-Standard UI Object Identification Using Layout-Weighted Similarity

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing robotic process automation (RPA) technologies struggle to reliably perform automated tasks on non-standard user interfaces (UIs) due to the inability to identify individual elements within these interfaces, leading to inconsistent performance.

Innovation Solution

A system and method for identifying and classifying non-standard UI elements by extracting objects from a screen image, determining object type candidates based on similarities with standard UI types, and applying object type-specific weight values to accurately identify the search target object.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If standard UI elements are used, then automation reliability is improved, but adaptability to non-standard UIs deteriorates

Engineering Contradiction:

Improveautomation reliabilityVSAvoidadaptability to non-standard UIs

Core Design Contradiction:

ReliabilityVSAdaptability or versatility

Solution Approach 1:

The system creates visual copies (screenshots) of non-standard UI elements and compares them against a database of standard UI element templates. By matching visual characteristics rather than relying on programmatic interfaces, the system can identify and automate non-standard UI elements with the same reliability as standard elements.

Inventive Principle:

Principle #26Copying

Solution Approach 2:

The system changes the identification parameters from programmatic UI element properties to visual image characteristics. By converting UI elements into image data and comparing visual parameters (color, shape, position, text content), the system achieves consistent identification across both standard and non-standard UI implementations.

Inventive Principle:

Principle #35Parameter changes

2Adaptability or versatility

If non-standard UI elements are used, then adaptability is improved, but automation reliability deteriorates

Engineering Contradiction:

Improveadaptability to non-standard UIsVSAvoidautomation reliability

Core Design Contradiction:

Adaptability or versatilityVSReliability

Solution Approach 1:

Inventive Principle:

Principle #26Copying

3Adaptability or versatility

If visual screenshot-based identification is used, then adaptability to non-standard UIs is improved, but measurement precision deteriorates

Engineering Contradiction:

Improveadaptability to nonstandard UIsVSAvoidobject identification precision

Core Design Contradiction:

Adaptability or versatilityVSMeasurement precision

Solution Approach 1:

The system segments the screenshot into multiple candidate regions and evaluates each independently against the template database. By dividing the visual identification task into discrete candidate evaluations with specific matching criteria, the system maintains precise identification while handling diverse nonstandard UI elements.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system provides feedback through confidence scores and matching results for each identified UI element. This feedback mechanism allows the system to adjust identification thresholds and improve precision by learning from matching outcomes, ensuring accurate identification even with visual-based methods.

Inventive Principle:

Principle #23Feedback

Data Source

PatentEP3992923B1System and method for identifying non-standard user interface object

Publication Date: 2026.02.25 SAMSUNG SDS CO LTD

Figure 1
Figure 2~3
Figure 4

AI summary

Provided is a non-standard user interface object identification system. An object candidate extractior extracts one or more objects from an image. A first similarity analyzer determines object type candidates of the one or more objects in accordance with similarities between the one or more objects and a standard user interface (UI) element. A second similarity analyzer selects object type-specific weight values in accordance with layout characteristics of the one or more objects and determines object types of the one or more objects using the object type candidates and the object type-specific weight values. An object identifier receives type and characteristic information of a search target object and identifies the search target object in accordance with characteristic information and the object types of the one or more objects.