Software Object Image Augmentation via UI Attribute Modification

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current image augmentation techniques for training computer vision machine learning models are inadequate for software objects, as they often generate unrealistic or irrelevant images that do not improve the predictiveness or generalizability of the models, particularly for recognizing software interfaces like windows, dialogs, and web pages.

Innovation Solution

A client computer system that iteratively selects and modifies attributes of software objects, such as UI elements, using methods like UI automation, HTML, and CSS modifications, to generate a large number of realistic and relevant augmented images for training, testing, and validation datasets.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Quantity of substance

If current image augmentation techniques are used for software objects, then the quantity of training images increases, but the quality and relevance of the images deteriorate

Engineering Contradiction:

Improvequantity of training imagesVSAvoidrelevance of augmented images

Core Design Contradiction:

Quantity of substanceVSReliability

Solution Approach 1:

The system modifies specific parameters of software object images such as window positions, dialog box locations, web page layouts, and UI element attributes to generate augmented images that maintain realism and relevance while increasing dataset quantity

Inventive Principle:

Principle #35Parameter changes

Solution Approach 2:

The system segments software interfaces into distinct components (windows, dialogs, web pages, UI elements) and applies targeted augmentation to each segment, ensuring that each component remains realistic and contextually appropriate while generating diverse training images

Inventive Principle:

Principle #1Segmentation

2Adaptability or versatility

If generic image augmentation is applied to software interfaces, then the diversity of training data increases, but the predictiveness of the model decreases

Engineering Contradiction:

Improvediversity of training dataVSAvoidpredictiveness of model

Core Design Contradiction:

Adaptability or versatilityVSReliability

Solution Approach 1:

The system applies different augmentation strategies to different local regions of software interfaces based on their specific characteristics, ensuring that each region's unique properties are preserved while achieving overall data diversity

Inventive Principle:

Principle #3Local quality

Solution Approach 2:

The system dynamically adjusts augmentation parameters based on the specific software object type and context, applying appropriate transformations that maintain the dynamic and interactive nature of software interfaces rather than using static generic augmentations

Inventive Principle:

Principle #15Dynamics

Data Source

PatentUS20230161466A1Image data augmentation using user interface element attributes

Publication Date: 2023.05.25 CITRIX SYSTEMS INC

AI summary

A computer system configured to augment images of software objects is provided. The computer system includes a memory and at least one processor coupled to the memory. The at least one processor is configured to iteratively select an attribute value from a predetermined set of attribute values; modify an attribute of a software object according to the attribute value; and generate a respective augmented image of the software object with the attribute modified according to the attribute value. The software object may comprise an executable software object.