Synthesized UI Control Detection for Privacy-Aware ML Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Detecting user interface control objects in images from applications behind firewalls or in virtual desktop environments is challenging due to lack of programmatic access and the difficulty in obtaining a large number of training images, especially in scenarios where sensitive information is involved, requiring a method that is privacy-aware and agnostic to application frameworks and computing environments.
Innovation Solution
A computer-implemented method that uses a machine learning model trained on an initial dataset of synthesized images to detect user interface control objects, which is updated with new images generated by processing screen images from application programs, employing data point generators to create new synthesized image sets while respecting privacy constraints by removing or replacing sensitive information.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a large number of real application screen images are collected for training, then the machine learning model detection accuracy is improved, but privacy security deteriorates due to sensitive information in the images
Solution Approach 1:
The patent creates synthesized copies of application screen images using a data point generator that replicates UI control objects in various contexts. Instead of using real sensitive images, the system generates synthetic training images that preserve the visual characteristics and control object patterns without containing any actual sensitive business information, thus maintaining detection accuracy while protecting privacy security
Solution Approach 2:
The patent extracts only the essential visual features and control object patterns from real application screens while removing all sensitive information. The data point generator captures UI control characteristics (buttons, text boxes, menus) without retaining any business-sensitive content, separating the useful training features from the harmful sensitive data
2Adaptability or versatility
If more training images are obtained from diverse applications, then the model's adaptability to different applications is improved, but the complexity of data collection and processing increases
Solution Approach 1:
The system performs self-service by automatically generating diverse training images through the data point generator without requiring manual collection from multiple applications. The generator autonomously creates varied UI scenarios by manipulating control objects in different contexts, eliminating the complex data collection process while maintaining high model adaptability
Solution Approach 2:
The patent prepares diverse training data in advance through synthesized image generation before model training begins. The data point generator pre-creates a comprehensive dataset with various UI configurations, control types, and layout patterns, so that when training starts, all necessary diverse data is already available, eliminating the need for complex ongoing data collection
3Object-affected harmful factors
If synthesized images are used for training instead of real images, then privacy protection is improved, but the initial model accuracy may deteriorate
Solution Approach 1:
The patent implements a dynamic two-stage training approach where the model is first trained on synthesized privacy-protected images to learn general UI patterns, then fine-tuned on a small set of real application images to achieve high accuracy. This dynamic progression allows the model to benefit from privacy protection during initial learning while still achieving high accuracy through subsequent adaptation
Solution Approach 2:
The system performs preliminary training on synthesized images to establish a solid foundation of UI control recognition patterns before introducing real images. This preliminary action on privacy-protected data prepares the model with generalizable knowledge, reducing the amount of sensitive real data needed and minimizing accuracy loss while maintaining strong privacy protection
Data Source
AI summary
A User Interface (UI) interface object detection system employs an initial dataset comprising a set of images, that may include synthesized images, to train a Machine Learning (ML) engine to generate an initial trained model. A data point generator is employed to generate an updated synthesized image set which is used to further train the ML engine. The data point generator may employ images generated by an application program as a reference by which to generate the updated synthesized image set. The images generated by the application program may be tagged in advance. Alternatively, or in addition, the images generated by the application program may be captured dynamically by a user using the application program.