Machine-Learned GUI Command Generation From Pixels Without APIs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing approaches for interacting with graphical user interfaces (GUIs) often require access to structured application representations like HTML sources or DOM trees, which are not always available, leading to performance drops when such information is absent, and they struggle with understanding GUIs visually without relying on explicit structural data.
Innovation Solution
A machine-learned sequence processing model is trained using pixel-based pretraining and fine-tuning to recognize and navigate GUIs, processing image data and natural language instructions to generate control commands without needing custom APIs, leveraging pre-trained models to understand GUI architecture implicitly.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If existing approaches use structured application representations (HTML sources, DOM trees), then performance is improved when such information is available, but performance drops when such information is absent and they cannot understand GUIs visually
Solution Approach 1:
The model is designed to process multiple types of input representations (pixel data, HTML sources, DOM trees) through a unified architecture. The pixel-based pretraining enables the model to understand GUIs visually, while fine-tuning on structured data allows it to leverage structured information when available. This multi-functional capability ensures reliable performance across diverse input formats without requiring custom APIs for each application.
2Measurement precision
If custom APIs are required for each application, then precise control is achieved, but device complexity and maintenance needs increase
Solution Approach 1:
The patent extracts and removes the requirement for custom application-specific APIs from the system architecture. By using pixel-based pretraining, the model directly processes visual information from GUIs without needing application-level interfaces. This extraction of the API dependency layer simplifies the overall system while maintaining precise control through visual understanding and fine-tuned command generation.
Solution Approach 2:
The unified model architecture serves multiple functions: it processes pixel data, understands GUI structure implicitly, generates control commands, and adapts to different applications through fine-tuning. This universal approach replaces the need for multiple custom APIs, reducing system complexity while maintaining precise control across diverse applications.
3Adaptability or versatility
If pixel-based pretraining is used, then adaptability to UI changes is improved and API access is eliminated, but training complexity increases
Solution Approach 1:
The patent applies preliminary action by performing pixel-based pretraining on a large dataset of GUI screenshots before fine-tuning on specific tasks. This pretraining phase establishes fundamental visual understanding of GUI structures, buttons, text fields, and navigation patterns. The pre-trained model serves as a robust foundation that adapts easily to UI changes, while the subsequent fine-tuning phase efficiently adapts this foundation to specific applications without requiring custom APIs.
Data Source
AI summary
An example method can include providing a natural language instruction and user interface image data to a machine-learned sequence processing model that is configured to process image data and generate commands for controlling the target computing device, wherein the machine-learned sequence processing model has parameters learned using an interface recognition objective based on an evaluation of an interface recognition output generated based on processing a rendered training interface from a pre-training dataset and an interface navigation objective based on an evaluation of a user interface command generated based on processing a rendered training interface from a fine-tuning dataset; receiving, from the machine-learned sequence processing model, a command indicating an interaction with the user interface to implement the natural language instruction; and generating, based on the command, a control signal configured to initiate the interaction.