AI-Driven UI Structure Grounding for Accurate Task Automation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current automation, productivity, and accessibility frameworks struggle to understand user interface structures accurately, leading to inaccurate results and hallucinations due to a lack of grounding in the structure of web and application layouts, necessitating application-specific training and failing to meet accessibility standards for users with disabilities.
Innovation Solution
A framework that analyzes user interfaces using a vision language model to identify tasks, trims irrelevant structural tags, and generates executable code via a completion language model to perform actions, without requiring integration into the target application, and includes moderation services to ensure content appropriateness.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If current AI models analyze DOM or source code to understand user interface structure, then automation and productivity tasks can be performed, but the models produce inaccurate results and hallucinations due to lacking understanding of application layouts
Solution Approach 1:
The patent introduces an intermediary component called 'UI structure grounding framework' that mediates between the raw DOM/source code and the AI model. This framework processes the UI structure information before presenting it to the AI model, thereby improving accuracy without requiring changes to the core model architecture. The intermediary translates and structures the UI information in a way that the model can understand accurately.
Solution Approach 2:
The system performs preliminary processing of UI structure information before it reaches the AI model. The UI structure grounding framework pre-processes DOM elements, organizes hierarchical relationships, and prepares structured representations in advance. This preliminary action ensures that when the AI model receives the information, it is already in an optimized format that prevents hallucinations and improves task identification accuracy.
2Measurement precision
If automation frameworks are tailored for each specific application with custom training, then accuracy for that application improves, but device complexity and development time increase significantly
Solution Approach 1:
The UI structure grounding framework is designed as a universal component that can work with multiple different applications and DOM structures without requiring custom training for each. It provides a standardized interface and processing methodology that adapts to various application types, thereby maintaining high accuracy across different targets while avoiding the complexity of application-specific customization.
Solution Approach 2:
The framework adjusts its processing parameters and structure representation dynamically based on the input DOM characteristics, rather than requiring retraining for each application. By changing parameters such as element selection criteria, hierarchy depth, and attribute prioritization, the system adapts to different applications while maintaining a single unified framework, thus reducing complexity.
3Loss of information
If complete DOM structure is provided to the AI model, then comprehensive information is available, but analysis time and computational resources increase
Solution Approach 1:
The UI structure grounding framework extracts only the relevant and essential portions of the DOM structure needed for task identification, rather than processing the complete DOM tree. It identifies and extracts key elements such as interactive components, hierarchical relationships, and critical attributes, thereby maintaining information completeness for task execution while significantly reducing the volume of data requiring AI model analysis and cutting processing time.
Data Source
Figure 1A
Figure 1B
Figure 2
AI summary
A data processing system implements obtaining an image of a user interface of an application and a textual representation of a structure of the user interface from a client-side proxy on a client device; analyzing the image of the user interface using a vision language model to obtain one or more task recommendations; analyzing the textual representation of the structure of the user interface and the one or more task recommendations using a completion language model to generate one or more workflows, each workflow being associated with a task recommendation of the one or more task recommendations and comprising one or more actions and executable code to perform the one or more actions in the application; providing the one or more workflows to the client-side proxy; causing the client-side proxy to present a representation of each task recommendation of the one or more task recommendations on the user interface;