Multimodal UI Automation for Text-and-Image Task Reasoning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing UI automation tools struggle with tasks that require reasoning over both text and images, such as interacting with map applications or text-lite applications, as they rely on a single modality and cannot effectively handle the combination of visual and textual information.
Innovation Solution
A framework using visual and text state information, written objectives and rules, and generative AI to automate UI interactions by capturing current state information, sending multimodal input to a generative AI system, receiving predicted actions, and implementing programmatic actions on the UI.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If single-modality UI automation tools are used, then device complexity is reduced, but the ability to handle complex tasks requiring both text and image reasoning deteriorates
Solution Approach 1:
The patent combines multiple modalities (visual information from screenshots and textual information from UI elements) into a unified multimodal input system that feeds into a large language model, enabling the system to handle complex UI tasks that require both image and text reasoning capabilities
Solution Approach 2:
The patent introduces a multimodal processing intermediary that converts UI screenshots and element information into a standardized format suitable for large language model processing, bridging the gap between visual UI representation and textual reasoning capabilities
2Productivity
If generative AI systems with multimodal input are implemented, then task solving capability improves, but computational resource consumption increases
Solution Approach 1:
The patent extracts only the essential visual and textual features from UI screenshots that are relevant for task completion, removing redundant information before processing through the generative AI system, thereby reducing computational resource consumption while maintaining automation efficiency
Solution Approach 2:
The patent performs preliminary processing of UI information (such as identifying relevant UI elements and extracting key textual content) before submitting to the generative AI system, preparing the data in advance to optimize processing efficiency and reduce computational overhead during execution
3Measurement precision
If comprehensive visual and textual information is captured, then measurement precision of UI state improves, but data processing time increases
Solution Approach 1:
The patent applies local quality processing by focusing computational resources on analyzing specific regions of interest in UI screenshots and prioritizing extraction of relevant textual information from key UI elements, rather than uniformly processing the entire screen, thus improving recognition accuracy while reducing overall processing time
Data Source
AI summary
Systems and methods provide user interface (UI) automation for multiple types of devices and applications. A computing device stores an objective for automated manipulation of a UI of a user device and conducts an iterative sequence to achieve the objective. The iterative sequence includes capturing current state information of the UI, wherein the current state information includes text and an image of the UI; sending multimodal input to a generative artificial intelligence (AI) system, wherein the multimodal input includes the image of the UI, a text prompt, and the objective; receiving, from the AI system, a predicted next action for the UI based on the multimodal input; initiating a programmatic action to implement the predicted next action on the UI; and recording the programmatic action. The computing device generates, based on the iterative sequence, a UI automation model associated with achieving the objective on the UI.