Mobile Agent Training Data via State Transition Graphs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The challenge in training mobile agents lies in the need for sufficient high-quality data, which is currently obtained manually, leading to labor-intensive and error-prone processes.
Innovation Solution
A method involving collecting data triples representing interaction behaviors, constructing a state transition graph, obtaining an interaction trajectory, and generating training data for mobile agents without manual intervention, utilizing a simulator and depth-first search strategy with multimodal large language models for semantic understanding.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If manual construction of training data is used, then data quality can be controlled, but labor intensity and errors increase
Solution Approach 1:
The system enables automatic self-generation of training data through simulator-based virtual interaction. The mobile agent autonomously explores the virtual environment, collects data triples, constructs state transition graphs, and generates training datasets without requiring manual annotation or human labor, thereby resolving the contradiction between data quality control and labor intensity
Solution Approach 2:
The patent replaces the manual mechanical process of data collection and annotation with an automated computational system. The simulator executes virtual interactions automatically, and the processing system automatically constructs graphs and generates data, substituting human manual operations with automated algorithms and systems
2Measurement precision
If manual construction of training data is used, then data accuracy can be ensured, but time consumption increases
Solution Approach 1:
The simulator-based system enables continuous automatic data generation without interruption. The mobile agent continuously explores the virtual environment, collects interactions, and generates training data in an uninterrupted automated flow, eliminating the time-consuming manual processes while maintaining data accuracy through systematic automated verification
Solution Approach 2:
The system performs preliminary actions by pre-establishing the virtual environment and pre-programming the mobile agent's exploration behavior. This allows the data generation process to proceed automatically and continuously without requiring manual intervention at any stage, significantly reducing time consumption while maintaining accuracy through predetermined accurate interaction protocols
Data Source
AI summary
A method for generating training data of a mobile agent relating to the technical field of artificial intelligence is provided. The method includes: collecting multiple data triples representing interaction behaviors in an application; each of the data triples including a first user interface state, an action, and a second user interface state; constructing a state transition graph based on the multiple data triples; obtaining an interaction trajectory based on the state transition graph; and generating training data of the mobile agent based on the interaction trajectory.