Virtual multi-purpose ai agent

The method generates an action database and trains a machine learning model to achieve optimal actions for graphical user interfaces, addressing the limitations of foundation models by ensuring meaningful and accurate state transitions.

WO2026238907A1PCT designated stage Publication Date: 2026-11-19MERCEDES BENZ GROUP AG
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/EP2026/064117
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2025-05-14
Filing Date
2026-05-13
Publication Date
2026-11-19

AI Technical Summary

Technical Problem

Existing AI agents for operating graphical user interfaces rely on foundation models that do not guarantee meaningful or correct outputs, particularly in mapping effects to causes, and lack efficient task planning.

Method used

A method involving the generation of an action database through analyzing initial and final states of a graphical user interface, training a machine learning model with this data, and performing a tiered search for optimal actions using semantic and supervised learning to achieve a desired final state.

Benefits of technology

Enables an AI agent to perform arbitrary tasks efficiently by avoiding heuristic planning, ensuring meaningful actions and accurate state transitions, even in complex scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure EP2026064117_19112026_PF_FP_ABST
    Figure EP2026064117_19112026_PF_FP_ABST
Patent Text Reader

Abstract

The invention relates to a method for executing an AI agent for automatically operating a graphical user interface displayed on a user computer, comprising the following steps: generating (S1) an action database by analyzing an initial state of the graphical user interface prior to an automatic action, and an end state after an action; performing (S2) a training process on a machine learning model using data from the action database; detecting (S4) a user request; automatically determining (S5) a desired end state; performing (S6) a graduated search for an optimal action in order to arrive at the desired end state.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] Mercedes-Benz Group AG

[0002] Virtual Multi-Purpose AI Agent

[0003] The invention relates to a method for implementing a computer agent for the automatic operation of a graphical user interface displayed on a user computer, and to a corresponding computer system.

[0004] AI agents are known in the prior art. In this context, WO 2023 / 064515 A1 concerns a system comprising: at least one processor; and a memory in which instructions are stored which, when executed by the at least one processor, cause the system to perform a series of operations, the series of operations comprising: obtaining agent information for a computer-controlled agent of a virtual environment; generating, based on the agent information, a series of candidate interactions for the computer-controlled agent, which include model outputs linked to a multimodal machine learning model;

[0005] Presenting the series of candidate interactions for user selection; receiving a selection of one or more candidate interactions; and generating a node in a dialog tree for each selected candidate interaction, containing natural language output for the selected candidate interaction.

[0006] Furthermore, methods are known in which user input is processed using a multimodal machine learning model to generate output. The user can then use this output to control an application.

[0007] WO 2023 / 064067 A1 concerns a system comprising: at least one processor; and a memory containing instructions which, when executed by the at least one processor, cause the system to perform a series of operations, the series of operations comprising: receiving user input in natural language in a video game application; determining a model output associated with a multimodal machine learning model based on the received user input; and executing at least part of the model output to control the functionality of the video game application.

[0008] AI agents suitable for operating a user's computer, such as interacting with a browser window, are typically based on large machine learning models, so-called foundation models, which primarily execute autoregressive probabilistic AI methods. However, there is no guarantee that the outputs of foundation models are always meaningful or correct. In particular, foundation models are not always able to accurately map effects to causes, even when reasoning is applied.

[0009] The object of the invention is to provide an improved AI agent that avoids the heuristic approach of planning and / or justifying a foundation model.

[0010] The invention is defined by the features of the independent claims. Advantageous further developments and embodiments are the subject of the dependent claims.

[0011] A first aspect of the invention relates to a method for executing an AI agent for the automatic operation of a graphical user interface displayed on a user computer, comprising the steps:

[0012] - Generating an action database by analyzing the initial state of the graphical user interface before each action and the final state of the graphical user interface after each action, where each action includes automatic operation of the graphical user interface;

[0013] - Performing a training process of a machine learning model with data from the actions database regarding a mapping of each initial state and action pair to a final state of the graphical user interface;

[0014] - Capturing a user request for operating the graphical user interface;

[0015] - Automatically determining a desired final state of the graphical user interface from the user request;

[0016] - Performing a tiered search for an optimal action to achieve the desired final state, wherein the tiered search includes a semantic search for the desired final state, in case of failure a semantic search by a user, in case of failure again a semantic search using the trained machine learning model, and in case of failure again a re-training process of the machine learning model and a new inference of the machine learning model.

[0017] In a first step, an action database is generated by repeatedly executing an analysis of the initial state of the graphical user interface before an action and its final state immediately after an action. Preferably, for this first step of the process, and analogous to the prior art, a visual and / or multimodal foundation model is applied to describe the screen content displayed by the graphical user interface, focusing on interaction possibilities. This image description can then be passed on to other systems, possibly reformulated to suit each system, such as a web search, the existing foundation model, or an action database, to ensure that interaction possibilities are meaningful in this state.In this context, the term "meaningful" means: A change in state is to be expected through the possibility of interaction, and the pair "initial state, action" is not yet included in the actions database.

[0018] These inspirations from different sources are aggregated by selecting a partially randomly chosen action—the weighting of the inspirations and their suggested actions can vary, for example, from a random approach to a rewarding approach if an inspiration disproportionately often leads to a sensible action. This action is then carried out by the agent.

[0019] After the action is executed, the new screen content of the graphical user interface is compared with the previous screen content, and any differences are passed to the Foundation Model, which interprets the change in screen content caused by the executed action. This process is executed as many times as needed and ultimately serves the agent to create an action database in which meaningful actions are stored.

[0020] The final state of the graphical user interface, resulting from the execution of an action, serves in particular as the initial state of a subsequent action, if one exists. The states (initial state and final state) can themselves comprise lists of previous states. The respective semantic interpretation of an initial state, final state, or state change can also include a list of previous interpretations. Furthermore, the executed action can be the difference between the initial state before the action and the final state after the action. The semantic interpretation of the state change can also be the difference between the interpretations of the initial state before the action and the final state after the action.

[0021] It is harmless that different actions can lead to semantically identical states of the graphical user interface. Special consideration can also be given to situations where two pairs of "initial state before the action" and "action," where one initial state is the final state of another such pair, can describe mutually inverse actions.

[0022] This concludes the exploration phase of the AI ​​agent for the time being, and a further goal is to achieve a specialization or generalization of such elements of a data set from the actions database by applying a machine learning model.

[0023] It is further planned to train a machine learning model from the collected action data using supervised learning in order to generate abstractions and specializations of actions. This involves attempting to predict the final state after execution of an action as an output, based on the input values ​​of the action itself, i.e., at least the pair "initial state before the action" and "action" as input variables. The training process teaches the model which properties or relationships of such input variables affect the output variable. Since this is a computer intelligence agent, and therefore a virtual agent, it is indeed possible to virtually execute the actions in their respective initial states before each action and to determine the respective final state after execution of each action.This makes it possible to generate an arbitrarily large dataset for the supervised learning of the machine learning model.

[0024] This trains a machine learning model using supervised learning, which can map cause to effect. This leads to the artificial generation of a large number of new possible actions, including those not previously performed, thus enabling the specialization of actions; and, furthermore, to the preservation of structures within the machine learning model that allow for generalizations of states and actions. When a user submits a query to the AI ​​agent, the AI ​​agent first performs a direct semantic search to determine whether any of the actions in the action database achieve the desired end state as a result of the executed action. If so, the AI ​​agent executes all actions that lead to this desired end state. If not, the user searches for a semantic match between the query and the end states in the action database.It may also be possible for the AI ​​agent to pre-filter the action database based on the query. If a semantic match is found between the query and the end states in the action database, the AI ​​agent executes all actions that lead to that end state. If not, it checks whether an action can be found using the machine learning model to achieve an end state that matches the user query. If this check is successful, the AI ​​agent executes all actions determined by the machine learning model that lead to this desired end state. If this also fails, the action database must be expanded again, and / or the machine learning model must be further trained and refined, for example, through retraining.

[0025] In certain situations, it is useful to generate the action database for specific computer applications in order to later find generalizations or specializations when training the machine learning model.

[0026] Instead of using a foundation model, especially an autoregressive probabilistic AI model, to heuristically approximate planning or justification, this approach, in its entirety, applies genuine planning. By using the first aspect of experience, more complex virtual user queries can be solved than with conventional approaches.

[0027] According to an advantageous embodiment, at least one of the following is stored in the actions database to generate the actions database:

[0028] - A description of the initial state of the graphical user interface before the action is executed; the very first state, in particular, has a value to be defined.

[0029] - A semantic interpretation of the initial state of the graphical user interface before the action is executed;

[0030] - The action performed;

[0031] - A description of the final state of the graphical user interface after the action has been executed;

[0032] - A semantic interpretation of the final state of the graphical user interface after the action has been executed;

[0033] - A semantic interpretation of the change of state through the execution of the action.

[0034] According to another advantageous embodiment, additional data is generated for the training priority of the machine learning model, which is not or not yet contained in the action database and which maps a respective pair of initial state and action to a final state of the graphical user interface.

[0035] According to another advantageous embodiment, a random action is performed once or several times on the graphical user interface to generate the action database.

[0036] According to another advantageous embodiment, the user request includes a verbal description.

[0037] According to another advantageous embodiment, the verbal description is a description of a desired final state of the graphical user interface.

[0038] According to another advantageous embodiment, the method further includes the step of extending the action database with newly generated actions of the machine learning model.

[0039] Another aspect of the invention relates to a computer system comprising an input unit, an output unit configured to display a graphical user interface, and a computing system, wherein the input unit is configured to capture a user request for operating the graphical user interface and forward it to the computing system, wherein the computing system is configured to calculate screen content for the graphical user interface and to control the output unit to display the screen content, and wherein the computing system is configured to adapt and execute a computer agent for automatically operating the graphical user interface by generating and / or adapting an action database through the computing system by performing an analysis of an initial state of the graphical user interface before an action and a final state after an action.wherein the action comprises automatic operation of the graphical user interface, and wherein the computing system performs a training process of a machine learning model with data from the action database regarding a mapping of a respective pair of initial state and action to a final state of the graphical user interface, and wherein the computing system determines a desired final state of the graphical user interface from a user request received at the input unit for the operation of the graphical user interface; wherein the computing system is further configured to perform a graded search for an optimal action to achieve the desired final state, wherein the graded search is a semantic search for the desired final state, and in the event of failure, a semantic search enabled by the computing system by a user,In the event of its failure, a semantic search using the trained machine learning model is performed, and if this also fails, the machine learning model is retrained and a new inference is made using the machine learning model.

[0040] Advantages and preferred further developments of the proposed computer system result from an analogous and substantive transfer of the above statements made in connection with the proposed procedure.

[0041] Further advantages, features and details will become apparent from the following description, in which - possibly with reference to the drawing - at least one embodiment is described in detail.

[0042] They show:

[0043] Fig. 1 : A method for carrying out a Cl agent according to an embodiment of the invention.

[0044] Fig. 2: A computer system configured to carry out the method according to Fig. 1 in accordance with an embodiment of the invention.

[0045] Figure 1 shows a method for executing an AI agent to automatically operate a graphical user interface displayed on a user's computer. The purpose of this method is to enable the creation of a virtual AI agent capable of performing arbitrary tasks on a user's computer. Performing any given task requires efficient or optimized task planning, which, unlike previously known approaches, is not achieved using foundation models in this method, but rather through a search algorithm in state space. The following steps are performed during the process and are explained in greater detail using the system shown in Figure 2:

[0046] - Generating S1 of an action database by repeatedly performing an analysis of an initial state of the graphical user interface before an action and an end state of the graphical user interface after an action, wherein the action includes automatic operation of the graphical user interface;

[0047] - Performing S2 of a training process of a machine learning model with data from the actions database regarding a mapping of a respective pair of initial state and action to a final state of the graphical user interface;

[0048] - Extend S3 of the actions database with newly generated actions from the machine learning model.

[0049] - Capture S4 of a user request for operating the graphical user interface;

[0050] - Automatically determine S5 a desired final state of the graphical user interface from the user request;

[0051] - Performing S6 a graded search for an optimal action to achieve the desired final state, where the graded search includes a semantic search for the desired final state, if this fails, a semantic search by a user, if this fails again, a semantic search using the trained machine learning model, and if this fails again, a further training of the machine learning model and a new inference of the machine learning model. This enables a AI agent that can perform arbitrary tasks on a user computer. Unlike known approaches, the planning of the user-defined task is not carried out using a foundation model, but rather a search algorithm in the state space is used. The activation of the proposed AI agent takes place in three phases:

[0052] - The first phase involves autonomously storing initial states and actions as experiences in final states.

[0053] - In the second phase, an algorithm is applied that can derive additional meaningful actions for each initial state from the experiences.

[0054] - The actual execution of the user request takes place in the third phase, which is made possible by the results of the first and second phases.

[0055] Figure 2 shows a computer system with an input unit 1, an output unit 2 designed to display a graphical user interface, and a computing system 3. A user request for operating the graphical user interface is captured at the input unit 1, such as a keyboard, but preferably a speech input unit, and forwarded to the computing system 3. The computing system 3 interacts with the screen content for the graphical user interface and controls the output unit 2 to display the screen content. The interaction of the computing system 3 with the graphical user interface is carried out by a computer intelligence agent, which is automatically adapted and executed as follows: In the first phase, the computer intelligence agent captures an initial state of the graphical user interface and interprets it semantically. Based on this interpretation, various suggestions are generated to determine possible appropriate actions.From these, a subset of meaningful actions is selected, and each action is executed to capture the resulting end state of the graphical user interface and interpret it semantically. Corresponding mappings from an initial state to an end state, along with the associated action, are stored in an action database. This completes the first phase, allowing the second phase to begin. Using the entries in the action database, a supervised learning dataset is created. The input set for this dataset consists of an initial state and the action executed on it, while the output is a final state that follows this action based on this initial state.During the training process, the machine learning model creates generalizations of these representations, enabling the model to infer further meaningful data points for actions. In the third phase of the AI ​​agent, a user request, and thus a verbal description of the desired interaction goal with the graphical user interface, is captured. The selection of the action is attempted in stages until one of the stages is successful. In the first stage, a semantic search is performed for a suitable end state. In the second stage, a user performs a semantic search instead. If either of these is successful, corresponding, potentially successful actions are prioritized and displayed based on the action database. However, if neither of these options is promising, a further semantic search is performed by the machine learning model, utilizing the identified relationships.If this also fails, the first phase must be repeated in a new iteration to expand the action database or to fine-tune the machine learning model.

[0056] Although the invention has been further illustrated and explained in detail by means of preferred embodiments, the invention is not limited by the disclosed examples, and other variations can be derived from them by a person skilled in the art without departing from the scope of protection of the invention. It is therefore clear that a multitude of possible variations exist. It is also clear that the embodiments mentioned as examples are truly only examples and are not to be understood in any way as limiting, for example, the scope of protection, the possible applications, or the configuration of the invention.Rather, the preceding description and the description of the figures enable the person skilled in the art to implement the exemplary embodiments in concrete terms, whereby the person skilled in the art, with knowledge of the disclosed inventive concept, can make various changes, for example with regard to the function or the arrangement of individual elements mentioned in an exemplary embodiment, without leaving the scope of protection defined by the claims and their legal equivalents, such as further explanations in the description.

Claims

Mercedes-Benz Group AG Patent claims 1. Method for executing a computer agent for the automatic operation of a graphical user interface displayed on a user's computer, comprising the steps: - Generating (S1) an action database by analyzing an initial state of the graphical user interface before a respective action and an end state of the graphical user interface after the respective action, wherein the respective action includes automatic operation of the graphical user interface; - Performing (S2) a training process of a machine learning model with data from the actions database regarding a mapping of a respective pair of initial state and action to a final state of the graphical user interface; - Capturing (S4) a user request for operating the graphical user interface; - automatic determination (S5) of a desired final state of the graphical user interface from the user request; - Performing (S6) a graded search for an optimal action to achieve the desired final state, wherein the graded search includes a semantic search for the desired final state, in case of failure a semantic search by a user, in case of failure again a semantic search using the trained machine learning model, and in case of failure again a re-training of the machine learning model and a new inference of the machine learning model.

2. Method according to claim 1, where, to generate the actions database, at least one of the following is stored in the actions database: - A description of the initial state of the graphical user interface before the action is executed; - A semantic interpretation of the initial state of the graphical user interface before the action is executed; - The action performed; - A description of the final state of the graphical user interface after the action has been executed; - A semantic interpretation of the final state of the graphical user interface after the action has been executed; - A semantic interpretation of the change of state through the execution of the action.

3. Method according to any one of the preceding claims, where, for the training priority of the machine learning model, additional data is generated that is not or not yet contained in the action database and that maps a respective pair of initial state and action to a final state of the graphical user interface.

4. Method according to any one of the preceding claims, where a random action is performed once or several times on the graphical user interface to generate the action database.

5. Method according to any one of the preceding claims, where the user request includes a verbal description.

6. Method according to claim 5, where the verbal description is a description of a desired final state of the graphical user interface.

7. Method according to any of the preceding claims, further showing the step: Extending (S3) the actions database with newly generated actions of the machine learning model.

8. Computer system comprising an input unit (1), an output unit (2) for displaying a graphical user interface, and a computing system (3), wherein the input unit (1) is configured to capture a user request for operating the graphical user interface and forward it to the computing system (3), wherein the computing system (3) is configured to calculate screen content for the graphical user interface and to control the output unit (2) for displaying the screen content, wherein the computing system (3) is configured to adapt and execute a computer agent for automatically operating the graphical user interface by generating and / or adapting an action database through the computing system (3) by performing an analysis of an initial state of the graphical user interface before an action and an end state after an action, wherein the action comprises automatic operation of the graphical user interface.and by the computing system (3) performing a training process of a machine learning model with data from the actions database with respect to a mapping of a respective pair of initial state and action to a final state of the graphical user interface, and by the computing system (3) determining a desired final state of the graphical user interface from a user request received at the input unit (1) for the operation of the graphical user interface; wherein the computing system (3) is further configured to perform a graded search for an optimal action to achieve the desired final state, wherein the graded search is a semantic search for the desired final state, in the event of failure a semantic search initiated by a user, in the event of failure again a semantic search using the trained machine learning model,and, if this fails, involves a further training process of the machine learning model and a new inference of the machine learning model.