Operation trajectory construction method, interaction method, device, and storage medium

Abstract

The present application relates to the technical field of computers, and discloses an operation trajectory construction method, an interaction method, a device, and a storage medium. The method comprises: acquiring a data set corresponding to an application, the data set comprising a control list corresponding to each page in the application and a description of the application, the control list being used for recording all executable controls on the page, and the description at least reciting operations corresponding to execution of the controls; determining a user interface transition graph on the basis of the data set; and inputting the user interface transition graph and the description into a large language model to obtain an operation trajectory used for completing a task. An agent obtained according to the method of embodiments of the present application can effectively complete complex operations, significantly improving the success rate of execution and enhancing user experience.

Description

Methods for constructing operation trajectories, interaction methods, devices, and storage media.

[0001] This application claims priority to Chinese Patent Application No. 202411717064.3, filed on November 26, 2024, entitled “Method for Constructing Operation Trajectory, Interaction Method, Device and Storage Medium”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of electronic device technology, and in particular to a method for constructing an operation trajectory, an interaction method, a device, and a storage medium. Background Technology

[0003] With the explosive growth of applications installed on electronic devices, users frequently use various apps to complete different tasks in their daily lives and work. However, with the increase in the number of apps and the complexity of their operations, users need to spend a lot of time and energy switching between different apps. This not only reduces the efficiency of users in completing certain functions but also increases the complexity and fatigue of operation.

[0004] To address this issue, mobile phone control agents have emerged. A control agent is an intelligent software tool or service that can simulate user actions and automatically perform tasks in the background. By responding to user commands in real time, the control agent can automatically complete a series of operations on the user's mobile phone, thereby greatly improving efficiency. This automation tool not only reduces the frequency of switching between different applications but also significantly reduces manual operation steps, providing users with a more convenient experience.

[0005] However, as application functions become increasingly complex, many tasks often require more than 20 steps to execute. This operational complexity presents significant challenges to the design and execution of operational agents. In complex workflows, operational agents may encounter various obstacles during execution, such as page loading failures, operation timeouts, or interface changes, leading to execution failures and impacting user experience, thus hindering the improvement of work efficiency. Summary of the Invention

[0006] To address the issue of operational intelligence agents easily failing to execute under the aforementioned complex operating environments, this application provides a method for constructing operational trajectories, an interaction method, a device, and a storage medium.

[0007] Firstly, this application provides an interaction method applied to an electronic device, comprising: receiving a first operation from a first user on the electronic device; responding to the first operation, recommending a first intelligent agent to the first user, the first intelligent agent being used to perform a first task; receiving a second operation from the first user on the electronic device; and responding to the second operation, presenting the operation trajectory of the first intelligent agent performing the first task. This facilitates the user's understanding of each step of task execution and allows for replacement or modification as needed.

[0008] As one embodiment of the first aspect, the operation trajectory of the first intelligent agent performing the first task is presented, including: the operation trajectory is presented in the form of natural language text according to the execution order, showing the multiple steps included in the operation trajectory. This allows users to modify it according to their needs.

[0009] As one embodiment of the first aspect, presenting the operation trajectory of the first intelligent agent performing a first task includes: displaying the operation trajectory through a first interface; the first interface further includes at least one control for modifying the operation trajectory. Users can operate on the space as needed, making it more flexible and convenient.

[0010] As an embodiment of the first aspect, modifying the operation trajectory includes at least one of the following: manually editing the operation trajectory; automatically optimizing the operation trajectory; and re-recording the operation trajectory.

[0011] As one embodiment of the first aspect, the second operation is a viewing operation, a selection operation, a confirmation operation, or a modification operation on the first intelligent agent.

[0012] As one embodiment of the first aspect, recommending a first intelligent agent to a first user includes: recommending multiple intelligent agents to the first user in an intelligent agent store, the multiple intelligent agents including the first intelligent agent. This facilitates the user in quickly finding the intelligent agent they need.

[0013] As one embodiment of the first aspect, in response to a first operation, recommending a first intelligent agent to a first user includes: identifying the first user's intent based on the first operation; and recommending the first intelligent agent to the first user according to the intent.

[0014] As one embodiment of the first aspect, the first operation is used to instruct an electronic device to perform a first task.

[0015] As one embodiment of the first aspect, the first operation is to enter a search term in the search box.

[0016] As one embodiment of the first aspect, the first operation is dialogue input in a smart assistant.

[0017] As one embodiment of the first aspect, the first operation is used to receive a message from a second user and recommend a first agent to the user, including: presenting the user with the first agent shared by the second user to the first user via a message.

[0018] As one embodiment of the first aspect, the first task is a multi-user collaborative task, and the message is used to request the first user and the second user to collaborate on the first task based on a first intelligent agent. In a second aspect, this application provides a method for constructing an operation trajectory, including:

[0019] Obtain the dataset corresponding to the application. The dataset includes a list of controls corresponding to multiple pages in the application and the application's instruction manual. The control list records the executable controls on the page, and the instruction manual describes at least the operations corresponding to the executed controls. Determine the user interface transition map based on the dataset. Input the user interface transition map and instruction manual into a large model, such as a large language model, to obtain the operation trajectory used to complete the target task.

[0020] According to the construction method of this application embodiment, since the user interface transfer map can be applied to multiple applications, cross-application operations can be achieved when arranging operation trajectories, no longer limited to a single application. Furthermore, the agent obtained through a large language model can effectively improve the success rate of completing the target task, thus enhancing the user experience.

[0021] As an embodiment of the second aspect of this application, determining the user interface transition map based on a dataset includes: launching an application and obtaining a list of controls corresponding to the current page; operating the controls one by one according to the operation of the controls described in the specification, and recording the control information and operation type corresponding to the current operation; saving the content corresponding to the new page that appears after each operation, obtaining a list of controls corresponding to the new page, and operating the controls in the list of controls corresponding to the new page one by one, until all control lists corresponding to the application are traversed;

[0022] By using the content and control list corresponding to the new page as nodes in the user interface transition graph, and the control information and operation type as edges, the user interface transition graph is obtained. This convenient process ensures that all pages corresponding to the application can be traversed, and that controls are executed effectively.

[0023] As an embodiment of the second aspect of this application, the method further includes: constructing a simulation verification environment to verify all operation trajectories; and retaining the operation trajectory that meets the expected execution requirements when the verified operation trajectory meets the expected values. Verification of the operation trajectory not only improves the accuracy and reliability of operation trajectory generation but also simplifies the operation trajectory creation process, enabling users to quickly and accurately generate and deploy operation trajectories, thereby improving system stability and user experience.

[0024] As an embodiment of the second aspect of this application, all operation trajectories are verified, including: in a simulation environment, based on the description of the operation trajectory, simulating the actual operation scenario, and executing the steps corresponding to the operation trajectory step by step; if each step of the current operation trajectory can be executed according to the preset requirements, then it is determined that the operation trajectory meets the expected execution requirements.

[0025] As an embodiment of the second aspect of this application, the operation trajectory that meets the expected execution requirements is reflowed and input into the large model. By repeatedly reflowing, the quality of the operation trajectory can be improved.

[0026] As an embodiment of the second aspect of this application, optimizing the operation trajectory further includes: acquiring historical operation behavior data for completing the target task and result data for executing the operation trajectory, wherein the result data includes the state generated during the execution of the operation trajectory and the action data during the execution of the operation trajectory; inputting the user's historical operation behavior data and result data into the trajectory optimization model to obtain the optimized operation trajectory. This optimization method results in higher operation trajectory accuracy and can meet the user's personalized needs.

[0027] As an embodiment of the second aspect of this application, the optimized operation trajectory is evaluated. When the operation trajectory completes the target task or completes the preset intermediate steps, the optimized result is labeled with a reward tag; when the operation trajectory encounters an error or fails to achieve the expected target task, the optimized result is labeled with a penalty tag. This further improves the quality of the operation trajectory.

[0028] As an embodiment of the second aspect of this application, the state includes: the content displayed on the current page corresponding to the application, the text entered by the user, or the activation state of the control.

[0029] As an embodiment of the second aspect of this application, motion data includes: clicking, swiping, or entering text.

[0030] As an embodiment of the second aspect of this application, the historical operation behavior data includes: the user's historical operation behavior and the historical operation behavior of other users, wherein the similarity between the historical operation behavior of other users and the user's historical operation behavior meets a preset value.

[0031] As an embodiment of the second aspect of this application, the application specification includes at least a summary of the user interface, a description of the functions of the controls, a page navigation diagram, and a task operation sequence.

[0032] Thirdly, this application also discloses an electronic device, including: a memory for storing instructions executed by one or more processors of the electronic device, and a processor for performing the method explained in the first aspect embodiment or performing the method explained in the second aspect embodiment.

[0033] Fourthly, this application also discloses a computer-readable storage medium, characterized in that the computer-readable storage medium stores instructions that, when executed on an electronic device, cause the electronic device to perform the method explained in the first aspect embodiment or the method explained in the second aspect embodiment.

[0034] Fifthly, this application also discloses a computer program product containing instructions, characterized in that, when the computer program product is run on an electronic device, it causes the electronic device to perform the method explained in the first aspect embodiment or the method explained in the second aspect embodiment. Attached Figure Description

[0035] Figure 1A shows schematic diagrams of mobile phone interfaces in some embodiments;

[0036] Figure 1B shows schematic diagrams of mobile phone interfaces in some other embodiments;

[0037] Figure 2 shows a flowchart illustrating the method for constructing the operation trajectory provided in an embodiment of this application;

[0038] Figure 3 shows a schematic diagram of the structure of the electronic device of this application;

[0039] Figure 4A illustrates the overall architecture for constructing and using the operation trajectory in an embodiment of this application;

[0040] Figure 4B shows the overall architecture for constructing and using the operation trajectory in another embodiment;

[0041] Figure 4C shows a flowchart of the construction and use of the operation trajectory in an embodiment of this application;

[0042] Figure 5 shows a flowchart of the method for constructing the operation trajectory according to an embodiment of this application;

[0043] Figure 6 shows a flowchart of the automated interactive trajectory traversal process of the UTG in an embodiment of this application;

[0044] Figure 7 shows a schematic diagram of the architecture of the application specification of this application embodiment;

[0045] Figure 8 shows a schematic diagram of the user interface transition map corresponding to the application of this application embodiment;

[0046] Figure 9 shows a framework diagram of the intelligent agent construction process according to an embodiment of this application;

[0047] Figure 10 shows a flowchart of the agent optimization method according to an embodiment of this application;

[0048] Figure 11 shows a flowchart of the estimation and recommendation method for the co-occurrence mechanism according to an embodiment of this application;

[0049] Figure 12 shows a flowchart of the interaction method according to an embodiment of this application;

[0050] Figure 13 illustrates a schematic diagram of the user interface interacting with a mobile phone according to an embodiment of this application;

[0051] Figure 14 illustrates a user interface diagram of interaction between a user and a mobile phone according to another embodiment of this application;

[0052] Figure 15 illustrates a schematic diagram of the interface of a mobile phone executing an intelligent agent according to an embodiment of this application.

[0053] Figure 16 illustrates a block diagram of a SoC according to an embodiment of this application. Detailed Implementation

[0054] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0055] The illustrative embodiments of this application include, but are not limited to, a method for constructing an operation trajectory, an interaction method, a device, a storage medium, and a program product.

[0056] To facilitate understanding of the technical solution of this application, the technical problem to be solved by this application will be explained first.

[0057] As described above, with the increasing number of applications on electronic devices and the growing complexity of their functions, it often requires more than 20 steps to control an intelligent agent to automatically perform tasks. This makes it easy for the intelligent agent to fail during the execution process, resulting in a reduced user experience.

[0058] To address the aforementioned issues, some embodiments employ instruction recording tools to record actual user operations. This involves recording each step of the user's operation in real-time while the user is using the phone, and then converting these operations into a set of instructions that the system can automatically execute—that is, an agent. The phone can then execute these agents to complete the user's pre-recorded steps and automatically perform specific tasks, thereby replacing manual operation.

[0059] Referring to Figure 1A, which illustrates a mobile phone interface in some embodiments, as shown in Figure 1A, in the mobile phone interface 10, the user can input the task command "order a coffee" in the input box 11. After selecting the operation to be performed, such as selecting the "teaching command" command, the user can manually complete the task of ordering a coffee. The command recording tool inside the phone will automatically record the user's operation and save the recording. When the user inputs the command "order a coffee" again via voice or other means, the phone will automatically complete the task command based on the previously recorded video, fulfilling the user's target task. This method allows users to record various operations, including clicking, swiping, inputting text, opening or closing applications, etc., providing users with a simple and intuitive way. The phone can achieve highly personalized automated tasks by mimicking human operating habits. Moreover, this command recording method can capture the user's specific operating habits. No matter how complex or special these operations are, the user only needs to operate according to their usual operating habits, and the command recording tool will automatically record these steps. When the user inputs the set command, the phone will automatically reproduce these operations. In this way, users can record many tasks that can be completed through touch screen operations, without being limited by preset functional modules, thereby meeting personalized needs.

[0060] While the aforementioned command recording tools offer advantages in terms of personalization and flexibility, they also have the following drawbacks:

[0061] First, in actual operation, the phone controls the execution of corresponding operations based on the cross-sectional coordinates of the recorded screen. However, different phones have different screen sizes, resolutions, and system versions, which may cause deviations in the execution of the user-recorded operation steps. For example, a click operation recorded on one device may result in a shift in the click position on another device due to different resolutions. For operations requiring high precision, this may lead to the task failing to execute correctly or even triggering erroneous operations, thereby reducing the accuracy and reliability of the intelligent agent's execution.

[0062] Second, when tasks require crossing multiple applications and involve complex logical judgments or conditional branches, instruction recording tools cannot solve these problems. For example, as shown in Figure 1A, the teaching command method is limited to learning operations within a single application (APP). Users cannot create cross-application operations during the recording process, nor can they handle scenarios that require multiple interactions or complex judgments, greatly reducing its practicality.

[0063] Third, for tasks with lengthy or repetitive processes, even slight user oversights during recording can reduce accuracy, resulting in an AI agent that fails to accurately reflect the complexities of the operation. Therefore, users must carefully execute each step to ensure recording accuracy and prevent the generated AI agent from malfunctioning. This necessitates high concentration and patience during recording. When modifications or adjustments are needed to a pre-recorded AI agent, the entire process often needs to be re-recorded. This repetitive work further increases the user's workload and reduces operational efficiency.

[0064] In other embodiments, users can create intelligent agents using function orchestration tools on electronic devices, such as the Huawei Skill Center. These orchestration tools allow users to create a complete task flow to obtain an intelligent agent by dragging and combining different function blocks or operation steps. These elements can include built-in system operations, such as opening an application, setting an alarm, or sending a message; or services provided by third-party applications, such as opening an application at a specific time or performing a series of operations when a certain event is triggered. By combining these function blocks in a preset logical manner, users can customize and create their own intelligent agents according to their needs and preferences, achieving automated operation of the mobile phone.

[0065] Referring to Figure 1B, which illustrates a mobile phone interface in some other embodiments, the mobile phone interface 20 includes fixed statements such as "When...execute..." and fixed functional modules. For example, under the "When" statement, users can input tasks such as "Good Morning Broadcast" or specific times such as "7:30 AM on a legal working day (skipping legal holidays)" in the scene card. Under the "Execute" statement, users can select fixed functional blocks such as "Connect Mobile Phone", "Designate Speaker", and "Play the following content". These functional modules can also include sub-modules. For example, under "Designate Speaker", there is a specific functional block such as "...play...wake-up ringtone". Users can select the time, such as 3 seconds, and the song name "Canon" in the editable part of the corresponding statement to complete the editing of the operation trajectory. This arrangement tool uses fixed modules, and users only need to drag and arrange functional blocks on the visual interface to complete the task creation without writing code. This allows ordinary users to easily create automated tasks and obtain edited intelligent agents.

[0066] However, while function orchestration tools offer users convenience in creating automated tasks, handling complex tasks, such as those spanning multiple applications, involves more intricate programming statements. For example, the logical relationships between function blocks, the use of conditional statements, and the setup of loops and branches can still be challenging for users without a programming or technical background to understand and use these tools. Furthermore, for complex agents, adding, deleting, or modifying a step may require users to re-examine the entire orchestration trajectory; otherwise, a slight oversight can lead to the agent failing to execute correctly or failing after completing a portion of the steps, thus failing to complete the target task. Moreover, in this orchestration method, the number and types of function blocks involved are predefined by the system or third-party applications, limiting the user's orchestration to these scenarios and failing to meet the user's need for more personalized agents.

[0067] To address the issues mentioned above regarding the limitations of automated operations using instruction recording methods, such as low recording accuracy, inability to execute complex operations across applications, and the potential failure of generated agents to execute correctly due to minor errors in user recording, as well as the high requirements for user orchestration capabilities, limited functional modules, and the tendency for complex agents to fail, this application provides a method for constructing operation trajectories. This method constructs operation trajectories by obtaining the application's user interface transition graph and a large-scale model. The operation trajectory records and describes the sequence of manipulations or operational steps required to complete a target task. These sequences or steps can be executed by an agent, meaning the agent can execute these operation trajectories to complete the user's target task. The user interface transition graph is a graphical model used to represent the states of the user interface and the transition relationships between interfaces. Each state represents a specific configuration or view on the user interface, and a transition represents the user's action of moving from one state to another.

[0068] Since the user interface transition graph can cover all applications and their corresponding functions, it can be used to obtain the operation trajectories for all tasks, which are then executed by agents. In the following text, an agent refers to an entity that completes a target task based on an operation trajectory. These agents can be pre-generated and stored locally or downloaded online. When a user inputs a command, these agents corresponding to the command are directly retrieved, and the target task corresponding to the command is completed based on the agents. This method of constructing operation trajectories does not impose requirements on user actions, is not limited by functional modules, and can also achieve cross-application operations, effectively completing complex operations. Furthermore, the success rate of execution of operation trajectories (agents) generated through a large model is greatly improved, enhancing the user experience.

[0069] In some embodiments of this application, the large model can be a large language model, a multimodal model, a speech model, etc. The following embodiments will use a large language model for illustration.

[0070] The method for constructing the operation trajectory according to the embodiments of this application will be described below with reference to the accompanying drawings.

[0071] Referring to Figure 2, which illustrates a flowchart of the method for constructing an operation trajectory according to an embodiment of this application, this method can be used in electronic devices, such as mobile phones, computers, servers, and other devices with computing capabilities. As shown in Figure 2, the method includes, but is not limited to, the following steps:

[0072] S210, Obtain the dataset corresponding to the application. The dataset includes a list of controls corresponding to multiple pages in the application and the application's instruction manual.

[0073] This step is the data construction process, mainly involving various data related to the target application, such as lists of controls corresponding to multiple pages in the application and the application's manual, to form a dataset.

[0074] Each application contains many different pages, and each page corresponds to a list of controls. The control list records all executable controls on the corresponding page—the basic elements for user interaction with the application. These elements help users perform specific operations or input information. The phone can identify and understand all executable elements on each page based on the control list. Control types include, but are not limited to: buttons, text fields, switches, sliders, dropdown lists, progress bars, labels, images, links, list views, and navigation bars.

[0075] The application's documentation at least describes the operations corresponding to the controls, such as clicking a button, entering text, or switching options. In some embodiments, the documentation also includes a summary of the UI interface, detailed descriptions of control functions, a diagram of page transitions, and operation sequences for common tasks. This documentation provides additional contextual support for the interaction trajectories generated by the UI Transition Graph (UTG), enabling the system to have a more comprehensive and in-depth understanding of the app's functionality.

[0076] S220, determine the user interface transition map based on the dataset.

[0077] A user interface transition graph is a graphical structure used to represent the transition relationships between different user interfaces within an application. It uses nodes to represent different states of the application (e.g., pages, windows) and edges to represent user action paths (e.g., clicks, swipes) between interfaces. It can be used to analyze and generate interaction trajectories within an application, helping automation tools understand and simulate user operation flows.

[0078] In some embodiments, the application's corresponding interface and the list of controls corresponding to the interface can be used as nodes in the user interface transition graph. The operations performed on each control, such as action type and control information, are used as edges in the user interface transition graph to form the user interface transition graph. The action type may include, but is not limited to, clicking a button, entering text, or toggling an option. The control information includes, but is not limited to, the control's name, location, and operation description.

[0079] S230 inputs the user interface transfer map and instruction manual into the large language model to obtain the operation trajectory used to complete the target task.

[0080] Large Language Models (LLMs) are pre-trained models capable of generating operational trajectories. Their training sets are derived from application datasets and instruction manuals within the broader context of big data. LLMs utilize input user interface transfer graphs and instruction manuals to automatically generate intelligent agents (operational trajectories) with task logic and operational steps. These agents can then be automatically executed by electronic devices upon receiving user commands, thus automating the user's target tasks.

[0081] Therefore, according to the operation trajectory construction method of this application embodiment, since the user interface transfer map can be applied to all applications, cross-application operations can be realized when arranging operation trajectories, no longer limited to a single application. Furthermore, the agent obtained through a large language model can effectively improve the success rate of completing the target task, enhance the user experience, and facilitate the application and promotion of this function.

[0082] It should be noted that the operational agent or intelligent agent in the embodiments of this application can be understood as an entity corresponding to an operation trajectory for completing a certain task. That is, the operation trajectory in this application can be understood as an intelligent agent that assists the user in completing the target task.

[0083] It is understood that the method for constructing the operation trajectory provided in this application can be executed by an electronic device, including but not limited to mobile phones, smart TVs, wearable devices, tablets, desktop computers, computers, wearable devices (e.g., smartwatches, smart glasses, helmets), virtual reality (VR) devices, augmented reality (AR) devices, wireless devices in industrial control, wireless devices in self-driving, wireless devices in smart grids, wireless devices in transportation safety, wireless devices in smart cities, wireless devices in smart homes, servers, and so on.

[0084] Understandably, for ease of description, the following text will refer to electronic devices as mobile phones.

[0085] Furthermore, this application does not limit the application scenarios of intelligent agents and their related agent stores. They can be widely applied to the following scenarios:

[0086] Daily life automation scenario: In this scenario, users can set up smart agents through their mobile phones to automatically complete daily tasks, such as automatically turning off airplane mode, playing music, checking the weather, and sending schedule reminders every morning.

[0087] Workflow optimization scenarios: In work scenarios, users can leverage intelligent agents to automate tasks such as handling emails, scheduling meetings, organizing work logs, or managing projects. For example, an agent can be set to automatically archive or mark specific emails as pending after receiving them, or automatically open a meeting application and remind the user at a certain time. Through these automated processes, users can focus on more creative and complex tasks, thereby improving work efficiency.

[0088] Smart Home Control Scenarios: The intelligent agent constructed using the method described in this application can be applied to scenarios involving the coordinated control of smart home devices. For example, a user can set up an intelligent agent to automatically turn on the air conditioner and water heater before returning home from get off work, and adjust the lighting based on location and time. Through deep integration with smart home devices, users can enjoy a seamless smart living experience.

[0089] Cross-application task integration scenarios: Users can create and download smart agents that integrate functions from multiple applications through the smart agent store on their mobile phones. For example, a smart agent can be created to simultaneously handle price comparison, order placement, and logistics tracking for online shopping, thereby reducing manual steps.

[0090] The hardware structure of the electronic device will be introduced first.

[0091] The following describes the method for constructing the interaction trajectory in the embodiments of this application, taking into account the specific structure of the electronic device.

[0092] Referring to Figure 3, Figure 3 shows a schematic diagram of the structure of the electronic device 100 of this application.

[0093] Electronic device 100 may include processor 110, external memory interface 120, internal memory 121, universal serial bus (USB) connector 130, charging management module 140, power management module 141, battery 142, antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193, display screen 194, and subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0094] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0095] Processor 110 may include one or more processing units, such as application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU). These different processing units may be independent devices or integrated into one or more processors.

[0096] The processor 110 can generate operation control signals based on the instruction opcode and timing signals to control the instruction fetching and execution.

[0097] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0098] In some embodiments, the processor 110 acquires a dataset corresponding to the application. The dataset includes a list of controls for each page in the application and a specification for the application. The control list records the executable controls on the page, and the specification at least describes the operations corresponding to the executed controls. The processor 110 can determine a user interface transition graph based on the dataset and input the user interface transition graph and specification into a large language model to obtain an operation trajectory, i.e., an intelligent agent, for completing the task.

[0099] In some embodiments, the processor 110 can verify the agent. That is, in a simulated environment, the processor 110 simulates actual user operation scenarios, verifying each step of the agent's operation as expected, and ensuring its stable and correct execution under various possible usage scenarios. Verification not only identifies potential problems with the agent in practical applications, facilitating timely correction, but also provides data support for agent optimization. Automated verification improves the accuracy and reliability of agent generation, eliminating the need for manual user adjustments, enabling users to quickly and accurately generate and deploy agents, thus improving system stability and user experience.

[0100] The wireless communication function of electronic device 100 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0101] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with tuning switches.

[0102] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 150 may be housed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0103] The wireless communication module 160 can provide solutions for wireless communication applications on the electronic device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0104] In some embodiments, both the mobile communication module 150 and the wireless communication module 160 can enable the electronic device 100 to communicate with the cloud. The processor 110 obtains a dataset from the cloud, processes the dataset to obtain a user interface transition map, and finally obtains an intelligent agent.

[0105] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0106] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniature LED, a microLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 100 may include one or N displays 194, where N is a positive integer greater than 1.

[0107] In some embodiments, the display screen 194 may display the labels of these agents, or the agents corresponding to various steps, and display the corresponding operations and interfaces when the agents are executed.

[0108] To facilitate understanding of the method for constructing and applying the operation trajectory in the embodiments of this application, the overall architecture diagram for constructing and using the operation trajectory in the embodiments of this application is first described below. Here, the operation trajectory can be understood as an intelligent agent.

[0109] Referring to Figure 4A, Figure 4A illustrates the overall architecture for constructing and using the operation trajectory according to an embodiment of this application. As shown in Figure 4A, this architecture includes a UTG generation and analysis module 411, an agent generation and editing module 412, an automated agent verification and production module 413, a reinforcement learning trajectory optimization module 414, an intelligent recommendation engine 415, an agent store 421, and a user interaction and feedback module 422. Among these modules, the UTG generation and analysis module 411, the agent generation and editing module 412, the automated agent verification and production module 413, the reinforcement learning trajectory optimization module 414, and the intelligent recommendation engine 415 can be located on a cloud-based device 410 for constructing and recommending agents. The agent store 421 and the user interaction and feedback module 422 can be located on a user terminal device 420 for receiving user instructions, displaying the corresponding agent, and executing the agent.

[0110] In some embodiments, as shown in FIG4B, another embodiment of the overall architecture for constructing and using operation trajectories is presented, in which the automated agent verification production platform 430, corresponding to the automated agent verification production module 413, is a separate device, and the other modules of this structure are the same as the module layout in FIG4A.

[0111] In other embodiments, these modules can be installed in the user terminal device to complete the construction of the intelligent agent and its interaction with the user. This application does not limit the device to which these modules belong.

[0112] The functions of each module are described below with reference to Figure 4A and Figure 4C. Figure 4C shows a flowchart of the construction and use of the operation trajectory in an embodiment of this application.

[0113] As shown in Figures 4A and 4C, the UTG generation and analysis module 411, which can also be understood as the APP automatic traversal tool in Figure 4B, generates a complete UTG by traversing the APP's user interface and records the possible operation paths. By analyzing the UTG, basic interaction trajectories can be provided to users, reducing the complexity of creating intelligent agents and lowering the operational threshold. Furthermore, this traversal tool can traverse all applications and obtain UTGs, thus enabling cross-application operations and accurately generating interaction trajectories even for complex operations. The processing implemented by this module corresponds to step ① in Figure 4C, where the APP automatic traversal tool traverses the APP to obtain the user interface transition map.

[0114] The agent generation and editing module 412 is used to create and edit custom agents based on the interactive trajectories provided by UTG. It can also visualize the steps of agent generation and display them on the interface. Users can customize the agent through screen recording or adjust and modify each step of the agent to better suit their needs. The processing implemented by this module corresponds to step ② in Figure 4C, which involves constructing the trajectory.

[0115] The automated agent verification production module 413 (or the automated agent verification production platform 430 in Figure 4B) is used to automate the functional and stability verification of agents. After the agent is generated, the platform performs comprehensive verification of the agent through simulated environments and various test scenarios to ensure that the agent can operate stably under various conditions. This platform reduces the cost and errors of manual testing through an automated verification process, thereby improving the reliability of the agent. The processing implemented by this module corresponds to step ③ in Figure 4C, automated verification.

[0116] The reinforcement learning trajectory optimization module 414 is used to optimize the agent's operation trajectory by analyzing user data on the agent's usage and employing reinforcement learning algorithms. Through continuous learning and adjustment, the system's execution efficiency and adaptability to the agent can be improved, enabling the agent to better meet the user's actual needs and provide more efficient automated services in complex scenarios. The processing implemented by this module corresponds to step ④ in Figure 4C.

[0117] The intelligent recommendation engine 415 analyzes users' operating habits, historical data, and preference information to provide personalized intelligent agent recommendations. Based on a co-occurrence mechanism, this engine can recommend intelligent agents related to users' frequently used operations or display other similar intelligent agents commonly used by users. This helps users quickly find intelligent agents that suit their needs. The processing implemented in this module corresponds to step ⑤ in Figure 4B, intelligent recommendation.

[0118] The Agent Store 421 is the core of the entire system, providing users with a platform to share and download agents. Users can not only download agents created by other users from the Agent Store, but also upload their own creations for others to use and evaluate. The Agent Store promotes communication and innovation among users, enabling the entire ecosystem to continuously evolve and enrich.

[0119] The user interaction and feedback module 422 is used to collect user feedback during the use of the intelligent agent and provide an interaction interface between the user and the system. Through continuous user feedback, the system can continuously improve the design of the intelligent agent and the recommendation mechanism, thereby improving user satisfaction and the overall performance of the system. The processing implemented by this module corresponds to step ⑥ in Figure 4B, user editing.

[0120] The method for constructing the operation trajectory provided in this application embodiment will be described in detail below with reference to the accompanying drawings. This method can be applied to electronic devices with the hardware structure shown in Figure 3, or to electronic devices with the software structure shown in Figure 4A. These hardware or software structures may have more or fewer components than shown in the figures, or combine certain components, or split certain components, or have different component arrangements, etc., in electronic devices with similar hardware and software structures.

[0121] Referring to Figure 5, Figure 5 shows a flowchart of a method for constructing an operation trajectory according to an embodiment of this application. This method can be executed by a mobile phone. As shown in Figure 5, the method for constructing the operation trajectory includes steps S510-S540.

[0122] S510, Data Construction.

[0123] This process involves data collection and organization. By collecting and organizing various types of data related to the target app, a complete production dataset is created. Data construction includes an automated interaction trajectory traversal process based on the User Interface Transition Graph (UTG) and comprehensive data from the app's documentation.

[0124] The automated interactive trajectory traversal process of the UTG in this application embodiment and the APP manual will be described in detail below.

[0125] First, the automated interactive trajectory traversal process of UTG will be explained.

[0126] In some embodiments, the automated interaction trajectory traversal process of UTG can also be understood as the traversal process of the control list corresponding to the page of the application.

[0127] Referring to Figure 6, Figure 6 shows a flowchart of the automated interactive trajectory traversal process of the UTG according to an embodiment of this application. As shown in Figure 6, the automated interactive trajectory traversal process includes S511-S516.

[0128] S511, open the application.

[0129] The application can be opened automatically based on user instructions, or it can be opened manually by the user. This application does not limit this method.

[0130] S512, Get the list of page controls.

[0131] By obtaining a list of controls, we can ensure that the phone can recognize and understand all the elements on each page, which can be executed. For example, when the user clicks the share function (element), the phone performs the click operation and opens the next screen.

[0132] After obtaining the list of controls, each control is operated on one by one using a breadth-first or depth-first strategy. For example, clicking a button, entering text, or toggling options.

[0133] S513, determine if the list is empty.

[0134] If the list is empty, it means that all controls in the list have been traversed, so execute S516 to end the traversal.

[0135] If the list is not empty, execute S514, click the control, and remove it from the list. Remove the controls that have been processed from the list, and then continue editing the other controls in the list until the list is empty, at which point the editing ends.

[0136] S515 determines whether the page has changed.

[0137] We determine whether we have entered a new page by observing changes in the page. If the page changes, it means there is a new page. At this point, we need to continue to retrieve the current page control list, return to S512, and traverse the controls on the current page until the page stops changing.

[0138] If the page does not change, then execute S516.

[0139] S516, traversal complete.

[0140] During the traversal of the aforementioned operation trajectory, for each operation on a control, the system records the control information and operation type involved in the current operation, and takes a screenshot of the newly appeared page to construct data. This data will be used to construct the UTG in the subsequent trajectory generation stage. The state after each operation, such as screenshots and control lists, is considered a node in the UTG, while each operation, such as action type and control information, forms an edge in the UTG.

[0141] Using the above convenient method, each application's corresponding page and its controls can be traversed to obtain the executable operation trajectory.

[0142] In some embodiments, the dataset may also include user-recorded videos. For example, when a user completes a target task, they may record a video of themselves completing the task via screen recording. This video serves as the basic operation trajectory and does not require UTG orchestration, allowing for a faster acquisition of the intelligent agent.

[0143] The following is a further explanation of the application's instruction manual.

[0144] Referring to Figure 7, which illustrates the architecture of the application specification of this application embodiment, the construction process of the application specification mainly includes three parts: data collection, annotation, and application specification construction. Operation data collection can be achieved through several methods: user-driven exploration based on page traversal, such as clicks and swipes; large-scale model-assisted exploration based on task samples, i.e., obtaining corresponding operations based on big data; operations obtained during the execution of the application operation agent; and manual collection, i.e., user demonstrations of trajectories and free user operations. This collected data is obtained by the event listening module, reported to the UI operation module, and a corresponding operation sequence is obtained. Simultaneously, screenshots of the corresponding application interface and the UI structured representation form a data pair of interface screenshots and structured representations, clarifying and refining the descriptive information of the functions, behaviors, and interface performance of each interaction module. This data ultimately generates the UI interface summary, control function descriptions, page jump relationship diagrams, and task operation sequences of the specification. The content recorded in the specification provides additional contextual support for the operation trajectory generated by UTG, making the system's understanding of the APP's functions more comprehensive and in-depth, thus improving the accuracy of the generated operation trajectory.

[0145] Now let's return to Figure 5 and continue explaining the other steps in Figure 5.

[0146] S520 determines the user interface transition map based on the constructed data.

[0147] In some embodiments, the application's corresponding interface and the list of controls corresponding to the interface can be used as nodes in the user interface transition graph. The operations performed on each control, such as the action type and control information, are used as edges in the user interface transition graph to form the user interface transition graph.

[0148] Referring to Figure 8, Figure 8 illustrates a schematic diagram of the user interface transition graph corresponding to the application in this embodiment of the application. As shown in Figure 8, each application corresponds to many application interfaces. The content corresponding to these interfaces is used as nodes, where the content of the interface can be in the form of screenshots. Operations on elements or functional modules in the interface are used as edges to draw the corresponding user interface transition graph of the entire application. All operation trajectories corresponding to the application can be obtained from the user interface graph. For example, as shown in Figure 8, the interface corresponding to the mobile phone desktop is taken as the starting point of the operation trajectory. After passing through step 1, intermediate interface 1, step 2, intermediate interface 2, and step 3, the user's target task is finally completed, that is, opening a functional interface of an application. This interface is taken as the ending point of the operation trajectory. The starting interface screenshot, intermediate interface screenshot, and ending interface are combined through the corresponding steps 1, 2, and 3 to obtain the operation trajectory (intelligent agent) corresponding to the task target. This method of obtaining operation trajectories not only systematically records all possible interaction paths of the user interface, but also provides rich data support for generating and verifying intelligent agents. Furthermore, for each application, all its corresponding interaction paths are traversed. In other words, any task in the application can form an intelligent agent, fully covering all possible operation paths, providing users with a more convenient and faster operation experience, increasing user dependence, and facilitating promotion and use.

[0149] S530 inputs the user interface transfer graph and instruction manual into the Big Prophecy model to obtain the intelligent agent.

[0150] Referring to Figure 9, which illustrates a framework diagram of the agent construction process according to an embodiment of this application, as shown in Figure 9, the UTG trajectory and the APP's instruction manual are input into a Large Language Model (LLM) as a dataset. The LLM can combine these datasets to understand and generate natural language text, expressing the operation trajectory for completing the target task step by step in text form. This text can be executed by a mobile phone or viewed by a user, thus obtaining an agent. This agent is an unverified agent, also known as a pseudo-agent. Compared to agents in the prior art, this type of agent has higher accuracy, effectively increasing the probability of successful task execution and improving the user experience. In some embodiments, a video recorded by the user of the completion of the target task can also be input into the LLM. The LLM can understand and generate text based on the video content. This text can be executed by a mobile phone or viewed by a user, thus obtaining an agent. This recording method allows for a faster acquisition of the desired agent.

[0151] In some embodiments, pseudo-intelligent agents can be verified in order to further improve the correctness and reliability of the intelligent agent.

[0152] Referring to Figures 5 and 9, step S540 verifies the agent.

[0153] In a simulated environment, the mobile phone system, based on the agent's description, simulates the user's actual operating scenarios, automatically verifying each step of the agent's operation to ensure its correctness and reliability. This ensures the agent can stably and accurately complete tasks under various possible usage scenarios. Furthermore, the verification of the agent identifies potential problems in real-world applications, allowing for timely resolution and adjustments to ensure each agent can be executed effectively to achieve the user's target tasks.

[0154] As shown in Figures 5 and 9, to further enhance the accuracy and diversity of agents, validated data, such as successfully validated agents, is fed back into the dataset. Through continuous iteration via this data feedback mechanism, the large language model is optimized, thereby improving its accuracy. This gradually enhances the system's intelligence level in generating and validating agents, ensuring higher reliability and accuracy. Furthermore, the resulting agents can be repeatedly validated to ensure the stability of the selected agents' functionality.

[0155] In some embodiments, as shown in Figure 9, the dataset may also include user logs. These user logs may include, but are not limited to, data such as user operating habits, the intelligent agent actually operated by the user, or the user's trajectory during self-operation. Integrating user logs into the production and training of a large language model allows for the creation of an intelligent agent that better adapts to the user's personalized needs based on the large oracle model, further improving the user experience.

[0156] In some embodiments of this application, in order to further optimize the execution efficiency and adaptability of the agent, the agent is iteratively optimized using a trajectory optimization model.

[0157] Referring to Figure 10, which illustrates a flowchart of an intelligent agent optimization method according to an embodiment of this application, the optimization can be performed by a mobile phone or a cloud server. As shown in Figure 10, the intelligent agent optimization method includes steps S1010-S1060.

[0158] S1010: Input the operation trajectory into the trajectory optimization model.

[0159] In the embodiments of this application, the trajectory optimization model is trained based on data such as user actions, the execution results of the agent, and the environmental state corresponding to the agent. User actions include, but are not limited to, swiping, clicking, the order of operations, and selecting a path to perform the task. The execution results of the agent include, but are not limited to, information such as whether the agent succeeded or failed, the percentage of completion, or a loading failure at a certain step.

[0160] In some embodiments, the trajectory optimization model may be a Bézier curve fitting model or a polynomial curve fitting model, etc. The optimization algorithm may be gradient descent, Kalman filter, or genetic algorithm, etc., and the embodiments of this application are not limited thereto.

[0161] It should be noted that in the embodiments of this application, in order to protect the privacy and security of users, sensitive user information will be de-identified during the collection process to ensure the security of user identity information.

[0162] S1020 outputs the optimized trajectory.

[0163] The optimized operation trajectory can be used directly, for example, it can be stored in the cloud for users to download and use when needed, i.e., intelligent agent deployment. Simultaneously, personalized data of users using the intelligent agent is collected, such as user operation habits and usage frequency. Based on this data, the trajectory optimization model is trained, and the intelligent agent obtained based on the trajectory optimization model can better meet users' personalized needs.

[0164] S1030, obtain the return data of current network users.

[0165] Among them, live user feedback data refers to data collected from the actual usage environment, which can include historical operational behavior data of users completing target tasks. Because this data reflects user behavior and feedback in real-world scenarios, it is crucial for the continuous optimization and improvement of the model. Live user feedback data can include user behavior data, such as clicks, swipes, or input; user feedback data, such as direct evaluations and ratings of the agent, complaints, and suggestions; system log data, such as records of user habitual operations, and which type of agent is used more frequently. In addition, it can also include result data of the execution trajectory, such as whether the agent succeeded or failed, the percentage of completion, or information about the status of a certain step.

[0166] S1040, Data Quality Filtering. This determines whether the data quality meets the required standards.

[0167] When the data meets the requirements, such as when a user uses an agent to share a contact using two different trajectories, and the agent is used 80% of the time or executes successfully, the data is considered acceptable. Then, S1050 is executed, and these data are used as training data to perform reinforcement learning training on the trajectory optimization model. This allows the optimized agent to perform more intelligently and efficiently in different environments and tasks, while improving the user's operating experience and efficiency.

[0168] If the usage rate of an agent is lower than a set value, such as 10%, or if the agent fails to execute, it will be eliminated, and S1060 will be executed to discard low-quality data.

[0169] Therefore, by further optimizing the intelligent agent, its execution efficiency and adaptability can be improved, thereby enhancing the user experience.

[0170] In one embodiment of this application, during the optimization process, the execution process of the intelligent agent can be abstracted into a state space. In this state space, each state represents the specific environmental situation or execution process of the agent at a certain moment. For example, a state can represent the content displayed on the current page, the text entered by the user, the activation state of a control, etc. By mapping the complex execution process to a state space, the system can more accurately understand and analyze the performance of the intelligent agent in different contexts.

[0171] In each defined state, the agent needs to select an optimal action (action data) to execute in order to achieve the desired effect. This action can be a common interactive operation such as clicking, swiping, or entering text. The action selection process not only affects the current execution effect but also has a chain reaction on the agent's subsequent behavior. Therefore, choosing the appropriate action is crucial.

[0172] Furthermore, during the agent's optimization process, to guide the agent towards the optimal direction, the system defines a corresponding reward mechanism for each action. The definition of the reward is determined by evaluating the agent's actual performance during execution. For example, when the agent successfully completes a task or achieves an intermediate goal, the system provides a positive reward, such as a positive label or a higher score (e.g., 8 points or above). Conversely, when the agent makes an error or fails to achieve the expected result, a penalty is imposed, such as a negative label or a lower score (e.g., below 5 points). By introducing a reward mechanism, the agent can continuously optimize its behavioral strategy through reinforcement learning, gradually learning the optimal path. For example, during training, the trajectory optimization model continuously adjusts the agent's behavioral strategy based on the defined reward information, gradually guiding it towards the optimal path.

[0173] The intelligent agent optimization method according to the embodiments of this application, through the design of data feedback, state definition, action selection, and reward mechanism, combined with repeated reinforcement learning training, enables the optimized intelligent agent to better adapt to various complex operation scenarios and provide users with a more intelligent and personalized service experience.

[0174] In one embodiment of this application, to further enhance the user experience, a co-occurrence mechanism is used to obtain the user's personalized preferences, and intelligent agents that meet the user's personalized needs are recommended. For example, by collecting the user's own log data, or log data of similar users, the user's personalized preferences are analyzed, and based on these preferences, intelligent agents that meet the user's personalized needs are recommended.

[0175] Referring to Figure 11, which illustrates a flowchart of the estimation and recommendation method for the co-occurrence mechanism according to an embodiment of this application, the recommendation scheme employs two calculation methods: the first is a high-frequency operation agent recommendation algorithm, which can be executed by the high-frequency operation agent recommendation module; the second is a similar user agent recommendation algorithm.

[0176] The first type, the recommendation of high-frequency operating agents, includes steps S1111-S1113.

[0177] S1111, User log data return.

[0178] In the high-frequency operation intelligent agent recommendation module, the mobile phone system first performs data feedback on the user's historical operation data. This feedback data includes, but is not limited to, frequently used operations, frequently used functions, and daily task processing flows in different mobile applications.

[0179] In the embodiments of this application, to ensure the security of user personal information and protect user privacy, strict anonymization processing is performed on the data at each stage during the data feedback process. This involves removing information directly associated with the user's identity while retaining necessary operational data and behavioral patterns. The set of these anonymized data can be represented as D = {d1, d2, ..., dn}, where each di represents a specific operational behavior data record.

[0180] S1112, Behavioral Mining.

[0181] In some embodiments, in-depth behavioral mining is performed on the anonymized data to analyze user operating habits and identify high-frequency operation patterns. Data mining algorithms, such as the Frequent Pattern Growth (FP-Growth) algorithm, can identify operation sequences (operation trajectories) with high frequency and strong correlation. Assuming a user's operation sequence is S_u = {o_1, o_2, ..., o_m}, the processor can determine high-frequency operation patterns by calculating the frequency f(o_i) of each operation o_i and the support "support"(S_u) of the sequence pattern. f(o_i) = ("count"(o_i)) / (∑_(j=1)^m) "count"(o_j)) "support"(S_u)=("count"(S_u)) / (∑_(k=1)^N "count"(S_k))

[0182] Here, "count"(o_i) represents the number of times operation o_i appears in all operation records, and N is the number of operation sequences for all users.

[0183] S1113, Agent Generation and Recommendation.

[0184] Based on the identified high-frequency operation patterns, the system automatically generates corresponding agent editing suggestions. Users can then operate based on these suggestions, quickly creating action agents that meet their needs. For example, for a frequently executed three-step operation S_u={o_1,o_2,o_3}, the system automatically generates an agent template. This template matches the user's personalized needs, allowing users to create their own agents through simple editing, greatly reducing the complexity of manual configuration.

[0185] The second method, similar user agent recommendation, includes steps S1121-S1123.

[0186] S1121, Feature Extraction. In the similar user agent recommendation module, the system first analyzes users' personalized preferences to extract user features from the log data. These preferences include users' behavioral habits, interests, frequently used applications and functions, etc. To ensure privacy and security, the system anonymizes this data and abstracts it into a feature vector F_u = {f_1, f_2, ..., f_k}, where each feature f_i corresponds to a user's preference dimension, such as usage frequency, preference type, etc. (Anonymization process follows.)

[0187] S1122, Similarity Calculation. The system uses feature vectors to calculate the similarity of personalized preferences between users. The similarity calculation method in this application embodiment may include cosine similarity and Euclidean distance. Taking cosine similarity as an example, the similarity "Sim"(u_1,u_2) can be expressed as: "Sim"(u_1,u_2)=(F_u1·F_u2) / (∥F_u1∥×∥F_u2∥)

[0188] Here, F_u1 and F_u2 are the feature vectors of users u_1 and u_2, respectively. The closer this similarity value is to 1 (the preset value), the more similar the personalized preferences between users are.

[0189] S1123, Agent Recommendation.

[0190] After calculating the similarity between users, the system uses clustering algorithms, such as K-means clustering, to group similar users and recommends corresponding agents based on their operational data and usage habits. For example, if a group of users frequently uses a specific function in an application and generates a specific agent template, the system will recommend these agents to the users, allowing them to quickly obtain personalized agents and thus improving the practicality and usability of the agents.

[0191] The recommendation algorithm of this application provides an agent suitable for personalized service recommendation scenarios. The system can automatically recommend suitable agents based on the user's operating habits and historical data. For example, for users who enjoy sports, the system can recommend automated sports-related tasks, such as automatically recording exercise data and sending reminders to drink water after exercise. This readily meets the user's personalized needs for agents.

[0192] It should be noted that throughout the entire recommendation process of the intelligent agent described above, the system consistently adheres to strict data privacy protection principles. All collected user data undergoes anonymization to ensure the removal of sensitive user identity information. Furthermore, encrypted transmission and storage mechanisms are employed during recommendation execution to prevent data leakage or unauthorized access. Additionally, users can choose not to have their information used; for example, when retrieving user log data, the mobile interface prompts the user about which information is being retrieved and its purpose. Users can also choose not to participate in data feedback and analysis, further enhancing their control over their personal data.

[0193] Therefore, through the collaborative work of the two recommendation modules, the trajectory recommendation strategy based on the co-occurrence mechanism can effectively solve the problems of insufficient personalization and low recommendation accuracy in traditional intelligent agent recommendation systems. The high-frequency operation intelligent agent recommendation module provides users with intelligent intelligent agent suggestions based on their own operating habits. The similar user intelligent agent recommendation module further improves the accuracy and practicality of recommendations by mining the commonalities among users. The recommendation method of this application embodiment not only optimizes the user experience but also ensures that user privacy and data security are fully protected throughout the entire process.

[0194] In the above embodiments, the construction process of the operation trajectory (smart agent) has been described in detail. In the following embodiments, the interaction method of the embodiments of this application will be described in detail with reference to the accompanying drawings.

[0195] Once the intelligent agents constructed using the above method are established, the mobile phone can store these intelligent agents, or they can be stored in the cloud. When stored in the cloud, the user can operate the mobile phone to input corresponding commands. The mobile phone will then download the corresponding intelligent agent from the cloud based on the user's commands and execute the intelligent agent to automatically complete the user's desired task.

[0196] In some embodiments, the process of constructing the operation trajectory and the device that actually executes the operation trajectory may be different electronic devices. The device that executes the operation trajectory is an electronic device that interacts directly with the user. For example, the electronic device may be, but is not limited to, a mobile phone, a smart TV, a wearable device, a tablet computer, a computer with wireless transceiver capabilities, a virtual reality (VR) device, an augmented reality (AR) device, a wireless device in industrial control, a wireless device in self-driving cars, etc.

[0197] The following embodiments use a mobile phone as an example for illustration.

[0198] Referring to Figure 12, Figure 12 shows a flowchart of the interaction method according to an embodiment of this application. This method can be executed by a mobile phone. As shown in Figure 12, the interaction method includes, but is not limited to, the following steps:

[0199] S1210, Receive user instruction (first operation).

[0200] In the embodiments of this application, user instructions are used to instruct electronic devices to perform target tasks. These user instructions can be voice instructions, touchscreen instructions, or text instructions, etc. The target task can be as described in the above application scenarios, such as automatically turning off airplane mode every morning, playing music, automatically processing emails, scheduling meetings, etc., and will not be elaborated further here.

[0201] In some embodiments, after receiving a user's voice command, the mobile phone can display the corresponding text content in the dialog box between the user and the voice assistant, or it can choose not to display it and simply reply to the user via voice. Taking the current display interface when the mobile phone receives a user's voice command as the main interface as an example, after receiving the user's voice command, the main interface will display a dialog box between the user and the voice assistant, and the dialog box will contain the content of the voice command.

[0202] In some embodiments, the first operation may be any of the following operations.

[0203] The first action can be a pre-defined action (a specific action), such as adding an item to a shopping cart in an application (selecting an item), or opening the application, such as a map application. Alternatively, the user might take their phone to a designated location. Based on these actions, the electronic device can determine the user's intent. For example, when a user selects an item, it can infer that the user wants to buy it and recommend a payment agent or directly display the payment interface. When a user opens a map application, it can infer that the user wants navigation and recommend possible addresses. When the user reaches a designated location, it can infer that the user might need a taxi and recommend a taxi agent or directly select a vehicle and ask the user for confirmation.

[0204] The first operation could be a selection of an agent within a specific application. This application could provide task-related agents, representing entities capable of executing actions to complete a target task. For example, in an agent store application, a user opens the application and selects an agent tag for sharing contacts. These agent tags could be displayed in recommended locations within the application's page based on the user's historical action data.

[0205] The first action could be entering keywords into the search area. For example, a user can search for a contact to share using the search function. At this point, the system can confirm that the user wants to share the contact and display recommended smart agent tags on the interface, or directly open the corresponding application and wait for sharing.

[0206] The first operation could be receiving an agent shared by another user device. For example, agent sharing between two devices.

[0207] The first operation can also be in the scenario of multi-device collaborative requests, where it receives operations from intelligent agents shared by other devices.

[0208] Through the above operations, intelligent agents can be provided to users in various scenarios, providing convenience and reducing the time wasted by user editing.

[0209] S1220, in response to the first operation, recommend an intelligent agent (first intelligent agent) to the first user, the intelligent agent being used to perform the first task.

[0210] After determining the user's target task (first task), the electronic device determines the target operation trajectory corresponding to the target task from the pre-stored operation trajectory.

[0211] The process of constructing the operation trajectory has been described in detail in the above embodiments. For details, please refer to the operation trajectory construction method in the above embodiments. It will not be repeated here.

[0212] In some embodiments, recommending an agent to a first user may include the following scenarios:

[0213] When a user opens the Smart Agent Store, multiple smart agents are recommended to the user. Among these smart agents, the user's preferred smart agent (the first smart agent) can be included.

[0214] In some scenarios, electronic devices recognize a user's intent based on their first action. For example, when a user adds an item to their shopping cart in a store, the electronic device recognizes that the user wants to purchase the item and, based on that intent, recommends an agent to open the payment page. In some embodiments, the first action is used to receive a message from a second user and present the user with an agent that the second user shared with the first user via the message.

[0215] The first task can be a multi-user collaborative task. This task requests the first user and the second user to collaborate on the first task based on the first intelligent agent. This enables the sharing of intelligent agents under multi-task collaboration.

[0216] In the embodiments of this application, the mobile phone can set a title or label for each target operation trajectory. Upon receiving a user instruction, the user instruction can be analyzed and compared with the label or title of the operation trajectory to determine the intelligent agent that matches the instruction.

[0217] S1230, receive the second operation.

[0218] In some embodiments, the second operation may be a viewing operation, a selection operation, a confirmation operation, or a modification operation on the intelligent agent.

[0219] S1240, in response to the second operation, presents the operation trajectory of the agent performing the first task.

[0220] This process can be executed by the phone after the user confirms the action. Alternatively, the user can set the default to no confirmation in the settings, so that the phone directly executes the found smart agent after receiving the instruction.

[0221] In some embodiments, presenting the operation trajectory of an intelligent agent performing a first task may include: displaying the operation trajectory through an interface (first interface); the interface may also include at least one control for modifying the operation trajectory. Modifying the operation trajectory may include any one of manually editing the operation trajectory, automatically optimizing the operation trajectory, or re-recording the operation trajectory.

[0222] In the embodiments of this application, the operation trajectory is displayed in text form according to the execution order. This display is more in line with the user's operating habits and is easier for the user to edit.

[0223] According to the interaction method of the embodiments of this application, since the intelligent agent can be executed across applications and has high execution accuracy and a high success rate, the user experience is greatly improved.

[0224] The interaction method of this application embodiment will be further described below with reference to the operation of the mobile phone interface.

[0225] Referring to Figure 13, which illustrates a user interface diagram of the interaction between a user and a mobile phone according to an embodiment of this application, as shown in Figure 13, the user can directly input commands (first commands) via voice. For example, the user can input the target task via voice: "Share contact information of Li Xiang to my WeChat™". After receiving the voice command, the mobile phone analyzes the voice command and determines the intelligent agent that matches the command. The dialog box 1311 of the main interface 1310 can display the text content corresponding to the voice command, "The following intelligent agents have been recommended for you", and can also provide user options for selection, such as "Execute" and "View". When the user clicks "View", interface 1320 is displayed. In interface 1320, the steps of the intelligent agent are displayed sequentially, for example, executing steps 1-6, with each step having a corresponding execution statement. The user can view the accuracy of these steps, and when needed, can click the "Edit" function (control) in interface 1320 to edit each step. The user can also click the "Auto Optimize" (control) in interface 1320 to optimize the intelligent agent, etc. The optimization process can be referred to the optimization process in Figure 10 above. Alternatively, users can click the "Re-record" function (control) in interface 1320 to manually complete the target task. The phone will record the user's actions, and the recorded video can be used to derive an intelligent agent based on a large language model, which will then be displayed again in the execution steps. This allows for a faster and more accurate retrieval of the desired intelligent agent. Furthermore, users can directly click the "Try it" function in interface 1320, and the phone will respond to the command, executing each step according to the intelligent agent's description to complete the user's target task. In other embodiments, users can also manually input commands, which can be text input or input through clicks, swipes, or other operations. For example, a user can open an intelligent agent store and select a suitable intelligent agent based on the store's recommendations.

[0226] Referring to Figure 14, which illustrates a user interface diagram of another embodiment of this application, the interaction between a user and a mobile phone is shown. As shown in Figure 14, the user opens the (second instruction) Agent Store, displaying interface 1410. In interface 1410, the mobile phone recommends agents to the user based on their personalized needs, i.e., the recommendation algorithm of the sharing mechanism mentioned above. For example, under the module corresponding to "You May Like," there are corresponding task tags such as "Post to Moments" and "Share Contacts to WeChat™." These task tags correspond to agents that can complete the corresponding target tasks. When the user clicks (second instruction) "Share Contacts to WeChat™," the mobile phone receives the click operation and displays interface 1420. Interface 1420 displays the same content as interface 1320, showing each step of the agent in sequence for easy viewing and editing by the user. When the user clicks "Try," the mobile phone is triggered to execute the above steps, completing each step according to the description of each step in the agent until the target task in the instruction is completed. The user can also click "Add" to add steps, etc.

[0227] In some embodiments, as shown in Figure 14, on the main interface 1410 of the Agent Store, when the desired agent is not available in the store, the user can add a new agent by clicking the "+" widget 1411. For example, based on the user's click, the phone can recommend more agents to provide the user with more choices.

[0228] The following section, with reference to the interface diagram, explains the steps and corresponding interface for executing a smart agent on a mobile phone.

[0229] Referring to Figure 15, which illustrates the interface diagram of a mobile phone executing an intelligent agent according to an embodiment of this application. As shown in Figure 15, taking the mobile phone receiving the user's voice command "Share Li Xiang's information to my WeChat™" as an example, after receiving the user's voice command (first command) as shown in Figures 13 and 14 above, the mobile phone automatically opens the Contacts in the desktop interface 1510, enters the Contacts interface 1520, selects "Li Xiang", and accesses the details interface 1530. After selecting "More Options", finding "Share Contacts" in interface 1540, the user enters interface 1550. In interface 1550, select "WeChat™" (a third-party application), and in the new interface 1560, select "Send to a friend". Enter the WeChat™ contacts interface 1570, select the contact's name "Li Xiang", and select "Send" in interface 1580. Then, enter the chat interface between Li Xiang and me 1590. At this point, the command to share Li Xiang's information to my WeChat™ is completed. The whole process is completed automatically by the mobile phone, saving users time and effort and greatly improving the user experience.

[0230] This application also provides an electronic device, including:

[0231] Memory, used to store instructions executed by one or more processors of the device, and

[0232] The processor is used to execute the methods explained in conjunction with Figures 2 to 15 in the above embodiments.

[0233] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the methods explained in Figures 2 to 15 of the above embodiments.

[0234] This application also provides a computer program product containing instructions that, when run on an electronic device, cause a processor to perform the methods explained in Figures 2 to 15 of the above embodiments.

[0235] Referring now to FIG. 16, a block diagram of a SoC (System on Chip) 1600 according to an embodiment of this application is shown. In FIG. 16, similar components have the same reference numerals. Additionally, dashed boxes represent optional features of more advanced SoCs. In FIG. 16, the SoC 1600 includes: an interconnect unit 1650 coupled to an application processor 1610; a system agent unit 1680; a bus controller unit 1690; an integrated memory controller unit 1640; a group or one or more coprocessors 1620, which may include integrated graphics logic, an image processor, an audio processor, and a video processor; a static random access memory (SRAM) unit 1630; and a direct memory access (DMA) unit 1660. In one embodiment, the coprocessor 1620 includes a dedicated processor, such as, for example, a network or communication processor, a compression engine, a general-purpose computing on GPU (GPGPU), a high-throughput many integrated core (MIC) processor, or an embedded processor, etc.

[0236] The static random access memory (SRAM) cell 1630 may include one or more computer-readable media for storing data and / or instructions. The computer-readable storage medium may store instructions, specifically temporary and permanent copies of those instructions. These instructions may include, when executed by at least one unit in the processor, causing the SoC 1600 to perform the methods explained in the above embodiments, specifically referring to the methods explained in Figures 2 and 15 of the above embodiments, which will not be repeated here.

[0237] The various embodiments of the mechanisms disclosed in this application can be implemented in hardware, software, firmware, or a combination of these implementation methods. Embodiments of this application can be implemented as computer programs or program code executable on a programmable system, the programmable system including at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0238] Program code can be applied to input instructions to execute the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, the processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application-specific integrated circuit (ASIC), or a microprocessor.

[0239] The program code can be implemented using a high-level procedural language or an object-oriented programming language to communicate with the processing system. Assembly language or machine language can also be used when needed. In fact, the mechanisms described in this application are not limited to any particular programming language. In either case, the language can be a compiled language or an interpreted language.

[0240] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored thereon on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, the instructions may be distributed via a network or through other computer-readable media. Therefore, machine-readable media may include any mechanism for storing or transmitting information in a machine-readable (e.g., computer-readable) form, including but not limited to floppy disks, optical disks, CD-ROMs, magneto-optical disks, read-only memory (ROM), random access memory (RAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic cards or optical cards, flash memory, or tangible machine-readable storage for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) using the Internet in the form of electrical, optical, acoustic, or other propagation signals. Therefore, machine-readable media include any type of machine-readable medium suitable for storing or transmitting electronic instructions or information in a machine-readable (e.g., computer-readable) form.

[0241] In the accompanying drawings, some structural or methodological features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be necessary. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. Furthermore, the inclusion of structural or methodological features in a particular figure does not imply that such features are required in all embodiments, and in some embodiments, these features may be omitted or may be combined with other features.

[0242] It should be noted that all units / modules mentioned in the device embodiments of this application are logical units / modules. Physically, a logical unit / module can be a physical unit / module, a part of a physical unit / module, or a combination of multiple physical units / modules. The physical implementation of these logical units / modules themselves is not the most important factor; the combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed in this application. Furthermore, to highlight the innovative aspects of this application, the above-described device embodiments of this application have not introduced units / modules that are not closely related to solving the technical problems proposed in this application. This does not mean that the above-described device embodiments do not contain other units / modules.

[0243] It should be noted that in the examples and description of this patent, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0244] Although this application has been illustrated and described with reference to certain preferred embodiments thereof, those skilled in the art should understand that various changes in form and detail may be made thereto without departing from the spirit and scope of this application.

Claims

1. An interaction method applied to an electronic device, characterized in that, include: Receive a first operation from a first user on the electronic device; In response to the first operation, a first intelligent agent is recommended to the first user, and the first intelligent agent is used to perform the first task; Receive a second operation from the first user on the electronic device; In response to the second operation, the operation trajectory of the first agent executing the first task is presented.

2. The method according to claim 1, characterized in that, The presentation of the operation trajectory of the first intelligent agent executing the first task includes: The operation trajectory is presented in natural language in the order of execution, comprising the multiple steps involved.

3. The method according to claim 1 or 2, characterized in that, The presentation of the operation trajectory of the first intelligent agent executing the first task includes: The operation trajectory is displayed on the first interface; The first interface also includes at least one control for modifying the operation trajectory.

4. The method according to claim 3, characterized in that, The modification of the operation trajectory includes at least one of the following: The operation trajectory can be manually edited; The operation trajectory is automatically optimized; as well as The operation trajectory was re-recorded.

5. The method according to any one of claims 1-4, characterized in that, The second operation is a viewing operation, selection operation, confirmation operation, or modification operation of the first intelligent agent.

6. The method according to any one of claims 1-5, characterized in that, The step of recommending the first intelligent agent to the first user includes: In the agent store, multiple agents are recommended to the first user, including the first agent.

7. The method according to any one of claims 1-5, characterized in that, The step of recommending a first intelligent agent to the first user in response to the first operation includes: Based on the first operation, identify the intent of the first user; Based on the stated intent, the first intelligent agent is recommended to the first user.

8. The method according to any one of claims 1-5, characterized in that, The first operation is used to instruct the electronic device to perform the first task.

9. The method according to any one of claims 1-5, characterized in that, The first operation is to enter a search term in the search box.

10. The method according to any one of claims 1-5, characterized in that, The first operation is dialogue input in the smart assistant.

11. The method according to any one of claims 1-5, characterized in that, The first operation is used to receive a message from a second user, and the step of recommending the first agent to the user includes: presenting the first agent shared by the second user to the first user through the message to the user.

12. The method according to claim 11, characterized in that, The first task is a multi-user collaborative task, and the message is used to request the first user to collaborate with the second user to execute the first task based on the first intelligent agent.

13. A method for constructing an operation trajectory, characterized in that, include: Obtain the dataset corresponding to the application. The dataset includes a list of controls corresponding to multiple pages in the application and a specification of the application. The list of controls is used to record the executable controls in the pages, and the specification at least describes the operation corresponding to the control. Based on the dataset, a user interface transition map is determined; By inputting the user interface transfer map and the instruction manual into the large model, an operation trajectory for completing the target task is obtained.

14. The method according to claim 13, characterized in that, Determining the user interface transition map based on the dataset includes: Obtain all pages of the application and a list of controls corresponding to each page, and use the content and control list corresponding to each page as nodes of the user interface transition graph; When transitioning between pages, obtain the corresponding control information and operation type, and use the control information and operation type as edges of the user interface transition graph to obtain the user interface transition graph.

15. The method according to claim 14, characterized in that, The step of obtaining all pages of the application and the list of controls corresponding to each page includes: Launch the application and retrieve the list of controls corresponding to the current page; Operate each control one by one according to the instructions described in the manual. The content corresponding to the new page that appears after each operation is saved, and the control list corresponding to the new page is obtained. The controls in the control list corresponding to the new page are operated one by one until the control lists corresponding to all applications of the application are traversed, so as to obtain all pages of the application and the control list corresponding to each page.

16. The method according to any one of claims 13-15, characterized in that, Also includes: A simulation verification environment is constructed to verify all the operation trajectories. If the verified operation trajectory meets the expected execution requirements, the operation trajectory that meets the expected value is retained.

17. The method according to claim 16, characterized in that, The verification of all the aforementioned operation trajectories includes: In a simulated environment, based on the description of the operation trajectory, the actual operation scenario is simulated, and the steps corresponding to the operation trajectory are executed step by step. If each step of the current operation trajectory can be executed according to the preset requirements, then the operation trajectory is determined to meet the expected execution requirements.

18. The method according to any one of claims 13-15, characterized in that, The operation trajectory that meets the expected execution requirements is input into the large model, and the operation trajectory is optimized.

19. The method according to claim 18, characterized in that, The optimization of the operation trajectory includes: Acquire historical operation behavior data for completing the target task and result data for executing the operation trajectory, wherein the result data includes the state generated during the execution of the operation trajectory and the action data during the execution of the operation trajectory; The user's historical operation behavior data and the result data are input into the trajectory optimization model to obtain the optimized operation trajectory.

20. The method according to claim 18, characterized in that, The method further includes: evaluating the optimized operation trajectory. When the operation trajectory completes the target task or completes the preset intermediate steps, the optimized result is labeled with a reward tag. When the operation trajectory encounters an error or fails to achieve the expected target task, the optimized result is labeled with a penalty tag.

21. The method according to claim 19, characterized in that, The states include: the content displayed on the current page corresponding to the application, the text entered by the user, or the activation state of the control.

22. The method according to claim 19, characterized in that, The action data includes: clicking, swiping, or entering text.

23. The method according to claim 19, characterized in that, The historical operation behavior data includes: the user's historical operation behavior and the historical operation behavior of other users, wherein the similarity between the historical operation behavior of other users and the historical operation behavior of the user meets a preset value.

24. The method according to claim 13, characterized in that, The application's specification includes at least one of the following: a summary of the user interface, a description of the functions of the controls, a page navigation diagram, and a task operation sequence.

25. An electronic device, comprising: A memory for storing instructions executed by one or more processors of an electronic device, and a processor for performing the method of any one of claims 1 to 12 or the method of any one of claims 13 to 24.

26. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on an electronic device, cause the electronic device to perform the method of any one of claims 1 to 12 or the method of any one of claims 13 to 24.

27. A computer program product containing instructions, characterized in that, When the computer program product is run on an electronic device, it causes the electronic device to perform the method of any one of claims 1 to 12 or the method of any one of claims 13 to 24.

Images