Intelligent interaction method and electronic device

Abstract

Provided in the present application are an intelligent interaction method and an electronic device. The intelligent assistant in the present application can directly interact with an application, and trigger, on the basis of acquired page data, the application to execute a voice command, thereby simplifying the interaction process and improving the communication efficiency. The method comprises: an intelligent assistant in an electronic device receiving a first voice command of a user, wherein the first voice command carries information of a first application, and the first application is an application currently displayed by the electronic device; the intelligent assistant requesting the acquisition of first page data from the first application on the basis of the first voice command; and the intelligent assistant triggering, on the basis of the acquired first page data, the first application to execute a first event corresponding to the first voice command.

Description

Intelligent interaction methods and electronic devices

[0001] This application claims priority to Chinese Patent Application No. 202411543635.6, filed with the State Intellectual Property Office of China on October 30, 2024, entitled “Intelligent Interaction Method and Electronic Device”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of terminal technology, and in particular to an intelligent interaction method and electronic device. Background Technology

[0003] With the development of terminal technology, electronic devices are equipped with intelligent assistants. When it is inconvenient for users to operate the electronic devices directly, they can control them by voice through the intelligent assistant. For example, while driving, if it is inconvenient for users to operate the screen of a smart cockpit with their hands, they can use the smart assistant in the smart cockpit to instruct the smart cockpit to launch navigation applications and perform navigation through voice commands.

[0004] During intelligent interaction, the intelligent assistant relies on the screenshot function of the accessibility service configured in the electronic device to capture a screenshot. Through image analysis, it obtains the interface information from the screenshot to determine the location of the control indicated by the voice command. The intelligent assistant then sends this control location to the accessibility service, which triggers the application to execute the event corresponding to that control.

[0005] It can be seen that the current implementation of intelligent interaction relies on accessibility services, the interaction process is cumbersome, and the image analysis process is complex and consumes a lot of power. Summary of the Invention

[0006] To address the aforementioned technical problems, this application provides an intelligent interaction method and electronic device. The technical solution provided by this application enables the intelligent assistant to directly interact with applications, triggering the application to execute voice commands based on acquired page data, thus simplifying the interaction process and improving communication efficiency.

[0007] To achieve the above-mentioned technical objectives, this application provides the following technical solution:

[0008] Firstly, a smart interaction method is provided, comprising: a smart assistant in an electronic device receiving a first voice command from a user, the first voice command carrying information about a first application, the first application being the application currently displayed on the electronic device; the smart assistant requesting first page data from the first application based on the first voice command; and the smart assistant triggering the first application to execute a first event corresponding to the first voice command based on the obtained first page data.

[0009] It should be understood that, with existing technologies, smart assistants cannot obtain application page data directly; instead, they need to infer application data through screenshots. This method has several drawbacks. In the method provided in this application, the smart assistant can interact directly with the application, obtaining page data more suited to voice command execution and avoiding the acquisition of redundant information. Furthermore, the smart interaction process does not rely on accessibility services, avoiding unnecessary memory and power consumption and improving the efficiency of smart interaction.

[0010] According to the first aspect, the first page data includes the control tree data of the first page; wherein, the control tree data is transmitted between the first application and the smart assistant via inter-process communication (IPC).

[0011] The control tree data includes, for example, the controls currently displayed on the screen, control identifiers, the hierarchical relationship of each control, the display position of the controls, and the display size of the controls.

[0012] According to the first aspect, or any implementation of the first aspect above, the smart assistant triggers the first application to execute the first event corresponding to the first voice command based on the acquired first page data, including: the smart assistant acquiring a first control in the first application indicated by the first voice command based on the acquired first page data; the smart assistant sending a first operation instruction to the first application, the first operation instruction triggering the first application to operate the first control to execute the first event corresponding to the first voice command.

[0013] The first control can be, for example, a control included in the currently displayed interface of the first application. For instance, if the first application is a chat application and the currently displayed interface is the chat interface, the first control could be, for example, an input box or a send control in the chat interface. There can be one or more first controls.

[0014] For example, the first voice command carries information about the first application, which may be that the first voice command explicitly records the name of a certain application. In some scenarios, the first voice command may be, for example, "Xiaoyi, transfer the video call from Changlian to the Huawei Smart Screen"; where Changlian is the first application.

[0015] For example, the first voice command carries information about the first application, which may be an implicit record of the name of an application. In some scenarios, the first voice command may be, for example, "Xiaoyi, what other movies has the main character of this movie acted in?"; wherein, the smart assistant can use the video application currently playing the movie (such as Huawei Video) as the first application to execute the method provided in the first aspect of this application. The first voice command may also be, for example, "Order me some fried chicken."

[0016] For example, while an electronic device is displaying a chat application (such as the first application), the smart assistant receives a first voice command from the user: "Send XX message." In response to this first voice command, the smart assistant requests first page data from the chat application. Based on this first page data, the smart assistant can determine that the first control indicated by the first voice command is an input box, and the control command is to enter XX message in the input box and send it. Then, the smart assistant can send an operation instruction to the chat application, which instructs the chat application to execute the control command. This control command, for example, is a command to implement a first event.

[0017] In this way, the intelligent assistant can interact directly with the application, obtaining page data that is more suitable for voice command execution and avoiding the acquisition of redundant information. Furthermore, the intelligent interaction process does not rely on accessibility services, avoiding unnecessary memory and power consumption and improving the efficiency of intelligent interaction.

[0018] It should be understood that the second voice command or other voice commands shown in the examples below can also be understood by analogy, and will not be elaborated further.

[0019] According to the first aspect, or any implementation of the first aspect above, the method further includes: an electronic device displaying a page of a second application; a smart assistant receiving a second voice command from a user, wherein the second voice command carries information about a third application; the smart assistant obtaining second page data of the second application based on the second voice command; and the smart assistant triggering the third application to execute a second event based on the second page data, the second event being related to the second page data.

[0020] In this way, the smart assistant can obtain the second page data of the second application through IPC communication with the currently displayed second application.

[0021] According to the first aspect, or any implementation of the first aspect above, the smart assistant triggers a third application to execute a second event based on the data from the second page, including: the smart assistant instructing the third application to execute the second event in the background based on the data from the second page; or, the smart assistant, in response to a second voice command, triggers the electronic device to display the third application. The smart assistant instructs the third application to execute the second event based on the data from the second page.

[0022] Thus, when voice commands involve multiple applications, the smart assistant can also achieve direct IPC communication with multiple applications, triggering corresponding application execution events, thereby improving communication efficiency.

[0023] According to the first aspect, or any implementation of the first aspect above, the smart assistant triggers the third application to execute a second event based on the data from the second page, including: the smart assistant requesting the third application to obtain the data from the third page; and the smart assistant sending a second operation instruction to the third application based on the data from the second page and the data from the third page, thereby triggering the third application to execute the second event.

[0024] In this way, the intelligent assistant obtains the page data of the currently displayed application based on voice commands, and triggers other applications to run in the foreground based on this page data to obtain the page data of those other applications. Thus, the intelligent assistant combines the page data obtained from multiple sources to trigger the corresponding application execution events. This direct interaction between applications achieves a highly efficient intelligent interaction process.

[0025] According to the first aspect, or any implementation of the first aspect above, the method further includes: in response to a page change in the first application, the smart assistant obtains the updated first page data of the first application.

[0026] In this way, the intelligent assistant can determine the execution status of the first voice command based on the latest page display of the first application, or continue executing the first voice command. Alternatively, upon receiving other voice commands subsequently, it can also execute the corresponding voice commands based on the updated first page data.

[0027] According to the first aspect, or any implementation of the first aspect above, the method further includes: if no third voice command from the user is received within a preset time, the smart assistant stops obtaining data from the first page from the first application.

[0028] For example, if the smart assistant does not receive any more voice commands from the user within a preset time, it can automatically exit.

[0029] In this way, the smart assistant can automatically stop the smart interaction process, avoiding long waiting times for user voice commands and increasing device power consumption.

[0030] According to the first aspect, or any implementation of the first aspect above, before the smart assistant requests the first page data from the first application based on the first voice command, the method further includes: the smart assistant requests the currently displayed first window from the window management system (WMS) based on the first voice command, the first window being used to display the first application. Requesting the first page data from the first application includes: the smart assistant obtaining the first page data through the interface of the first window.

[0031] In some examples, the electronic device is equipped with a WMS (Windows Management System) to manage and control the display and interaction of multiple windows on the screen. The smart assistant, based on a first voice command, requests the WMS to retrieve the currently displayed first window, which is used to display the first application. This first window is, for example, the focus window. The WMS can provide the smart assistant with the interface of the first window, through which the smart assistant retrieves the first page data.

[0032] In this context, ordinary users control the currently displayed first application or instruct the first application to be launched via voice commands, thereby gaining control over the first application.

[0033] In this way, the smart assistant can obtain the first application currently being displayed through WMS, and thus obtain the first page data of the first application.

[0034] Secondly, a smart interaction method is provided, comprising: a smart assistant in an electronic device receiving a first voice command from a user, the first voice command carrying first information of a first application and second information of a second application, wherein the first application is the application currently displayed on the electronic device. Based on the first voice command, the smart assistant requests first page data from the first application. Based on the first page data, the smart assistant triggers the second application to execute a first event corresponding to the first voice command; wherein the first event is related to the first page data.

[0035] The first information of the first application carried in the first voice command can be information that directly instructs the first application, or information associated with the first application. The second information of the second application carried in the first voice command can be information that directly instructs the second application, or information associated with the second application.

[0036] In this way, users can control multiple applications through a single voice command. Alternatively, they can control other applications based on the page data of the currently displayed application.

[0037] According to the second aspect, the first page data includes the control tree data of the first page; wherein, the control tree data is transmitted between the first application and the smart assistant via inter-process communication (IPC).

[0038] In this way, the intelligent assistant can directly achieve IPC communication with the first application without having to go through the accessibility service, thus improving the efficiency of intelligent interaction.

[0039] According to the second aspect, or any implementation of the second aspect above, the smart assistant triggers the second application to execute the first event corresponding to the first voice command based on the data on the first page, including: the smart assistant instructing the second application to execute the first event in the background based on the data on the first page; or, the smart assistant responds to the first voice command and triggers the electronic device to display the second application. The smart assistant instructs the second application to execute the first event based on the data on the first page.

[0040] According to the second aspect, or any implementation of the second aspect above, the smart assistant triggers the second application to execute the first event based on the data from the first page, including: the smart assistant requesting the second application to obtain the data from the second page; and the smart assistant sending a second operation instruction to the second application based on the data from the first page and the data from the second page, thereby triggering the second application to execute the first event.

[0041] According to the second aspect, or any implementation of the second aspect above, the method further includes: in response to a page change in the first application, the smart assistant obtains the updated first page data of the first application.

[0042] According to the second aspect, or any implementation of the second aspect above, the method further includes: if the user's second voice command is not received within a preset time, the smart assistant stops obtaining data from the first page from the first application.

[0043] According to the second aspect, or any implementation of the second aspect above, before the smart assistant requests the first page data from the first application based on the first voice command, the method further includes: the smart assistant requests the currently displayed first window from the window management system (WMS) based on the first voice command, the first window being used to display the first application. Requesting the first page data from the first application includes: the smart assistant obtaining the first page data through the interface of the first window.

[0044] Thirdly, a smart interaction method is provided, applied to an electronic device. The electronic device includes a smart assistant and a first application. The method includes: the smart assistant receiving a first voice command from a user, the first voice command carrying information about the first application, the first application being the application currently displayed on the electronic device. Based on the first voice command, the smart assistant sends a first page data retrieval request to the first application. In response to the received first page data retrieval request, the first application sends first page data to the smart assistant. Based on the retrieved first page data, the smart assistant triggers the first application to execute a first event corresponding to the first voice command.

[0045] In this way, by requesting data from the first page, IPC communication between the smart assistant and the first application is achieved, enabling the acquisition of first-page data that is more suitable for voice command execution and avoiding the acquisition of redundant information. Furthermore, the smart interaction process does not rely on accessibility services, avoiding unnecessary memory and power consumption and improving the efficiency of smart interaction.

[0046] According to the third aspect, the first page data includes the control tree data of the first page; wherein, the control tree data is transmitted between the first application and the smart assistant via inter-process communication (IPC).

[0047] According to the third aspect, or any implementation of the third aspect above, the intelligent assistant triggers the first application to execute the first event corresponding to the first voice command based on the acquired first page data, including: the intelligent assistant acquiring a first control in the first application indicated by the first voice command based on the acquired first page data; the intelligent assistant sending a first operation instruction to the first application; and in response to the first operation instruction, the first application operating the first control to execute the first event corresponding to the first voice command.

[0048] According to the third aspect, or any implementation of the third aspect above, the method further includes: an electronic device displaying a page of a second application; a smart assistant receiving a second voice command from a user, wherein the second voice command carries information about a third application; the smart assistant sending a second page data retrieval request to the second application based on the second voice command; in response to the received second page data retrieval request, the second application sending the currently displayed second page data to the smart assistant; and the smart assistant triggering the third application to execute a second event based on the second page data.

[0049] In this way, the smart assistant can also obtain data from the second page through IPC communication with the second application, thereby improving the efficiency of smart interaction.

[0050] According to the third aspect, or any implementation of the third aspect above, the smart assistant triggers the third application to execute a second event based on the data from the second page, including: the smart assistant sending a first execution command to the third application based on the data from the second page; and the third application executing the second event in the background in response to the received first execution command. Alternatively, the smart assistant triggers the electronic device to display the third application in response to a second voice command. The smart assistant then sends a second execution command to the third application based on the data from the second page. The third application executing the second event in response to the received second execution command.

[0051] According to the third aspect, or any implementation of the third aspect above, the smart assistant sends a second execution command to the third application based on the second page data; in response to the received second execution command, the third application executes a second event, including: the smart assistant sending a third page data retrieval request to the third application; in response to the received third page data retrieval request, the third application sending the currently displayed third page data to the smart assistant; the smart assistant sending a second operation instruction to the third application based on the second page data and the third page data; and in response to the second operation instruction, the third application executes the second event.

[0052] In this way, the intelligent assistant can also obtain data from the third page through IPC communication with the third application, thereby improving the efficiency of intelligent interaction.

[0053] According to the third aspect, or any of the above implementations of the third aspect, the method further includes: in response to page changes in the first application, the smart assistant obtains the updated first page data of the first application.

[0054] According to the third aspect, or any implementation of the third aspect above, the method further includes: if no third voice command from the user is received within a preset time, the smart assistant stops obtaining data from the first page from the first application.

[0055] According to the third aspect, or any implementation of the third aspect above, the electronic device further includes a window management system (WMS). The smart assistant sends a first page data retrieval request to the first application based on a first voice command. In response to the received first page data retrieval request, the first application sends the currently displayed first page data to the smart assistant, including: the smart assistant requests the WMS to retrieve the currently displayed first window based on the first voice command; the first window is used to display the first application. The smart assistant retrieves the first page data of the first application through the interface of the first window.

[0056] For the technical effects of the third aspect and any of its implementation methods, please refer to the technical effects of the first aspect and any of its implementation methods mentioned above, which will not be repeated here.

[0057] Fourthly, a smart interaction method is provided, applied to an electronic device. The electronic device includes a smart assistant, a first application, and a second application. The method includes: the smart assistant receiving a first voice command from a user, the first voice command carrying first information of the first application and second information of the second application, the first application being the application currently displayed on the electronic device. Based on the first voice command, the smart assistant sends a first page data retrieval request to the first application. In response to the received first page data retrieval request, the first application sends the currently displayed first page data to the smart assistant. Based on the retrieved first page data, the smart assistant triggers the second application to execute a first event corresponding to the first voice command, wherein the first event is related to the first page data.

[0058] In this way, by requesting data from the first page, IPC communication between the smart assistant and the first application is achieved, enabling the acquisition of first-page data that is more suitable for voice command execution and avoiding the acquisition of redundant information. Furthermore, the smart interaction process does not rely on accessibility services, avoiding unnecessary memory and power consumption and improving the efficiency of smart interaction.

[0059] Furthermore, when voice commands involve multiple applications, the smart assistant can also trigger the corresponding application execution events, thereby improving communication efficiency.

[0060] According to the fourth aspect, the first page data includes the control tree data of the first page; wherein, the control tree data is transmitted between the first application and the smart assistant through inter-process communication (IPC).

[0061] According to the fourth aspect, or any implementation of the fourth aspect above, the smart assistant triggers the second application to execute the first event corresponding to the first voice command based on the acquired first page data, including: the smart assistant sending a first execution command to the second application based on the first page data; and the second application executing the first event in the background in response to the received first execution command. Alternatively, the smart assistant, in response to the first voice command, triggers the electronic device to display the second application. The smart assistant then sends a second execution command to the second application based on the first page data. The second application executing the first event in response to the received second execution command.

[0062] According to the fourth aspect, or any implementation of the fourth aspect above, the smart assistant triggers the second application to execute the first event based on the data from the first page, including: the smart assistant sending a request to the second application to retrieve the second page data; in response to the received request, the second application sending the currently displayed second page data to the smart assistant; the smart assistant sending a second operation instruction to the second application based on the first page data and the second page data; and in response to the second operation instruction, the second application executing the first event.

[0063] According to the fourth aspect, or any of the above implementations of the fourth aspect, the method further includes: in response to page changes in the first application, the smart assistant obtains the updated first page data of the first application.

[0064] According to the fourth aspect, or any implementation of the fourth aspect above, the method further includes: if the user's second voice command is not received within a preset time, the smart assistant stops obtaining data from the first page from the first application.

[0065] According to the fourth aspect, or any implementation of the fourth aspect above, before the smart assistant requests the first page data from the first application based on the first voice command, the method further includes: the smart assistant requests the currently displayed first window from the window management system (WMS) based on the first voice command, the first window being used to display the first application. Requesting the first page data from the first application includes: the smart assistant obtaining the first page data through the interface of the first window.

[0066] For the technical effects of the fourth aspect and any of its implementation methods, please refer to the technical effects of the second aspect and any of its implementation methods mentioned above, which will not be repeated here.

[0067] Fifthly, an electronic device is provided. The electronic device includes a processor and a memory. The memory is coupled to the processor and stores computer program code, including computer instructions. When the processor reads the computer instructions from the memory, the electronic device executes the following: A smart assistant in the electronic device receives a first voice command from a user, the first voice command carrying information about a first application, which is the application currently displayed on the electronic device. Based on the first voice command, the smart assistant requests first page data from the first application. Based on the obtained first page data, the smart assistant triggers the first application to execute a first event corresponding to the first voice command.

[0068] According to the fifth aspect, the first page data includes the control tree data of the first page; wherein, the control tree data is transmitted between the first application and the smart assistant via inter-process communication (IPC).

[0069] According to the fifth aspect, or any implementation of the fifth aspect above, the intelligent assistant triggers the first application to execute the first event corresponding to the first voice command based on the acquired first page data, including: the intelligent assistant acquiring the first control in the first application indicated by the first voice command based on the acquired first page data; the intelligent assistant sending a first operation instruction to the first application, the first operation instruction triggering the first application to operate the first control to execute the first event corresponding to the first voice command.

[0070] According to the fifth aspect, or any implementation of the fifth aspect above, when the processor reads computer instructions from memory, it also causes the electronic device to execute: the electronic device displays a page of a second application; the intelligent assistant receives a second voice command from the user, wherein the second voice command carries information about a third application; the intelligent assistant obtains second page data of the second application based on the second voice command; the intelligent assistant triggers the third application to execute a second event based on the second page data, the second event being related to the second page data.

[0071] According to the fifth aspect, or any implementation of the fifth aspect above, the smart assistant triggers the third application to execute the second event based on the data on the second page, including: the smart assistant instructing the third application to execute the second event in the background based on the data on the second page; or, the smart assistant responds to the second voice command and triggers the electronic device to display the third application. The smart assistant instructs the third application to execute the second event based on the data on the second page.

[0072] According to the fifth aspect, or any implementation of the fifth aspect above, the smart assistant triggers the third application to execute a second event based on the data on the second page, including: the smart assistant requesting the third application to obtain the data on the third page; and the smart assistant sending a second operation instruction to the third application based on the data on the second page and the data on the third page, thereby triggering the third application to execute the second event.

[0073] According to the fifth aspect, or any implementation of the fifth aspect above, when the processor reads computer instructions from memory, it also causes the electronic device to perform: in response to a page change in the first application, the intelligent assistant obtains the updated first page data of the first application.

[0074] According to the fifth aspect, or any implementation of the fifth aspect above, when the processor reads computer instructions from memory, it also causes the electronic device to execute: if a third voice command from the user is not received within a preset time, the smart assistant stops obtaining data from the first page of the first application.

[0075] According to the fifth aspect, or any implementation of the fifth aspect above, when the processor reads computer instructions from memory, it also causes the electronic device to execute: The intelligent assistant, based on a first voice command, requests the window management system (WMS) to obtain the currently displayed first window, which is used to display a first application. Requesting the first application to obtain first page data includes: the intelligent assistant obtaining the first page data through the interface of the first window.

[0076] Sixthly, an electronic device is provided. The electronic device includes a processor and a memory. The memory is coupled to the processor and stores computer program code, including computer instructions. When the processor reads the computer instructions from the memory, the electronic device executes the following: A smart assistant in the electronic device receives a first voice command from a user. The first voice command carries first information of a first application and second information of a second application, wherein the first application is the application currently displayed on the electronic device. Based on the first voice command, the smart assistant requests first page data from the first application. Based on the first page data, the smart assistant triggers the second application to execute a first event corresponding to the first voice command; wherein the first event is related to the first page data.

[0077] According to the sixth aspect, the first page data includes the control tree data of the first page; wherein, the control tree data is transmitted between the first application and the smart assistant via inter-process communication (IPC).

[0078] According to the sixth aspect, or any implementation of the sixth aspect above, the smart assistant triggers the second application to execute the first event corresponding to the first voice command based on the data on the first page, including: the smart assistant instructing the second application to execute the first event in the background based on the data on the first page; or, the smart assistant responding to the first voice command triggers the electronic device to display the second application. The smart assistant instructs the second application to execute the first event based on the data on the first page.

[0079] According to the sixth aspect, or any implementation thereof, the smart assistant triggers the second application to execute the first event based on the data from the first page, including: the smart assistant requesting the second application to obtain the data from the second page; and the smart assistant sending a second operation instruction to the second application based on the data from the first page and the data from the second page, thereby triggering the second application to execute the first event.

[0080] According to the sixth aspect, or any implementation of the sixth aspect above, when the processor reads computer instructions from memory, it also causes the electronic device to perform: in response to a page change in the first application, the intelligent assistant obtains the updated first page data of the first application.

[0081] According to the sixth aspect, or any implementation of the sixth aspect above, when the processor reads computer instructions from memory, it also causes the electronic device to execute: if a second voice command from the user is not received within a preset time, the smart assistant stops retrieving data from the first page of the first application.

[0082] According to the sixth aspect, or any implementation thereof, when the processor reads computer instructions from memory, it further causes the electronic device to execute: The intelligent assistant, based on a first voice command, requests the window management system (WMS) to obtain the currently displayed first window, which is used to display a first application. Requesting the first application to obtain first page data includes: the intelligent assistant obtaining the first page data through the interface of the first window.

[0083] A seventh aspect provides an electronic device. The electronic device includes a processor and a memory. The memory is coupled to the processor and is used to store computer program code, including computer instructions. When the processor reads the computer instructions from the memory, the electronic device causes it to perform the method of the third aspect or any embodiment of the third aspect.

[0084] Eighthly, an electronic device is provided. The electronic device includes a processor and a memory. The memory is coupled to the processor and is used to store computer program code, including computer instructions. When the processor reads the computer instructions from the memory, the electronic device causes it to perform the method of the fourth aspect or any embodiment of the fourth aspect.

[0085] A ninth aspect provides an electronic device that performs the function of the method described in the first aspect and any of its possible implementations; or, the electronic device performs the function of the method described in the second aspect and any of its possible implementations; or, the electronic device performs the function of the method described in the third aspect and any of its possible implementations; or, the electronic device performs the function of the method described in the fourth aspect and any of its possible implementations. This function can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-described functions.

[0086] A tenth aspect is provided: a computer-readable storage medium storing a computer program (also referred to as instructions or code) that, when executed by an electronic device, causes the electronic device to perform the method of the first aspect or any embodiment thereof; or causes the electronic device to perform the method of the second aspect or any embodiment thereof; or causes the electronic device to perform the method of the third aspect or any embodiment thereof; or causes the electronic device to perform the method of the fourth aspect or any embodiment thereof.

[0087] Eleventhly, a computer program product is provided, which, when run on an electronic device, causes the electronic device to perform the method of the first aspect or any one of the embodiments of the first aspect; or causes the electronic device to perform the method of the second aspect or any one of the embodiments of the second aspect; or causes the electronic device to perform the method of the third aspect or any one of the embodiments of the third aspect; or causes the electronic device to perform the method of the fourth aspect or any one of the embodiments of the fourth aspect.

[0088] In a twelfth aspect, a circuit system is provided, the circuit system including a processing circuit configured to perform the method of the first aspect or any embodiment of the first aspect; or, the processing circuit configured to perform the method of the second aspect or any embodiment of the second aspect; or, the processing circuit configured to perform the method of the third aspect or any embodiment of the third aspect; or, the processing circuit configured to perform the method of the fourth aspect or any embodiment of the fourth aspect.

[0089] In a thirteenth aspect, a chip system is provided, including at least one processor and at least one interface circuit, wherein the at least one interface circuit is configured to perform transceiver functions and send instructions to the at least one processor, wherein when the at least one processor executes the instructions, the at least one processor executes the method of the first aspect or any one of the embodiments of the first aspect; or, the at least one processor executes the method of the second aspect or any one of the embodiments of the second aspect; or, the at least one processor executes the method of the third aspect or any one of the embodiments of the third aspect; or, the at least one processor executes the method of the fourth aspect or any one of the embodiments of the fourth aspect.

[0090] The technical effects of the aforementioned aspects can be referenced from each other, and will not be elaborated further here. Attached Figure Description

[0091] Figure 1 is a schematic diagram of the accessible service interaction scenario provided in an embodiment of this application;

[0092] Figure 2 is a schematic diagram of the interface provided in an embodiment of this application;

[0093] Figure 3 is a schematic diagram of the communication system in which the intelligent interaction method provided in the embodiments of this application is applied;

[0094] Figure 4 is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of this application;

[0095] Figure 5 is a schematic diagram of the software structure of the electronic device provided in the embodiment of this application;

[0096] Figure 6 is a schematic flowchart of the intelligent interaction method provided in an embodiment of this application;

[0097] Figure 7 is a schematic diagram of a scenario where a schedule is added via a smart assistant according to an embodiment of this application;

[0098] Figure 8 is a schematic diagram of inter-module interaction provided in an embodiment of this application;

[0099] Figure 9 is a schematic diagram of inter-module interaction provided in an embodiment of this application;

[0100] Figure 10 is a schematic diagram of a scenario where notes are added via a smart assistant, as provided in an embodiment of this application.

[0101] Figure 11 is a second schematic diagram of a scenario where notes are added via a smart assistant, as provided in an embodiment of this application.

[0102] Figure 12 is a schematic diagram of the intelligent interaction method provided in the embodiment of this application (II).

[0103] Figure 13 is a schematic diagram of the intelligent interaction method provided in the embodiment of this application;

[0104] Figure 14 is a schematic diagram of the structure of the electronic device provided in the embodiment of this application. Detailed Implementation

[0105] The technical solutions of the embodiments of this application are described below with reference to the accompanying drawings. In the description of the embodiments of this application, the terminology used in the following embodiments is for the purpose of describing specific embodiments only and is not intended to be a limitation of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to include expressions such as “one or more,” unless the context clearly indicates otherwise. It should also be understood that in the following embodiments of this application, “at least one” and “one or more” refer to one or more (including two).

[0106] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized. The term "connection" includes direct connections and indirect connections, unless otherwise stated. "First" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated.

[0107] In the embodiments of this application, the words "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of the words "exemplarily" or "for example" is intended to present the relevant concepts in a specific manner.

[0108] In some embodiments, the smart assistant can enable intelligent interaction between the user interface (UI) and the electronic device. For example, the smart assistant can receive voice commands from the user instructing the user to operate the electronic device. Based on these voice commands, the smart assistant can invoke accessibility services to obtain interface information of the current UI. In this way, the smart assistant can obtain the controls displayed on the UI interface indicated by the voice command, and trigger the application to execute the corresponding events of those controls through the accessibility service. In some examples, the electronic device is a smart screen, smart cockpit, or mobile phone. In some examples, the smart assistant is a smart assistant application installed on the electronic device, such as Xiaoyi (Xiaoyi's AI assistant).

[0109] In some examples, in response to a user's voice command, as shown in Figure 1, the smart assistant requests the accessibility service to take a screenshot of the current UI. The smart assistant then obtains the screenshot and processes it using image analysis technology to extract interface information. In some examples, this interface information includes information about the controls displayed on the current screen, such as their position and size. Then, as shown in Figure 1, the smart assistant uses this interface information to obtain the location information of the control indicated by the user's voice command and sends this location information and the operation method to the accessibility service. In this way, the accessibility service can instruct the application to perform an operation on the corresponding control.

[0110] For example, as shown in Figure 2, when the electronic device is displaying a chat interface, it receives a user's voice command "Send XXX message". In response to this voice command, the smart assistant obtains a screenshot of the currently displayed interface through accessibility services and uses image analysis technology to obtain the location information of the input box 21 displayed on the current interface. Then, the smart assistant can send this location information and the input content indicated by the voice command to the accessibility service. In this way, the accessibility service can instruct the chat application to input and display the required input content in the input box 21 and send the input content.

[0111] In other examples, in response to a user's voice command, as shown in Figure 1, the intelligent assistant requests accessibility service information about the currently displayed interface. The multiple controls displayed on the interface can form a control tree. A control tree, in graphical user interface programming, is a structured representation used to manage and organize interface elements, such as controls. Control trees are typically presented in a tree structure, with the root node representing the entire application or window, and child nodes representing specific interface controls, such as buttons, text boxes, and lists. The accessibility service can view all accessibility nodes, such as controls, on the current interface through code. For example, the accessibility service can obtain the root node information in the currently focused window using the code "AccessibilityNodeInforootNode = info.getRootInActivewindow()", traverse the node tree using the code "traverseNode(rootNode)", output the node information using the code "private void traverseNode(AccessibilityNodeInfo node)", and traverse child nodes using the code "traverseNode(childNode)". In this way, the accessibility service can output interface information. Based on the interface information and voice commands, the intelligent assistant can instruct the application corresponding to the currently focused window to execute the corresponding voice commands through accessibility services.

[0112] It can be seen that current intelligent interaction processes rely on accessibility services. These accessibility services primarily provide functions for people with disabilities, and their functionality does not fully match that of intelligent assistants. Therefore, during intelligent interaction, the information provided by accessibility services to the intelligent assistant cannot be customized, resulting in the transmission of some unnecessary information, consuming power and memory, and accessibility services may also be unable to provide some information needed by the intelligent assistant.

[0113] It should be understood that there is inter-process communication (IPC) between intelligent assistants, accessibility services, and applications. Data transmission via IPC involves long links, high power consumption, and low efficiency. Identifying interface information through screenshots also suffers from significant image recognition overhead.

[0114] Furthermore, for some privacy-sensitive interfaces, current electronic devices do not allow screenshots to obtain private data, or screenshots may obscure the private data. Therefore, smart assistants cannot provide a good intelligent interactive experience for users in these scenarios.

[0115] Therefore, this application provides an intelligent interaction method in which the intelligent assistant can directly interact with the application, triggering the application to execute voice commands based on the acquired interface information, simplifying the interaction process and improving communication efficiency. Furthermore, the direct interaction between the intelligent assistant and the application eliminates redundant information exchange, reducing data transmission power consumption and lowering the development difficulty of the intelligent assistant.

[0116] Figure 3 is a schematic diagram of a communication system for the intelligent interaction method provided in this application embodiment. As shown in Figure 3, the communication system includes an electronic device 100 and a server 200.

[0117] In some embodiments, the electronic device 100 may be a smart screen, smart cockpit, mobile phone, tablet computer, personal computer (PC), ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), wearable device, artificial intelligence (AI) device, or other terminal device. The operating system installed on the electronic device 100 may include, but is not limited to, those that allow for the installation of a smart screen, smart cockpit, mobile phone, tablet computer, personal computer (PC), ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), wearable device, artificial intelligence (AI) device, etc. Alternatively, other operating systems may be used. This application does not limit the specific type of electronic device 100 or the operating system installed on it.

[0118] Optionally, server 200 can be a device or server with computing capabilities, such as a cloud server or a network server. Server 200 can be a single server, a server cluster consisting of multiple servers, or a cloud computing service center.

[0119] In some embodiments, the electronic device 100 is equipped with a smart assistant, and the server 200 is a smart assistant server used to support the cloud computing function of the smart assistant.

[0120] For example, Figure 4 shows a schematic diagram of a hardware structure of an electronic device 100.

[0121] Electronic device 100 may include processor 110, external memory interface 120, internal memory 121, universal serial bus (USB) interface 130, charging management module 140, power management module 141, battery 142, antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, audio module 170, sensor module 180, button 190, motor 191, indicator 192, camera 193, display screen 194, and subscriber identification module (SIM) card interface 195, etc.

[0122] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0123] Processor 110 may include one or more processing units, such as application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU). These different processing units may be independent devices or integrated into one or more processors.

[0124] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0125] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI) and a display serial interface (DSI). In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to enable the electronic device 100 to capture images. The processor 110 and the display screen 194 communicate via the DSI interface to enable the electronic device 100 to display images.

[0126] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with tuning switches.

[0127] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0128] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel can be manufactured using a liquid crystal display (LCD), such as an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a mini-LED, a micro-LED, a micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 100 may include one or N displays 194, where N is a positive integer greater than 1.

[0129] The audio module 170 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 170 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 170 may be located in the processor 110, or some functional modules of the audio module 170 may be located in the processor 110. In some embodiments, the audio module 170 may include, for example, a speaker, a receiver, a microphone, a headphone jack, etc.

[0130] The software system of electronic device 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses a layered operating system as an example to exemplify the software structure of electronic device 100.

[0131] Figure 5 is a software structure block diagram of an electronic device 100 according to an embodiment of this application.

[0132] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the operating system is divided into four layers, from top to bottom: the application layer, the application framework layer, the system libraries, and the kernel layer.

[0133] The application layer can include a series of application packages.

[0134] As shown in Figure 5, the application package may include applications such as smart assistant, chat, calendar, notes, contacts, gallery, map, call, and video.

[0135] The intelligent assistant is used to enable intelligent interaction between the user and the electronic device. For example, the intelligent assistant can enable the user to interact with the UI interface of the electronic device 100 through voice, thereby controlling the electronic device 100.

[0136] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0137] As shown in Figure 5, the application framework layer may include a window management system (WMS), a content provider, a view system, a phone manager, a resource manager, a notification manager, etc.

[0138] A window management system is used to manage window programs. It can obtain the screen size, determine if a status bar is present, lock the screen, and capture the screen, among other things.

[0139] Content providers store and retrieve data, making that data accessible to applications. This data may include videos, images, audio, made and received phone calls, browsing history and bookmarks, phone books, etc.

[0140] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon could include views for displaying text and views for displaying images.

[0141] The phone manager is used to provide communication functions for electronic device 100. For example, it manages call status (including connection and disconnection).

[0142] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and more.

[0143] The notification manager allows applications to display notifications in the status bar. These notifications can be used to convey informational messages and can disappear automatically after a short pause, requiring no user interaction. The notification manager can also display notifications as icons or scrollbar text in the top status bar, such as notifications from background applications, or as dialog boxes on the screen.

[0144] System libraries can include multiple functional modules. For example: surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), etc.

[0145] The Surface Manager is used to manage the display subsystem and provides the blending of 2D and 3D layers for multiple applications.

[0146] The media library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, and AMR.

[0147] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0148] A 2D graphics engine is a graphics engine for 2D drawing.

[0149] The kernel layer is the layer between hardware and software. The kernel layer contains at least the display driver, camera driver, audio driver, and sensor driver.

[0150] Figure 6 is a flowchart illustrating an intelligent interaction method provided in an embodiment of this application. It should be noted that this method is not limited to the specific order described in Figure 6 and below. It should be understood that in other embodiments, the order of some steps in this method can be interchanged according to actual needs, or some steps can be omitted or deleted. The method includes the following steps:

[0151] S601, the intelligent assistant receives the user's voice commands.

[0152] In some embodiments, the electronic device receives user voice through a voice system and sends the user voice to a smart assistant. For example, the user voice may include the user's voice commands. For instance, as shown in Figure 4, the electronic device captures the user's voice commands through an audio module and sends the voice commands to the smart assistant.

[0153] S602, the smart assistant sends a voice command recognition request to the server.

[0154] In some embodiments, in response to a received voice command, the smart assistant sends a voice command recognition request to the server, which requests the server to recognize the voice command received by the smart assistant. In some examples, the voice command recognition request carries the voice command itself.

[0155] For example, the smart assistant sends an automatic speech recognition (ASR) text command to the server, which carries the voice command, thereby triggering the server to convert the voice command into a text command, so that the smart assistant can recognize and execute it.

[0156] S603, The server sends voice commands to the smart assistant and receives a recognition response.

[0157] In some embodiments, after receiving a voice command from a smart assistant, the server can process the voice command using a speech recognition algorithm to obtain the corresponding text command. The server can then send the obtained text command back to the smart assistant. In some examples, the server sends a voice command recognition response to the smart assistant, and includes the text command in that response.

[0158] For example, the voice command recognition response is, for instance, the statement “Directives[]”, through which the server instructs the smart assistant to execute a text command that corresponds to the user’s voice command, thereby realizing the conversion of voice commands.

[0159] For example, if a user's voice command is "Order me some fried chicken," then the aforementioned Directives statement includes two parts: one part instructs the user to open the food delivery app, and the other part instructs the user to retrieve page data. This allows the smart assistant to subsequently open the food delivery app and, by retrieving the app's page data, place an order for fried chicken.

[0160] In other embodiments, the electronic device supports a smart assistant to analyze voice commands on the device side. In this case, steps S602 and S603 are optional. After obtaining the user's voice command, the smart assistant can directly analyze the voice command and obtain the corresponding voice recognition result.

[0161] In some embodiments, the electronic device is equipped with a service based on speech-to-text (STT) technology. The electronic device can also process voice commands through this service and output the processing results to a smart assistant. For example, after receiving a voice command, the electronic device may not directly hand it over to the smart assistant for processing, but instead send the voice command to a service based on STT technology for processing. STT technology is a technique for converting speech into written text.

[0162] S604, the smart assistant obtains the focus window, and the application corresponding to the focus window.

[0163] The focus window refers to the interface element or application window that the user is currently interacting with. Specifically, the focus window is the window through which the operating system receives user input, which may include input via voice, touch, or keyboard.

[0164] In some embodiments, a user's voice commands are typically used to control the application displayed in the currently focused window of an electronic device. The smart assistant, based on the received voice command response from the server, can then determine the application displayed in the focused window and designate that application as the current focus application requiring interaction.

[0165] For example, in the scenario shown in Figure 2, while the electronic device is displaying a chat application, it receives a user's voice command "send XX message". Then, the smart assistant, through interaction with the server, recognizes the response based on the received voice command and determines that the focus application displayed in the focus window is the chat application.

[0166] In other embodiments, the user's voice commands are also used to instruct the electronic device to launch a new application. The smart assistant, based on the received voice command and its response, can launch the application instructed by the user's voice command and display it in the focus window, designating it as the currently active application for interaction.

[0167] For example, if a user's voice command is "Order me some fried chicken," then the smart assistant, through interaction with the server, can identify the response based on the received voice command and determine whether to launch the food delivery app. Once launched, the food delivery app window will be the focus window.

[0168] In some embodiments, a window management system (WMS) is a component of an operating system or graphical user interface used to manage and control the display and interaction of multiple windows on the screen of an electronic device. For example, a WMS can be used to implement functions such as window creation and destruction, focus management, event handling, window layout, and window switching. For instance, a WMS can be used to create new windows and destroy windows that are no longer needed. Each window can correspond to an ability instance, and window changes include changes to the ability instance. Windows include full-screen windows, pop-ups, etc. Furthermore, a WMS can be used to determine which window is currently active and to receive user input (such as voice input events); that is, a WMS can be used to determine the focus window.

[0169] In other examples, for some other operating systems, each window may correspond to an activity instance, or a UIViewController instance.

[0170] For example, after receiving a user's voice command, the smart assistant registers a change event in the focus window with the WMS. This allows the smart assistant to obtain the focus window API from the WMS. Furthermore, after the focus window changes, the smart assistant can also obtain the updated focus window API from the WMS. This enables subsequent interaction between the smart assistant and the focused application via the focus window API.

[0171] In some embodiments, after receiving a user's voice command or after receiving a voice command recognition response, the intelligent assistant may create and display a modal window, which can intuitively demonstrate the voice interaction process to the user.

[0172] For example, as shown in Figure 7(a), during the display of the chat interface on the electronic device, a user voice command is received: "Remember the chat information on the screen and add my speech to the calendar." The smart assistant sends this voice command to the server. In response to the received voice command recognition response from the server, as shown in Figure 7(b), the smart assistant creates and displays a modal window 71, and displays the recognized voice command on the modal window 71. In this way, the user can determine the current voice command recognition status of the electronic device through the content displayed in the modal window 71.

[0173] In some examples, electronic devices display a modal window at the top of the current display interface via pop-ups or other methods for easy user viewing. While the electronic device displays the modal window, this window is not the focus window; the focus window remains the display window of the application corresponding to the user's voice command. For example, in the scenario shown in Figure 7, the focus window is still the chat application window. This facilitates subsequent interaction between the smart assistant and the focus application displayed in the focus window.

[0174] S605, the smart assistant sends a page data retrieval request to the application.

[0175] S606: The application sends page data to the smart assistant and receives a response.

[0176] In some embodiments, the user's intention in issuing voice commands is to interact with the UI of the focused application, such as manipulating the displayed content, including text and images. After identifying the focused application, the smart assistant can send a page data retrieval request to that application. This request retrieves the page data of the currently displayed interface to determine the content the user intends to interact with.

[0177] Correspondingly, after receiving a page data retrieval request from the smart assistant, the focused application can provide the smart assistant with the page data of the currently displayed interface.

[0178] In some examples, the focused application can be the application that provides the focused window. The focused application changes based on the user's actions. For example, if an electronic device displays the chat window of a chat application, the chat application is the focused application; if the electronic device switches from displaying the chat interface of a chat application to displaying the shopping interface of a shopping application in response to the user's switching action, the focused application will also become the shopping application.

[0179] In some examples, the focused application can receive user voice commands, and the page data includes the page data of the interface displayed in the focused window. In other examples, the focused application is an application launched by the intelligent assistant in response to a user voice command, and the page data includes the page data of the interface displayed in the focused window after the application is launched and displayed.

[0180] In some examples, the graphical user interface (GUI) editing process utilizes a control tree to manage and organize the structured representation of interface elements. The control tree organizes controls through parent-child relationships, forming a hierarchical structure. For example, a window control can have multiple child controls, such as menu bars and toolbars. Each control in the control tree can register an event detector. When the user interacts with a control, the event propagates along the tree structure, from child controls to parent controls, thus enabling event responses. Therefore, the page data requested by the intelligent assistant from the application includes control tree data. This control tree data includes, for example, the controls currently displayed, their identifiers, their hierarchical relationships, their display positions, and their display sizes. Subsequently, based on this page data, the intelligent assistant can determine the controls that the user's voice commands instruct, as well as the controls that can respond to voice commands.

[0181] In some embodiments, the control tree can dynamically add, delete, or modify controls, supporting dynamic interface updates. For example, during the display of a focused application, events such as page content switching or the start / end of page animations may occur. The smart assistant can then obtain the latest page data through interaction with the focused application. For instance, the smart assistant can register a page data update event with the focused application. In response to a page data update, the focused application can then send the updated page data back to the smart assistant.

[0182] In some embodiments, during the design of an application's user graphical interface (GUI), developers can use native controls to build the GUI or utilize graphics provided by certain third-party frameworks. These third-party frameworks include, for example, Flutter, Qt, Weex, or other frameworks. Flutter and Qt frameworks are used to build user interfaces and render graphics, each with its own unique rendering implementation and architecture. For instance, Flutter uses the Skia graphics library as its underlying rendering engine. Skia is an open-source 2D graphics library that supports multiple platforms, such as... And the desktop. During application interface display, Flutter draws directly to the screen using Skia, rather than relying on the platform's native controls. Flutter's user interface consists of a tree-structured set of widgets. Similarly, Qt provides Qt Quick, a high-performance graphical user interface framework based on the Qt meta-object language (QML), specifically designed for building user interfaces. Qt Quick, for example, can use OpenGL for hardware-accelerated rendering. Qt Widgets are C++-based user interface components.

[0183] In some examples, the user graphical interface is rendered using native controls and the aforementioned example framework. The control tree data corresponding to this graphical interface includes data from the native controls and / or widgets drawn by third-party frameworks for each node. The focused application can obtain the data of each native control, but may not be able to directly obtain the data of the widgets drawn by third-party frameworks. Therefore, the focused application can obtain a screenshot of the area where the widgets drawn by third-party frameworks are located. In some examples, if the display interface includes third-party controls, such as widgets drawn by third-party frameworks, the page data fed back by the focused application to the smart assistant includes screenshot data corresponding to the display area of ​​the third-party controls. Subsequently, after obtaining the page data, the smart assistant can also obtain information about each control on the current display interface through image data analysis of the screenshot data, thereby determining the content controlled by the user's voice commands.

[0184] S607, the smart assistant sends page data to the server.

[0185] In some embodiments, the smart assistant receives a response based on the received data, which may include page data from the application. The smart assistant can then send this page data to the server. Correspondingly, the server receives the page data sent by the smart assistant, triggering the server to analyze the page data based on previously acquired voice commands.

[0186] S608, The server sends voice command execution instructions to the smart assistant.

[0187] In some embodiments, the server can determine the control to be operated by the user's voice instruction based on the received page data and the text commands previously obtained by analyzing voice commands. This control corresponds to the currently displayed content of the focused application. The server can then generate a voice command execution instruction, which carries the identifier of the control operated by the voice command and the corresponding control command. The server can then send this voice command execution instruction to the smart assistant. The smart assistant can then receive the voice command execution instruction.

[0188] For example, as shown in Figure 7, based on the received page data, the server can determine that the control indicated by the current user's voice command is the text content 72 displayed on the interface, and the control command is to generate a corresponding schedule based on the text content 72 and add the schedule to the calendar application. Then, the server can send a voice command execution instruction carrying the identifier of the text content 72 and the corresponding control command to the smart assistant.

[0189] In other embodiments, the electronic device supports a smart assistant analyzing page data on the device side. In this case, steps S607 and S608 are optional. After obtaining the page data, the smart assistant can directly analyze the page data in conjunction with the text command corresponding to the voice command and obtain the corresponding voice command execution instruction.

[0190] S609, the smart assistant sends operation commands to the application.

[0191] In some embodiments, after receiving a voice command execution instruction, the smart assistant obtains the control and application of the instruction operation and executes the corresponding control command. The application corresponding to the voice command execution instruction can be the currently displayed focused application or other applications.

[0192] In some examples, as shown in Figure 8, the intelligent assistant obtains the focus window for the voice command instruction and the focus application displayed in the focus window through the window management system (WMS). Then, the intelligent assistant obtains the control tree data sent by the focus application and generates corresponding operation instructions, which instruct the focus application to execute the user's voice command.

[0193] For example, as shown in Figure 2, when the electronic device is displaying a chat application, it receives a user's voice command "Send XX message". In response to this voice command, the smart assistant determines that the focus application is the chat application, the control indicated by the voice command is input box 21, and the control command is to enter XX message in input box 21 and send it. Then, the smart assistant can send an operation instruction to the focus application, which is used to instruct the focus application to execute the control command.

[0194] In other examples, as shown in Figure 9, the smart assistant obtains the focus window supported by the voice command and the focus application displayed in the focus window, such as application 1, through the Window Management System (WMS). Then, the smart assistant obtains the control tree data sent by the focus application and generates corresponding operation instructions. These instructions are used to instruct other applications, such as application 2, to execute the user's voice commands. The smart assistant can execute some of the voice commands, while other applications execute the remaining voice commands.

[0195] Optionally, other applications can execute the user's voice commands in the background. Alternatively, other applications can execute the user's voice commands in the foreground, i.e., the electronic device switches another application as the focus application. Optionally, as shown in step S604 above, the smart assistant has registered the focus window change event with the WMS. Then, after the focus window changes, the smart assistant can obtain the updated focus window API fed back by the WMS. In this way, the smart assistant can establish a connection with the new focus application through the updated focus window API, thereby obtaining the page data of the new focus application so that the new focus can execute operation instructions.

[0196] For example, as shown in Figure 7(b), in response to a user's voice command, the smart assistant obtains the text content 72 displayed on the interface as the control indicated by the user's voice command. The control command is to generate a corresponding schedule based on the text content 72 and add the schedule to the calendar application. The smart assistant can then obtain the text content 72 based on the obtained page data and determine the text content required to add the schedule, such as the meeting topic and meeting time. Afterward, the smart assistant can instruct the launch of the calendar application; for example, the smart assistant can send a calendar application launch request to the desktop to launch the calendar application. After the calendar application is launched, the smart assistant can send an operation instruction to the calendar application, carrying the obtained text content required to add the schedule in the operation instruction. Then, in response to this operation instruction, as shown in Figure 7(c), the calendar application can add the schedule on the corresponding date. The calendar application can be launched in the background, meaning it is not visible to the user during the schedule addition process. Subsequently, after the schedule is added, the smart assistant can obtain the schedule addition result. As shown in Figure 7(b), the smart assistant can display the schedule addition results in the modal window 71, such as the calendar has been added and the specific schedule content added, which helps users understand the execution results of voice commands.

[0197] For example, as shown in Figure 10, while the electronic device is displaying a chat application, it receives the user's voice command "take notes" through the voice system. After receiving this voice command, the smart assistant can identify the currently focused application, such as the chat application. The smart assistant can then establish a connection with the chat application and obtain the control tree data provided by the chat application. Based on the voice command, the smart assistant determines that the user intends to add the chat application's display content, such as chat content, to the note application. The smart assistant can then trigger the launch of the note application to add the note content. After obtaining the chat content, the smart assistant needs to determine the insertion position of the chat content within the note application. After the note application is launched, the electronic device displays the note application in the focused window, the smart assistant establishes a connection with the note application, obtains the control tree data provided by the note application to determine the insertion position of the chat content, and sends corresponding operation instructions to the note application. For example, as shown in Figure 11(a), the smart assistant obtains the chat content 111. As shown in Figure 11(b), the note application is, for example, a memo application, and the focused window displays the memo application's homepage. The smart assistant then obtains the identifier of the add control 112, determines the control command as creating a new memo, and adds the memo content as chat content 111. Afterward, the smart assistant can send corresponding operation instructions to the memo application, instructing it to operate the add control 112 to create a new memo and add the memo content as chat content 111 to the newly created memo. As shown in Figure 11(c), after adding the memo content, the memo application can switch the corresponding display page and send the switched page data to the smart assistant so that the smart assistant can determine the execution status of the operation instructions. A modal window corresponding to the smart assistant can also be displayed in Figures 11(a), (b), or (c), but is not shown in the current figure.

[0198] In some embodiments, within a preset time range after receiving the aforementioned voice command, the smart assistant may also receive new voice commands from the user. Then, the smart assistant can repeatedly execute steps S601-S609 to respond to the user's voice commands. The preset time range may be, for example, 20 seconds, 30 seconds, etc.

[0199] In some embodiments, the smart assistant may automatically exit if it does not receive any more user voice commands within a preset time. Alternatively, the smart assistant may exit if it detects a user-instructed exit action, such as a voice command to exit. After exiting, the smart assistant stops acquiring events related to window changes and page data updates.

[0200] In this way, the intelligent assistant can interact directly with the application, obtaining page data that is more suitable for voice command execution and avoiding the acquisition of redundant information. Furthermore, the intelligent interaction process does not rely on accessibility services, avoiding unnecessary memory and power consumption and improving the efficiency of intelligent interaction.

[0201] Figure 12 is a flowchart illustrating another intelligent interaction method provided in an embodiment of this application. It should be noted that this method is not limited to the specific order described in Figure 12 and below. It should be understood that in other embodiments, the order of some steps in this method can be interchanged according to actual needs, or some steps can be omitted or deleted. The method includes the following steps:

[0202] S1201. The smart assistant in the electronic device receives a first voice command from the user. The first voice command carries information about a first application, which is the application currently displayed on the electronic device.

[0203] For example, the first application is the focus application, and the user can control the first application through a first voice command. In this way, even when it is inconvenient for the user to directly operate the electronic device's display screen, they can still control the first application through the first voice command.

[0204] In some embodiments, the information of the first application carried in the first voice command may be information that directly indicates the first application, or information associated with the first application.

[0205] For example, the first voice command might be "Order me some fried chicken using a food delivery app," which directly includes the name of the food delivery app. Another example is "Order me some fried chicken," which doesn't directly include the name of the food delivery app, but only food delivery apps can perform the corresponding restaurant reservation function; that is, the first voice command carries information related to the food delivery app.

[0206] Optionally, other aspects of step S1201 can be found in the relevant content of steps S601-S603 in the above embodiments.

[0207] S1202: The smart assistant requests data from the first page from the first application based on the first voice command.

[0208] In some embodiments, the smart assistant sends the received first voice command to a server, which analyzes the first voice command to obtain the first application controlled by the first voice command. Alternatively, the smart assistant analyzes the first voice command on the device side to obtain the first application controlled by the first voice command.

[0209] In some examples, after identifying a first application, the smart assistant may request page data from that application. For instance, based on a first voice command, the smart assistant sends a request to the first application to retrieve first page data. In response to the received request, the first application sends the first page data to the smart assistant. This first page data may be, for example, the page data currently being displayed by the first application.

[0210] In some examples, the first page data includes the control tree data for the first page. This control tree data is transmitted between the first application and the smart assistant via IPC communication.

[0211] In this way, the intelligent assistant can directly achieve IPC communication with the first application without having to go through the accessibility service, thus improving the efficiency of intelligent interaction.

[0212] In some embodiments, a typical user controls the currently displayed first application or instructs the user to launch the first application via voice commands, thereby gaining control over the first application. Upon receiving the first voice command, the smart assistant needs to determine the currently displayed first application and the interaction method with it.

[0213] In some examples, the electronic device is equipped with a WMS (Windows Management System) to manage and control the display and interaction of multiple windows on the screen. The smart assistant, based on a first voice command, requests the WMS to retrieve the currently displayed first window, which is used to display the first application. This first window is, for example, the focus window. The WMS can provide the smart assistant with the interface of the first window, through which the smart assistant retrieves the first page data.

[0214] In some examples, the application that a user wants to control via the first voice command is the focused application, not other non-focused applications. Therefore, when an electronic device displays multiple applications, each application is shown in its own window, with only one window being the focus window and the others being non-focused. The smart assistant can determine the focus window through the WMS (Windows Management System), thus identifying the first application displayed in the focus window that the user currently wants to control via voice.

[0215] Optionally, other aspects of step S1202 can be found in the relevant content of steps S604-S606 in the above embodiments.

[0216] S1203: The smart assistant triggers the first event corresponding to the first voice command executed by the first application based on the data obtained from the first page.

[0217] In some embodiments, after obtaining the data of the first page, the smart assistant can combine the analysis results of the first voice command to determine the first event that the first voice command instructs the first application to perform, thereby triggering the first application to perform the first event.

[0218] In some examples, the smart assistant triggers a first application to execute a first event corresponding to a first voice command based on the acquired first page data. This includes: the smart assistant acquiring a first control in the first application indicated by the first voice command based on the acquired first page data; the smart assistant sending a first operation instruction to the first application, which triggers the first application to operate the first control to execute the first event corresponding to the first voice command.

[0219] For example, as shown in Figure 2, when the electronic device is displaying a chat application (such as the first application), the smart assistant receives a first voice command from the user: "Send XX message". In response to this first voice command, the smart assistant requests first page data from the chat application. Based on this first page data, the smart assistant can determine that the first control indicated by the first voice command is input box 21, and the control command is to input XX message in input box 21 and send it. Then, the smart assistant can send an operation instruction to the chat application, which instructs the chat application to execute the control command. This control command, for example, is a command to implement a first event.

[0220] In this way, the intelligent assistant can interact directly with the application, obtaining page data that is more suitable for voice command execution and avoiding the acquisition of redundant information. Furthermore, the intelligent interaction process does not rely on accessibility services, avoiding unnecessary memory and power consumption and improving the efficiency of intelligent interaction.

[0221] In some embodiments, the electronic device displays a page of a second application. A smart assistant receives a second voice command from the user. This second voice command carries information about a third application. Based on the second voice command, the smart assistant retrieves second page data from the second application. Based on the second page data, the smart assistant triggers the third application to execute a second event.

[0222] In some examples, the smart assistant retrieves second page data of a second application based on a second voice command, including: the smart assistant sending a second page data retrieval request to the second application based on the second voice command; and in response to the received second page data retrieval request, the second application sending the currently displayed second page data to the smart assistant.

[0223] In this way, the smart assistant can obtain the second page data of the second application through IPC communication with the currently displayed second application.

[0224] In some examples, before the third application executes the second event, the third application is in any of the following states: not started, started but running in the background, or started but not displayed in the focused window.

[0225] For example, the second voice command is "Remember the chat information on the screen and add my speech to the corresponding schedule in the calendar application." This second voice command directly carries information from the calendar application (such as a third application).

[0226] For example, the second voice command is "Remember the chat information on the screen and add my speech to the calendar." It can be seen that this second voice command carries information about the third application, but the calendar application can be identified based on "calendar," thus identifying the third application.

[0227] Thus, when voice commands involve multiple applications, the smart assistant can also achieve direct IPC communication with multiple applications, triggering corresponding application execution events, thereby improving communication efficiency.

[0228] In some examples, the smart assistant triggers a third application to perform a second event based on data from the second page, including: the smart assistant instructs the third application to perform the second event in the background based on data from the second page.

[0229] For example, based on the data on the second page, the smart assistant sends a first execution command to the third application. In response to the received first execution command, the third application executes a second event in the background.

[0230] For example, as shown in Figure 7(b), in response to a user's voice command, the smart assistant obtains the text content 72 displayed on the interface as the control indicated by the user's voice command. The control command is to generate a corresponding schedule based on the text content 72 and add the schedule to the calendar application. The smart assistant can then obtain the text content 72 based on the obtained page data and determine the text content required to add the schedule, such as the meeting topic and meeting time. Afterward, the smart assistant can instruct the calendar application to launch in the background; for example, the smart assistant can send a calendar application launch request to the desktop to launch the calendar application via the desktop. After the calendar application launches, the smart assistant can send an operation instruction (such as a first execution command) to the calendar application, carrying the obtained text content required to add the schedule in the operation instruction. Then, in response to this operation instruction, as shown in Figure 7(c), the calendar application can add the schedule on the corresponding date. The calendar application can be launched in the background, meaning that the user cannot see the schedule being added.

[0231] In this way, the intelligent assistant obtains the page data of the currently displayed application based on voice commands, and triggers other applications to execute corresponding events in the background based on the page data. Through direct interaction between applications, a highly efficient intelligent interaction process is achieved.

[0232] In other examples, the smart assistant triggers a third application to perform a second event based on data from the second page. This includes the smart assistant responding to a second voice command by triggering an electronic device to display a third application. The smart assistant instructs the third application to perform a second event related to the second page data.

[0233] For example, the smart assistant sends a second execution command to a third application based on data from the second page. In response to the received second execution command, the third application executes a second event.

[0234] In some examples, the smart assistant requests data from a third-party application, specifically a third-page application. Based on the data from the second and third pages, the smart assistant sends a second action command to the third application, triggering the third application to execute a second event.

[0235] For example, the smart assistant sends a request to a third application to retrieve data from a third page. In response to the received request, the third application sends the currently displayed data from the third page to the smart assistant. Then, based on the data from the second and third pages, the smart assistant sends a second operation command to the third application. In response to the second operation command, the third application executes a second event.

[0236] For example, as shown in Figure 10, the smart assistant determines, based on the voice command, that the user intends to add content from the chat application, such as chat content, to the note-taking application. The smart assistant can then trigger the launch of the note-taking application to add the note content. After obtaining the chat content (such as data from the second page), the smart assistant needs to determine the insertion position of the chat content within the note-taking application. After the note-taking application launches, the electronic device displays the note-taking application, and the window displaying the note-taking application is the focus window. Subsequently, the smart assistant can obtain the control tree data provided by the note-taking application (such as data from the third page) to determine the insertion position of the chat content within the note-taking application and send the corresponding operation command to the note-taking application. For example, as shown in Figure 11(a), the smart assistant obtains chat content 111. As shown in Figure 11(b), the note-taking application is, for example, a memo application, and the focus window displays the memo application's homepage. The smart assistant then obtains the identifier of the add control 112, determines the control command to be "create memo," and adds the memo content as chat content 111. Afterwards, the smart assistant can send corresponding operation instructions to the memo application, instructing the memo application to add control 112 to create a new memo, and add memo content as chat content 111 to the newly created memo. As shown in Figure 11(c), after adding memo content, the memo application can switch the corresponding display page.

[0237] In this way, the intelligent assistant obtains the page data of the currently displayed application based on voice commands, and triggers other applications to run in the foreground based on this page data to obtain the page data of those other applications. Thus, the intelligent assistant combines the page data obtained from multiple sources to trigger the corresponding application execution events. This direct interaction between applications achieves a highly efficient intelligent interaction process.

[0238] In some embodiments, in response to a page change in the first application, the smart assistant obtains the updated first page data sent by the first application.

[0239] In this way, the intelligent assistant can determine the execution status of the first voice command based on the latest page display of the first application, or continue executing the first voice command. Alternatively, upon receiving other voice commands subsequently, it can also execute the corresponding voice commands based on the updated first page data.

[0240] In some embodiments, if no third voice command from the user is received within a preset time, the smart assistant stops obtaining data from the first page of the first application.

[0241] For example, if the smart assistant does not receive any more voice commands from the user within a preset time, it can automatically exit.

[0242] In this way, the smart assistant can automatically stop the smart interaction process, avoiding long waiting times for user voice commands and increasing device power consumption.

[0243] Optionally, other aspects of step S1203 can be found in the relevant aspects of steps S607-S609 in the above embodiments.

[0244] In some scenarios, the first voice command mentioned above may also directly carry information about the first application and information about another application that needs to be controlled.

[0245] Figure 13 is a flowchart illustrating another intelligent interaction method provided in an embodiment of this application. It should be noted that this method is not limited to the specific order described in Figure 13 and below. It should be understood that in other embodiments, the order of some steps in this method can be interchanged according to actual needs, or some steps can be omitted or deleted. The method includes the following steps:

[0246] S1301, The smart assistant in the electronic device receives a first voice command from the user. The first voice command carries first information of a first application and second information of a second application. The first application is the application currently displayed on the electronic device.

[0247] The first information of the first application carried in the first voice command can be information that directly instructs the first application, or information associated with the first application. The second information of the second application carried in the first voice command can be information that directly instructs the second application, or information associated with the second application.

[0248] In this way, users can control multiple applications through a single voice command. Alternatively, they can control other applications based on the page data of the currently displayed application.

[0249] S1302. The smart assistant requests data from the first page from the first application based on the first voice command.

[0250] In some examples, the first page data includes the control tree data of the first page; wherein, the control tree data is transferred between the first application and the smart assistant via IPC communication.

[0251] In this way, the intelligent assistant can directly achieve IPC communication with the first application without having to go through the accessibility service, thus improving the efficiency of intelligent interaction.

[0252] S1303, The smart assistant triggers the second application to execute the first event corresponding to the first voice command based on the obtained first page data. The first event is related to the first page data.

[0253] In some embodiments, the smart assistant triggers the second application to execute the first event corresponding to the first voice command based on the data on the first page, including: the smart assistant instructs the second application to execute the first event in the background based on the data on the first page.

[0254] For example, based on the data obtained from the first page, the smart assistant triggers the second application to execute the first event corresponding to the first voice command. This includes: the smart assistant sending the first execution command to the second application based on the data from the first page. In response to the received first execution command, the second application executes the first event in the background.

[0255] In this way, the intelligent assistant obtains the page data of the currently displayed application based on voice commands, and triggers other applications to execute corresponding events in the background based on the page data. Through direct interaction between applications, a highly efficient intelligent interaction process is achieved.

[0256] In other embodiments, the smart assistant, in response to a first voice command, triggers the electronic device to display a second application. Based on data from the first page, the smart assistant instructs the second application to perform a first event.

[0257] For example, in response to a first voice command, the smart assistant triggers the electronic device to display a second application. Based on the data from the first page, the smart assistant sends a second execution command to the second application. In response to the received second execution command, the second application executes the first event.

[0258] In some examples, the smart assistant triggers a second application to execute a first event based on data from the first page, including: the smart assistant requesting the second application to retrieve data from the second page; and the smart assistant sending a second operation command to the second application based on the first and second page data to trigger the second application to execute the first event.

[0259] For example, the smart assistant sends a request to a second application to retrieve data from the second page. In response to the received request, the second application sends the currently displayed data from the second page to the smart assistant. Based on the data from the first and second pages, the smart assistant sends a second operation command to the second application. In response to the second operation command, the second application executes a first event.

[0260] In this way, the intelligent assistant obtains the page data of the currently displayed application based on voice commands, and triggers other applications to run in the foreground based on this page data to obtain the page data of those other applications. Thus, the intelligent assistant combines the page data obtained from multiple sources to trigger the corresponding application execution events. This direct interaction between applications achieves a highly efficient intelligent interaction process.

[0261] Optionally, other contents of steps S1301-S1303 can be referred to the relevant contents of steps S1201-S1203 in the above embodiments.

[0262] In some solutions, multiple embodiments of this application can be combined, and the combined solution can be implemented. Optionally, some operations in the processes of each method embodiment may be combined, and / or the order of some operations may be changed. Furthermore, the execution order between the steps of each process is merely exemplary and does not constitute a limitation on the execution order between steps; other execution orders are also possible. It is not intended to indicate that the execution order is the only possible order in which these operations can be performed. Those skilled in the art will conceive of various ways to reorder the operations described herein. In addition, it should be noted that the process details involved in one embodiment of this document are similarly applicable to other embodiments, or different embodiments may be combined.

[0263] Furthermore, some steps in the method embodiments can be equivalently replaced with other possible steps. Alternatively, some steps in the method embodiments may be optional and can be deleted in certain use cases. Or, other possible steps may be added to the method embodiments.

[0264] Furthermore, the various method embodiments can be implemented individually or in combination.

[0265] The intelligent interactive display method provided by the embodiments of this application has been described in detail above with reference to Figures 6-13. The electronic device provided by the embodiments of this application is described in detail below with reference to Figure 14.

[0266] In one possible design, Figure 14 is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. As shown in Figure 14, the electronic device 1400 may include a transceiver unit 1401 and a processing unit 1402. The electronic device 1400 can be used to implement the functions of the electronic device involved in the above method embodiments.

[0267] Optionally, the transceiver unit 1401 is used to support the electronic device 1400 in executing S601, S602, S603, S607 and S608 in FIG. 6; and / or, to support the electronic device 1400 in executing S1201 in FIG. 12; and / or, to support the electronic device 1400 in executing S1301 in FIG. 13.

[0268] Optionally, the processing unit 1402 is used to support the electronic device 1400 in executing S604, S605, S606 and S609 in FIG. 6; and / or, to support the electronic device 1400 in executing S1202 and S1203 in FIG. 12; and / or, to support the electronic device 1400 in executing S1302 and S1303 in FIG. 13.

[0269] Optionally, the transceiver unit 1401 may include a receiving unit and a transmitting unit, which may be implemented by a transceiver or transceiver-related circuit components, and may be a transceiver or transceiver module. The operation and / or function of each unit in the electronic device 1400 are respectively to implement the corresponding process of the display intelligent interaction method described in the above method embodiments. All relevant content of each step involved in the above method embodiments can be referred to the functional description of the corresponding functional unit, and for the sake of brevity, it will not be repeated here.

[0270] Optionally, the electronic device 1400 shown in FIG14 may further include a storage unit (not shown in FIG14) storing a program or instructions. When the transceiver unit 1401 and the processing unit 1402 execute the program or instructions, the electronic device 1400 shown in FIG14 can perform the display intelligent interaction method described in the above method embodiments.

[0271] The technical effects of the electronic device 1400 shown in Figure 14 can be referred to the technical effects of the display intelligent interaction method described in the above method embodiments, and will not be repeated here.

[0272] In addition to being in the form of electronic device 1400, the technical solution provided in this application can also be a functional unit or chip in an electronic device, or a device used in conjunction with an electronic device.

[0273] This application also provides a chip system, including: a processor coupled to a memory, the memory being used to store programs or instructions, wherein when the program or instructions are executed by the processor, the chip system implements the methods in any of the above method embodiments.

[0274] Optionally, the chip system may contain one or more processors. These processors can be implemented in hardware or software. When implemented in hardware, the processor can be a logic circuit, an integrated circuit, etc. When implemented in software, the processor can be a general-purpose processor, implemented by reading software code stored in memory.

[0275] Optionally, the chip system may contain one or more memories. The memory may be integrated with the processor or disposed separately from it; this application embodiment does not limit this. For example, the memory may be a non-transient processor, such as a read-only memory (ROM), which may be integrated with the processor on the same chip or disposed separately on different chips. This application embodiment does not specifically limit the type of memory or the arrangement of the memory and processor.

[0276] For example, the chip system may be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a micro controller unit (MCU), a programmable logic device (PLD), or other integrated chips.

[0277] It should be understood that each step in the above method embodiments can be completed by integrated logic circuits in the processor hardware or by instructions in software form. The method steps disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or being executed by a combination of hardware and software modules in the processor.

[0278] This application also provides a computer-readable storage medium storing a computer program. When the computer program is run on a computer, it causes the computer to perform the aforementioned steps to implement the intelligent interaction method in the above embodiments.

[0279] This application also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned steps to implement the intelligent interaction method described in the above embodiments.

[0280] In addition, this application also provides an apparatus. Specifically, the apparatus may be a component or module, and may include one or more processors and a memory connected together. The memory is used to store a computer program. When the computer program is executed by one or more processors, the apparatus performs the intelligent interaction methods described in the above-described method embodiments.

[0281] The apparatus, computer-readable storage medium, computer program product, or chip provided in the embodiments of this application are all used to execute the corresponding methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects of the corresponding methods provided above, and will not be repeated here.

[0282] The steps of the methods or algorithms described in conjunction with the embodiments of this application can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium well known in the art. An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an application-specific integrated circuit (ASIC).

[0283] Through the above description of the embodiments, those skilled in the art will clearly understand that, for the sake of convenience and brevity, the division of the above functional modules is only used as an example. In practical applications, the above functions can be assigned to different functional modules as needed; that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device, and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0284] In the several embodiments provided in this application, it should be understood that the disclosed methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of modules or units is only a logical functional division, and there may be other division methods in actual implementation; for example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of modules or units may be electrical, mechanical or other forms.

[0285] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0286] Computer-readable storage media include, but are not limited to, any of the following: USB flash drive, portable hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, and other media capable of storing program code.

[0287] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An intelligent interaction method, characterized in that, The method includes: The intelligent assistant in the electronic device receives a first voice command from the user, the first voice command carrying information about a first application, the first application being the application currently displayed on the electronic device; The smart assistant requests the first page data from the first application based on the first voice command; The smart assistant triggers the first application to execute the first event corresponding to the first voice command based on the data obtained from the first page.

2. The method according to claim 1, characterized in that, The first page data includes the control tree data of the first page; wherein the control tree data is transmitted between the first application and the smart assistant via inter-process communication (IPC).

3. The method according to claim 1 or 2, characterized in that, The smart assistant, based on the acquired data from the first page, triggers the first application to execute a first event corresponding to the first voice command, including: The smart assistant obtains the first control in the first application that is operated according to the first voice command based on the first page data obtained; The smart assistant sends a first operation instruction to the first application, which triggers the first application to operate the first control to execute the first event corresponding to the first voice command.

4. The method according to any one of claims 1-3, characterized in that, The method further includes: The electronic device displays a page of the second application; The intelligent assistant receives a second voice command from the user; wherein the second voice command carries information about a third application. The smart assistant obtains the second page data of the second application based on the second voice command; The smart assistant triggers the third application to execute a second event based on the data on the second page, and the second event is related to the data on the second page.

5. The method according to claim 4, characterized in that, The smart assistant triggers the third application to execute a second event based on the data from the second page, including: The smart assistant instructs the third application to execute the second event in the background based on the data from the second page; or, The smart assistant responds to the second voice command and triggers the electronic device to display the third application; The smart assistant instructs the third application to perform the second event based on the data on the second page.

6. The method according to claim 5, characterized in that, The smart assistant triggers the third application to execute the second event based on the data from the second page, including: The smart assistant requests data from the third application to obtain data from the third page. The smart assistant sends a second operation instruction to the third application based on the second page data and the third page data, so as to trigger the third application to execute the second event.

7. The method according to any one of claims 1-6, characterized in that, The method further includes: In response to a change in the page of the first application, the smart assistant obtains the updated first page data of the first application.

8. The method according to any one of claims 1-7, characterized in that, Before the smart assistant requests the first page data from the first application based on the first voice command, the method further includes: The smart assistant requests the window management system (WMS) to obtain the currently displayed first window based on the first voice command. The first window is used to display the first application. The step of requesting the first page data from the first application includes: The smart assistant obtains the data from the first page through the interface of the first window.

9. An intelligent interaction method, characterized in that, The method includes: The intelligent assistant in the electronic device receives a first voice command from the user. The first voice command carries first information of a first application and second information of a second application. The first application is the application currently displayed by the electronic device. The smart assistant requests the first page data from the first application based on the first voice command; The smart assistant triggers the second application to execute a first event corresponding to the first voice command based on the data on the first page; wherein the first event is related to the data on the first page.

10. The method according to claim 9, characterized in that, The first page data includes the control tree data of the first page; wherein the control tree data is transmitted between the first application and the smart assistant via inter-process communication (IPC).

11. The method according to claim 9 or 10, characterized in that, The smart assistant triggers the second application to execute the first event corresponding to the first voice command based on the data from the first page, including: The smart assistant instructs the second application to execute the first event in the background based on the data from the first page; or, The smart assistant responds to the first voice command and triggers the electronic device to display the second application; The smart assistant instructs the second application to perform the first event based on the data on the first page.

12. The method according to claim 11, characterized in that, The smart assistant triggers the second application to execute the first event based on the data from the first page, including: The smart assistant requests data from the second application to retrieve the second page. The smart assistant sends a second operation instruction to the second application based on the data from the first page and the data from the second page, so as to trigger the second application to execute the first event.

13. The method according to any one of claims 9-12, characterized in that, The method further includes: In response to a change in the page of the first application, the smart assistant obtains the updated first page data of the first application.

14. The method according to any one of claims 9-13, characterized in that, Before the smart assistant requests the first page data from the first application based on the first voice command, the method further includes: The smart assistant requests the window management system (WMS) to obtain the currently displayed first window based on the first voice command. The first window is used to display the first application. The step of requesting the first page data from the first application includes: The smart assistant obtains the data from the first page through the interface of the first window.

15. An electronic device, characterized in that, include: A processor and a memory, the memory being coupled to the processor, the memory being used to store computer program code, the computer program code including computer instructions, which, when the processor reads the computer instructions from the memory, cause the electronic device to perform the method as described in any one of claims 1-8; or cause the electronic device to perform the method as described in any one of claims 9-14.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a computer program that, when executed on an electronic device, causes the electronic device to perform the method as described in any one of claims 1-8; or causes the electronic device to perform the method as described in any one of claims 9-14.

17. A computer program product, characterized in that, When the computer program product is run on a computer, it causes the computer to perform the method as described in any one of claims 1-8; or causes the computer to perform the method as described in any one of claims 9-14.

Images