Interaction method and apparatus, and computer device and storage medium

Abstract

The present application relates to an interaction method and apparatus, and a computer device and a storage medium. The method comprises: on the basis of multi-modal information related to a target vehicle, determining a plurality of target intents related to a user; starting a dialog service flow, which comprises: for each of the plurality of target intents, sending an instruction, which corresponds to the target intent, to an application, to which an atomic service function for implementing the target intent belongs, among at least one application, such that the application calls, on the basis of the instruction, the atomic service function for implementing the target intent; and providing, to a user, information used for guiding user interaction, wherein the information for guiding user interaction comprises: a calling result corresponding to the atomic service function for implementing the target intent.

Description

An interaction method, apparatus, computer device, and storage medium

[0001] Cross-references to related applications

[0002] This application claims priority to Chinese Patent Application No. 2024116359130, filed on November 15, 2024, entitled "An Interactive Method, Apparatus, Computer Equipment and Storage Medium", the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of vehicles, specifically to an interaction method, apparatus, computer device, and storage medium. Background Technology

[0004] With the rapid development of intelligent technologies, in-vehicle voice interaction systems have become an important component of modern automobiles. In recent years, breakthroughs in artificial intelligence, natural language processing, and speech recognition technologies have significantly improved the performance of in-vehicle voice interaction systems. These technologies enable vehicles to more accurately understand the driver's voice commands and execute corresponding operations through voice feedback, greatly enhancing convenience and safety during driving. The development of in-vehicle voice technology not only simplifies traditional manual operation but also enhances driving safety by reducing driver distraction.

[0005] Currently, in-vehicle voice interaction systems can recognize and process multiple intentions expressed in a single sentence. In other words, the system can recognize multiple needs or intentions expressed by a user in a single sentence and distribute these intentions to different in-vehicle applications for processing.

[0006] However, current in-vehicle voice interaction systems simply distribute multi-intent commands to various applications for execution when processing them. The execution results of multiple commands are not displayed on the interface, making it impossible for users to intuitively see the operation results corresponding to each command. Summary of the Invention

[0007] One of the purposes of this application is to provide an interaction method, device, computer equipment, and storage medium to address the problem of how to improve the convenience of interaction between a vehicle and its users.

[0008] To achieve the above objectives, the technical solution adopted in this application is as follows:

[0009] An interaction method includes: determining multiple target intents related to a user based on multimodal information related to a target vehicle; initiating a dialogue service flow, the dialogue service flow including: for each of the multiple target intents, sending an instruction corresponding to the target intent to at least one application to which an atomic service function for implementing the target intent belongs, so that the application invokes the atomic service function for implementing the target intent according to the instruction; and providing the user with information to guide user interaction, the information including: the invocation result corresponding to the atomic service function for implementing the target intent.

[0010] In one possible implementation, the method further includes: providing the user with information to guide user interaction, including: displaying the call results corresponding to the atomic service function used to achieve the target intent in the dialogue service flow interface, or playing voice indicating the call results corresponding to the atomic service function used to achieve the target intent.

[0011] In one possible implementation, the information used to guide user interaction further includes: information indicating recommended atomic service functions corresponding to the atomic service functions used to achieve the target intent; providing users with information to guide user interaction also includes: displaying user interface components of the recommended atomic service functions corresponding to the atomic service functions used to achieve the target intent in the dialogue service flow interface, or playing voice indicating the recommended atomic service functions corresponding to the atomic service functions used to achieve the target intent.

[0012] In one possible implementation, multiple target intentions related to the user are determined based on multimodal information related to the target vehicle, including: using a target model on a cloud server to determine multiple target intentions related to the user based on multimodal information related to the target vehicle.

[0013] In one possible implementation, the method further includes: after providing the user with information to guide user interaction, in response to the user's operation on the corresponding information item in the information to guide user interaction, invoking the atomic service function corresponding to the corresponding information item.

[0014] An interactive device includes: an intent determination unit, configured to determine multiple target intents related to a user based on multimodal information related to a target vehicle; an initiation unit, configured to initiate a dialogue service flow, the dialogue service flow including: for each of the multiple target intents, sending an instruction corresponding to the target intent to at least one application to which an atomic service function for implementing the target intent belongs, so that the application invokes the atomic service function for implementing the target intent according to the instruction; and a guidance unit, configured to provide the user with information for guiding user interaction, the information for guiding user interaction including: the invocation result corresponding to the atomic service function for implementing the target intent.

[0015] A computer device includes: a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the interaction method of the first aspect or any corresponding embodiment described above.

[0016] A vehicle includes a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the interaction method of the first aspect or any corresponding embodiment described above.

[0017] A computer-readable storage medium storing computer instructions for causing a computer to perform the interactive method of the first aspect described above or any corresponding embodiment thereof.

[0018] A computer program product includes computer instructions for causing a computer to perform the interactive method described in the first aspect or any of its corresponding embodiments.

[0019] The beneficial effects of this application are:

[0020] By processing multimodal data and determining multiple target intents related to the user based on multimodal information associated with the target vehicle, a dialogue service flow is initiated. This flow includes sending the corresponding instruction to at least one application containing the atomic service function used to implement the target intent for each intent. This eliminates the need for the user to perform a series of operations on the vehicle's infotainment screen to trigger the execution of the intent, simplifying the user experience and improving the convenience of vehicle-user interaction. Simultaneously, information guiding user interaction is provided, including the call results corresponding to the atomic service function used to implement the target intent, allowing the user to intuitively see each call result. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0022] Figure 1 is a schematic diagram of an example architecture of an interactive system that can be used to execute the interactive methods provided in the embodiments of this application;

[0023] Figure 2 is a flowchart illustrating the interaction method provided in an embodiment of this application;

[0024] Figure 3 is a schematic diagram illustrating an example of determining target intent;

[0025] Figure 4 is a schematic diagram of the effect of calling the corresponding atomic service function of the navigation APP on the target vehicle to achieve the target intent;

[0026] Figure 5 is a schematic diagram illustrating the effect of calling the corresponding atomic service function of the music APP on the target vehicle to achieve the target intent;

[0027] Figure 6 is a schematic diagram illustrating the effect of calling the corresponding atomic service function of the music APP on the target vehicle to achieve the target intent;

[0028] Figure 7 is a flowchart illustrating an example of the interaction between the target vehicle and the user;

[0029] Figure 8 is a schematic diagram of the hardware structure of a computer device according to an embodiment of this application. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0031] Referring to Figure 1, it shows a schematic diagram of an example architecture of an interactive system that can be used to perform the interactive methods provided in the embodiments of this application.

[0032] An example architecture of an interactive system that can be used to execute the interactive methods provided in the embodiments of this application includes: a data acquisition unit, a data processing unit, and an atomic service function unit.

[0033] The data acquisition unit collects voice data from user interactions, images of the target vehicle's internal and external environments, and sensor data from the target vehicle. The data processing unit includes a voice data processing module, an image processing module, a sensor data processing module, an encoding module, and a target model. Atomized service functions include the atomic service functions of each app on the target vehicle that possesses atomic service capabilities. An atomic service function can be understood as the smallest functional unit, capable of performing a single operation.

[0034] The voice data processing module can perform speech recognition on the user's interactive speech to obtain the speech recognition results. The voice data processing unit can use a sound source localization algorithm to determine the user's location within the target vehicle. The voice data processing unit can also identify the user by matching the user's interactive voiceprint features with preset voiceprint features. The preset voiceprint features are those of users whose voiceprint features have been previously recorded.

[0035] The image processing module can process images of the interior environment of the target vehicle to determine the user's age group, gender, seating position, and identity information. The image processing unit may include a Driver Monitoring System (DMS) and an Occupant Monitoring System (OMS). The Driver Monitoring System monitors the driver's gaze direction in real time. The Occupant Monitoring System can process images of the vehicle's interior environment captured by the vehicle's cameras to determine the passengers' gender, age, gestures, and seating position.

[0036] The sensor data processing module can process sensor data collected by the target vehicle's sensors to determine information related to the target vehicle's driving and weather-related information about the target vehicle's location. Driving-related information may include the target vehicle's speed, remaining fuel level, or remaining battery charge. Weather-related information about the target vehicle's location may include the temperature and weather type, such as rain or snow.

[0037] The encoding module is used to encode the multimodal information related to the target vehicle to obtain a representation of the multimodal information related to the target vehicle.

[0038] It should be noted that the target model can be on the target vehicle or on a cloud server.

[0039] The target model is used to receive representations of multimodal information related to the target vehicle and predict the representation of the target intent. Thus, the target intent is determined. The target model is a trained neural network. Before the interaction method provided in this application embodiment is executed for the first time, the target model is trained using training samples and annotation operations on the training samples.

[0040] As an example, the target model is a pre-trained language model. As another example, the pre-trained language model used as the target model is a Large Language Model (LLM). When the input to the target model is natural language text corresponding to natural language speech, the pre-trained language model performs Natural Language Processing (NLP) tasks.

[0041] Training the target model allows it to learn: which information from the target model's input to predict actions that satisfy the user's interactive voice instructions, and how to determine the target intent. Training samples for the target model can include training information related to the multimodal information relevant to the target vehicle. For any given piece of information, if the multimodal information relevant to the target vehicle includes that information, then that information is classified as belonging to the multimodal information relevant to the target vehicle. The labeled operations of the target model's training samples can be understood as: the expected output operations of the target model when the training samples are input. During training, the loss between the operations predicted by the target model and the labeled operations of the training samples can be calculated, and the parameters of the target model can be updated based on this loss.

[0042] Referring to Figure 2, it shows a schematic diagram of an example of the interaction method provided in an embodiment of this application.

[0043] Step S201: Based on the multimodal information related to the target vehicle, determine multiple target intentions related to the user.

[0044] The multimodal information related to the target vehicle includes: the speech recognition results of the user's interactive voice, the parameter information of the target vehicle, the external environment information of the target vehicle, the user's action information, the user's seating location information, the user's attribute information, the user's identity information, information related to the APP on the target vehicle determined during the historical time period of using the APP on the target vehicle, navigation information related to the target vehicle, music information related to the target vehicle, and point of interest information related to the target vehicle. Among them, the point of interest information related to the target vehicle indicates the point of interest reached by the target vehicle during the target historical time period and / or the point of interest reached by the target vehicle multiple times.

[0045] The speech recognition result of the user's interactive voice is obtained by performing speech recognition on the user's interactive voice. Specifically, performing speech recognition on the user's interactive voice refers to performing speech recognition on the voice data of the user's interactive voice.

[0046] As an example, the parameter information of the target vehicle may include: information related to the driving of the target vehicle and information related to the cockpit of the target vehicle.

[0047] As an example, information related to the driving of the target vehicle includes: the target vehicle's speed, the target vehicle's remaining fuel, or the target vehicle's remaining battery charge.

[0048] As an example, information related to the target vehicle's cabin includes: the brightness of the vehicle's infotainment screen, the volume of the vehicle's audio system, the temperature of the vehicle's air conditioning, the airflow of the vehicle's air conditioning, and the angle of the vehicle's seats.

[0049] The external environment information of the target vehicle refers to information related to the external environment of the target vehicle.

[0050] As an example, the external environment information of the target vehicle includes: weather-related information about the area where the target vehicle is located, driving safety-related information about the road the target vehicle is currently on, and points of interest (POIs) near the target vehicle.

[0051] As an example, weather-related information for the area where the target vehicle is located includes the temperature, the type of weather, etc.

[0052] As an example, information related to the driving safety of the target vehicle on the road it is currently on includes one or more of the following: information indicating whether the road is slippery, information indicating whether the road is icy, information indicating whether the road is bumpy, information indicating whether the road is a high-altitude road, etc.

[0053] As an example, user action information includes at least some of the following: user gestures, the direction the user's hand is pointing, the direction the user's gaze is in, and the user's actions.

[0054] As an example, user attribute information includes: the user's age group, the user's gender, etc.

[0055] User identity information is used to identify the user's identity.

[0056] As an example, a user's identity information includes: the user's identifier.

[0057] The user's seating position information indicates the user's seating location. As an example, the user's seating position is one of the following: driver's seat, front passenger seat, rear left seat, rear middle seat, or rear right seat.

[0058] Navigation information related to the target vehicle may include: the origin of the current navigation to the target vehicle, the destination of the current navigation to the target vehicle, and the navigation route of the current navigation to the target vehicle.

[0059] Music information related to the target vehicle may include: the playlist to which the currently playing song belongs, and the playback progress of the song in that playlist. The playback progress indicates which song has been played so far.

[0060] The end time of the historical period in which the app on the target vehicle was used is before the current time. The app on the target vehicle was used within the historical period in which the app was used.

[0061] As an example, for a navigation app on a target vehicle, the multimodal information related to the target vehicle includes: information related to the navigation app determined during the historical time period of its use. This information includes: the navigation route determined during the historical time period of the app's use, the name of the starting point of the navigation route determined during the historical time period of the app's use, and the name of the destination of the navigation route determined during the historical time period of the app's use.

[0062] As an example, for a music app on a target vehicle, the multimodal information related to the target vehicle includes: information related to the music app on the target vehicle determined during the historical time period of using the music app on the target vehicle. Information related to the music app on the target vehicle includes: identifiers of identified recommended songs.

[0063] Referring to Figure 3, which shows a schematic diagram of an example of determining a target intent.

[0064] The user speaks an interactive voice command, triggering the target vehicle to acquire multimodal information related to the vehicle. The user's interactive voice command is the voice used for interaction. This voice command can be natural language speech, reflecting the user's desired action from the target vehicle. To acquire this multimodal information, the system acquires data collected by the target vehicle, including the user's interactive voice command, sensor data, images of the target vehicle's internal environment, and images of the target vehicle's external environment. The data collected by the target vehicle is processed to obtain the multimodal information related to the target vehicle. This multimodal information includes at least some of the following: the speech recognition result of the user's interactive voice command, the target vehicle's parameter information, the target vehicle's external environment information, the user's action information, the user's attribute information, the user's identity information, and information related to the app on the target vehicle determined during the historical time period of app usage on the target vehicle.

[0065] Each piece of information in the multimodal information related to the target vehicle is encoded to obtain a representation of the multimodal information related to the target vehicle.

[0066] The representation of multimodal information related to the target vehicle is input into the target model, and the target model outputs the target intent.

[0067] In step S201, multimodal information related to the target vehicle can be input into the target model, and the target model can output the target intent.

[0068] The target model can be on the target vehicle or on a cloud server.

[0069] It should be noted that the target model can use a portion of the multimodal information related to the target vehicle to predict the target's intent.

[0070] As an example, the interior temperature of the target vehicle is high, and the user's hand is pointing towards the window and making a downward gesture, indicating the user's intention to open the window.

[0071] As an example, a user says the interactive voice command, "Play songs suitable for people of my age." The intended message is to play songs suitable for users in the age group to which this user belongs.

[0072] As an example, a user says the interactive voice command, "Move your seat back a little." The intended action is to move the user's seat back.

[0073] Step S202: Start the dialogue service flow. The dialogue service flow includes: for each of the multiple target intents, sending the instruction corresponding to the target intent to at least one application to which the atomic service function for implementing the target intent belongs, so that the application calls the atomic service function for implementing the target intent according to the instruction.

[0074] In the embodiments of the invention, each APP with atomic service functions on the target vehicle can have multiple atomic service functions. When the dialogue flow service is started, for each of the multiple target intents, the instruction corresponding to the target intent can be sent to at least one application to which the atomic service function for implementing the target intent belongs, so that the application can call the atomic service function to implement the target intent according to the instruction.

[0075] As an example, the vehicle center app on the target vehicle has atomic service functions such as an atomic service function for opening windows and an atomic service function for closing windows. The air conditioning app on the target vehicle has atomic service functions such as an atomic service function for turning on the air conditioning, an atomic service function for turning off the air conditioning, and an atomic service function for adjusting the air conditioning temperature.

[0076] Once the target intent is determined, the dialogue service flow can be initiated, invoking the atomic service functions used to achieve the target intent. After invoking the atomic service functions used to achieve the target intent, the invocation results corresponding to the atomic service functions used to achieve the target intent can be displayed in the dialogue service flow interface, which is displayed on the in-vehicle infotainment screen of the target vehicle.

[0077] The result of an atomic service function used to achieve a target intent is either the result obtained through that atomic service function or an indication that the target intent has been completed. For example, if the atomic service function used to achieve the target intent is an "Air Conditioner Turn On" atomic service function in the air conditioning app of the target vehicle, then the result of the call to that atomic service function is a text indicating that the air conditioning has been turned on. If the atomic service function used to achieve the target intent is an atomic service function for searching POIs in the search app of the target vehicle, then the result of the call to that atomic service function is the search result obtained by calling the atomic service function for searching POIs in the search app of the target vehicle.

[0078] As an example, a user speaks a voice command related to the music app on the target vehicle. The target model determines the target intent related to the music app on the target vehicle and invokes the corresponding atomic service function of the music app on the target vehicle to realize the target intent.

[0079] Referring to Figure 4, it shows a schematic diagram of the effect of calling the corresponding atomic service function of the navigation APP on the target vehicle to achieve the target intent.

[0080] When the user interacts with the target vehicle's interaction system for the tth time, the user says the interactive voice command "Go to... subway station to pick someone up, then go to..., find a place to eat," thus determining two target intentions: generating a navigation route through... subway station and the "..." in "then go to...", and searching for multiple restaurants near the "..." in "then go to...". A dialogue service flow is initiated, calling the atomic service function of the navigation app on the target vehicle to generate the navigation route through... subway station and the "..." in "then go to...". The atomic service function of the search app on the target vehicle to search for POIs is also called to search for multiple restaurants near the "..." in "then go to...". When the user interacts with the target vehicle's interaction system for the (t+1)th time, the user says, "Are there any hot pot restaurants?" The user's intention is to find a hot pot restaurant from multiple restaurants. This initiates a dialogue service flow, calling the atomic service function of the search app on the target vehicle for searching Points of Interest (POIs). During this (t+1)th interaction, the dialogue service flow interface displays, "These 5 hot pot restaurants are all pretty good." When the user interacts with the target vehicle's interaction system for the (t+2)th time, the user says, "Choose one with good parking." This initiates a dialogue service flow, calling the atomic service function of the search app on the target vehicle for searching POIs. Based on the surrounding environment information of the target vehicle, the atomic service function of the search app finds a hot pot restaurant with good parking from the 5 restaurants. When the user interacts with the target vehicle's interaction system for the t+3rd time, the user says the interactive voice "Just select...". The target intent is to navigate to the "..." in "Just select...". The dialogue service flow is started, and the atomic service function of the navigation APP on the target vehicle is called to generate a navigation route from the target vehicle's current location to the "..." in "Just select...".

[0081] Referring to Figure 5, it shows a schematic diagram of the effect of calling the corresponding atomic service function of the music APP on the target vehicle to achieve the target intent.

[0082] When the user interacts with the target vehicle's interaction system for the t-th time, the user says the interactive voice prompt "Which song is the most famous for '…'?" The target intent is to search for the name of the most famous song for "…". A dialogue service flow is initiated, calling the atomic service function of the music app on the target vehicle to search for the name of the most famous song for "…" from the knowledge base. When the user interacts with the target vehicle's interaction system for the t+1-th time, the user says the interactive voice prompt "What is the background of its creation?" The target intent is to search for the background of the most famous song for "…". A dialogue service flow is initiated, calling the atomic service function of the music app on the target vehicle to search for background music knowledge from the knowledge base. When the user interacts with the target vehicle's interaction system for the t+2-th time, the user says the interactive voice prompt "In which album?" The target intent is to search for the album to which the song belongs.

[0083] Initiate a dialogue service flow, invoking the atomized service function of the music app on the target vehicle to search for the album name of the hit song "..." from the knowledge base. During the (t+2)th interaction between the user in the target vehicle and the vehicle's interaction system, the user utters the interactive voice command, "What other songs are in this album?" The target intent is: to find the names of other songs in this album besides the hit song "...". Initiate a dialogue service flow, invoking the atomized service function of the music app on the target vehicle to search for the names of other songs in this album besides the hit song "...". During the (t+3)th interaction between the user in the target vehicle and the vehicle's interaction system, the user utters the interactive voice command, "Add these songs to the playlist." The target intent is to add other songs to the playlist.

[0084] Initiate a dialogue service flow and call the atomic service function of the music app on the target vehicle to add other songs to the playlist.

[0085] In step S203, information is provided to the user to guide user interaction. This information includes the call result corresponding to the atomic service function used to achieve the target intent.

[0086] Information used to guide user interaction can be played via voice or displayed as text in the dialogue service flow interface.

[0087] In this embodiment, by processing multimodal data and determining multiple target intents related to the user based on multimodal information associated with the target vehicle, a dialogue service flow is initiated. This dialogue service flow includes: for each target intent, the application to which the atomic service function corresponding to at least one application used to implement the target intent belongs is invoked, thus realizing the target intent. Users do not need to perform a series of operations on the vehicle's infotainment screen to trigger an action with their intended intent, simplifying user operations and improving the convenience of vehicle-user interaction. Simultaneously, information is provided to guide user interaction, including the call results corresponding to the atomic service functions used to realize the target intent, allowing users to intuitively see each call result.

[0088] In one possible implementation, providing users with information to guide user interaction includes: displaying the call results corresponding to the atomic service functions used to achieve the target intent in the dialogue service flow interface, or playing voice prompts indicating the call results corresponding to the atomic service functions used to achieve the target intent.

[0089] As an example, a user becomes interested in a neighborhood the target vehicle is passing through while it is driving. The user says, "What neighborhood is ahead?" The target intent is to search for the name of the neighborhood ahead of the target vehicle, and the result is "Neighborhood ahead is A". The search app on the target vehicle uses its atomic service function for searching Points of Interest (POIs) to search for the name of the neighborhood ahead. The user then says, "How much does it cost to rent a three-bedroom apartment?" The target intent is to search for the rent of a three-bedroom apartment in the neighborhood ahead of the target vehicle, and the result is "Rental fee: xxx yuan".

[0090] In one possible implementation, the information used to guide user interaction further includes: information indicating recommended atomic service functions corresponding to the atomic service functions used to achieve the target intent; providing users with information to guide user interaction also includes: displaying user interface components of the recommended atomic service functions corresponding to the atomic service functions used to achieve the target intent in the dialogue service flow interface, or playing voice indicating the recommended atomic service functions corresponding to the atomic service functions used to achieve the target intent.

[0091] In addition to the call results corresponding to the atomic service functions used to achieve the target intent, the information provided to guide user interaction also includes information indicating recommended atomic service functions that correspond to the atomic service functions used to achieve the target intent. This information can be played via voice or displayed on the dialogue service flow interface.

[0092] To determine the recommended atomic service functions corresponding to the atomic service functions used to achieve the target intent, the relationships between atomic service functions can be pre-defined. Other atomic service functions related to the atomic service function used to achieve the target intent are then identified as the recommended atomic service functions corresponding to the atomic service function used to achieve the target intent. Alternatively, the target model can predict the recommended atomic service functions corresponding to the atomic service function used to achieve the target intent based on the relationships between atomic service functions.

[0093] As an example, the target intent is to recommend songs. The atomic service function used to implement this intent is the song search service function of the music app on the target vehicle. A dialog service flow is initiated, calling the song search service function of the music app on the target vehicle to search for recommended songs. The corresponding recommendation atomic service function used to implement the target intent is the music app's atomic service function for setting sound effects for the recommended songs.

[0094] Referring to Figure 6, it shows a schematic diagram of the effect of calling the corresponding atomic service function of the music APP on the target vehicle to achieve the target intent.

[0095] The objective is to recommend a second song after the first song in each of the currently recommended songs has finished playing. A dialogue service flow is initiated, invoking the atomic service function of the music app on the target vehicle to search for the recommended second song. This second song is different from the first song. The dialogue service flow interface displays "The recommended songs have finished playing, and some new songs have been recommended for you." The "recommended songs" in this message refer to the first song. A new song identifier is displayed on the dialogue service flow interface. This new song refers to the second song.

[0096] In one possible implementation, after providing the user with information to guide user interaction, a feedback operation corresponding to the corresponding information item is executed in response to the user's operation on the corresponding information item in the information used to guide user interaction.

[0097] Feedback actions can be outputs generated based on the user's interactive voice and the results of their actions, used to convey information to the user. Specifically, after providing the user with information to guide their interaction, a feedback action corresponding to that information item is executed in response to the user's action on that item.

[0098] In one possible implementation, the user's action could be clicking on a corresponding information item in the information used to guide user interaction or speaking a voice command for the corresponding information item in the information used to guide user interaction.

[0099] In one possible implementation, if a user's operation on a corresponding information item in the information used to guide user interaction is an operation on information related to the recommended atomic service function corresponding to the atomic service function used to achieve the target intent, the feedback operation corresponding to the corresponding information item is to call the recommended atomic service function corresponding to the atomic service function used to achieve the target intent.

[0100] As an example, the atomic service function used to achieve the target intent is the atomic service function for activating the seat massage function in a seat massage app. The recommended atomic service function corresponding to the atomic service function for activating the seat massage function is used to set the parameter values ​​for the seat massage function's operating parameters. The corresponding information items in the information used to guide user interaction include: the parameter values ​​for the seat massage function's operating parameters. When the user speaks the parameter values ​​for the seat massage function's operating parameters, the recommended atomic service function corresponding to the atomic service function for activating the seat massage function is activated.

[0101] Referring to Figure 7, it shows a flowchart illustrating an example of the interaction between the target vehicle and the user.

[0102] Multimodal information related to the target vehicle is input into the target model, which outputs the target intent. The skill distribution function of the target vehicle's voice app sends the corresponding instruction to each of at least one target intent to the app containing the atomic service function used to implement the target intent in the vehicle center app, navigation app, music app, etc. The app containing the atomic service function used to implement the target intent calls the atomic service function used to implement the target intent according to the instruction corresponding to the target intent, and obtains the call result corresponding to the atomic service function used to implement the target intent. When providing information to guide user interaction, in this example, the UI component of the call result corresponding to the atomic service function used to implement the target intent and the recommended atomic service function corresponding to the atomic service function used to implement the target intent is displayed in the dialogue service flow interface. In this example, the user's operation on the corresponding information item in the information used to guide user interaction is the user's operation on the information related to the recommended atomic service function corresponding to the atomic service function used to implement the target intent in the information used to guide user interaction. In this example, the feedback operation corresponding to the relevant information item is to call the recommended atomic service function corresponding to the atomic service function used to achieve the target intent.

[0103] In this embodiment, the need for users inside a vehicle to frequently search for additional points of interest (POIs) that meet specific criteria from a pool of already searched POIs is considered. POIs that satisfy the conditions indicated by the speech recognition results of the user's interactive speech can be found from a pool of searched POIs, eliminating the need for the user to manually determine which POIs meet the criteria. This simplifies the user's operation and improves the user experience.

[0104] This embodiment also provides an interactive device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0105] This application also provides an interactive device for implementing the above-described method embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "unit" can refer to a combination of software and / or hardware that performs a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated. The devices in this application are presented in the form of functional units, where a functional unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above-described functions.

[0106] An interactive device, the interactive device comprising:

[0107] The intent determination unit is used to determine multiple target intents related to the user based on multimodal information related to the target vehicle.

[0108] The startup unit is used to start a dialogue service flow, which includes: for each of a plurality of target intents, sending an instruction corresponding to the target intent to at least one application to which the atomic service function for implementing the target intent belongs, so that the application invokes the atomic service function for implementing the target intent according to the instruction.

[0109] The guidance unit is used to provide users with information to guide user interaction. This information includes the call results corresponding to the atomic service functions used to achieve the target intent.

[0110] Furthermore, the guiding unit is also used for:

[0111] The dialog service flow interface displays the call results corresponding to the atomic service functions used to achieve the target intent, or plays the voice prompts indicating the call results corresponding to the atomic service functions used to achieve the target intent.

[0112] Furthermore, the information used to guide user interaction also includes: information indicating recommended atomic service functions corresponding to atomic service functions used to achieve the target intent; the guidance unit is further used for:

[0113] The dialog service flow interface displays user interface components or playback instructions for recommended atomic service functions corresponding to the atomic service functions used to achieve the target intent.

[0114] Furthermore, the intent determination unit is used to determine multiple target intents related to the user based on the target vehicle's multimodal information using the target model on the cloud server.

[0115] Furthermore, the interactive device also includes an execution unit, configured to perform a feedback operation corresponding to the corresponding information item in response to a user's operation on a corresponding information item in the information used to guide user interaction.

[0116] In this embodiment, the interactive device is presented in the form of a functional unit. Here, a functional unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0117] Referring to Figure 8, which is a schematic diagram of the hardware structure of a computer device according to an embodiment of this application, the computer device includes one or more processors 10, a memory 20, and interfaces for connecting the various components, including high-speed interfaces and low-speed interfaces. The various components communicate with each other using different buses and can be installed on a common motherboard or otherwise as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to the interface). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple devices can be connected, each providing some necessary operations (e.g., as a server array, a set of blade servers, or a multiprocessor system). The processor 10 can be a central processing unit, a network processor, or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip can be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device can be a complex programmable logic device (CLP), a field-programmable gate array (FPGA), a general-purpose array logic (GPRS), or any combination thereof. The memory 20 stores instructions executable by at least one processor 10 to cause the processor 10 to perform the methods shown in the above embodiments. The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on vehicle usage, etc. Furthermore, the memory 20 may include high-speed random access memory and non-transient memory, such as at least one disk storage device, flash memory device, or other non-transient solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, which can be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof. The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk, or solid-state drive; the memory 20 may also include combinations of the above types of memory. The computer device also includes an input device 30 and an output device 40. The processor 10, memory 20, input device 30, and output device 40 may be connected via a bus or other means. The input device 30 can receive input digital or character information, and generate key signal inputs related to user settings and function control of the computer device, such as touch screen, keypad, mouse, trackpad, touchpad, joystick, one or more mouse buttons, trackball, joystick, etc.The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED), and a haptic feedback device (e.g., a vibration motor). The display device includes, but is not limited to, liquid crystal displays, light-emitting diodes, displays, and plasma displays. In some alternative embodiments, the display device may be a touchscreen.

[0118] This application also provides a computer-readable storage medium. The methods described in this application can be implemented in hardware or firmware, or implemented as recordable on a storage medium, or implemented as computer code downloaded over a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and subsequently stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the methods shown in the above embodiments are implemented.

[0119] A portion of the embodiments of this application can be applied as a computer program product, such as computer program instructions. When executed by a computer, these instructions, through the operation of the computer, can invoke or provide the methods and / or technical solutions according to this application. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, and installation package files. Accordingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions; the computer compiling the instructions and then executing the corresponding compiled program; the computer reading and executing the instructions; or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0120] The above embodiments are merely preferred embodiments provided to fully illustrate this application, and the scope of protection of this application is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on this application are all within the scope of protection of this application.

Claims

1. An interaction method, characterized in that, The method includes: Based on multimodal information related to the target vehicle, determine multiple target intentions related to the user; Initiate a dialogue service flow, which includes: For each of the multiple target intents, the instruction corresponding to the target intent is sent to at least one application to which the atomic service function for implementing the target intent belongs, so that the application invokes the atomic service function for implementing the target intent according to the instruction. Provide users with information to guide user interaction, including the call results corresponding to atomic service functions used to achieve the target intent.

2. The interaction method of claim 1, wherein, Providing users with information to guide user interaction includes: The dialog service flow interface displays the call results corresponding to the atomic service functions used to achieve the target intent, or plays the voice prompts indicating the call results corresponding to the atomic service functions used to achieve the target intent.

3. The interaction method of claim 2, wherein, Information used to guide user interaction also includes: information indicating recommended atomic service functions corresponding to atomic service functions used to achieve the target intent; and information provided to the user to guide user interaction, including: The dialog service flow interface displays user interface components or playback instructions for recommended atomic service functions corresponding to the atomic service functions used to achieve the target intent.

4. The interaction method of claim 1, wherein, Based on multimodal information related to the target vehicle, determine multiple target intents related to the user, including: By utilizing the target model on the cloud server, multiple target intentions related to the user are determined based on multimodal information related to the target vehicle.

5. The interaction method according to any one of claims 1-4, characterized in that, The method further includes: After providing information to the user to guide user interaction, in response to the user's action on the corresponding information item in the information used to guide user interaction, a feedback action corresponding to the corresponding information item is executed.

6. An interactive device, characterized by The device includes: The intent determination unit is used to determine multiple target intents related to the user based on multimodal information related to the target vehicle. A startup unit is used to start a dialogue service flow, which includes: for each of a plurality of target intents, sending an instruction corresponding to the target intent to at least one application to which an atomic service function for implementing the target intent belongs, so that the application invokes the atomic service function for implementing the target intent according to the instruction; The guidance unit is used to provide users with information to guide user interaction. This information includes the call results corresponding to the atomic service functions used to achieve the target intent.

7. A computer device, comprising: include: A memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, the processor executing the computer instructions to perform the interactive method of any one of claims 1 to 5.

8. A vehicle characterized by comprising: The vehicle includes a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the interactive method of any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium has stored thereon computer instructions for causing a computer to perform the interaction method of any one of claims 1 to 5.

10. A computer program product, characterised in that, The computer readable storage medium has stored thereon computer instructions for causing a computer to perform the interaction method of any one of claims 1 to 5.

Images