Interaction method and apparatus, and computer device and storage medium

Abstract

The present application relates to an interaction method and apparatus, and a computer device and a storage medium. The method comprises: on the basis of multi-modal information related to a target vehicle, determining a target intent related to a user; and starting a dialog service flow, wherein the dialog service flow comprises: sending an instruction, which corresponds to the target intent, to an application to which an atomic service function for implementing the target intent belongs, so that the application invokes, on the basis of the instruction, the atomic service function for implementing the target intent.

Description

An interaction method, apparatus, computer device, and storage medium

[0001] Cross-reference to related applications

[0002] This application claims priority to Chinese Patent Application No. 2024116358975, filed on November 15, 2024, entitled “An Interactive Method, Apparatus, Computer Equipment and Storage Medium”, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of vehicles, specifically to an interaction method, apparatus, computer device, and storage medium. Background Technology

[0004] With the rapid development of artificial intelligence technology, voice interaction has become an important component of modern smart devices, especially in the field of smart cars. It greatly enhances the convenience and safety of users during driving, allowing them to complete a series of operations such as navigation settings, music playback, phone calls, and window control through simple voice commands, effectively reducing driver distraction while driving.

[0005] Most smart cars on the market today are equipped with voice interaction systems based on speech recognition and natural language processing technologies. These systems can recognize users' voice commands and convert them into corresponding operation commands, enabling quick responses to functions such as playing a specified song, adjusting the air conditioning temperature, and checking the weather.

[0006] However, while vehicles offer a wide variety of built-in voice functions, users often struggle to fully utilize all available features. After the initial basic voice commands, further adjustments or exploration of other functions typically require users to exit the voice interaction interface and instead navigate to the touchscreen to find and operate specific pages—a cumbersome process that can easily disrupt the driving flow. Summary of the Invention

[0007] One of the purposes of this application is to provide an interaction method, device, computer equipment, and storage medium to address the problem of how to improve the convenience of interaction between a vehicle and its users.

[0008] To achieve the above objectives, the technical solution adopted in this application is as follows:

[0009] An interaction method includes: determining a user-related target intent based on multimodal information related to a target vehicle; initiating a dialogue service flow, the dialogue service flow including: sending an instruction corresponding to the target intent to an application to which an atomic service function for implementing the target intent belongs, so that the application invokes the atomic service function for implementing the target intent according to the instruction.

[0010] In one possible implementation, the application for distribution in the target vehicle sends the identifier of the recommended atomic service function to the middleware.

[0011] In one possible implementation, middleware is used to receive the identifier of the recommended atomic function corresponding to the atomic service function used to achieve the target intent; based on the identifier of the recommended atomic function, the interface component of the recommended atomic function is found; and the user interface component of the recommended atomic function is displayed in the dialog service flow interface.

[0012] In one possible implementation, the number of recommended atomic functions is multiple; in the dialog service flow interface, the controls for displaying recommended atomic functions include: combining multiple interface components to obtain an interface component combination; and displaying the interface component combination.

[0013] In one possible implementation, the above method also includes: invoking the recommendation atomic function to which the user-selected user interface component belongs.

[0014] In one possible implementation, invoking the recommended atomic function to which the user-selected user interface component belongs includes: invoking the recommended atomic function to which the user-selected user interface component belongs through the software development kit of the recommended atomic function to which the user-selected user interface component belongs.

[0015] In one possible implementation, the above method further includes: in response to detecting an operation that recommends an option for a visible-to-speakable operation, invoking the corresponding atomic service function on the business side.

[0016] In one possible implementation, the target intent related to the user is determined based on multimodal information related to the target vehicle, including: using a target model on a cloud server to determine the target intent based on multimodal information related to the target vehicle.

[0017] An interactive device includes: an intent determination unit for determining a target intent related to a user based on multimodal information related to a target vehicle; and an initiation unit for initiating a dialogue service flow, the dialogue service flow including: sending an instruction corresponding to the target intent to an application to which an atomic service function for implementing the target intent belongs, so that the application invokes the atomic service function for implementing the target intent according to the instruction.

[0018] A computer device includes: a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the interaction method of the first aspect or any corresponding embodiment described above.

[0019] A vehicle comprising:

[0020] The memory and the processor are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the interaction method of the first aspect or any corresponding embodiment described above.

[0021] A computer-readable storage medium storing computer instructions for causing a computer to perform the interactive method of the first aspect described above or any corresponding embodiment thereof.

[0022] A computer program product includes computer instructions for causing a computer to perform the interactive method described in the first aspect or any of its corresponding embodiments.

[0023] The beneficial effects of this application are:

[0024] Based on multimodal information related to the target vehicle, the target intent related to the user is determined. A dialogue service flow is initiated, which includes sending instructions corresponding to the target intent to the application containing the atomic service function used to implement the target intent. This allows the application to invoke the atomic service function to achieve the target intent, eliminating the need for the user to perform a series of operations on the vehicle's screen to trigger the intended action, thus simplifying user operations and improving the convenience of vehicle-user interaction. Simultaneously, multimodal information related to the target vehicle is acquired, and the target intent is determined based on the target model and the multimodal information related to the target vehicle. The multimodal information related to the target vehicle has a strong correlation with the user's intent. The determined target intent is highly accurate, and the atomic service function is invoked relatively accurately. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0026] Figure 1 is a schematic diagram of an example architecture of an interactive system that can be used to execute the interactive methods provided in the embodiments of this application;

[0027] Figure 2 is a flowchart illustrating the interaction method provided in an embodiment of this application;

[0028] Figure 3 is a schematic diagram illustrating an example of determining target intent;

[0029] Figure 4 is a flowchart of an example of determining target intent;

[0030] Figure 5 is a flowchart of an example of determining the target intent related to the navigation APP on the target vehicle;

[0031] Figure 6 is a flowchart of an example of determining target intent related to a music app on a target vehicle;

[0032] Figure 7 is a schematic diagram of an example of recommended songs and sound effects;

[0033] Figure 8 is a system architecture diagram of an example of the interaction between the target vehicle and the user;

[0034] Figure 9 shows the identifiers and components of the atomic service function;

[0035] Figure 10 is a schematic diagram of the hardware structure of a computer device according to an embodiment of this application. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0037] Referring to Figure 1, it shows a schematic diagram of an example architecture of an interactive system that can be used to perform the interactive methods provided in the embodiments of this application.

[0038] An example architecture of an interactive system that can be used to execute the interactive methods provided in the embodiments of this application includes: a data acquisition unit, a data processing unit, and an atomic service function unit.

[0039] The data acquisition unit collects voice data from user interactions, images of the target vehicle's internal and external environments, and sensor data from the target vehicle. The data processing unit includes a voice data processing module, an image processing module, a sensor data processing module, an encoding module, and a target model. Atomized service functions include the atomic service functions of each app on the target vehicle that possesses atomic service capabilities. An atomic service function can be understood as the smallest functional unit, capable of performing a single operation.

[0040] The voice data processing module can perform speech recognition on the user's interactive speech to obtain the speech recognition results. The voice data processing unit can use a sound source localization algorithm to determine the user's location within the target vehicle. The voice data processing unit can also identify the user by matching the user's interactive voiceprint features with preset voiceprint features. The preset voiceprint features are those of users whose voiceprint features have been previously recorded.

[0041] The image processing module can process images of the interior environment of the target vehicle to determine the user's age group, gender, seating position, and identity information. The image processing unit may include a Driver Monitoring System (DMS) and an Occupant Monitoring System (OMS). The Driver Monitoring System monitors the driver's gaze direction in real time. The Occupant Monitoring System can process images of the vehicle's interior environment captured by the vehicle's cameras to determine the passengers' gender, age, gestures, and seating position.

[0042] The sensor data processing module can process sensor data collected by the target vehicle's sensors to determine information related to the target vehicle's driving and weather-related information about the target vehicle's location. Driving-related information may include the target vehicle's speed, remaining fuel level, or remaining battery charge. Weather-related information about the target vehicle's location may include the temperature and weather type, such as rain or snow.

[0043] The encoding module is used to encode the multimodal information related to the target vehicle to obtain a representation of the multimodal information related to the target vehicle.

[0044] It should be noted that the target model can be on the target vehicle or on a cloud server.

[0045] The target model is used to receive representations of multimodal information related to the target vehicle and predict the representation of the target intent. Thus, the target intent is determined. The target model is a trained neural network. Before the interaction method provided in this application embodiment is executed for the first time, the target model is trained using training samples and annotation operations on the training samples.

[0046] As an example, the target model is a pre-trained language model. As another example, the pre-trained language model used as the target model is a Large Language Model (LLM). When the input to the target model is natural language text corresponding to natural language speech, the pre-trained language model performs Natural Language Processing (NLP) tasks.

[0047] Training the target model allows it to learn: which information from the target model's input to predict actions that satisfy the user's interactive voice instructions, and how to determine the target intent. Training samples for the target model can include training information related to the multimodal information relevant to the target vehicle. For any given piece of information, if the multimodal information relevant to the target vehicle includes that information, then that information is classified as belonging to the multimodal information relevant to the target vehicle. The labeled operations of the target model's training samples can be understood as: the expected output operations of the target model when the training samples are input. During training, the loss between the operations predicted by the target model and the labeled operations of the training samples can be calculated, and the parameters of the target model can be updated based on this loss.

[0048] Referring to Figure 2, it shows a flowchart of the interaction method provided in an embodiment of this application.

[0049] In step S201, the target intent related to the user is determined based on the multimodal information related to the target vehicle.

[0050] The multimodal information related to the target vehicle includes: the speech recognition results of the user's interactive voice, the parameter information of the target vehicle, the external environment information of the target vehicle, the user's action information, the user's seating location information, the user's attribute information, the user's identity information, information related to the APP on the target vehicle determined during the historical time period of using the APP on the target vehicle, navigation information related to the target vehicle, music information related to the target vehicle, and point of interest information related to the target vehicle. Among them, the point of interest information related to the target vehicle indicates the point of interest reached by the target vehicle during the target historical time period and / or the point of interest reached by the target vehicle multiple times.

[0051] The speech recognition result of the user's interactive voice is obtained by performing speech recognition on the user's interactive voice. Specifically, performing speech recognition on the user's interactive voice refers to performing speech recognition on the voice data of the user's interactive voice.

[0052] As an example, the parameter information of the target vehicle may include: information related to the driving of the target vehicle and information related to the cockpit of the target vehicle.

[0053] As an example, information related to the driving of the target vehicle includes: the target vehicle's speed, the target vehicle's remaining fuel, or the target vehicle's remaining battery charge.

[0054] As an example, information related to the target vehicle's cabin includes: the brightness of the vehicle's infotainment screen, the volume of the vehicle's audio system, the temperature of the vehicle's air conditioning, the airflow of the vehicle's air conditioning, and the angle of the vehicle's seats.

[0055] The target vehicle's external environment information refers to information related to the target vehicle's external environment.

[0056] As an example, the external environment information of the target vehicle includes: weather-related information about the area where the target vehicle is located, driving safety-related information about the road the target vehicle is currently on, and points of interest (POIs) near the target vehicle.

[0057] As an example, weather-related information for the area where the target vehicle is located includes the temperature, the type of weather, etc.

[0058] As an example, information related to the driving safety of the target vehicle on the road it is currently on includes one or more of the following: information indicating whether the road is slippery, information indicating whether the road is icy, information indicating whether the road is bumpy, information indicating whether the road is a high-altitude road, etc.

[0059] As an example, user action information includes at least some of the following: user gestures, the direction the user's hand is pointing, the direction the user's gaze is in, and the user's actions.

[0060] As an example, user attribute information includes: the user's age group, the user's gender, etc.

[0061] User identity information is used to identify the user's identity.

[0062] As an example, a user's identity information includes: the user's identifier.

[0063] The user's seating position information indicates the user's seating location. As an example, the user's seating position is one of the following: driver's seat, front passenger seat, rear left seat, rear middle seat, or rear right seat.

[0064] Navigation information related to the target vehicle may include: the origin of the current navigation to the target vehicle, the destination of the current navigation to the target vehicle, and the navigation route of the current navigation to the target vehicle.

[0065] Music information related to the target vehicle may include: the playlist to which the currently playing song belongs, and the playback progress of the song in that playlist. The playback progress indicates which song has been played so far.

[0066] The end time of the historical period in which the app on the target vehicle was used is before the current time. The app on the target vehicle was used within the historical period in which the app was used.

[0067] As an example, for a navigation app on a target vehicle, the multimodal information related to the target vehicle includes: information related to the navigation app determined during the historical time period of its use. This information includes: the navigation route determined during the historical time period of the app's use, the name of the starting point of the navigation route determined during the historical time period of the app's use, and the name of the destination of the navigation route determined during the historical time period of the app's use.

[0068] As an example, for a music app on a target vehicle, the multimodal information related to the target vehicle includes: information related to the music app on the target vehicle determined during the historical time period of using the music app on the target vehicle. Information related to the music app on the target vehicle includes: identifiers of identified recommended songs.

[0069] Referring to Figure 3, which shows a schematic diagram of an example of determining a target intent.

[0070] The user speaks an interactive voice command, triggering the target vehicle to acquire multimodal information related to the vehicle. The user's interactive voice command is the voice used for interaction. This voice command can be natural language speech, reflecting the user's desired action from the target vehicle. To acquire this multimodal information, the system acquires data collected by the target vehicle, including the user's interactive voice command, sensor data, images of the target vehicle's internal environment, and images of the target vehicle's external environment. The data collected by the target vehicle is processed to obtain the multimodal information related to the target vehicle. This multimodal information includes at least some of the following: the speech recognition result of the user's interactive voice command, the target vehicle's parameter information, the target vehicle's external environment information, the user's action information, the user's attribute information, the user's identity information, and information related to the app on the target vehicle determined during the historical time period of app usage on the target vehicle.

[0071] Each piece of information in the multimodal information related to the target vehicle is encoded to obtain a representation of the multimodal information related to the target vehicle.

[0072] The representation of multimodal information related to the target vehicle is input into the target model, and the target model outputs the target intent.

[0073] In step S201, multimodal information related to the target vehicle can be input into the target model, and the target model can output the target intent.

[0074] The target model can be on the target vehicle or on a cloud server.

[0075] It should be noted that the target model can use a portion of the multimodal information related to the target vehicle to predict the target's intent.

[0076] As an example, the interior temperature of the target vehicle is high, and the user's hand is pointing towards the window and making a downward gesture, indicating the user's intention to open the window.

[0077] As an example, a user says the interactive voice command, "Play songs suitable for people of my age." The intended message is to play songs suitable for users in the age group to which this user belongs.

[0078] As an example, a user says the interactive voice command, "Move your seat back a little." The intended action is to move the user's seat back.

[0079] Referring to Figure 4, a flowchart of an example of determining target intent is shown.

[0080] The interior environment image of the target vehicle indicates the internal environment of the target vehicle. Processing the interior environment image of the target vehicle can yield information such as the user's age group and gestures. The exterior environment image of the target vehicle indicates the external environment of the target vehicle. Processing sensor data and the exterior environment image and sensor data yields information about the exterior environment of the target vehicle. The exterior environment information of the target vehicle can include: weather-related information, i.e., weather-related information about the region where the target vehicle is located; information related to vehicle driving safety, i.e., road-related information about the target vehicle's current location; and nearby points of interest (POIs). Processing the user's interactive voice data yields the user's seating location and the speech recognition results of the user's interactive voice.

[0081] It should be noted that the various information shown in Figure 4 represents the multimodal information related to the target vehicle that may be used to determine the target's intent. The target model can use a portion of the multimodal information related to the target vehicle to predict the target's intent. In other words, only the multimodal information related to the target vehicle is provided to the target model, and the target model decides which information from the multimodal information related to the target vehicle to use to determine the target's intent.

[0082] Each piece of information in the multimodal information related to the target vehicle is encoded to obtain a representation of the multimodal information related to the target vehicle.

[0083] The multimodal information related to the target vehicle is input into the target model, and the target model outputs a representation of the target intent. Thus, the target intent is determined.

[0084] Referring to Figure 5, it shows a flowchart of an example of determining the target intent associated with a navigation app on a target vehicle.

[0085] The user speaks the interactive voice command, "What kind of community is that?". The speech recognition result of the user's interactive voice command, obtained from the multimodal information related to the target vehicle, is "What kind of community is that?". The external environment information of the target vehicle in the multimodal information related to the target vehicle includes: multiple points of interest (POIs) near the target vehicle, and the supporting facilities for those POIs. The user's action information in the multimodal information related to the target vehicle includes: the direction the user's hand is pointing. The target intent is: to find the name of the POI with the type "community" in the direction the user is pointing from the multiple POIs near the target vehicle.

[0086] Referring to Figure 6, it shows a flowchart of an example of determining a target intent related to a music app on a target vehicle.

[0087] When the target vehicle is traveling along the navigation route provided in the navigation information associated with the target vehicle, the user utters the interactive voice command "Play a song." Multiple users inside the target vehicle, including children, are processed. The interior image of the target vehicle is processed to determine the age group to which the users belong. The destination in the navigation information associated with the target vehicle is an amusement park. Based on the children's age group and the destination (amusement park), the target model determines the target intent: to play cheerful children's songs.

[0088] In step S202, a dialogue service flow is initiated. The dialogue service flow includes sending an instruction corresponding to the target intent to the application to which the atomic service function used to implement the target intent belongs, so that the application invokes the atomic service function used to implement the target intent according to the instruction.

[0089] In this embodiment of the invention, each app on the target vehicle that has atomic service functions can have multiple atomic service functions. When the dialogue flow service is started, instructions for realizing the target intent can be sent to the corresponding application, so that the application can call the atomic service functions for realizing the target intent according to the instructions.

[0090] As an example, the vehicle center app on the target vehicle has atomic service functions such as an atomic service function for opening windows and an atomic service function for closing windows. The air conditioning app on the target vehicle has atomic service functions such as an atomic service function for turning on the air conditioning, an atomic service function for turning off the air conditioning, and an atomic service function for adjusting the air conditioning temperature.

[0091] Once the target intent is determined, the dialogue service flow can be initiated, invoking the atomic service functions used to achieve the target intent. After invoking the atomic service functions used to achieve the target intent, the invocation results corresponding to the atomic service functions used to achieve the target intent can be displayed in the dialogue service flow interface, which is displayed on the in-vehicle infotainment screen of the target vehicle.

[0092] The result of an atomic service function used to achieve a target intent is either the result obtained through that atomic service function or an indication that the target intent has been completed. For example, if the atomic service function used to achieve the target intent is an "Air Conditioner Turn On" atomic service function in the air conditioning app of the target vehicle, then the result of the call to that atomic service function is a text indicating that the air conditioning has been turned on. If the atomic service function used to achieve the target intent is an atomic service function for searching POIs in the search app of the target vehicle, then the result of the call to that atomic service function is the search result obtained by calling the atomic service function for searching POIs in the search app of the target vehicle.

[0093] As an example, a user becomes interested in a neighborhood the target vehicle is passing through while it is driving. The user says, "What neighborhood is ahead?" The target intent is to search for the name of the neighborhood ahead of the target vehicle. The search app on the target vehicle uses its atomic service function for searching Points of Interest (POIs) to search for the name of the neighborhood ahead. The user then says, "How much does it cost to rent a three-bedroom apartment?" The target intent is to search for the rent of a three-bedroom apartment in the neighborhood ahead of the target vehicle. The real estate agency app on the target vehicle uses its atomic service function for searching rent to search for the rent of a three-bedroom apartment in the neighborhood ahead. Finally, the user says, "Call to inquire about the situation." The target intent is to call the relevant personnel on the real estate transaction platform. The Bluetooth phone app uses its atomic call function to call the relevant personnel on the real estate transaction platform.

[0094] As an example, a user speaks a voice command related to the music app on the target vehicle. The target model determines the target intent related to the music app on the target vehicle and invokes the corresponding atomic service function of the music app on the target vehicle to realize the target intent.

[0095] In this embodiment, based on multimodal information related to the target vehicle, a user-related target intent is determined. A dialogue service flow is initiated, which includes sending an instruction corresponding to the target intent to the application containing the atomic service function used to implement the target intent. This allows the application to invoke the atomic service function based on the instruction, thus realizing the target intent. The user does not need to perform a series of operations on the vehicle's screen to trigger an action with the user's intent, simplifying the user's operation and improving the convenience of vehicle-user interaction. Simultaneously, multimodal information related to the target vehicle is acquired, and the target intent is determined based on the target model and the multimodal information related to the target vehicle. The multimodal information related to the target vehicle has a strong correlation with the user's intent. The determined target intent is highly accurate, and the atomic service function is invoked relatively accurately.

[0096] In one possible implementation, the above method further includes: the application for distribution in the target vehicle sending the identifier of the recommended atomic service function to the middleware.

[0097] Recommended atomic service functions are those associated with the atomic service function used to achieve the target intent, which are then invoked. Each recommended atomic service function is uniquely identified by its identifier. When the application of the target vehicle decides to distribute a recommended atomic service function, it needs to send the identifier of the recommended atomic service function to the middleware. The middleware acts as a bridge, receiving service requests from the application and scheduling and coordinating the corresponding recommended atomic service functions based on those requests.

[0098] Referring to Figure 7, which shows a schematic diagram of an example of recommended songs and sound effects.

[0099] The user says, "Recommend a song." This initiates a dialogue service flow, invoking the atomic service function of the music app on the target vehicle to search for recommended songs. The recommended song is then displayed on the vehicle's infotainment screen.

[0100] The atomic service function used to achieve the target intent corresponds to the recommendation atomic service function, which is the atomic service function of the music APP on the target vehicle used to set the sound effects of recommended songs.

[0101] In this embodiment, by analyzing the atomic service functions that the user has already invoked, the middleware can more accurately understand the user's preferences and needs. The middleware can intelligently recommend other related atomic service functions, eliminating the need for the user to search and filter atomic service functions themselves. This improves the efficiency of the user in obtaining and using services and enhances the convenience of vehicle-user interaction.

[0102] In one possible implementation, the method further includes: receiving, via middleware, the identifier of a recommended atomic function corresponding to the atomic service function used to achieve the target intent; identifying the UI component of the recommended atomic function based on its identifier; and displaying the UI component of the recommended atomic function in the dialog service flow interface.

[0103] The user interface component for the recommended atomic service function corresponding to the atomic service function used to achieve the target intent may include: the name of the recommended atomic service function corresponding to the atomic service function used to achieve the target intent, and parameters related to the recommended atomic service function corresponding to the atomic service function used to achieve the target intent.

[0104] As an example, the target intent is to recommend songs. The atomic service function used to implement this intent is the song search service function of the music app on the target vehicle. A dialog service flow is initiated, calling the song search service function of the music app on the target vehicle to search for recommended songs. The corresponding recommendation atomic service function used to implement the target intent is the music app's atomic service function for setting sound effects for the recommended songs.

[0105] After the application used for distribution in the target vehicle sends the identifier of the recommended atomic service function to the middleware, the middleware receives the identifier of the recommended atomic function corresponding to the atomic service function used to realize the target intent. Based on the identifier of the recommended atomic function, the middleware can find the user interface component of the recommended atomic function corresponding to the identifier of the recommended atomic function, and then display the control of the recommended atomic function in the dialog service flow interface.

[0106] Users can invoke the recommended atomic service function corresponding to the atomic service function that achieves the target intent by clicking the user interface component corresponding to the recommended atomic service function, or by speaking instructions to invoke the recommended atomic service function corresponding to the atomic service function that achieves the target intent.

[0107] In this embodiment, after receiving the identifier of the recommended atomic function corresponding to the atomic service function used to achieve the target intent using the middleware, the user interface component of the recommended atomic function is found through the identifier of the recommended atomic function. Then, the user clicks on the user interface component or calls the recommended atomic service function corresponding to the atomic service function used to achieve the target intent by voice, which facilitates the user and the vehicle to continue to interact and improves the convenience of the interaction between the vehicle and the user.

[0108] In one possible implementation, the number of recommended atomic functions is multiple. The controls for displaying recommended atomic functions in the dialog service flow interface include: combining multiple user interface components to obtain a user interface component combination; and displaying the user interface component combination.

[0109] There are multiple recommended atomic functions. The middleware, based on data changes, first processes the atomic function components corresponding to the voice request input by the current user to the voice application, and then combines the user interface components corresponding to each recommended atomic function to obtain a user interface component combination. The user interface combination is then displayed. Then, the user interface components of the recommended atomic service functions corresponding to the atomic service functions that realize the target intent are displayed. This allows users to invoke the recommended atomic service functions that realize the target intent by clicking on the user interface components of the recommended atomic service functions that realize the target intent, or by speaking instructions to invoke the recommended atomic service functions that realize the target intent.

[0110] Referring to Figure 8, which shows a system architecture diagram of an example of the interaction between the target vehicle and the user.

[0111] Multimodal information related to the target vehicle is input into the target model, which outputs the target intent. The skill distribution function of the target vehicle's voice app will, for each of at least one target intent, send the corresponding instruction to the app belonging to the atomic service function used to implement the target intent in the target vehicle's vehicle center app, navigation app, music app, etc. The app belonging to the atomic service function used to implement the target intent calls the atomic service function used to implement the target intent according to the instruction corresponding to the target intent, obtains the call result corresponding to the atomic service function used to implement the target intent, and sends the call result to the middleware. The middleware displays the UI component or combination of multiple UI components corresponding to the call result. In response to detecting the user's operation on a certain UI component, it generates the corresponding operation instruction and sends the instruction corresponding to the target intent to the app of the corresponding UI component of the target vehicle and the app of the corresponding UI component's atomic service function used to implement the target intent.

[0112] Please refer to Figure 9, which shows the identifier and component diagram of the atomic service function.

[0113] The system receives voice commands from users or event triggers to determine the target intent corresponding to the multimodal information. For each target intent, atomic functions are combined to determine the target atomic function. The voice application distributes vertical skills to relevant applications, such as navigation, music, and vehicle center apps, through the voice SDK. The application side of each application processes the voice commands, executes the command functions, and synchronizes the corresponding data to a component in the Kubernetes cluster (Vertical Pod Autoscaler, VPA) used to automatically adjust Pod resource requests (CPU and memory) through the CUI middleware SDK. Based on data changes, the middleware first combines the atomic functions corresponding to the current user's voice request to the voice application and displays them on the interface through a template UI. Then, it requests the cloud-based large model or makes corresponding recommendations based on the offline large model. Users can click or speak to select the recommended option, which calls the business side to execute the corresponding function. The combination of atomic functions can be determined by marking each atomic function interface with a unique identifier (id), and then associating the atomic UI display with the corresponding identifier. For example: Air conditioner switch ID—Air conditioner switch control interface—Air conditioner switch UI component; Air conditioner temperature ID—Air conditioner switch read interface—Air conditioner temperature UI component; Air conditioner ** ID—Air conditioner ** read / write interface—Air conditioner ** UI component. Another example: Music information ID—Music content read interface—Music content UI component; Sound effect information ID—Sound effect content read interface—Sound effect content UI component; Yet another example: Other atomic capability IDs—Read / write interface—UI component, etc.

[0114] In this embodiment, by combining UI components, multiple related atomic functions can be integrated into one interface, allowing users to see all recommended functions at a glance, facilitating continued interaction between users and the vehicle, and improving the convenience of interaction between the vehicle and the user.

[0115] In one possible implementation, the recommended atomic function of the user interface component selected by the user is invoked.

[0116] Users can send selection operations for recommended atomic functions by clicking the mouse, typing on the keyboard, touching the screen, or using voice. Correspondingly, the computer device can respond to the selection operation by determining the recommended atomic function to which the user interface component selected by the user belongs from the combination of user interface components, and then calling the recommended atomic function to which the user interface component selected by the user belongs.

[0117] As an example, imagine a user driving a smart car who wants to play music using the in-car entertainment system. The user can select a user interface component (such as a "play music" button) or speak a corresponding voice command through the smart cockpit's user interface (e.g., a touchscreen or voice assistant). When the user selects or triggers this user interface component, the appropriate recommendation atomic function is invoked to execute the user's request. For example, if the user selects the "play music" button, a recommendation atomic function responsible for music playback is invoked, which selects a song from the user's music library and begins playback. Similarly, if the user requests to play a specific song or artist through a voice assistant, the corresponding recommendation atomic function is invoked to recognize the user's voice command and execute the playback operation accordingly.

[0118] In this embodiment, users can select appropriate interface components from the recommended atomic functions to interact with according to their needs and preferences, thereby obtaining services that best meet their needs and improving user satisfaction.

[0119] In one possible implementation, the above-mentioned invocation of the recommended atomic function to which the user-selected user interface component belongs includes: invoking the recommended atomic function to which the user-selected user interface component belongs through the software development kit of the recommended atomic function to which the user-selected user interface component belongs.

[0120] A Software Development Kit (SDK) provides a set of software tools, interfaces, libraries, etc., to simplify the software development, testing, and deployment process. In this embodiment, the SDK is used to receive and execute recommended atomic functions belonging to the user-selected user interface component. Specifically, the middleware in the SDK integrates the SDK for all atomic functions. Specifically, the SDK corresponding to the target atomic function is matched from the middleware's SDK library, and the recommended atomic function belonging to the user-selected user interface component is invoked according to that SDK.

[0121] In this embodiment, calling the recommended atomic function of the user interface component selected by the user through the SDK of the recommended atomic function of the user interface component can significantly improve development efficiency and reduce development costs.

[0122] In one possible implementation, in response to detecting an operation that recommends an option for a visible-to-speakable operation, the corresponding atomic service function is executed by invoking the business side.

[0123] When a user is detected to be performing an action on the recommended option for the "Visible and Sayable" operation, the corresponding atomic service function can be executed automatically by calling the business side.

[0124] As an example, a user is driving a smart car and browsing various function options through the smart cockpit's touchscreen interface. The smart cockpit system supports "what you see is what you speak," meaning the user can perform corresponding operations by speaking the names of the options visible on the screen. Specifically, the smart cockpit's touchscreen interface displays a series of function options, such as "navigate to destination," "play music," and "adjust air conditioning temperature." Voice recognition technology detects when the user speaks the name of a visible option on the screen, such as "play music." Once the target intent is confirmed (i.e., the "play music" option is spoken and selected), it invokes the atomic function associated with the "play music" function to realize that target intent.

[0125] In this embodiment, by operating on the recommended options that can be viewed and spoken, the corresponding atomic service function is executed by calling the business side. Users do not need to manually click the button on the touch screen; they only need to speak the name of the visible option to perform the operation, which improves the convenience and efficiency of the operation.

[0126] This application also provides an interactive device for implementing the above-described method embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "unit" can refer to a combination of software and / or hardware that performs a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated. The devices in this application are presented in the form of functional units, where a functional unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above-described functions.

[0127] An interactive device includes: an intent determination unit, configured to determine a target intent related to a user based on multimodal information related to a target vehicle; and an initiation unit, configured to initiate a dialogue service flow, the dialogue service flow including: sending an instruction corresponding to the target intent to an application to which an atomic service function for implementing the target intent belongs, so that the application invokes the atomic service function for implementing the target intent according to the instruction.

[0128] Furthermore, the aforementioned apparatus also includes: an identifier sending unit, for the target vehicle's application for distribution to send the identifier of the recommended atomic service function to the middleware.

[0129] Furthermore, the above-mentioned apparatus also includes: an identifier receiving unit, used to receive, using middleware, an identifier of a recommended atomic function corresponding to an atomic service function used to achieve a target intent; a user interface component lookup unit, used to look up the user interface component of the recommended atomic function based on the identifier of the recommended atomic function; and a user interface component display unit, used to display the user interface component of the recommended atomic function in the dialog service flow interface.

[0130] Furthermore, it is recommended that there be multiple atomic functions, and a user interface component display unit, which is used to combine multiple user interface components to obtain a user interface component combination; and to display the user interface component combination.

[0131] Furthermore, the above-mentioned device also includes: a first atomic function invocation unit, used to invoke the recommended atomic function to which the user interface component selected by the user belongs.

[0132] Furthermore, the first atomic function invocation unit is used to invoke the recommended atomic function of the user interface component selected by the user through the software development kit of the recommended atomic function of the user interface component selected by the user.

[0133] Furthermore, the aforementioned apparatus also includes: a second atomic function invocation unit, used to invoke the corresponding atomic service function of the business terminal in response to detecting an operation that recommends an option for a visible-to-speakable operation.

[0134] Furthermore, the intent determination unit is used to determine the user's target intent based on multimodal information related to the target vehicle, using the target model on the cloud server.

[0135] Referring to Figure 10, which is a schematic diagram of the hardware structure of a computer device according to an embodiment of this application, the computer device includes one or more processors 10, a memory 20, and interfaces for connecting the various components, including high-speed interfaces and low-speed interfaces. The various components communicate with each other using different buses and can be installed on a common motherboard or otherwise as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to the interface). In some alternative embodiments, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple devices can be connected, each providing some necessary operations (e.g., as a server array, a set of blade servers, or a multiprocessor system). The processor 10 can be a central processing unit, a network processor, or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip can be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device can be a complex programmable logic device (CLP), a field-programmable gate array (FPGA), a general-purpose array logic (GPRS), or any combination thereof. The memory 20 stores instructions executable by at least one processor 10 to cause the processor 10 to perform the methods shown in the above embodiments. The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on vehicle usage, etc. Furthermore, the memory 20 may include high-speed random access memory and non-transient memory, such as at least one disk storage device, flash memory device, or other non-transient solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, which can be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof. The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk, or solid-state drive; the memory 20 may also include combinations of the above types of memory. The computer device also includes an input device 30 and an output device 40. The processor 10, memory 20, input device 30, and output device 40 may be connected via a bus or other means. The input device 30 can receive input digital or character information, and generate key signal inputs related to user settings and function control of the computer device, such as touch screen, keypad, mouse, trackpad, touchpad, joystick, one or more mouse buttons, trackball, joystick, etc.The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED), and a haptic feedback device (e.g., a vibration motor). The display device includes, but is not limited to, liquid crystal displays, light-emitting diodes, displays, and plasma displays. In some alternative embodiments, the display device may be a touchscreen.

[0136] This application also provides a computer-readable storage medium. The methods described in this application can be implemented in hardware or firmware, or implemented as recordable on a storage medium, or implemented as computer code downloaded over a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and subsequently stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the methods shown in the above embodiments are implemented.

[0137] A portion of the embodiments of this application can be applied as a computer program product, such as computer program instructions. When executed by a computer, these instructions, through the operation of the computer, can invoke or provide the methods and / or technical solutions according to this application. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, and installation package files. Accordingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions; the computer compiling the instructions and then executing the corresponding compiled program; the computer reading and executing the instructions; or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0138] The above embodiments are merely preferred embodiments provided to fully illustrate this application, and the scope of protection of this application is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on this application are all within the scope of protection of this application.

Claims

1. An interaction method, characterized in that, The method includes: Based on multimodal information related to the target vehicle, determine the target intent related to the user; Initiate a dialogue service flow, which includes sending an instruction corresponding to a target intent to the application to which the atomic service function for implementing the target intent belongs, so that the application invokes the atomic service function for implementing the target intent according to the instruction.

2. The interaction method of claim 1, wherein, The method further includes: The application used for distribution in the target vehicle sends the identifier of the recommended atomic service function to the middleware.

3. The interaction method of claim 2, wherein, The method further includes: Utilize middleware to receive the identifier of the recommended atomic function corresponding to the atomic service function used to achieve the target intent; Based on the identifier of the recommended atomic functions, find the user interface components of the recommended atomic functions; The user interface components for recommending atomic functions are displayed in the dialogue service flow interface.

4. The interaction method of claim 3, wherein, The recommended number of atomic functions is multiple; the controls for displaying the recommended atomic functions in the dialog service flow interface include: Combine multiple user interface components to obtain a user interface component composition; This displays the combination of user interface components.

5. The interaction method of claim 4, wherein, The method further includes: Invoke the recommended atomic function of the user interface component selected by the user.

6. The interaction method of claim 5, wherein, The invocation of the recommended atomic function to which the user-selected user interface component belongs includes: The recommended atomic function of the user interface component is invoked through the software development kit of the recommended atomic function of the user interface component selected by the user.

7. The interaction method of claim 1, wherein, The method further includes: In response to detecting an action that recommends an option for a visible-to-speakable action, the corresponding atomic service function is executed by calling the business side.

8. The interaction method of claim 1, wherein, The step of determining the user-related target intent based on multimodal information related to the target vehicle includes: By utilizing the target model on the cloud server, and based on multimodal information related to the target vehicle, the target intent related to the user is determined.

9. An interactive device, characterized by The device includes: The intent determination unit is used to determine the target intent related to the user based on multimodal information related to the target vehicle; A startup unit is used to start a dialogue service flow, which includes sending an instruction corresponding to a target intent to the application to which the atomic service function for implementing the target intent belongs, so that the application invokes the atomic service function for implementing the target intent according to the instruction.

10. A computer device, comprising: include: A memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, the processor executing the computer instructions to perform the interactive method of any one of claims 1 to 8.

11. A vehicle characterized by comprising: include: A memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, the processor executing the computer instructions to perform the interactive method of any one of claims 1 to 8.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the interactive method of any one of claims 1 to 8.

13. A computer program product, characterised in that, Includes computer instructions for causing a computer to perform the interactive method of any one of claims 1 to 8.

Images