Method for operating a virtual assistant in a vehicle, and information technology system

Abstract

The invention relates to a method for operating a virtual assistant in a vehicle (1), wherein the virtual assistant controls a vehicle function and / or brings about the outputting of information to a user (2), at least taking into account a context for assisting the user (2) as determined for the vehicle (1). The method according to the invention is characterized in that interactions carried out by the user (2) are detected, wherein a trigger module (3) classifies the interactions so as to generate interaction information; the context at least of the vehicle (1) is detected by a context module (4) and provided by the context module (4) in the form of context information; when a new interaction is detected, the interaction information generated for this interaction and the context information available in the process are supplied to a conclusion module (5) that processes the interaction information and context information with a suitably trained large basic model (6) and outputs, as a processing result, conclusion information comprising control instructions, able to be processed by a computing unit (7) in the vehicle (1), for controlling the vehicle function and / or output information for output to the user (2) via output means belonging to the vehicle.

Description

[0001] Mercedes-Benz Group AG

[0002] Method for operating a virtual assistant in a vehicle and information technology system

[0003] The invention relates to a method for operating a virtual assistant in a vehicle according to the type defined in more detail in the preamble of claim 1 and to an information technology system for carrying out the method.

[0004] Using a wide variety of devices such as smartphones, tablets, smart home devices, and the like, information can be accessed from the internet or functions can be operated, such as switching smart home lighting on and off, managing a shopping list, setting an alarm, and so forth. The number and range of usable functions are constantly increasing, allowing users to rely on such digital tools for support in more and more situations. While users clearly benefit from the increasing availability of new functions, this also creates the problem that it is impossible for individuals to maintain a comprehensive overview of all available features. Consequently, the risk increases that functionalities will be developed that are either not used or underutilized because users are unaware of their existence or how to operate them.

[0005] Voice assistants have become established for assisting users. A voice assistant is able to process spoken language from a user using methods of computational linguistics, also known as natural language processing (NLP). This allows it to capture the intention and semantic content contained in a voice command. Artificial intelligence, particularly based on artificial neural networks, is usually used to process the corresponding voice input. The voice assistant can provide information to the user or control functions. German patent DE 102022 209 925 A1 discloses the cloud-based management of user accounts, user profiles, and user devices in connection with a vehicle. A user can save individual settings for a vehicle in their user profile. For example, the user can save a preferred seat configuration in their user profile.When the user approaches the vehicle, this is detected, and the user's profile is automatically loaded from a server and activated in the vehicle. The vehicle is then configured according to the information stored in the user profile.

[0006] Furthermore, DE 102018 133670 A1 discloses a method and a device for generating control signals to assist vehicle occupants. First, the user is identified. A user-specific, rule-based data system, stored externally in the vehicle, is transferred to the vehicle via a suitable interface. This rule-based data system contains a wide variety of relationships for different vehicle contexts and actions to be performed in each context. Suitable control signals are output for the actions assigned to the context to trigger their execution in the vehicle. A confidence value can be assigned to the rules stored in the data system, indicating the probability that the determined rule result actually corresponds to the user's preference in the given situation.Before actions are executed in the vehicle, they can first be presented to the user, who then assesses whether or not they wish to perform the corresponding action. This can be used to automatically adapt the rule-based data system, thereby increasing the confidence level of the respective rules for future situations. If a sufficiently high confidence level is achieved, the resulting action can be executed automatically. While the methods disclosed in this publication allow for convenient user assistance, defining the rules underlying the rule-based data system requires significant effort. Furthermore, a comparatively large amount of user data from historical interactions must be collected and processed to learn user preferences.Due to its rule-based operation, the system implemented in the vehicle will always behave the same way in the same vehicle context. This can negatively impact the user experience, as the user can predict the system's behavior. Furthermore, DE 102021 132 143 A1 discloses a method for operating a motor vehicle and a motor vehicle itself. Data relating to the motor vehicle, its environment, and / or a user are collected. If this data meets certain trigger conditions, instructions for using vehicle functions in a specific way are issued to the user. After the user performs a predefined operating action, the vehicle function in question is activated.

[0007] Furthermore, DE 102017216 916 A1 discloses a method for operating a motor vehicle control device to offer a driver a function selection and a control device. In this method, at least one selectable vehicle function is provided in a function selection displayed on an output device, and this function is executed after selection by the user. The output of the function selection can be dynamically adapted depending on the driving situation.

[0008] Furthermore, DE 102018211 973 A1 discloses a proactive, context-based provision of service recommendations in vehicles. This involves collecting data that describes a user's vehicle usage. This data is then analyzed using artificial intelligence to generate a service recommendation tailored to the user.

[0009] Furthermore, DE 102022 131 646 A1 discloses a method for providing a proactive recommendation message from a vehicle's digital assistant to the vehicle's driver, as well as a computer-readable medium, a system, and a vehicle. This involves recording the user interaction between the driver and a vehicle software application, along with any contextual information present during that interaction. Based on the frequency with which the vehicle user interacts with the software application, and taking the contextual information into account, a usage scenario is determined. A recommendation message appropriate to this usage scenario is then issued to the driver.

[0010] Furthermore, WO 2017 / 067853 A1 discloses a vehicle user advice system. The present invention is based on the objective of providing an improved method for operating a virtual assistant in a vehicle, which avoids the disadvantages mentioned above.

[0011] According to the invention, this problem is solved by a method for operating a virtual assistant in a vehicle with the features of claim 1. Advantageous embodiments and further developments, as well as an information technology system for carrying out the method, are described in the dependent claims.

[0012] A generic method for operating a virtual assistant in a vehicle, wherein the virtual assistant controls a vehicle function and / or causes the output of information to the user, taking into account at least a context determined for the vehicle, provides that interactions performed by the user are recorded, wherein a trigger module classifies the interactions to generate interaction information; the context of at least the vehicle is recorded by a context module and provided by the context module in the form of context information;wherein, according to the invention, when a new interaction is detected, the interaction information generated for this interaction and the context information present are fed to an inference module, which processes the interaction information and context information with a suitably trained large base model and outputs inference information as a processing result, comprising control commands processable by a computing unit in the vehicle for controlling the vehicle function and / or output information for output to the user via vehicle-specific output means; and wherein the large base model outputs different inference information for the same input data, taking into account a random default value.

[0013] The inventive method is based on the idea of ​​providing a flexible virtual assistant in the vehicle. This assistant is not based on predefined rules; rather, its behavior is determined by a suitably trained large foundation model. Foundation models are also referred to as basic models or "foundation models." A large foundation model is thus a "Large Foundation Model." A large foundation model represents a computer model in the field of artificial intelligence. The foundation model is trained using machine learning to solve a given task. During the training process, the underlying artificial neural network processes extensive amounts of data to acquire general knowledge of the world. This gives the foundation model the ability to evaluate situations based on common sense.This allows for appropriate actions to be performed in all possible situations, even without the need to define fixed rules. The data used for training can be obtained, for example, via the internet.

[0014] A well-known category or subcategory of specially adapted large base models are large language models, also known as Large Language Models (LLMs). Large language models are used, for example, in the context of chatbots or voice dialogue systems. However, a large base model is more flexible because it is not only trained to solve a specific problem (in this case, language interaction) but can also solve more general problems. Through specialized training, a large base model can also be trained to artificially generate images or videos, create control commands for robots, and so on. This allows for the creation of further subcategories of large base models.

[0015] Due to its usable general world knowledge and ability to evaluate situations based on common sense, the inference module is capable of issuing appropriate control commands or actions in the respective situation.

[0016] The goal is to provide output information. Since no fixed rules are used, different content can be generated for similar or identical situations, ensuring sufficient variety. The virtual assistant can be used by a wide range of people without prior specific training. Once the large base model is trained, it requires no further adjustments to perform its tasks satisfactorily. This eliminates the need for complex and costly further development work.

[0017] First, user interactions are recorded using the trigger module. The user can be detected using a variety of sensors. For example, the user can be visually captured using cameras. Camera images can be processed using artificial intelligence, particularly machine vision algorithms. This allows characteristic image content to be identified and classified. Furthermore, the user's facial expressions and / or gestures can be recognized and analyzed in camera images. The vehicle can also be equipped with sensors that allow the generation of depth information. These include, for example, radar sensors, motion detectors, and the like. Such sensors can be used to locate the limbs of vehicle occupants within the vehicle and determine their pose. This also allows inferences to be made about the execution of specific operating gestures.Furthermore, the vehicle interior can be acoustically recorded, for example, using microphones. This allows for the capture of sounds spoken by the user, as well as noises emitted by the user, such as groans, grunts, cries of anger, sighs, belches, and the like. Additionally, the user can interact with a wide variety of human-machine interfaces in the vehicle, such as buttons, switches, rotary push-buttons, touch-sensitive displays, and the like. The way the user interacts with a graphical user interface can be recorded. Furthermore, the way the user operates the steering wheel, the accelerator, brake, and / or clutch pedals, whether the user fastens a seatbelt in its designated holder, and similar actions can be recorded and analyzed.The trigger module can be connected to a fieldbus system in the vehicle, such as a CAN bus, an Ethernet data line, or similar, to acquire the relevant information. This allows data to be accessed from sensors in the vehicle or read from control units of vehicle subsystems.

[0018] The trigger module is so named because, when corresponding interaction information is generated, it triggers the execution of the subsequent process steps of the method according to the invention. This means that each time the trigger module detects a relevant user interaction, the context is determined, at least for the vehicle, and this information is fed to the inference module for generating the inference information. The trigger module classifies the user's interaction, assigning each interaction a unique identifier, such as "fasten seatbelt," "adjust side mirror," "accelerate sharply," and the like. To classify the interactions, the trigger module can include information about the source of the corresponding data describing the user's interaction.For example, if a corresponding signal is provided by a seatbelt buckle, the trigger module can determine whether the user has fastened or unfastened their seatbelt. The trigger module can also detect that the user is braking hard by reading information from a control unit associated with the braking system. To classify the user's interactions, the trigger module can employ machine learning methods, such as processing data using artificial neural networks. Such an artificial neural network is trained to process a wide variety of data fed to the trigger module and to perform the appropriate classification. Various training methods are suitable for this purpose, such as supervised learning.

[0019] The context module functions similarly to the trigger module, except that, at least in the first execution loop or iteration, it does not record user interaction but processes all data relevant to determining the vehicle's context. This can include sensor data provided by the vehicle's environmental sensors, such as cameras, external microphones, radar sensors, ultrasonic sensors, LiDARe, and similar devices. Using this sensor data, the vehicle's surroundings can be captured, allowing for the detection of static and dynamic objects. Furthermore, the relative distance between the vehicle and these objects can be determined.For example, the system can determine whether the road surface is wet, the outside temperature, whether the sun is low and could blind the driver, the traffic volume, the type of environment the vehicle is currently in, and similar information. To assess the context of relevant information, the vehicle can also retrieve information from external sources, such as the internet. For instance, traffic reports or the current weather forecast can be accessed this way. The current date and time can also be used to assess the context. A navigation route programmed into the vehicle's navigation system can also be used to evaluate the context.Information describing the vehicle's condition can also be considered for context assessment, such as oil temperature, fuel level, trunk load, the number of passengers, longitudinal and / or lateral acceleration acting on the vehicle, and the like. This information can be output by the context module in its raw form as context information, or it can be prepared for processing by the inference module. For example, the data can be structured and / or the file format changed. The information can also be summarized in a text field. Such data preparation can also be performed using machine learning methods.

[0020] The interaction information and the context information available at the moment of generation are now fed to the inference module. The inference module can execute the large base model itself or communicate with an external computing unit that executes the large base model. Thus, the processing of the information fed to the large base model is possible both within and outside the vehicle. Preferably, the large base model is executed on a server or a server cluster, particularly in the form of a high-performance computing cluster. This allows for timely data processing by particularly complex machine learning models, which is relevant for time-critical applications. The corresponding inference information can then be communicated back to the vehicle. If necessary, a less complex large base model executed locally in the vehicle can be used.The quality of the results may decrease, however, the function of the virtual assistant is possible even without a communicative connection to the computing system.

[0021] Depending on the situation, the inference information contains control commands and / or output information. This allows vehicle functions to be automatically controlled according to the situation and / or information to be output to the user. Visual, acoustic, and haptic output methods can be used for this purpose. For example, text can be displayed on a screen in the vehicle or spoken aloud via computer-generated speech. Artificially generated pictograms, images, photos, animations, videos, and the like, or those read from a data storage device, can also be displayed on a screen. Sounds such as beeps or warning tones can also be emitted via loudspeakers. Furthermore, haptic feedback, such as tactile feedback or vibrations, can be provided to the user using appropriate actuators integrated into user-touched vehicle components.As previously described, the invention provides that the large base model outputs different inference information for the same input data, taking into account a random predefined value. An artificial neural network is characterized by a wide variety of parameters. For example, weighting factors and bias factors are assigned to the neurons of the artificial neural network. The processing method of the artificial neural network can also be adjusted by the so-called temperature value. A high temperature value leads to more complex and creative outputs, while a low temperature value results in a factual and structured output of results. This temperature value can be interpreted as a random predefined value. This makes it possible to generate different outputs for the same input data, thereby preventing monotony in the behavior of the virtual assistant.This means the virtual assistant behaves differently in similar or identical situations, thereby enhancing the user experience. This prevents the virtual assistant from becoming boring or predictable for the user, as it exhibits a new behavior each time.

[0022] An advantageous further development of the method according to the invention provides that a reaction action performed by the user in response to the activation of the vehicle function and / or the output of information is recorded, wherein a personalization module classifies the reaction actions to generate reaction information and, by comparing the reaction information with the respective correlating contextual and inference information, identifies the user's personality traits and stores them in the form of personality information. Thus, the user's acceptance of the actions performed by the virtual assistant can be determined and its degree evaluated.

[0023] The personalization module functions similarly to the triggering module. The reactions are subsequent user interactions to corresponding actions of the virtual assistant. For example, the virtual assistant might lower the target temperature for the vehicle's passenger compartment. In response, the user raises the target temperature again by entering a corresponding manual command. Taking the current context into account, the personalization module then determines, for instance, that the user is wearing a T-shirt, i.e., lightly dressed. This allows the system to store the association that, given the current outside temperature of, say, 24°C, the vehicle's climate control should maintain the interior temperature and not lower it further.For example, the only personality trait that can be stored is the information "user XY gets cold easily".

[0024] For example, the virtual assistant could automatically initiate the playback of a specific song by a preferred artist. A joyful exclamation from the user could then be detected as a reaction. Taking the context into account, the personalization module could, for instance, determine that the user enjoys listening to songs with a relatively high BPM, such as a drum and bass track, when merging onto a highway.

[0025] The personalization module allows the virtual assistant to take the user's personal preferences into account in its behavior. Unlike established approaches, this method does not train an underlying artificial neural network on the user's specific characteristics. Instead, it feeds additional input data describing the user's preferences into the large base model. This eliminates the need for continuous training of the underlying artificial neural network. This reduces the technical effort and also allows for flexible and thus universally applicable use of the underlying artificial neural network for a wide variety of users.

[0026] For example, personality information can be stored as text. Generally, all valid file formats and data structures are suitable. The relationship between the context and the user's reaction does not need to be explicitly stored, as this information is implicitly contained within the personality information in the form of the personality trait. This reduces the storage space required for storing the personality information.

[0027] Ideally, at least one of the following personality traits of the user is identified: likes, dislikes, and / or habits. This allows for a particularly comprehensive description of the user's personality, so that the virtual assistant is highly likely to perform actions appropriate to the user in any given situation.

[0028] A further advantageous embodiment of the method according to the invention provides that the large base model takes personality information into account when generating the inference information. As already mentioned, the virtual assistant can consider personality traits when determining the actions to be performed. This allows the personality information to be used advantageously. Personality information or personality traits can not only be determined automatically by analyzing the user's reactions, but can also optionally be manually specified by the user. For example, users can provide a textual description of their preferences, dislikes, and / or habits via a suitable human-machine interface, such as a keyboard or voice command, which is then recorded and stored accordingly.

[0029] Personality information can be added to the large base model in the form of contextual information or as an additional data set.

[0030] A further advantageous embodiment of the method according to the invention provides that the large base model assigns a confidence level to each control command, and the computing unit automatically executes the respective control command if the underlying confidence level exceeds a defined confidence threshold or if the computing unit otherwise prompts the user to execute the control command. The control command is only executed if the user confirms the request with an action. This prevents the user from experiencing a loss of control due to the automatic action of the virtual assistant. Only those control commands for which a high confidence level is determined are implemented automatically. This means that, with a high degree of probability, the user actually desires the corresponding activation of the vehicle function by the control command.Conversely, if confidence is low, the risk increases that the user will be dissatisfied with the corresponding control of the vehicle function. Such undesired use of vehicle functions is thus prevented. By considering whether the user confirms or rejects the request, the personalization module can generate relevant information and store it as personality traits in the user's profile. This allows for a more reliable assessment of the confidence with which control commands are evaluated.

[0031] According to a further advantageous embodiment of the method according to the invention, the context module is further provided for by retrieving context information from at least one end device that is communicatively coupled to the vehicle. Thus, the context module can not only consider information relating to the vehicle context but also access other data sources. This has already been described in the context of the user's smart home automation. For example, the settings used in the morning for the smart lighting in the user's house can be transferred to the vehicle. The mobile end device could, for example, be the user's smartphone. Corresponding settings for the smart home lighting can be read from an app running on the mobile end device.The mobile device can be connected to the vehicle wirelessly, for example via Wi-Fi, Bluetooth, NFC or similar, or via a wired connection, for example via USB cable or Ethernet cable.

[0032] It is also preferable for at least one device to be connected to the vehicle via the internet. This allows, in particular, the connection of devices located at a greater distance from the vehicle. The vehicle can be equipped with a telecommunications unit that allows internet access via mobile network or, if a Wi-Fi hotspot is nearby, via Wi-Fi. This enables not only communication with an external server for providing the main base model, but also direct communication with the user's smart home devices.

[0033] An information technology system of the generic type, comprising a vehicle with a computing unit, is further developed according to the invention in that the computing unit is configured to operate a virtual assistant according to a method described above. Accordingly, the computing unit has at least read access to a computer-readable storage medium containing machine-interpretable instructions which, when executed by a processor of the computing unit, cause it to execute the method according to the invention. The vehicle can be any road vehicle such as a car, truck, van, bus, or the like. Generally, it could also be a rail vehicle, watercraft, or aircraft.

[0034] An advantageous further development of the information technology system according to the invention provides for an external computing unit, wherein the vehicle and the computing unit are in communication with each other, and wherein the inference module is integrated into the computing unit. This facilitates the external processing of interaction and context information by the large base model on a corresponding server. In general, it is also conceivable that the trigger module, context module, and / or personalization module are provided completely or partially on the computing unit. This reduces the processing effort by the computing unit in the vehicle, although it increases the amount of data to be communicated with the external computing unit.

[0035] Further advantageous embodiments of the inventive method for operating a virtual assistant in a vehicle and of an information technology system for carrying out the method also result from the exemplary embodiments, which are described in more detail below with reference to the figures.

[0036] This shows:

[0037] Fig. 1 shows a schematic top view of a vehicle with a virtual assistant operated according to a method according to the invention;

[0038] Fig. 2 is a schematic top view of the vehicle shown in Fig. 1 in an alternative embodiment; and

[0039] Fig. 3 shows a schematic sequence of an interaction between a user and the virtual assistant.

[0040] The use of virtual assistants, such as voice dialogue systems, has proven its worth in everyday life. Figure 1 shows a representation of a vehicle 1 according to the invention, comprising a computing unit 7 for providing such an assistance function. However, according to the invention, the virtual assistant goes beyond the functionality of a voice dialogue system, since interaction with a user 2 is not only possible via voice, but also via a wide variety of interaction modalities.

[0041] In vehicle 1, a trigger module 3, a context module 4, and an inference module 5 are implemented in the computing unit 7. Furthermore, vehicle 1 communicates with an external computing unit 9 via a telecommunications unit 10. In the advantageous embodiment shown, interaction with a large base model 6 is possible via the external computing unit 9. The large base model 6 can be executed on the external computing unit 9 or on an external server or server network. The external computing unit 9 can act as an intermediary. Generally, it is also conceivable that the large base model 6 is executed on the computing unit 7 by the inference module 5 itself, or that the inference module 5 is integrated into the external computing unit 9.

[0042] Interactions performed by user 2 are recorded in vehicle 1, with the trigger module 3 classifying the interactions to generate interaction information. The context module 4, in turn, captures a context for vehicle 1 and provides this in the form of context information. As soon as a new interaction is captured by the trigger module 3, the trigger module 3 initiates the subsequent steps of a method according to the invention for operating the virtual assistant. The context is then captured, and the context information, as well as the interaction information associated with the interaction, is fed to the inference module 5 for processing by the large base model 6. The large base model 6 is characterized by suitable training, enabling it to possess general world knowledge and thus to evaluate situations based on common sense.Thus, the large base model 6 "knows" in each situation which action should be performed with which vehicle function in order to assist user 2.

[0043] As a result, the large base model 6, and thus the inference module 5, provides inference information, including control commands for controlling a vehicle function that can be processed by the computing unit 7, and / or output information for output to the user 2 via vehicle-specific output devices (not shown in detail). The large base model 6 is trained to generate the respective output data in a file format that can be processed by the vehicle 1.

[0044] In general, it is also conceivable that the functionalities of the computing unit 7 are distributed across several physical computer systems in vehicle 1.

[0045] Figure 2 shows an alternative embodiment of the vehicle 1 in which a personalization module 8 is integrated into the processing pipeline between the user 2 and the context module 4. Reaction actions performed by the user 2 in response to the activation of the vehicle function and / or the output of information are recorded and classified by the personalization module 8 to generate reaction information. By comparing the reaction information with the respective correlating context information and inference information, personality traits of the user 2 can be identified and stored in the form of personality information. This personality information can be fed to the large base model 6 separately or included in the context information and considered for the generation of inference information.The response of user 2 can in turn affect the context and / or trigger a new interaction.

[0046] Figure 3 shows an embodiment of a possible interaction between the virtual assistant and the user 2. In step 301, the user 2 enters the vehicle 1 and fastens their seatbelt. Fastening the seatbelt can be considered a trigger. Accordingly, the trigger module 3 outputs interaction information describing that a seatbelt has been engaged in the designated buckle. This information can be tagged with a suitable identifier, such as: "User has fastened their seatbelt".

[0047] In response, steps 302 and 303 are executed. In step 302, the context module 4 aggregates information describing the context of vehicle 1 and generates contextual information from this in step 303. Context module 4 captures, for example, the current time and day of the week, the location of vehicle 1, the current weather and outside temperature, the current settings of vehicle components, settings of user 2's smart home devices, user 2's current preferences and behavior patterns, entries in user 2's digital calendar, and similar information. This information is then fed to the inference module 5, which, in step 304, processes the information using the large base model 6.

[0048] For example, the large base model 6 determines that in step 305, based on user 2's preferences and their otherwise relaxed morning routine, calming music from a favorite artist should be played. In step 306, the vehicle's ambient lighting should be set to a pulsating, bright blue with a low flashing frequency. Blue is chosen because it was the color set in user 2's smart home lighting that morning. Since there is no rush and the usual morning coffee has been skipped, step 307 suggests a stop at a local bakery to buy a coffee. A confidence level can be assigned to the possible outputs in steps 305 to 307, describing the probability that the generated output is actually desired by user 2.For steps 305 and 306, the confidence level is higher than a defined confidence threshold, so the respective actions are executed automatically. For step 307, only a medium level of confidence is determined, so a navigation route to the bakery is not automatically set; instead, a suggestion to buy a coffee is displayed to user 2. For all other functions, step 308 indicates such a low level of confidence that no action is initiated by the virtual assistant.

[0049] In another embodiment, user 2 might perform a shoulder check while driving in the middle lane of a highway. This is detected by visually sensing user 2. By reading relevant information from a data bus of vehicle 1, the context module 4 determines that vehicle 1 is traveling at a relatively high speed in the middle lane of the highway and is approaching a vehicle ahead. Processing this information, the large base model 6 determines user 2's intention to overtake. Accordingly, the inference module 5 issues a command to automatically activate vehicle 1's left turn signal.Optionally, a message can be displayed on a screen in vehicle 1 indicating that the turn signal has been activated, and user 2 can be informed accordingly via computer-generated speech. This prevents user 2 from steering vehicle 1 into the left lane without activating the turn signal.

[0050] In another embodiment, an audio signal recorded by a microphone capturing the ambient noise in the vehicle interior detects that user 2 is drumming on the steering wheel of vehicle 1 in a specific rhythm. As contextual information, at least the personality information generated by the personalization module 8 is transmitted to the inference module 5 along with the interaction information. The personality information includes a description of the music artists favored by user 2, in particular, favorite songs. The large base model 6 identifies songs by the artists favored by user 2 that match the rhythm of the drumming and selects one of the songs for automatic playback via the infotainment system of vehicle 1. To prevent music from being played unintentionally in vehicle 1, user 2 can be proactively asked beforehand whether or not the corresponding song should be played.The song will only play if User 2 confirms this. User 2 can do this, for example, by pressing a corresponding confirmation button on a touch-sensitive display or by giving a corresponding voice command.

[0051] In another embodiment, user 2 steers alternately left and right, even though vehicle 1 is traveling on a straight road. Here, the ambient light and the current time can be considered as context. For example, if the situation occurs during the day in sunshine, the large base model 6 could determine that user 2 is in a good mood and is weaving for the sheer joy of driving. However, if it is determined that the situation occurs at night, particularly considering a navigation route taken after starting the journey from a bar, the large base model 6 could determine that user 2 is weaving due to intoxication. In the first situation, no further action is required, allowing user 2 to continue controlling vehicle 1.In the second situation, however, at least a semi-automated takeover of control for vehicle 1 can occur, bringing vehicle 1 to a controlled stop. Optionally, a telephone call can also be automatically initiated to a taxi company to summon a taxi for user 2.

Claims

Mercedes-Benz Group AG Patent claims 1. Method for operating a virtual assistant in a vehicle (1), wherein the virtual assistant controls a vehicle function and / or causes the output of information to the user (2), taking into account at least a context determined for the vehicle (1) to assist a user (2), wherein interactions performed by the user (2) are recorded, wherein a trigger module (3) classifies the interactions to generate interaction information; the context of at least the vehicle (1) is recorded by a context module (4) and provided by the context module (4) in the form of context information;characterized in that, when a new interaction is recorded, the interaction information generated for this interaction and the context information available are fed to an inference module (5), which processes the interaction information and context information with a suitably trained large base model (6) and outputs inference information as a processing result, comprising control commands for controlling the vehicle function that can be processed by a computing unit (7) in the vehicle (1) and / or output information for output to the user (2) via vehicle-specific output means; wherein the large base model (6) outputs different inference information for the same input data, taking into account a random predefined value.

2. Method according to claim 1, characterized in that a reaction action performed by the user (2) in response to the activation of the vehicle function and / or the output of the information is recorded, wherein a personalization module (8) uses the reaction actions to generate Response information is classified and, by comparing the response information with the respective correlating context information and inference information, personality traits of the user (2) are identified and stored in the form of personality information.

3. Method according to claim 2, characterized in that at least one of the following personality traits is identified: likes, dislikes and / or habits of the user.

4. Method according to claim 2 or 3, characterized in that the large base model (6) takes the personality information into account in the generation of the inference information.

5. Method according to one of claims 1 to 4, characterized in that the large basic model (6) assigns a confidence level to the respective control commands and the computing unit (7) automatically executes a respective control command if the underlying confidence level exceeds a defined confidence threshold or otherwise the computing unit (7) causes a request to be sent to the user (2) to execute the control command, and only executes the control command if the user (2) confirms the request by performing an operation.

6. Method according to one of claims 1 to 5, characterized in that the context module (4) retrieves context information from at least one terminal device communicatively coupled with the vehicle (1).

7. Method according to claim 6, characterized in that at least one terminal device is connected to the vehicle (1) via the Internet.

8. Information technology system comprising a vehicle (1) with a Computing unit (7), characterized in that the computing unit (7) is configured to operate a virtual assistant according to a method according to one of claims 1 to 7.

9. Information technology system according to claim 8, characterized by a vehicle-external computing device (9), wherein the vehicle (1) and the computing device (9) are in communication connection to each other, and wherein the inference module (5) is integrated into the computing device (9).

Images