Device, operation method of device, and computer-readable non-volatile recording medium having recorded thereon program for performing operation method of device

Abstract

A device is disclosed. The device, according to one embodiment of the present disclosure, may comprise: a memory; and one or more processors for receiving motion information of a user, obtaining lower-level vehicle events on the basis of the received motion information and event specification information, inferring, from the obtained lower-level vehicle events and through a large language model, an upper-level vehicle event and a vehicle function corresponding to the upper-level vehicle event, and generating an automation rule based on the inferred upper-level vehicle event and the inferred vehicle function, and storing the generated automation rule in the memory.

Description

A computer-readable non-volatile recording medium storing a device, a method of operating the device, and a program for performing the method of operating the device.

[0001] The present disclosure relates to a technology for generating and proposing user-customized automated rules based on a large language model.

[0002] Recently, in-vehicle infotainment systems (hereinafter IVI systems) often include service suggestion systems to provide various services and functions to users.

[0003] Generally, such service suggestion systems are configured based on fixed operational rules that provide a combination of specific function APIs (Application Programming Interfaces) according to situational conditions predefined by the vehicle manufacturer. For example, specific functions may be automatically recommended or executed based on the vehicle's driving status, location information, or time of day.

[0004] However, because this method is structured such that services are unilaterally proposed based on scenarios predefined by the vehicle company, there is a problem in that unnecessary services are recommended that do not exactly match individual user preferences or actual situations.

[0005] In particular, there are limitations in providing a personalized user experience because it fails to reflect dynamic information such as users' driving patterns, habits, and preferences.

[0006] Furthermore, the automation rules and suggestion scenarios configured within the system are statically defined, making it difficult to change or optimize them in real-time based on user feedback or environmental changes.

[0007] This limits ultimate personalization, and users may feel uncomfortable with or ignore the service recommendations provided by the system.

[0008] Therefore, there is a growing need for improved technology capable of dynamically generating rules based on user-specific requirements and situations, and configuring customized proposal services.

[0009] The purpose of the present disclosure may be to dynamically generate automation rules for complex conditions based on user requirements and contexts by utilizing a large language model (LLM).

[0010] The purpose of the present disclosure may be to identify user needs based on conversation within a vehicle and to propose and improve situation-appropriate automation rules in real time.

[0011] The purpose of the present disclosure may be to detect underlying user needs in advance based on user behavior patterns and situations, and to provide suitable suggestions before the user makes a request.

[0012] The purpose of the present disclosure may be to improve and optimize rules in real time by reflecting user feedback.

[0013] An apparatus according to one embodiment of the present disclosure may include a memory; and one or more processors that receive user operation information, acquire lower vehicle events based on the received operation information and event specification information, infer upper vehicle events and vehicle functions corresponding to the upper vehicle events from the acquired lower vehicle events through a large language model, generate automation rules based on the inferred upper vehicle events and the inferred vehicle functions, and store the generated automation rules in the memory.

[0014] A method of operation of a device according to one embodiment of the present disclosure may include: receiving user operation information; acquiring lower vehicle events based on the received operation information and event specification information; inferring upper vehicle events and vehicle functions corresponding to the upper vehicle events from the acquired lower vehicle events through a large language model; and generating automation rules based on the inferred upper vehicle events and the inferred vehicle functions and storing the generated automation rules.

[0015] A computer-readable non-volatile recording medium storing a program for performing a method of operation of a device according to one embodiment of the present disclosure, wherein the method of operation may include: receiving user operation information; acquiring lower vehicle events based on the received operation information and event specification information; inferring a higher vehicle event and a vehicle function corresponding to the higher vehicle event from the acquired lower vehicle events through a large language model; and generating an automation rule based on the inferred higher vehicle event and the inferred vehicle function and storing the generated automation rule.

[0016] According to various embodiments of the present disclosure, there is an advantage in that automation rules can be easily generated through light conversation or natural language input, even if the user is not well aware of the system's functions, context, operable APIs, etc.

[0017] According to various embodiments of the present disclosure, rather than a simple rule-based system, complex automation rules that dynamically consider various events and situations are generated, thereby providing the effect of providing more meaningful operations.

[0018] According to various embodiments of the present disclosure, since a function or service that meets the corresponding need can be suggested in a timely manner before the user recognizes or inputs a command, convenience of use can be improved.

[0019] According to various embodiments of the present disclosure, automation rules can be immediately modified and improved by reflecting user feedback in real time, which has the advantage of allowing the user to flexibly adjust the system to meet their changing needs.

[0020] Figure 1 is a drawing illustrating an example of the exterior and interior of a vehicle.

[0021] Figure 2 is a diagram illustrating the architecture of a signal processing system for vehicles.

[0022] FIG. 3a is a drawing illustrating an example of the arrangement of a vehicle display device inside a vehicle.

[0023] FIG. 3b is a drawing illustrating another example of the arrangement of a vehicle display device inside a vehicle.

[0024] Figure 4 is an example of an internal block diagram of the vehicle of Figure 1.

[0025] FIG. 5 is a block diagram illustrating the configuration of a server according to one embodiment of the present disclosure.

[0026] FIGS. 6a to 6d are drawings for explaining the configuration of a system according to an embodiment of the present disclosure.

[0027] FIG. 7 is a drawing for explaining the configuration of a preemptive service module according to one embodiment of the present disclosure.

[0028] FIG. 8 is a flowchart illustrating a method of operation of a device according to one embodiment of the present disclosure.

[0029] FIG. 9 is a flowchart for explaining a method of operation of a vehicle according to another embodiment of the present disclosure.

[0030] FIGS. 10a and FIGS. 10b are drawings illustrating a process for explaining an automation rule according to an embodiment of the present disclosure.

[0031] FIG. 11 is a flowchart for explaining a method of operation of a vehicle according to another embodiment of the present disclosure.

[0032] FIGS. 12a to 12c are drawings illustrating examples of generating and proposing automation rules under the initiative of a user according to an embodiment of the present disclosure.

[0033] FIGS. 13a and 13b are drawings illustrating an example in which a vehicle automatically proposes automation rules and improves automation rules based on user feedback according to an embodiment of the present disclosure.

[0034] The present disclosure will be described in more detail below with reference to the drawings.

[0035] The suffixes "module" and "part" for components used in the following description are assigned solely for the ease of drafting this specification and do not inherently confer any particularly significant meaning or role. Accordingly, the terms "module" and "part" may be used interchangeably.

[0036] Figure 1 is a drawing illustrating an example of the exterior and interior of a vehicle.

[0037] Referring to the drawing, the vehicle (200) is operated by a plurality of wheels (103FR, 103FL, 103RL,...) that rotate by a power source, and a steering wheel (150) for controlling the direction of travel of the vehicle (200).

[0038] Meanwhile, the vehicle (200) may further be equipped with a camera (195), etc., for acquiring an image of the front of the vehicle.

[0039] Meanwhile, the vehicle (200) may be equipped with a plurality of displays (180a, 180b) for displaying images, information, etc. inside.

[0040] In FIG. 1, a cluster display (180a) and an AVN (Audio Video Navigation) display (180b) are exemplified as multiple displays (180a, 180b). Other displays such as a HUD (Head Up Display) are also possible.

[0041] Meanwhile, the AVN (Audio Video Navigation) display (180b) may also be named the Center Information Display.

[0042] Meanwhile, the vehicle (200) described in this specification may be a concept that includes all of the following: a vehicle equipped with an engine as a power source, a hybrid vehicle equipped with an engine and an electric motor as a power source, an electric vehicle equipped with an electric motor as a power source, etc.

[0043] Figure 2 is a diagram illustrating the architecture of a signal processing system for vehicles.

[0044] Referring to the drawing, the architecture (300a) of the vehicle signal processing system can correspond to a zone-based architecture.

[0045] Accordingly, sensor devices and processors inside the vehicle may be placed in each of the multiple zones (Z1 to Z4), and a signal processing device (170a) including a vehicle communication gateway (GWDa) may be placed in the central area of ​​the multiple zones (Z1 to Z4).

[0046] Meanwhile, the signal processing device (170a) may additionally include an autonomous driving control module (ACC), a cockpit control module (CPG), etc., in addition to the vehicle communication gateway (GWDa).

[0047] The vehicle communication gateway (GWDa) within the signal processing device (170a) may be a High Performance Computing (HPC) gateway.

[0048] That is, the signal processing device (170a) of FIG. 2 is an integrated HPC and can exchange data with an external communication module (not shown) or a processor (not shown) in a plurality of zones (Z1 to Z4).

[0049] FIG. 3a is a drawing illustrating an example of the arrangement of a vehicle display device inside a vehicle.

[0050] Referring to the drawing, the vehicle interior may be equipped with a cluster display (180a), an AVN (Audio Video Navigation) display (180b), a rear seat entertainment display (180c, 180d), a rearview mirror display (not shown), etc.

[0051] FIG. 3b is a drawing illustrating another example of the arrangement of a vehicle display device inside a vehicle.

[0052] A vehicle display device (100) according to an embodiment of the present disclosure may include a plurality of displays (180a to 180b) and a signal processing device (170) that performs signal processing for displaying images, information, etc. on the plurality of displays (180a to 180b) and outputs an image signal to at least one display (180a to 180b).

[0053] Among the plurality of displays (180a to 180b), the first display (180a) is a cluster display (180a) for displaying driving status, operation information, etc., and the second display (180b) may be an AVN (Audio Video Navigation) display (180b) for displaying vehicle operation information, navigation map, various entertainment information or video.

[0054] The signal processing device (170) has a processor (175) inside and can execute a first virtualization machine to a third virtualization machine (not shown) on a hypervisor (not shown) within the processor (175).

[0055] A second virtualization machine (not shown) operates for the first display (180a), and a third virtualization machine (not shown) can operate for the second display (180b).

[0056] Meanwhile, the first virtualization machine (not shown) within the processor (175) can be controlled to set up a shared memory (508) based on a hypervisor (505) for the same data transmission to the second virtualization machine (not shown) and the third virtualization machine (not shown). Accordingly, the same information or the same image can be synchronized and displayed on the first display (180a) and the second display (180b) within the vehicle.

[0057] Meanwhile, the first virtualization machine (not shown) within the processor (175) shares at least a portion of the data with the second virtualization machine (not shown) and the third virtualization machine (not shown) for data sharing processing. Accordingly, data can be shared and processed by multiple virtualization machines for multiple displays within the vehicle.

[0058] Meanwhile, the first virtualization machine (not shown) within the processor (175) can receive and process wheel speed sensor data of the vehicle and transmit the processed wheel speed sensor data to at least one of the second virtualization machine (not shown) or the third virtualization machine (not shown). Accordingly, the wheel speed sensor data of the vehicle can be shared with at least one virtualization machine, etc.

[0059] Meanwhile, the vehicle display device (100) according to the embodiment of the present disclosure may further include a rear seat entertainment display (180c) for displaying driving status information, simple navigation information, various entertainment information or images.

[0060] The signal processing device (170) can control the RSE display (180c) by running a fourth virtualization machine (not shown) in addition to the first to third virtualization machines (not shown) on a hypervisor (not shown) within the processor (175).

[0061] Accordingly, various displays (180a to 180c) can be controlled using a single signal processing device (170).

[0062] Meanwhile, some of the multiple displays (180a to 180c) operate under a Linux OS, and others can operate under a Web OS.

[0063] A signal processing device (170) according to an embodiment of the present disclosure can control displays (180a to 180c) operating under various operating systems (OS) to synchronize and display the same information or the same image.

[0064] Meanwhile, FIG. 3b illustrates that a vehicle speed indicator (212a) and a vehicle interior temperature indicator (213a) are displayed on a first display (180a), a home screen (222) including a plurality of applications, a vehicle speed indicator (212b), and a vehicle interior temperature indicator (213b) is displayed on a second display (180b), and a second home screen (222b) including a plurality of applications and a vehicle interior temperature indicator (213c) is displayed on a third display (180c).

[0065] Figure 4 is an example of an internal block diagram of the vehicle of Figure 1.

[0066] Referring to the drawings, a vehicle (200) according to an embodiment of the present disclosure may be equipped with a lamp drive unit (751), a steering drive unit (752), a brake drive unit (753), a power source drive unit (754), a suspension drive unit (756), an air conditioning drive unit (757), a window drive unit (758), a seat drive unit (761), and a signal processing device (170).

[0067] Meanwhile, the vehicle (200) may further be equipped with an ECU (770), a plurality of sensor devices (SN), and a plurality of communication modules (EMa~EMd).

[0068] Meanwhile, the vehicle (200) according to the embodiment of the present disclosure may further be equipped with a vehicle display device (100).

[0069] A vehicle display device (100) according to an embodiment of the present disclosure may include an input unit (110), a communication device (120) for communication with an external device, a plurality of communication modules (EMa~EMd) for internal communication, a memory (140), a signal processing device (170), a plurality of displays (180a~180c), an audio output unit (185), and a power supply unit (190).

[0070] Multiple communication modules (EMa~EMd) can be placed in each of the multiple zones (Z1~Z4) of FIG. 2, for example.

[0071] Meanwhile, the signal processing device (170) may have a communication switch (736b) inside for data communication with each communication module (EM1~EM4).

[0072] Each communication module (EM1~EM4) can perform data communication with a plurality of sensor devices (SN), ECU (770), or area signal processing device (170Z).

[0073] Meanwhile, a plurality of sensor devices (SN) may include a camera (195), lidar (196), radar (197), or position sensor (198).

[0074] The input unit (110) may be equipped with physical buttons, pads, etc. for button input, touch input, etc.

[0075] Meanwhile, the input unit (110) may be equipped with a microphone (not shown) for user voice input.

[0076] The communication device (120) can exchange data wirelessly with a mobile terminal (800) or a server (900).

[0077] In particular, the communication device (120) can wirelessly exchange data with the vehicle driver's mobile terminal. Various data communication methods are possible as wireless data communication methods, such as Bluetooth, WiFi, WiFi Direct, and APiX.

[0078] The communication device (120) can receive weather information, road traffic condition information, for example, TPEG (Transport Protocol Expert Group) information from a mobile terminal (800) or a server (900). To this end, the communication device (120) may be equipped with a mobile communication module (not shown).

[0079] Meanwhile, the communication device (120) can exchange data wirelessly with an adjacent vehicle.

[0080] For example, the communication device (120) can exchange vehicle messages wirelessly with an adjacent vehicle through V2X (Vehicle-to-everything) communication.

[0081] A plurality of communication modules (EM1~EM4) can receive sensor data, etc. from an ECU (770), a sensor device (SN), or a region signal processing device (170Z), and transmit the received sensor data to the signal processing device (170).

[0082] Here, the sensor data may include at least one of vehicle direction data, vehicle location data (GPS data), vehicle angle data, vehicle speed data, vehicle acceleration data, vehicle tilt data, vehicle forward / reverse data, battery data, fuel data, tire data, vehicle lamp data, vehicle interior temperature data, and vehicle interior humidity data.

[0083] Such sensor data can be obtained from a heading sensor, a yaw sensor, a gyro sensor, a position module, a vehicle forward / reverse sensor, a wheel sensor, a vehicle speed sensor, a vehicle body inclination sensor, a battery sensor, a fuel sensor, a tire sensor, a steering sensor based on steering wheel rotation, a vehicle interior temperature sensor, a vehicle interior humidity sensor, etc.

[0084] Meanwhile, the position module may include a GPS module or a position sensor (198) for receiving GPS information.

[0085] Meanwhile, at least one of the multiple communication modules (EM1 to EM4) can transmit location information data sensed from a GPS module or a location sensor (198) to a signal processing device (170).

[0086] Meanwhile, at least one of the plurality of communication modules (EM1 to EM4) can receive vehicle front image data, vehicle side image data, vehicle rear image data, and obstacle distance information around the vehicle from a camera (195), lidar (196), radar (197), etc., and transmit the received information to a signal processing device (170).

[0087] The memory (140) can store various data for the overall operation of the vehicle display device (100), such as a program for processing or controlling the signal processing device (170).

[0088] For example, memory (140) can store data regarding a hypervisor, a first virtualization machine to a third virtualization machine, for execution within a processor (175).

[0089] The audio output unit (185) converts an electrical signal from the signal processing device (170) into an audio signal and outputs it. To do this, a speaker or the like may be provided.

[0090] The power supply unit (190) can supply power necessary for the operation of each component under the control of the signal processing unit (170). In particular, the power supply unit (190) can receive power from a battery inside the vehicle, etc.

[0091] The signal processing device (170) controls the overall operation of each unit within the vehicle display device (100) or vehicle (200).

[0092] For example, the signal processing device (170) may include a processor (175) that performs signal processing for a vehicle display (180a, 180b).

[0093] The processor (175) can run a first virtualization machine to a third virtualization machine (not shown) on a hypervisor (not shown) within the processor (175).

[0094] Among the first to third virtual machines (not shown), the first virtual machine (not shown) may be named a Server Virtual Machine, and the second to third virtual machines (not shown) may be named a Guest Virtual Machine.

[0095] For example, a first virtualization machine (not shown) within a processor (175) can receive sensor data from a plurality of sensor devices, such as vehicle sensor data, location information data, camera image data, audio data, or touch input data, and process or modify it to output it.

[0096] In this way, by performing most of the data processing in the first virtualization machine (not shown), 1:N data sharing becomes possible.

[0097] As another example, the first virtualization machine (not shown) can directly receive and process CAN data, Ethernet data, audio data, radio data, USB data, and wireless communication data for the second virtualization machine to the third virtualization machine (not shown).

[0098] And, the first virtualization machine (not shown) can transmit the processed data to the second virtualization machine to the third virtualization machine (not shown).

[0099] Accordingly, among the first to third virtualization machines (not shown), only the first virtualization machine (not shown) receives sensor data, communication data, or external input data from a plurality of sensor devices and performs signal processing, thereby reducing the signal processing burden on other virtualization machines and enabling 1:N data communication, which enables synchronization when sharing data.

[0100] Meanwhile, the first virtualization machine (not shown) can control the sharing of the same data with the second virtualization machine (not shown) and the third virtualization machine (not shown) by writing data to the shared memory (508).

[0101] For example, the first virtualization machine (not shown) can record vehicle sensor data, the location information data, the camera image data, or the touch input data in a shared memory (508) and control the sharing of the same data with the second virtualization machine (not shown) and the third virtualization machine (not shown). Accordingly, data sharing in a 1:N manner becomes possible.

[0102] Ultimately, by performing most of the data processing on the first virtualization machine (not shown), 1:N data sharing becomes possible.

[0103] Meanwhile, the first virtualization machine (not shown) within the processor (175) can control the second virtualization machine (not shown) and the third virtualization machine (not shown) to set up a shared memory (508) based on the hypervisor (505) for the same data transmission.

[0104] Meanwhile, the signal processing device (170) can process various signals such as audio signals, video signals, and data signals. To this end, the signal processing device (170) can be implemented in the form of a System On Chip (SOC).

[0105] FIG. 5 is a block diagram illustrating the configuration of a server according to one embodiment of the present disclosure.

[0106] Referring to FIG. 5, the server (900) may refer to a device that trains an artificial neural network using a machine learning algorithm or uses a trained artificial neural network. Here, the server (900) may be composed of multiple servers to perform distributed processing and may be defined as a 5G network.

[0107] The server (900) may be included as part of the vehicle (200) and may perform at least some of the AI ​​processing together.

[0108] The server (900) may include a communication interface (510), memory (530), a learning processor (540), and a processor (560).

[0109] The communication interface (510) can transmit and receive data with the vehicle (200) or an external device.

[0110] The memory (530) may include a model storage unit (531). The model storage unit (531) may store a model (or artificial neural network, 531a) that is being learned or has been learned through the learning processor (540).

[0111] The learning processor (540) can train the artificial neural network (531a) using training data. The training model may be used while mounted on the server (900) of the artificial neural network, or it may be used while mounted on an external device such as a vehicle (200).

[0112] The learning model may be implemented in hardware, software, or a combination of hardware and software. If part or all of the learning model is implemented in software, one or more instructions constituting the learning model may be stored in memory (530).

[0113] The processor (560) can use a learning model to infer a result value for new input data and generate a response or control command based on the inferred result value.

[0114] FIGS. 6a to 6c are drawings for explaining the configuration of a system according to an embodiment of the present disclosure.

[0115] The system (6000) of Fig. 6a can be referred to as an IVEX (In Vehicle Experience) system.

[0116] Referring to FIG. 6a, the system (6000) may include an edge sensor group (6100), a recognizer (6200), an insightor (6300), and an illustrator (6400).

[0117] The edge sensor group (6100) may be included in the plurality of sensor devices (SN) of FIG. 2.

[0118] The recognizer (6200) and the insightor (6300) may be included in the signal processing device (170) of the vehicle display device (100) of FIG. 4. In particular, the recognizer (6200) and the insightor (6300) may be included in the processor (175) of the signal processing device (170).

[0119] In another embodiment, the recognizer (6200) and the insightor (6300) may be included in the cockpit control module (CPG) of FIG. 2.

[0120] An illustrator (6400) may be included in the vehicle display device (100) of FIG. 2 and FIG. 4.

[0121] The recognizer (6200) can recognize the vehicle situation based on a set of sensing data received from the edge sensor group (6100). The edge sensor group (6100) may include vehicle internal sensors (6110) and external sensors (6120).

[0122] The vehicle interior sensors (6110) are sensors placed inside the vehicle (200) and may include a front camera, lidar, radar, interior camera, vehicle interior temperature sensor, and vehicle interior humidity sensor.

[0123] The external sensors (6120) are sensors placed on the exterior of the vehicle (200) and may include an external camera, lidar, radar, heading sensor, yaw sensor, gyro sensor, position module, vehicle forward / reverse sensor, wheel sensor, vehicle speed sensor, vehicle body inclination sensor, battery sensor, fuel sensor, tire sensor, steering sensor based on steering wheel rotation, etc. The position module may include a GPS module or a position sensor (198) for receiving GPS information.

[0124] The sensing data set may include an internal vehicle data set and an external vehicle data set.

[0125] The vehicle interior data set may be a data set sensed by the vehicle interior sensors (6110). The vehicle interior data set may include at least one of vehicle interior temperature data, vehicle interior humidity data, battery data, fuel data, vehicle lamp data, tire data or vehicle interior image data, and audio data received through a microphone.

[0126] The vehicle external data set may be a data set sensed by external sensors (6120). The vehicle external data set may include at least one of vehicle location data (GPS), vehicle direction data, vehicle angle data, vehicle speed data, vehicle acceleration data, vehicle tilt data, whether the vehicle is moving forward or backward, front image data, or rear image data.

[0127] The recognizer (6200) can perform preprocessing and calibration on the sensing data set and can obtain vehicle internal information (6210) and vehicle external information (6220) based on the preprocessed and calibrated sensing data set.

[0128] The vehicle interior information (6210) may include a vehicle interior data set or information about the vehicle interior situation obtained from the vehicle interior data set.

[0129] The vehicle external information (6220) may include a vehicle external data set or information about the vehicle external situation obtained from the vehicle external data set.

[0130] The recognizer (6200) can recognize a vehicle situation based on at least one of vehicle internal information (6210) or vehicle external information (6220), and can generate vehicle situation information for the recognized vehicle situation.

[0131] Vehicle situation information may include at least one of vehicle external situation information indicating a situation regarding the exterior of the vehicle (200), vehicle internal situation information indicating a situation regarding the interior of the vehicle (200), or user situation information indicating a situation regarding a user inside the vehicle (200). User situation information may be included in the vehicle internal situation information.

[0132] User situation information may include information about the user's face identifier (Face ID), distraction, gaze, position, gesture, emotional state, whether they are drinking or taking drugs, drowsiness, and other actions inside the vehicle (200).

[0133] The recognizer (6200) can transmit the generated vehicle situation information to the insightor (6300).

[0134] The insightor (6300) can generate a response result based on vehicle situation information received from the recognizer (6200).

[0135] The insightor (6300) may include a multimodal context engine (6310), a context controller (6320), a safety assistance engine (6330), a user context engine (6340), an AI orchestrator (6350), and an AI model set (6360).

[0136] The multimodal context engine (6310) can generate multimodal context data based on vehicle situation information received from the recognizer (6200). The multimodal context data may include at least one of text data or image data describing the vehicle situation generated based on the vehicle situation information.

[0137] The context controller (6320) may be a component included in the multimodal context engine (6310) or provided separately from the multimodal context engine (6310).

[0138] When the context controller (6320) receives a user query from the AI ​​orchestrator (6350), it can execute a process of generating multimodal context data. The user query may be a speech recognition result corresponding to a voice command spoken by the user.

[0139] The safety assist engine (6330) can determine whether the current situation is a safe situation or a dangerous situation based on vehicle situation information received from the recognizer (6200).

[0140] If the safety assist engine (6330) determines that the current situation is a dangerous situation, it can transmit driver assistance control commands and warning notification output commands to the safety application (6440) of the illustrator (6400).

[0141] The user context engine (6340) can generate user context data based on user information. User information may be referred to as a user persona. User information may include at least one of the user's nationality, age, gender, occupation, personality, or psychological type (Myers-Briggs Type Indicator, MBTI).

[0142] The AI ​​orchestrator (6350) can generate a prompt based on at least one of multimodal context data or user context data. The AI ​​orchestrator (6350) can transmit the generated prompt to at least one of a plurality of AI models included in the AI ​​model set (6360).

[0143] The AI ​​model set (6360) may include multiple AI models. Each AI model may output an inference result in response to a prompt received from the AI ​​orchestrator (6350) and may transmit the inference result to the AI ​​orchestrator (6350).

[0144] The AI ​​orchestrator (6350) can generate additional prompts based on inference results and can transmit the additional prompts to at least one AI model in the set of AI models (6360). The AI ​​model that receives the additional prompts can output additional inference results in response to the additional prompts and can transmit the additional inference results to the AI ​​orchestrator (6350).

[0145] The AI ​​orchestrator (6350) can obtain an inference result or additional inference result received from the AI ​​model as a response result, and can output the obtained response result to an illustrator (6400).

[0146] The illustrator (6400) can output service information or perform driving control functions based on the response result output from the insightor (6300).

[0147] The illustrator (6400) may include a multimodal output encoder (6410), a visual interface (6420), an audio interface (6430), and a safety application (6440).

[0148] The multimodal output encoder (6410) can encode the response result output from the insightor (6300) and output the encoded response result data to the visual interface (6420) or audio interface (6430).

[0149] The visual interface (6420) or audio interface (6430) may be referred to as an output interface.

[0150] The visual interface (6420) can display response result data output from the multimodal output encoder (6410) or an image based on the response result data. The visual interface (6420) may include at least one of the plurality of displays (180a to 180c) of FIG. 4.

[0151] The visual interface (6420) can display an Augmented Reality (AR) image based on response result data or a Mixed Reality (MR) image based on response result data.

[0152] The audio interface (6430) can output response result data in the form of audio. The audio interface (6430) may be included in the audio output unit (185) of FIG. 4.

[0153] The safety application (6440) can perform Advanced Driver Assistance System (ADAS) control according to the response result received from the driver assistance control command or the insightor (6300), and can output a warning notification according to the warning notification output command.

[0154] The safety application (6440) may be included in the electronic control unit (770) of FIG. 4 or executed by the electronic control unit (770).

[0155] FIG. 6b is a drawing for explaining the configuration of an insightor according to another embodiment of the present disclosure.

[0156] In FIG. 6b, the multimodal context engine (6310) may include a context controller (6320).

[0157] Referring to FIG. 6b, the insightor (6300) may include a multimodal context engine (6310), an AI orchestrator (6350), and a multimodal LLM (6361).

[0158] The multimodal context engine (6310) may include a multimodal signal adapter (6311), a multimodal indexer (6312), a multimodal context buffer (6313), a multimodal context retriever (6314), a multimodal context descriptor (6315), a multimodal event monitor (6316), and a context controller (6320).

[0159] The multimodal signal adapter (6311) can generate pre-processed multimodal data by filtering, cleaning, synchronizing, and reformulating data received from various sensors or vehicle situation information received from a recognizer (6200).

[0160] At least one of the following can be input to the multimodal signal adapter (6311): audio data received through a microphone, front image data captured through a front camera, ADAS information obtained from the front image data or vehicle sensors, location data, IVI (In-Vehicle Infotainment) system data, IVI display information, DMS (Driver Monitoring System) information / IMS (Interior Monitoring System) information based on image data captured through an interior camera, and biometric data obtained from a biometric sensor.

[0161] The multimodal indexer (6312) can generate multimodal processing data by dividing preprocessed multimodal data into chunks and can index the multimodal processing data. The multimodal indexer (6312) can obtain an encoding vector or keyword representing the attribute (or meaning) of the multimodal processing data divided into chunks as an index.

[0162] The multimodal context buffer (6313) can store multimodal processing data and an index corresponding to the multimodal processing data.

[0163] The multimodal context retriever (6314) can search for multimodal processing data most relevant to the user query through an index from the multimodal context buffer (6313) and can select the searched multimodal processing data as a context candidate for the creation of multimodal context data.

[0164] The multimodal context descriptor (6315) can reconfigure multimodal processing data selected as a context candidate into multimodal context data having a prompt form that can be interpreted by the multimodal LLM (6361).

[0165] The multimodal event monitor (6316) can monitor whether an index matching the trigger condition is entered when a trigger condition for multimodal data is registered. If an index matching the trigger condition is entered, the multimodal event monitor (6316) can generate a trigger event to generate a proactive service query.

[0166] The context controller (6320) can control the overall operation of the multimodal context engine (6310). When the context controller (6320) receives a user query from the AI ​​orchestrator (6350), it can execute the process of generating multimodal context data. When the context controller (6320) receives a trigger event from the multimodal event monitor (6316), it can generate a preemptive service query.

[0167] The AI ​​orchestrator (6350) can generate a prompt by combining a user query and a multimodal context, and can transmit the generated prompt to a multimodal LLM (6361). The user query may be a query in the form of recognized text based on a voice command spoken by the user. The voice command may be received through a microphone, and the voice command may be converted into text through an Automatic Speech Recognition (ASR) process. The user query may be text converted through an Automatic Speech Recognition (ASR) process.

[0168] The multimodal LLM (6361) may be an example of an AI model included in the set of AI models (6360) of FIG. 6a. The multimodal LLM (6361) may output an inference result from a prompt received from the AI ​​orchestrator (6350) and may transmit the inference result to the AI ​​orchestrator (6350).

[0169] The inference result may include a result representing a response to the prompt.

[0170] For example, the inference result may include an API call result regarding whether an API call corresponding to a driving assistance function was successfully performed, and a feedback generation result regarding whether feedback corresponding to the API call was successfully generated.

[0171] FIG. 6c is a drawing for explaining the configuration of an AI orchestrator according to one embodiment of the present disclosure.

[0172] Referring to FIG. 6c, the AI ​​orchestrator (6350) may include a task arbitrator (6351), a prompt manager (6352), a sub-agent set (6353), a knowledge DB (6354), a workflow controller (6355), and a tool set / adapter set (6356).

[0173] The task arbitrator (6351) can determine one of the multiple sub-agents based on the voice recognition result and the multimodal context data output from the multimodal context engine (6310).

[0174] The prompt manager (6352) can generate a prompt for the operation of a determined sub-agent. The prompt manager (6352) can generate a prompt based on multimodal context data and user queries. The prompt manager (6352) can generate a prompt based on information about the functions that the determined sub-agent can perform, multimodal context data, the results of previously performed functions, conversation history, and user queries.

[0175] The sub-agent set (6353) may include multiple sub-agents. For example, the sub-agent set (6353) may include a navigation agent for navigation services, a vehicle function agent for providing vehicle functions, a telephony agent for automated telephone answering services, and a Q&A agent for providing response services to queries.

[0176] A sub-agent determined by the task arbitrator (6351) may call a cloud AI model or an on-device AI model assigned to it. The cloud model or on-device model may be a Large Language Model (LLM).

[0177] A sub-agent can call a cloud AI model or an on-device AI model assigned to it to obtain inference results corresponding to a prompt from the model.

[0178] Based on the obtained inference result, the sub-agent can call the knowledge DB (6354) to provide additional information, obtain additional information from the knowledge DB (6354), specify the name of the function to be executed and the parameter value of the function, and determine the feedback text to be provided to the user.

[0179] The sub-agent can transmit to the workflow controller (6355) a parsing result generated by parsing the inference result received from the model, which includes additional information called from the knowledge DB (6354), the name of the function to be executed, the parameter values ​​of the function, and a feedback phrase.

[0180] The knowledge DB (6354) can store additional information and information about functions. The information about functions may include the name of the function and the parameter values ​​of the function.

[0181] The workflow controller (6355) can generate control commands to perform the corresponding function based on the parsing results and can store the results of the conversation and function performed by the AI ​​model and sub-agent.

[0182] The tool set / adapter set (6356) can call an API corresponding to a control command generated by the workflow controller (6355). The tool set / adapter set (6356) can convert the control command into an execution command of an actual function or an API call command that the IVI system can understand and execute, and can execute the converted command.

[0183] FIG. 6d is a drawing for explaining the configuration of an insightor according to another embodiment of the present disclosure.

[0184] The insightor (6300-1) may be included in the signal processing unit (170) of the vehicle display device (100) of FIG. 4. The insightor (6300-1) may be included in the processor (175) of the signal processing unit (170).

[0185] The insightor (6300-1) may be included in the cockpit control module (CPG) of FIG. 2.

[0186] Referring to FIG. 6d, an insightor (6300-1) according to another embodiment of the present disclosure can generate a response result based on vehicle situation information received from a recognizer (6200).

[0187] The insightor (6300) may include a multimodal context engine (6310), a context controller (6320), a safety assistance engine (6330), a user context engine (6340), an AI orchestrator (6350), an AI model set (6360), and a preemptive service module (7000).

[0188] The description of each component of the multimodal context engine (6310), context controller (6320), safety assistance engine (6330), user context engine (6340), AI orchestrator (6350), and AI model set (6360) is replaced with the description of FIG. 6a.

[0189] The preemptive service module (7000) can generate automation rules for automating vehicle functions based on user action information and can update the automation rules based on user feedback.

[0190] The preemptive service module (7000) can generate dynamic automation rules through user requests and complex event analysis by linking with the AI ​​model set (6660), and can execute the generated automation rules.

[0191] FIG. 7 is a drawing for explaining the configuration of a preemptive service module according to one embodiment of the present disclosure.

[0192] Referring to FIG. 7, the preemptive service module (7000) may include a conversational interface (7010), a context monitor (7020), a preemptive rule generator (7030), and an auto-executor (7040).

[0193] The conversational interface (7010) can convert the user's voice into text. The conversational interface (7010) may be an agent-type user interface that outputs audio feedback and visual feedback to the user. The conversational interface (7010) may be a chatbot-type user interface capable of conversing with the user.

[0194] The context monitor (7020) can receive multimodal context data from the multimodal context engine (6310). The multimodal context data may include at least one of an internal situation event representing the situation inside the vehicle, an external situation event representing the situation outside the vehicle, a user's action based on the internal situation event or the external situation event, or a reaction based on the internal situation event or the external situation event.

[0195] The context monitor (7020) can provide multimodal context data accumulated over a certain period to the preemptive rule generator (7030) upon receiving a user request.

[0196] The context monitor (7020) can provide multimodal context data accumulated over a certain period to the preemptive rule generator (7030) when an event registered in a rule (or automation rule) occurs. Here, the event may be a lower vehicle event or a higher vehicle event described later.

[0197] The preemptive rule generator (7030) can generate automation rules based on user action information.

[0198] The preemptive rule generator (7030) may include a complex event processing module (7031), a rule generation module (7032), and a rule recommendation module (7033).

[0199] The complex event processing module (7031) can obtain a higher vehicle event and a vehicle function matching the higher vehicle event based on a plurality of lower vehicle events when a user request or a user's IVI (In-Vehicle Infotainment) operation occurs (when user operation information is received).

[0200] The complex event processing module (7031) can transmit multiple sub-vehicle events to the AI ​​orchestrator (6350). The AI ​​orchestrator (6350) can generate a prompt requesting inference of a parent vehicle event and a vehicle function corresponding to the parent vehicle event based on the multiple sub-vehicle events according to the received command, and can transmit the generated prompt to the LLM (6361).

[0201] The LLM (6631) can output an inference result using a prompt received from the AI ​​orchestrator (6350). The inference result may include a parent vehicle event and a vehicle function that matches the parent vehicle event.

[0202] The complex event processing module (7031) can receive upper vehicle events and vehicle functions inferred by the LLM (6361) through the AI ​​orchestrator (6350). The complex event processing module (7031) can receive automation rules inferred by the LLM (6361) through the AI ​​orchestrator (6350).

[0203] The rule generation module (7032) can generate an automation rule (7050) representing API driving logic that performs a matching vehicle function when a higher vehicle event occurs through the LLM (6631).

[0204] The rule generation module (7032) can generate a prompt that instructs the LLM (6631) to perform a matching vehicle function when a higher vehicle event occurs.

[0205] The rule recommendation module (7033) can suggest a vehicle function (or service) matched to the upper vehicle event to the user when an upper vehicle event defined in the automation rule (7050) occurs.

[0206] The rule recommendation module (7033) can suggest vehicle functions (or services) matched to upper vehicle events on the visual interface (6420) through the chatbot application.

[0207] The automatic executor (7040) can execute code or scripts that constitute the automation rule (7050) to perform vehicle functions corresponding to higher-level vehicle events or provide vehicle services through APIs. APIs may be interfaces for executing vehicle functions or services inferred based on automation in conjunction with an external system.

[0208] FIG. 8 is a flowchart illustrating a method of operation of a device according to one embodiment of the present disclosure.

[0209] Additionally, although the embodiment of FIG. 8 is described as being performed by the processor (175) of the vehicle (200), it may also be performed by the processor (560) of the cockpit control module (CPG) or the server (900). The processor (175) may be provided in multiple numbers.

[0210] Referring to FIG. 8, the processor (175) of the vehicle (200) can receive user operation information (S801).

[0211] User operation information may include various operations, state changes, biological reactions, etc., performed or maintained by the user inside the vehicle (200).

[0212] User action information may include one or more of the following: input actions performed by the user through the IVI (In-Vehicle Infotainment) system (e.g., screen touch, button click, gesture swipe, navigation destination setting, audio control operation), voice commands spoken through the conversation interface (7010), information on the user's operation of physical interfaces within the vehicle (200) (e.g., steering wheel, pedal, defog button, etc.), or vehicle occupancy status (e.g., whether the driver or passenger is seated, whether the seatbelt is fastened, information on whether the user is present based on the seat sensor).

[0213] User operation information may further include the user's history of maintaining or changing air conditioning settings, such as indoor temperature set values, whether the cooling / heating mode has been changed, and the circulation mode status.

[0214] The user's motion information may further include the driver's biological or behavioral response information, such as forward gaze deviation, duration of eye closure, drowsiness-like patterns, etc.

[0215] User action information may further include user reactions to notifications or warnings provided by the system.

[0216] The processor (175) can obtain sub-vehicle events based on user operation information and event specification information (S803).

[0217] The processor (175) can acquire multiple sub-vehicle events based on user operation information and event specification information received from the multimodal context engine (6310).

[0218] In one embodiment, the event specification information may be information for representing an event detected inside the vehicle (200) or outside the vehicle (200). The event specification information may be information generated based on vehicle interior information (6210) and vehicle interior information (6220).

[0219] Event specification information may include one or more of an event identifier for identifying the event, conditions for the occurrence of the event, time of occurrence of the event, type of sensor that detected the event, location of occurrence of the event, or content of the event.

[0220] The preemptive rule generator (7030) can generate sub-vehicle events based on user action information and can also generate sub-vehicle events based on event specification information.

[0221] The preemptive rule generator (7030) may generate sub-vehicle events based on multimodal context data provided from the context monitor (7020). The preemptive rule generator (7030) may generate multiple sub-vehicle events based on at least one of user action information, event specification information, or multimodal context data provided from the context monitor (7020).

[0222] The processor (175) can infer a higher vehicle event and a vehicle function corresponding to the higher vehicle event from multiple lower vehicle events obtained through the LLM (6361) (S805).

[0223] The processor (175) can input multiple lower vehicle events into the LLM (6361) to obtain upper vehicle events and vehicle functions that match the upper vehicle events.

[0224] The processor (175) can generate automation rules based on inferred higher vehicle events and vehicle functions, and can store the generated automation rules in memory (140) (S807).

[0225] The processor (175) can generate an automation rule to resolve the upper vehicle event using the upper vehicle event, vehicle function, and available API information inferred through the LLM (6361).

[0226] The processor (175) can obtain code or script as an automation rule that performs vehicle functions corresponding to the upper vehicle event when the upper vehicle event occurs through the LLM (6361).

[0227] In one embodiment, the automation rule may be in the form of a script and code, or a file or prompt in a form that the automatic executor (7040) can read and execute. Vehicle functions or services may be performed by API driving logic included in the code / script.

[0228] FIG. 9 is a flowchart for explaining a method of operation of a vehicle according to another embodiment of the present disclosure.

[0229] FIG. 9 may be an embodiment that embodies FIG. 8.

[0230] Referring to FIG. 9, the processor (175) can obtain user operation information (S901).

[0231] The processor (175) can determine whether the user has requested a vehicle function according to specific conditions based on the acquired user operation information (S903).

[0232] If the user's action information includes a voice command, the processor (175) can obtain an analysis result indicating the intent of the voice command through a voice recognition module. Based on the analysis result of the obtained voice command, the processor (175) can determine whether the user has requested a vehicle function according to specific conditions.

[0233] In one embodiment, the processor (175) can determine that the user has requested a vehicle function according to a specific condition when the operation information is a voice command and the analysis result of the voice command expresses a specific condition and requests the execution of a corresponding vehicle function.

[0234] In another embodiment, the processor (175) may determine that the user has not requested a vehicle function according to a specific condition if the operation information is a voice command and the analysis result of the voice command indicates that there is no clear intention to execute a vehicle function or there is no association between a specific condition and a vehicle function.

[0235] If the processor (175) determines that the user has not requested a vehicle function according to a specific condition based on the acquired user's operation information, it can acquire sub-vehicle events related to the user's operation information based on the event specification information (S905).

[0236] For example, if the processor (175) indicates that the user's motion information is in a high-speed driving state, it can obtain a precipitation detection event and a high-speed road detection event, which is driving on a high-speed road and is related to the user's high-speed driving state, as sub-vehicle events.

[0237] The processor (175) can infer upper vehicle events and vehicle functions corresponding to upper vehicle events from user action information and acquired lower vehicle events through the LLM (6631) (S907).

[0238] The processor (175) can generate a prompt using user operation information and acquired lower vehicle events. The prompt may include commands requesting upper vehicle events and vehicle functions corresponding to the upper vehicle events using user operation information and acquired lower vehicle events.

[0239] For example, the processor (175) can generate a prompt requesting a vehicle function to resolve a higher vehicle event and a higher vehicle event through lower vehicle events including motion information indicating a high-speed driving state and a precipitation detection event / highway detection event.

[0240] The processor (175) can input the generated prompt into the LLM (6631) to obtain the upper vehicle event and the vehicle function corresponding to the upper vehicle event.

[0241] The processor (175) can determine whether there is no automation rule corresponding to a higher vehicle event and a vehicle function corresponding to the higher vehicle event (S909).

[0242] The processor (175) can determine whether an automation rule, which is a rule for performing a vehicle function that resolves a higher vehicle event and a higher vehicle event, is stored in memory (140). If the processor (175) does not store an automation rule in memory (140), it can determine that the automation rule does not exist, and if the automation rule is stored in memory (140), it can determine that the automation rule exists.

[0243] If the processor (175) determines that there is no corresponding automation rule, it can create an automation rule for function automation and register the created automation rule (S911).

[0244] An automation rule may represent code or a script representing API execution logic based on combination conditions of sub-vehicle events. An automation rule may be code or a script containing API execution logic that is called to perform a matching vehicle function when a combination of sub-vehicle events satisfies a pre-set condition.

[0245] The automation rule may be code or a script containing API driving logic that is called to perform a matching vehicle function when a combination of user action information and sub-vehicle events satisfies a preset condition.

[0246] The processor (175) can store the generated automation rule in memory (140). The processor (175) can store the generated automation rule by matching it to a user (or user account).

[0247] If the processor (175) determines that there is no corresponding automation rule, it may suggest to the user the registration of an automation rule. The processor (175) may display a notification suggesting the registration of an automation rule created through the visual interface (6420).

[0248] The processor (175) can register at least one of the lower vehicle events and the upper vehicle events matched thereto in the context monitor (7020) or memory (140) (S913).

[0249] Subsequently, the context monitor (7020) can determine whether a previously registered upper vehicle event is detected based on an event received from the multimodal context engine (6310).

[0250] Meanwhile, if the processor (175) determines that there is an automation rule corresponding to a higher vehicle event and a vehicle function corresponding to the higher vehicle event (S909), it can output rule description information explaining the corresponding automation rule (S915).

[0251] The processor (175) can display rule description information explaining the corresponding rule through a visual interface (6420).

[0252] Rule description information may be text describing the currently occurring parent vehicle event and the vehicle function performed in response to the occurrence of the parent vehicle event.

[0253] Meanwhile, if the processor (175) determines that the user has requested a vehicle function according to a specific condition, it can obtain sub-vehicle events corresponding to the specific condition based on the event specification information (S917).

[0254] For example, assume a case where a user speaks a voice command saying, "The glass window is foggy, please remove the fog."

[0255] The processor (175) can determine that the analysis result of the voice command is a request for a frost removal function due to a visual obstruction condition. The processor (175) can obtain sub-vehicle events corresponding to the visual obstruction condition based on the event specification information.

[0256] For example, sub-vehicle events may include an event where the difference between the vehicle interior temperature and the vehicle exterior temperature is 20 degrees or more, an event where the indoor humidity is 65% or more, and an event where the visibility of the windshield is reduced.

[0257] The processor (175) can infer upper vehicle events and vehicle functions corresponding to upper vehicle events from lower vehicle events obtained through the LLM (6631) (S919).

[0258] The processor (175) can generate a prompt using the analysis results of operation information (e.g., the analysis results of voice commands) and lower vehicle events, and can input the generated prompt into the LLM (6631). The LLM (6631) can infer a higher vehicle event and a vehicle function that matches the higher vehicle event in response to the input prompt.

[0259] For example, LLM (6631) can infer a defogging function to resolve a frost occurrence event from a prompt including a request for frost removal due to a visibility obstruction situation, an event where the difference between the vehicle interior temperature and the vehicle exterior temperature is 20 degrees or more, an event where the indoor humidity is 65% or more, and an event where the front windshield visibility is obstructed.

[0260] The processor (175) can determine whether the inferred vehicle function requires a change in the operation method of the LLM (6631) (S921).

[0261] In one embodiment, the mode of operation of the LLM (6631) may be any one of the conversational tone of the response output by the LLM (6631), the length of the response, the language of the response, and the depth of the response.

[0262] For example, the processor (175) may determine that a change in the operation method of the LLM (6631) is required if the inferred vehicle function includes a defogging function and a function to switch to a friendly tone of voice.

[0263] As another example, the processor (175) may determine that if the inferred vehicle function includes only a defog function, it does not require a change in the operation method of the LLM (6631).

[0264] If the processor (175) determines that the inferred vehicle function requires a change in the operation method of the LLM (6631), it can generate an automation rule to change the operation method of the LLM (6631) and register the generated automation rule (S923).

[0265] The automation rule may be a rule defined to perform a vehicle function matching the upper vehicle event when it is determined that an upper vehicle event has been detected.

[0266] In one embodiment, the processor (175) may store the generated automation rule in memory (140). The automation rule may be stored by matching it to a specific user.

[0267] FIGS. 10a and FIGS. 10b are drawings illustrating a process for explaining an automation rule according to an embodiment of the present disclosure.

[0268] In particular, FIG. 10a is a diagram illustrating the process of generating automation rules by recommendation, and FIG. 10b is a diagram illustrating the process of generating automation rules under the initiative of a user.

[0269] FIG. 10a assumes that the user's operation information (1001) is determined not to request a vehicle function according to a specific condition.

[0270] Referring to FIG. 10a, the processor (175) can generate a prompt using a high-speed driving state (1001) of a vehicle (200) representing user action information and a high-speed driving event / precipitation detection event (1002) representing lower vehicle events. The processor (175) can input the generated prompt into an LLM (6631) to obtain from the LLM (6631) a high-speed driving risk event (1003) representing a higher vehicle event and a vehicle speed automatic deceleration function (1004) to resolve the high-speed driving risk event (1003).

[0271] The processor (175) can generate a first automation rule (1005) including API driving logic that is called to perform an automatic vehicle speed reduction function (1004) when a high-speed driving state (1001) and a high-speed driving event / precipitation detection event (1002) occur.

[0272] Subsequently, the processor (175) may control the electronic control unit (770) to reduce the speed of the vehicle (200) by executing the first automation rule (1005) when the high-speed driving state (1001) and the high-speed driving event / precipitation detection event (1002) are satisfied. The electronic control unit (770) may control one or more of the brake drive unit (753) or the power source drive unit (754) to reduce the speed of the vehicle (200) according to the first automation rule (1005).

[0273] Thus, according to an embodiment of the present disclosure, fundamental user needs are detected in advance from user behavior and situations through an LLM, and a suitable vehicle control service can be provided before the user requests it.

[0274] FIG. 10b assumes that the user utters a voice command saying "The windshield is foggy, remove the frost," and that the user's action information (1011) is determined to be requesting a vehicle function according to specific conditions.

[0275] Referring to FIG. 10b, the processor (175) can generate a prompt using a request for frost removal (1011) due to frost formation on the windshield, which represents user action information, and an external temperature drop event / indoor-outdoor temperature difference event (1012), which represents lower vehicle events. The processor (175) can input the generated prompt into the LLM (6631) to obtain from the LLM (6631) a visibility obstruction event (1013) due to frost formation, which represents upper vehicle events, and a defogging function (1014) to resolve the visibility obstruction event (1013) due to frost formation.

[0276] The processor (175) can generate a second automation rule (1015) that includes API driving logic called to perform a defogging function (1014) when an external temperature drop event / an indoor-outdoor temperature difference event (1012) is detected.

[0277] Subsequently, when a high-speed driving state (1001) and a high-speed driving event / precipitation detection event (1002) are detected, the processor (175) can execute a second automation rule (1015) to control the electronic control unit (770) to perform a defogging function. The electronic control unit (770) can control the air conditioning drive unit (757) to perform a defogging function.

[0278] As such, according to an embodiment of the present disclosure, a user can easily create necessary rules through voice commands, and rules reflecting requirements that the user was unaware of can be provided.

[0279] FIG. 11 is a flowchart for explaining a method of operation of a vehicle according to another embodiment of the present disclosure.

[0280] In particular, FIG. 11 is a diagram illustrating the process of detecting a higher vehicle event, proposing an automation rule corresponding to the higher vehicle event, and updating the automation rule based on user feedback.

[0281] Referring to FIG. 11, the processor (175) can determine whether a registered event is detected (S1101).

[0282] The processor (175) can determine that a registered event has been detected when a higher vehicle event matching multiple lower vehicle events is detected.

[0283] When a registered event is detected, the processor (175) can acquire a vehicle function based on an automation rule corresponding to the detected event (S1103) and output a suggestion feedback for a service suggestion for the acquired vehicle function (S1105).

[0284] The suggestion feedback may be feedback that suggests providing a vehicle function that matches the top vehicle event, based on an automation rule, when a top vehicle event is detected.

[0285] The processor (175) can display suggestion feedback through a visual interface (6420).

[0286] When the processor (175) receives an acceptance input from a user agreeing to the proposal feedback (S1107), it can execute an automation rule corresponding to the detected upper vehicle event (S1109).

[0287] The processor (175) can receive user acceptance input via voice command and execute automation rules according to the received acceptance input. The processor (175) can control the electronic control unit (770) to perform vehicle functions according to the execution of the automation rules.

[0288] The processor (175) may not agree to the suggestion feedback, receive adjustment input for separate user detailed adjustment, or update the automation rule if API constraints exist (S1111).

[0289] In one embodiment, the processor (175) may update the automation rule upon receiving a adjustment input for fine-tuning the automation rule. For example, it is assumed that the automation rule is a rule that provides a mail notification function related to a travel schedule, and the fine-tuning input is a voice command that adds a function to set a destination at the time of receiving the mail and to reply whether to attend.

[0290] The processor (175) can update the automation rule to add a destination setting function and an attendance reply function to the mail notification function of the automation rule according to a voice command that adds a destination setting function and an attendance reply function at the time of receiving the mail.

[0291] In another embodiment, the processor (175) can update the automation rule if there are API constraints or policy conditions. For example, it is assumed that the automation rule is a rule that provides an automatic audio playback function that plays an audio playback application when boarding the vehicle (200), and the API constraint is that when boarding the vehicle (200), there is no function to play a specific album, and the list of last played audio is playable.

[0292] The processor (175) can update the automation rule to add the last song list auto-play function to the song auto-play function.

[0293] As such, according to an embodiment of the present disclosure, automation rules can be immediately modified and improved by reflecting user feedback in real time. Through this, the user can flexibly adjust the system to meet their changing needs.

[0294] FIGS. 12a to 12c are drawings illustrating examples of generating and proposing automation rules under the initiative of a user according to an embodiment of the present disclosure.

[0295] FIGS. 12a to 12c show examples of conversation screens including text that recognizes a voice command spoken by a user and text that indicates a response output through an LLM (6631) in response to the voice command.

[0296] In particular, FIGS. 12a to 12c may be embodiments applied when user operation information in S903 of FIG. 9 requests a function according to specific conditions.

[0297] Referring to FIG. 12a, the vehicle (200) can display a first conversation screen (1210) through a visual interface (6420). When setting a destination, the processor (175) can receive a first text command (1211) corresponding to a voice command having an analysis result asking for the reason for recommending a different route when a route different from the usual route is recommended. Based on the analysis result of the first text command (1211), the processor (175) can determine that a vehicle function (a function providing the reason for recommending a different route) is requested according to a specific condition (recommendation of a different route).

[0298] The processor (175) can obtain accident events, construction events, and traffic congestion events existing on the existing path as sub-vehicle events.

[0299] The processor (175) can obtain a function to provide a recommendation reason for a different path that matches a higher vehicle event called a path change event and a path change event from lower vehicle events obtained through the LLM (6631).

[0300] The processor (175) can output a first response (1212) representing feedback corresponding to a first text command (1211), output a second text command (1213) corresponding to a voice input representing a voice output limiting function, and output a second response (1214) representing feedback corresponding to the second text command (1213).

[0301] Accordingly, the processor (175) can generate an automation rule that calls API driving logic to provide a recommendation reason function for another path and a voice output limit function when a path change event occurs.

[0302] FIG. 12b can display a second conversation screen (1220) through the visual interface (6420) of the vehicle (200). The processor (175) can display a text command (1221) upon receiving a voice command having an analysis result requesting a conversation in a friendly tone when the user is riding alone.

[0303] The processor (175) may determine that the received text command (1221) is a request to change the operation method of the LLM (6631). The processor (175) may set the conversational tone of the response output by the LLM (6631) according to the text command (1221) to a friendly tone.

[0304] The processor (175) can create and register an automation rule that makes the conversational tone of the response output by the LLM (6631) operate in a friendly tone when the user is alone in the vehicle (200).

[0305] Referring to FIG. 12c, the vehicle (200) can display a third conversation screen (1210) through a visual interface (6420).

[0306] The processor (175) can output a first text command (1231) corresponding to a voice command to turn on the recirculation function when the vehicle (200) enters a tunnel. Based on the analysis result of the first text command (1231), the processor (175) can determine that a vehicle function (recirculation function on) is requested according to a specific condition (when entering a tunnel).

[0307] The processor (175) can obtain light reduction events, map data-based tunnel location recognition events, and GPS signal weakening events as sub-vehicle events.

[0308] The processor (175) can obtain a higher vehicle event called a tunnel entry event and a betting cycle function that matches the tunnel entry event from lower vehicle events obtained through the LLM (6631).

[0309] The processor (175) may output a response (1232) indicating feedback corresponding to the first text command (1231). The response (1232) may be text inquiring whether the recirculation function is enabled. The processor (175) may output the response (1232) if the automation rule for performing the recirculation function according to the tunnel entry event is not stored in memory (140).

[0310] The processor (175) can display a second text command (1233) in accordance with a voice command indicating consent.

[0311] The processor (175) can generate an automation rule that calls API driving logic to provide a betting cycle function when a tunnel entry event occurs.

[0312] FIGS. 13a and 13b are drawings illustrating an example in which a vehicle automatically proposes automation rules and improves automation rules based on user feedback according to an embodiment of the present disclosure.

[0313] FIGS. 13a and FIGS. 13b show examples of conversation screens including text that recognizes a voice command spoken by a user and text that indicates a response output through the LLM (6631) in response to the voice command.

[0314] In particular, FIGS. 13a and FIGS. 13b may be drawings embodying an embodiment of FIG. 11.

[0315] Referring to FIG. 13a, the processor (175) can display a first conversation screen (1310) through a visual interface (6420) to propose automation rules.

[0316] In FIG. 13a, the automation rule may be a rule that performs an automatic destination setting function, which automatically sets the destination of the vehicle (200) to a location included in an email upon the occurrence of a schedule-related email reception event.

[0317] The top vehicle event is a schedule-related email reception event, and the vehicle function matching the top vehicle event may be the automatic destination setting function.

[0318] The processor (175) can display a first response (1311) inquiring about notifying an email regarding a travel schedule on the first conversation screen (1310). After that, the processor (175) can display a first text command (1312) on the first conversation screen (1310) in response to a voice command requesting a notification of an email regarding a travel schedule according to the first response (1311).

[0319] The processor (175) can display a second response (1313) on the first dialog screen (1310) for a proposal of an automation rule (1313) indicating that the automatic destination setting function of the vehicle (200) can be activated in response to the first text command (1312).

[0320] The processor (175) can display a second text command (1314) on the first conversation screen (1310) according to a voice command indicating a request to accept the proposal of the first automation rule (1313) and to add a reply function indicating attendance to the automatic destination setting function of the first automation rule (1313).

[0321] The processor (175) can create and register a second automation rule that includes an attendance reply function in addition to the automatic destination setting function according to the second text command (1314). The processor (175) can display a third response (1315) related to the creation of the second automation rule on the first conversation screen (1310).

[0322] The second automation rule may be a rule that performs an automatic destination setting function, which automatically sets the destination of the vehicle (200) to a location included in an email in accordance with the occurrence of a schedule-related email reception event.

[0323] Referring to FIG. 13b, the processor (175) can display a second conversation screen (1320) through a visual interface (6420) to propose automation rules.

[0324] The automation rule may be a rule that performs the function of executing a music playback application when a vehicle boarding event occurs.

[0325] The processor (175) can display a first response (1321) suggesting that a sound source playback application be executed when boarding a vehicle on the second conversation screen (1320).

[0326] The processor (175) may accept the first response (1321) and display a first text command (1322) in response to a voice command to play an album with a like button. The first text command (1322) may be a request to add a function to play an album that the user has liked when the audio playback application is launched.

[0327] If there is an API restriction that prevents a specific album from being played in the corresponding audio playback application, the processor (175) can display a second response (1323) on the second dialog screen (1320) indicating a function that can be provided (a function to play the last played list) while notifying the restriction.

[0328] The processor (175) may change the existing automation rule according to a second text command (1324) corresponding to a voice command indicating acceptance. The changed automation rule may be a rule that plays the last played list while performing a sound playback application when a vehicle boarding event occurs.

[0329] As such, according to the embodiments of FIGS. 13a and 13b, automation rules that meet the requirements can be suggested in a timely manner before the user recognizes or inputs a command, and automation rules can be updated according to user feedback, thereby improving user convenience and providing a customized service.

[0330] An apparatus according to one embodiment of the present disclosure may include a memory (140); and one or more processors (175) that receive user operation information, acquire lower vehicle events based on the received operation information and event specification information, infer upper vehicle events and vehicle functions corresponding to the upper vehicle events from the acquired lower vehicle events through a large language model, generate automation rules based on the inferred upper vehicle events and the inferred vehicle functions, and store the generated automation rules in the memory.

[0331] The above automation rule may be code or a script that performs the vehicle function when the above higher vehicle event occurs.

[0332] One or more processors (175) can obtain the code or the script through the large language model.

[0333] The above device may further include an output interface, and the one or more processors (175) may determine whether an automation rule corresponding to the detected upper vehicle event is stored in the memory when an upper vehicle event is detected, and if the automation rule is stored in the memory, output a notification indicating that the vehicle function corresponding to the automation rule can be provided through the output interface.

[0334] When the above one or more processors (175) receive user acceptance input for the vehicle function, they execute the automation rule to provide the vehicle function, and when they receive feedback to reflect additional functions for the vehicle function, they update the automation rule to reflect the additional functions and store the updated automation rule in the memory (140).

[0335] When the above one or more processors (175) receive user acceptance input for the vehicle function, they execute the automation rule to provide the vehicle function, receive feedback to reflect additional functions for the vehicle function, and if there are constraints on the received feedback, update the automation rule to reflect additional functions other than the additional functions, and store the updated automation rule in the memory (140).

[0336] The above operation information may include one or more of the following: input operations performed through the IVI (In-Vehicle Infotainment) system, voice commands, information on user operation of a physical interface, or vehicle occupancy status information.

[0337] The above event specification information is information for expressing an event detected inside or outside the device, and the above event specification information may include one or more of an event identifier for identifying the event, a condition for the occurrence of the event, a time of occurrence of the event, a type of sensor that detected the event, a location where the event occurred, or the content of the event.

[0338] One or more processors (175) may generate an automation rule for changing the operation method of the big language model based on the inferred vehicle function being a function for changing the operation method of the big language model, and store the generated automation rule in the memory.

[0339] The above device

[0340] It can be a vehicle (200) or a CDC (Cockpit domain controller).

[0341] The functions of the elements disclosed in the present invention may be implemented using circuits or processing circuits comprising general-purpose processors, special-purpose processors, integrated circuits, ASICs (Application-Specific Integrated Circuits), conventional circuits, and / or combinations thereof. A processor may be defined as a processing circuit or circuit comprising transistors and other circuits.

[0342] In the present invention, circuits, units, or means may be hardware designed or programmed to perform a specified function. The hardware may be the hardware disclosed in the present invention or other known hardware programmed or configured to perform a specified function. Where the hardware is a processor that can be considered a type of circuit, the circuits, means, or units may be a combination of hardware and software, and the software may constitute the hardware and / or the processor.

[0343] The above-described disclosure can be implemented as computer-readable code on a medium on which a program is recorded. A computer-readable medium includes all types of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable media include a Hard Disk Drive (HDD), a Solid State Disk (SSD), a Silicon Disk Drive (SDD), ROM, RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc. Additionally, the computer may include a processor (175).

[0344] Although preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above. Various modifications are possible by those skilled in the art without departing from the essence of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical spirit or perspective of the present disclosure.

Claims

1. In the device, Memory; and One or more processors comprising receiving user operation information, acquiring lower vehicle events based on the received operation information and event specification information, inferring upper vehicle events and vehicle functions corresponding to the upper vehicle events from the acquired lower vehicle events through a large language model, generating automation rules based on the inferred upper vehicle events and the inferred vehicle functions, and storing the generated automation rules in the memory. device.

2. In Paragraph 1, The above automation rule is Code or script that performs the vehicle function when the above-mentioned upper vehicle event occurs device.

3. In Paragraph 2, The above one or more processors Obtaining the code or script through the above-mentioned massive language model device.

4. In Paragraph 1, Includes additional output interfaces, The above one or more processors When a higher-level vehicle event is detected, it is determined whether an automation rule corresponding to the detected higher-level vehicle event is stored in the memory, and if the automation rule is stored in the memory, a notification is output through the output interface indicating that the vehicle function corresponding to the automation rule can be provided. device.

5. In Paragraph 4, The above one or more processors When user acceptance input regarding the above vehicle function is received, the above automation rule is executed to provide the above vehicle function, and When feedback is received to reflect additional functions regarding the above vehicle functions, the automation rule is updated to reflect the additional functions, and the updated automation rule is stored in the memory. device.

6. In Paragraph 4, The above one or more processors When user acceptance input regarding the above vehicle function is received, the above automation rule is executed to provide the above vehicle function, and Receive feedback to reflect additional functions regarding the above vehicle functions, and if there are constraints on the received feedback, update the automation rule to reflect additional functions other than the above additional functions, and store the updated automation rule in the memory. device.

7. In Paragraph 1, The above operation information Includes one or more of information regarding input actions performed through an IVI (In-Vehicle Infotainment) system, voice commands, user operations on a physical interface, or vehicle occupancy status information. device.

8. In Paragraph 1, The above event specification information is Information for representing an event detected inside or outside the device, and The above event specification information is Includes one or more of an event identifier for identifying the above event, conditions for the occurrence of the above event, time of occurrence of the above event, type of sensor that detected the above event, location of occurrence of the above event, or content of the above event. device.

9. In Paragraph 1, The above one or more processors Based on the fact that the above-mentioned inferred vehicle function is a function for changing the operation method of the above-mentioned large language model, an automation rule for changing the operation method of the above-mentioned large language model is generated, and the generated automation rule is stored in the memory. device.

10. In Paragraph 1, The above device Vehicle or CDC (Cockpit domain controller) device.

11. In the method of operating the device, Step of receiving user action information; A step of acquiring sub-vehicle events based on the received operation information and event specification information; A step of inferring a higher-level vehicle event and a vehicle function corresponding to the higher-level vehicle event from the lower-level vehicle events obtained through a large language model; and The step of generating an automation rule based on the inferred higher-level vehicle event and the inferred vehicle function, and storing the generated automation rule. Method of operation of the device.

12. In Paragraph 11, The above automation rule is Code or script that performs the vehicle function when the above-mentioned upper vehicle event occurs Method of operation of the device.

13. In Paragraph 12, The step of obtaining the code or the script through the above-mentioned giant language model is further included. Method of operation of the device.

14. In Paragraph 11, A step of determining whether an automation rule corresponding to the detected higher vehicle event is stored when a higher vehicle event is detected; If the above automation rule is stored, the method further includes the step of outputting a notification indicating that the vehicle function corresponding to the above automation rule is available. Method of operation of the device.

15. In Paragraph 14, When user acceptance input regarding the above vehicle function is received, a step of executing the above automation rule to provide the above vehicle function; and If feedback is received to reflect additional functions regarding the above vehicle functions, the method further includes the step of updating the automation rule to reflect said additional functions and storing the updated automation rule. Method of operation of the device.

16. In Paragraph 14, A step of providing the vehicle function by executing the automation rule when receiving user acceptance input regarding the vehicle function; A step of receiving feedback to reflect additional functions for the above vehicle functions; and If there are constraints on the received feedback, the method further includes the step of updating the automation rule to reflect the additional function and other additional functions, and saving the updated automation rule. Method of operation of the device.

17. In Paragraph 11, The above operation information Includes one or more of information regarding input actions performed through an IVI (In-Vehicle Infotainment) system, voice commands, user operations on a physical interface, or vehicle occupancy status information. Method of operation of the device.

18. In Paragraph 11, The above event specification information is Information for representing an event detected inside or outside the device, and The above event specification information is Includes one or more of an event identifier for identifying the above event, conditions for the occurrence of the above event, time of occurrence of the above event, type of sensor that detected the above event, location of occurrence of the above event, or content of the above event. Method of operation of the device.

19. In Paragraph 11, The method further includes the step of generating an automation rule for changing the operation method of the large language model based on the fact that the inferred vehicle function is a function for changing the operation method of the large language model, and storing the generated automation rule. Method of operation of the device.

20. A computer-readable non-volatile recording medium having a program for performing a method of operating a device, The above method of operation Step of receiving user action information; A step of acquiring sub-vehicle events based on the received operation information and event specification information; A step of inferring a higher-level vehicle event and a vehicle function corresponding to the higher-level vehicle event from the lower-level vehicle events obtained through a large language model; and The step of generating an automation rule based on the inferred higher-level vehicle event and the inferred vehicle function, and storing the generated automation rule. Non-volatile recording media.

Images