Apparatus, operation method of apparatus, and computer-readable non-volatile recording medium on which program for performing operation method of apparatus is recorded

Abstract

An apparatus according to one embodiment of the present disclosure may comprise: an output interface; a memory for storing pieces of multi-modal processed data of a preset time unit, generated on the basis of pieces of multi-modal data; and one or more processors for acquiring a voice command, retrieving, from the memory, multi-modal processed data related to the acquired voice command, generating a prompt on the basis of the retrieved multi-modal processed data and the acquired voice command, acquiring a response result from the prompt through an artificial intelligence model, and outputting the acquired response result through the output interface.

Description

A computer-readable non-volatile recording medium storing a device, a method of operating the device, and a program for performing the method of operating the device.

[0001] The present disclosure relates to on-device context management technology for intelligent multimodal interaction within a vehicle.

[0002] A vehicle AI (Artificial Intelligence) agent is an agent that enables intelligent interaction between the driver and the vehicle.

[0003] In the case of existing vehicle AI agents, a large amount of multimodal information, such as ADAS (Advanced Driver Assistance System) information, DMS (Driver Monitoring System) information, and IMS (Interior Monitoring System) information, is utilized as a context for understanding the situation, but there was a fundamental limitation that this could only be utilized for results recognized by the solutions.

[0004] Recent AI agents based on Large Language Models (LLM) are capable of supporting multimodal processing by receiving and interpreting video and audio simultaneously; however, since most require the transmission of data streams to cloud servers, there has been a problem of relatively high data costs.

[0005] In addition, directly inputting data streams into LLM significantly increases the input tokens and computational load, resulting in a problem where API usage fees (or cloud computing costs) become very high.

[0006] Due to these problems, there were limitations in use scenes where automotive AI agents utilize various multimodal data to understand the user's environment and situation and interact intelligently with the user.

[0007] The purpose of the present disclosure may be to enable an AI agent to effectively utilize information inside and outside a vehicle to flexibly interact with a user.

[0008] The purpose of the present disclosure may be to process and manage various vehicle multimodal data in real time into context data suitable for interpretation by a generative AI model.

[0009] The objective of the present disclosure may be to reduce cloud computing costs by equipping a vehicle with a generative AI model and directly outputting a response to context data.

[0010] An apparatus according to one embodiment of the present disclosure may include: an output interface; a memory for storing multimodal processing data of a preset time unit generated based on multimodal data; and one or more processors for acquiring a voice command, searching for multimodal processing data related to the acquired voice command from the memory, generating a prompt based on the searched multimodal processing data and the acquired voice command, acquiring a response result from the prompt through an artificial intelligence model, and outputting the acquired response result through the output interface.

[0011] A method of operation of an apparatus according to one embodiment of the present disclosure may include: storing multimodal processing data of a preset time unit generated based on multimodal data; acquiring a voice command; searching for multimodal processing data related to the acquired voice command; generating a prompt based on the searched multimodal processing data and the acquired voice command; acquiring a response result from the prompt through an artificial intelligence model; and outputting the acquired response result.

[0012] A computer-readable non-volatile recording medium storing a program for performing a method of operation of a device according to one embodiment of the present disclosure, wherein the method of operation may include: storing multimodal processing data of a preset time unit generated based on multimodal data; acquiring a voice command; searching for multimodal processing data associated with the acquired voice command; generating a prompt based on the searched multimodal processing data and the acquired voice command; acquiring a response result from the prompt through an artificial intelligence model; and outputting the acquired response result.

[0013] According to various embodiments of the present disclosure, even in cases where there are objects or situations that ADAS / DMS / IMS cannot recognize, it is possible to support flexible interpretation of multimodal data by utilizing generative AI models.

[0014] According to various embodiments of the present disclosure, by providing an optimized context based on user queries on a device, the problems of transmission costs and generative AI model costs for multimodal data can be mitigated.

[0015] According to various embodiments of the present disclosure, by continuously storing multimodal data and supporting exploration thereof, relevant context can be provided even for objects or past situation information that have already been passed at the time of user utterance.

[0016] According to various embodiments of the present disclosure, various proactive services can be automatically provided according to the input conditions of multimodal data.

[0017] Figure 1 is a drawing illustrating an example of the exterior and interior of a vehicle.

[0018] Figure 2 is a diagram illustrating the architecture of a signal processing system for vehicles.

[0019] FIG. 3a is a drawing illustrating an example of the arrangement of a vehicle display device inside a vehicle.

[0020] FIG. 3b is a drawing illustrating another example of the arrangement of a vehicle display device inside a vehicle.

[0021] Figure 4 is an example of an internal block diagram of the vehicle of Figure 1.

[0022] FIG. 5 is a block diagram illustrating the configuration of a server according to one embodiment of the present disclosure.

[0023] FIGS. 6a to 6c are drawings for explaining the configuration of a system according to an embodiment of the present disclosure.

[0024] FIG. 7 is a flowchart illustrating a method of operation of a device according to one embodiment of the present disclosure.

[0025] FIG. 8 is a flowchart illustrating the process of generating indexed multimodal processing data based on multimodal data according to one embodiment of the present disclosure.

[0026] FIGS. 9a to 9f are drawings illustrating a process of outputting a response result based on a user's voice command and multimodal context data according to an embodiment of the present disclosure.

[0027] FIGS. 10a to 10d are drawings illustrating a process of outputting a response result based on a user's voice command and multimodal context data according to another embodiment of the present disclosure.

[0028] FIGS. 11a to 11d are drawings illustrating a process of outputting a response result based on a user's voice command and multimodal context data according to another embodiment of the present disclosure.

[0029] FIG. 12 is a flowchart illustrating a method of operation of a device according to another embodiment of the present disclosure.

[0030] FIGS. 13a to 13c are drawings illustrating a process of generating a preemptive service query and outputting a response result when a trigger event occurs according to an embodiment of the present disclosure.

[0031] The present disclosure will be described in more detail below with reference to the drawings.

[0032] The suffixes "module" and "part" for components used in the following description are assigned solely for the ease of drafting this specification and do not inherently confer any particularly significant meaning or role. Accordingly, the terms "module" and "part" may be used interchangeably.

[0033] Figure 1 is a drawing illustrating an example of the exterior and interior of a vehicle.

[0034] Referring to the drawing, the vehicle (200) is operated by a plurality of wheels (103FR, 103FL, 103RL,...) that rotate by a power source, and a steering wheel (150) for controlling the direction of travel of the vehicle (200).

[0035] Meanwhile, the vehicle (200) may further be equipped with a camera (195), etc., for acquiring an image of the front of the vehicle.

[0036] Meanwhile, the vehicle (200) may be equipped with a plurality of displays (180a, 180b) for displaying images, information, etc. inside.

[0037] In FIG. 1, a cluster display (180a) and an AVN (Audio Video Navigation) display (180b) are exemplified as multiple displays (180a, 180b). Other displays such as a HUD (Head Up Display) are also possible.

[0038] Meanwhile, the AVN (Audio Video Navigation) display (180b) may also be named the Center Information Display.

[0039] Meanwhile, the vehicle (200) described in this specification may be a concept that includes all of the following: a vehicle equipped with an engine as a power source, a hybrid vehicle equipped with an engine and an electric motor as a power source, an electric vehicle equipped with an electric motor as a power source, etc.

[0040] Figure 2 is a diagram illustrating the architecture of a signal processing system for vehicles.

[0041] Referring to the drawing, the architecture (300a) of the vehicle signal processing system can correspond to a zone-based architecture.

[0042] Accordingly, sensor devices and processors inside the vehicle may be placed in each of the multiple zones (Z1 to Z4), and a signal processing device (170a) including a vehicle communication gateway (GWDa) may be placed in the central area of ​​the multiple zones (Z1 to Z4).

[0043] Meanwhile, the signal processing device (170a) may additionally include an autonomous driving control module (ACC), a cockpit control module (CPG), etc., in addition to the vehicle communication gateway (GWDa).

[0044] The vehicle communication gateway (GWDa) within the signal processing device (170a) may be a High Performance Computing (HPC) gateway.

[0045] That is, the signal processing device (170a) of FIG. 2 is an integrated HPC and can exchange data with an external communication module (not shown) or a processor (not shown) in a plurality of zones (Z1 to Z4).

[0046] FIG. 3a is a drawing illustrating an example of the arrangement of a vehicle display device inside a vehicle.

[0047] Referring to the drawing, the vehicle interior may be equipped with a cluster display (180a), an AVN (Audio Video Navigation) display (180b), a rear seat entertainment display (180c, 180d), a rearview mirror display (not shown), etc.

[0048] FIG. 3b is a drawing illustrating another example of the arrangement of a vehicle display device inside a vehicle.

[0049] A vehicle display device (100) according to an embodiment of the present disclosure may include a plurality of displays (180a to 180b) and a signal processing device (170) that performs signal processing for displaying images, information, etc. on the plurality of displays (180a to 180b) and outputs an image signal to at least one display (180a to 180b).

[0050] Among the plurality of displays (180a to 180b), the first display (180a) is a cluster display (180a) for displaying driving status, operation information, etc., and the second display (180b) may be an AVN (Audio Video Navigation) display (180b) for displaying vehicle operation information, navigation map, various entertainment information or video.

[0051] The signal processing device (170) has a processor (175) inside and can execute a first virtualization machine to a third virtualization machine (not shown) on a hypervisor (not shown) within the processor (175).

[0052] A second virtualization machine (not shown) operates for the first display (180a), and a third virtualization machine (not shown) can operate for the second display (180b).

[0053] Meanwhile, the first virtualization machine (not shown) within the processor (175) can be controlled to set up a shared memory (508) based on a hypervisor (505) for the same data transmission to the second virtualization machine (not shown) and the third virtualization machine (not shown). Accordingly, the same information or the same image can be synchronized and displayed on the first display (180a) and the second display (180b) within the vehicle.

[0054] Meanwhile, the first virtualization machine (not shown) within the processor (175) shares at least a portion of the data with the second virtualization machine (not shown) and the third virtualization machine (not shown) for data sharing processing. Accordingly, data can be shared and processed by multiple virtualization machines for multiple displays within the vehicle.

[0055] Meanwhile, the first virtualization machine (not shown) within the processor (175) can receive and process wheel speed sensor data of the vehicle and transmit the processed wheel speed sensor data to at least one of the second virtualization machine (not shown) or the third virtualization machine (not shown). Accordingly, the wheel speed sensor data of the vehicle can be shared with at least one virtualization machine, etc.

[0056] Meanwhile, the vehicle display device (100) according to the embodiment of the present disclosure may further include a rear seat entertainment display (180c) for displaying driving status information, simple navigation information, various entertainment information or images.

[0057] The signal processing device (170) can control the RSE display (180c) by running a fourth virtualization machine (not shown) in addition to the first to third virtualization machines (not shown) on a hypervisor (not shown) within the processor (175).

[0058] Accordingly, various displays (180a to 180c) can be controlled using a single signal processing device (170).

[0059] Meanwhile, some of the multiple displays (180a to 180c) operate under a Linux OS, and others can operate under a Web OS.

[0060] A signal processing device (170) according to an embodiment of the present disclosure can control displays (180a to 180c) operating under various operating systems (OS) to synchronize and display the same information or the same image.

[0061] Meanwhile, FIG. 3b illustrates that a vehicle speed indicator (212a) and a vehicle interior temperature indicator (213a) are displayed on a first display (180a), a home screen (222) including a plurality of applications, a vehicle speed indicator (212b), and a vehicle interior temperature indicator (213b) is displayed on a second display (180b), and a second home screen (222b) including a plurality of applications and a vehicle interior temperature indicator (213c) is displayed on a third display (180c).

[0062] Figure 4 is an example of an internal block diagram of the vehicle of Figure 1.

[0063] Referring to the drawings, a vehicle (200) according to an embodiment of the present disclosure may be equipped with a lamp drive unit (751), a steering drive unit (752), a brake drive unit (753), a power source drive unit (754), a suspension drive unit (756), an air conditioning drive unit (757), a window drive unit (758), a seat drive unit (761), and a signal processing device (170).

[0064] Meanwhile, the vehicle (200) may further be equipped with an ECU (770), a plurality of sensor devices (SN), and a plurality of communication modules (EMa~EMd).

[0065] Meanwhile, the vehicle (200) according to the embodiment of the present disclosure may further be equipped with a vehicle display device (100).

[0066] A vehicle display device (100) according to an embodiment of the present disclosure may include an input unit (110), a communication device (120) for communication with an external device, a plurality of communication modules (EMa~EMd) for internal communication, a memory (140), a signal processing device (170), a plurality of displays (180a~180c), an audio output unit (185), and a power supply unit (190).

[0067] Multiple communication modules (EMa~EMd) can be placed in each of the multiple zones (Z1~Z4) of FIG. 2, for example.

[0068] Meanwhile, the signal processing device (170) may have a communication switch (736b) inside for data communication with each communication module (EM1~EM4).

[0069] Each communication module (EM1~EM4) can perform data communication with a plurality of sensor devices (SN), ECU (770), or area signal processing device (170Z).

[0070] Meanwhile, a plurality of sensor devices (SN) may include a camera (195), lidar (196), radar (197), or position sensor (198).

[0071] The input unit (110) may be equipped with physical buttons, pads, etc. for button input, touch input, etc.

[0072] Meanwhile, the input unit (110) may be equipped with a microphone (not shown) for user voice input.

[0073] The communication device (120) can exchange data wirelessly with a mobile terminal (800) or a server (900).

[0074] In particular, the communication device (120) can wirelessly exchange data with the vehicle driver's mobile terminal. Various data communication methods are possible as wireless data communication methods, such as Bluetooth, WiFi, WiFi Direct, and APiX.

[0075] The communication device (120) can receive weather information, road traffic condition information, for example, TPEG (Transport Protocol Expert Group) information from a mobile terminal (800) or a server (900). To this end, the communication device (120) may be equipped with a mobile communication module (not shown).

[0076] Meanwhile, the communication device (120) can exchange data wirelessly with an adjacent vehicle.

[0077] For example, the communication device (120) can exchange vehicle messages wirelessly with an adjacent vehicle through V2X (Vehicle-to-everything) communication.

[0078] A plurality of communication modules (EM1~EM4) can receive sensor data, etc. from an ECU (770), a sensor device (SN), or a region signal processing device (170Z), and transmit the received sensor data to the signal processing device (170).

[0079] Here, the sensor data may include at least one of vehicle direction data, vehicle location data (GPS data), vehicle angle data, vehicle speed data, vehicle acceleration data, vehicle tilt data, vehicle forward / reverse data, battery data, fuel data, tire data, vehicle lamp data, vehicle interior temperature data, and vehicle interior humidity data.

[0080] Such sensor data can be obtained from a heading sensor, a yaw sensor, a gyro sensor, a position module, a vehicle forward / reverse sensor, a wheel sensor, a vehicle speed sensor, a vehicle body inclination sensor, a battery sensor, a fuel sensor, a tire sensor, a steering sensor based on steering wheel rotation, a vehicle interior temperature sensor, a vehicle interior humidity sensor, etc.

[0081] Meanwhile, the position module may include a GPS module or a position sensor (198) for receiving GPS information.

[0082] Meanwhile, at least one of the multiple communication modules (EM1 to EM4) can transmit location information data sensed from a GPS module or a location sensor (198) to a signal processing device (170).

[0083] Meanwhile, at least one of the plurality of communication modules (EM1 to EM4) can receive vehicle front image data, vehicle side image data, vehicle rear image data, and obstacle distance information around the vehicle from a camera (195), lidar (196), radar (197), etc., and transmit the received information to a signal processing device (170).

[0084] The memory (140) can store various data for the overall operation of the vehicle display device (100), such as a program for processing or controlling the signal processing device (170).

[0085] For example, memory (140) can store data regarding a hypervisor, a first virtualization machine to a third virtualization machine, for execution within a processor (175).

[0086] The audio output unit (185) converts an electrical signal from the signal processing device (170) into an audio signal and outputs it. To do this, a speaker or the like may be provided.

[0087] The power supply unit (190) can supply power necessary for the operation of each component under the control of the signal processing unit (170). In particular, the power supply unit (190) can receive power from a battery inside the vehicle, etc.

[0088] The signal processing device (170) controls the overall operation of each unit within the vehicle display device (100) or vehicle (200).

[0089] For example, the signal processing device (170) may include a processor (175) that performs signal processing for a vehicle display (180a, 180b).

[0090] The processor (175) can run a first virtualization machine to a third virtualization machine (not shown) on a hypervisor (not shown) within the processor (175).

[0091] Among the first to third virtual machines (not shown), the first virtual machine (not shown) may be named a Server Virtual Machine, and the second to third virtual machines (not shown) may be named a Guest Virtual Machine.

[0092] For example, a first virtualization machine (not shown) within a processor (175) can receive sensor data from a plurality of sensor devices, such as vehicle sensor data, location information data, camera image data, audio data, or touch input data, and process or modify it to output it.

[0093] In this way, by performing most of the data processing in the first virtualization machine (not shown), 1:N data sharing becomes possible.

[0094] As another example, the first virtualization machine (not shown) can directly receive and process CAN data, Ethernet data, audio data, radio data, USB data, and wireless communication data for the second virtualization machine to the third virtualization machine (not shown).

[0095] And, the first virtualization machine (not shown) can transmit the processed data to the second virtualization machine to the third virtualization machine (not shown).

[0096] Accordingly, among the first to third virtualization machines (not shown), only the first virtualization machine (not shown) receives sensor data, communication data, or external input data from a plurality of sensor devices and performs signal processing, thereby reducing the signal processing burden on other virtualization machines and enabling 1:N data communication, which enables synchronization when sharing data.

[0097] Meanwhile, the first virtualization machine (not shown) can control the sharing of the same data with the second virtualization machine (not shown) and the third virtualization machine (not shown) by writing data to the shared memory (508).

[0098] For example, the first virtualization machine (not shown) can record vehicle sensor data, the location information data, the camera image data, or the touch input data in a shared memory (508) and control the sharing of the same data with the second virtualization machine (not shown) and the third virtualization machine (not shown). Accordingly, data sharing in a 1:N manner becomes possible.

[0099] Ultimately, by performing most of the data processing on the first virtualization machine (not shown), 1:N data sharing becomes possible.

[0100] Meanwhile, the first virtualization machine (not shown) within the processor (175) can control the second virtualization machine (not shown) and the third virtualization machine (not shown) to set up a shared memory (508) based on the hypervisor (505) for the same data transmission.

[0101] Meanwhile, the signal processing device (170) can process various signals such as audio signals, video signals, and data signals. To this end, the signal processing device (170) can be implemented in the form of a System On Chip (SOC).

[0102] FIG. 5 is a block diagram illustrating the configuration of a server according to one embodiment of the present disclosure.

[0103] Referring to FIG. 5, the server (900) may refer to a device that trains an artificial neural network using a machine learning algorithm or uses a trained artificial neural network. Here, the server (900) may be composed of multiple servers to perform distributed processing and may be defined as a 5G network.

[0104] The server (900) may be included as part of the vehicle (200) and may perform at least some of the AI ​​processing together.

[0105] The server (900) may include a communication interface (510), memory (530), a learning processor (540), and a processor (560).

[0106] The communication interface (510) can transmit and receive data with the vehicle (200) or an external device.

[0107] The memory (530) may include a model storage unit (531). The model storage unit (531) may store a model (or artificial neural network, 531a) that is being learned or has been learned through the learning processor (540).

[0108] The learning processor (540) can train the artificial neural network (531a) using training data. The training model may be used while mounted on the server (900) of the artificial neural network, or it may be used while mounted on an external device such as a vehicle (200).

[0109] The learning model may be implemented in hardware, software, or a combination of hardware and software. If part or all of the learning model is implemented in software, one or more instructions constituting the learning model may be stored in memory (530).

[0110] The processor (560) can use a learning model to infer a result value for new input data and generate a response or control command based on the inferred result value.

[0111] FIGS. 6a to 6c are drawings for explaining the configuration of a system according to an embodiment of the present disclosure.

[0112] The system (6000) of Fig. 6a can be referred to as an IVEX (In Vehicle Experience) system.

[0113] Referring to FIG. 6a, the system (6000) may include an edge sensor group (6100), a recognizer (6200), an insightor (6300), and an illustrator (6400).

[0114] The edge sensor group (6100) may be included in the plurality of sensor devices (SN) of FIG. 2.

[0115] The recognizer (6200) and the insightor (6300) may be included in the signal processing device (170) of the vehicle display device (100) of FIG. 4. In particular, the recognizer (6200) and the insightor (6300) may be included in the processor (175) of the signal processing device (170).

[0116] In another embodiment, the recognizer (6200) and the insightor (6300) may be included in the cockpit control module (CPG) of FIG. 2.

[0117] An illustrator (6400) may be included in the vehicle display device (100) of FIG. 2 and FIG. 4.

[0118] The recognizer (6200) can recognize the vehicle situation based on a set of sensing data received from the edge sensor group (6100). The edge sensor group (6100) may include vehicle internal sensors (6110) and external sensors (6120).

[0119] The vehicle interior sensors (6110) are sensors placed inside the vehicle (200) and may include a front camera, lidar, radar, interior camera, vehicle interior temperature sensor, and vehicle interior humidity sensor.

[0120] The external sensors (6120) are sensors placed on the exterior of the vehicle (200) and may include an external camera, lidar, radar, heading sensor, yaw sensor, gyro sensor, position module, vehicle forward / reverse sensor, wheel sensor, vehicle speed sensor, body inclination detection sensor, battery sensor, fuel sensor, tire sensor, steering sensor based on steering wheel rotation, etc. The position module may include a GPS module or a position sensor (198) for receiving GPS information.

[0121] The sensing data set may include an internal vehicle data set and an external vehicle data set.

[0122] The vehicle interior data set may be a data set sensed by the vehicle interior sensors (6110). The vehicle interior data set may include at least one of vehicle interior temperature data, vehicle interior humidity data, battery data, fuel data, vehicle lamp data, tire data or vehicle interior image data, and audio data received through a microphone.

[0123] The vehicle external data set may be a data set sensed by external sensors (6120). The vehicle external data set may include at least one of vehicle location data (GPS), vehicle direction data, vehicle angle data, vehicle speed data, vehicle acceleration data, vehicle tilt data, whether the vehicle is moving forward or backward, front image data, or rear image data.

[0124] The recognizer (6200) can perform preprocessing and calibration on the sensing data set and can obtain vehicle internal information (6210) and vehicle external information (6220) based on the preprocessed and calibrated sensing data set.

[0125] The vehicle interior information (6210) may include a vehicle interior data set or information about the vehicle interior situation obtained from the vehicle interior data set.

[0126] The vehicle external information (6220) may include a vehicle external data set or information about the vehicle external situation obtained from the vehicle external data set.

[0127] The recognizer (6200) can recognize a vehicle situation based on at least one of vehicle internal information (6210) or vehicle external information (6220), and can generate vehicle situation information for the recognized vehicle situation.

[0128] Vehicle situation information may include at least one of vehicle external situation information indicating a situation regarding the exterior of the vehicle (200), vehicle internal situation information indicating a situation regarding the interior of the vehicle (200), or user situation information indicating a situation regarding a user inside the vehicle (200). User situation information may be included in the vehicle internal situation information.

[0129] User situation information may include information about the user's face identifier (Face ID), distraction, gaze, position, gesture, emotional state, whether they are drinking or taking drugs, drowsiness, and other actions inside the vehicle (200).

[0130] The recognizer (6200) can transmit the generated vehicle situation information to the insightor (6300).

[0131] The insightor (6300) can generate a response result based on vehicle situation information received from the recognizer (6200).

[0132] The insightor (6300) may include a multimodal context engine (6310), a context controller (6320), a safety assistance engine (6330), a user context engine (6340), an AI orchestrator (6350), and an AI model set (6360).

[0133] The multimodal context engine (6310) can generate multimodal context data based on vehicle situation information received from the recognizer (6200). The multimodal context data may include at least one of text data or image data describing the vehicle situation generated based on the vehicle situation information.

[0134] The context controller (6320) may be a component included in the multimodal context engine (6310) or provided separately from the multimodal context engine (6310).

[0135] When the context controller (6320) receives a user query from the AI ​​orchestrator (6350), it can execute a process of generating multimodal context data. The user query may be a speech recognition result corresponding to a voice command spoken by the user.

[0136] The safety assist engine (6330) can determine whether the current situation is a safe situation or a dangerous situation based on vehicle situation information received from the recognizer (6200).

[0137] If the safety assist engine (6330) determines that the current situation is a dangerous situation, it can transmit driver assistance control commands and warning notification output commands to the safety application (6440) of the illustrator (6400).

[0138] The user context engine (6340) can generate user context data based on user information. User information may be referred to as a user persona. User information may include at least one of the user's nationality, age, gender, occupation, personality, or psychological type (Myers-Briggs Type Indicator, MBTI).

[0139] The AI ​​orchestrator (6350) can generate a prompt based on at least one of multimodal context data or user context data. The AI ​​orchestrator (6350) can transmit the generated prompt to at least one of a plurality of AI models included in the AI ​​model set (6360).

[0140] The AI ​​model set (6360) may include multiple AI models. Each AI model may output an inference result in response to a prompt received from the AI ​​orchestrator (6350) and may transmit the inference result to the AI ​​orchestrator (6350).

[0141] The AI ​​orchestrator (6350) can generate additional prompts based on inference results and can transmit the additional prompts to at least one AI model in the set of AI models (6360). The AI ​​model that receives the additional prompts can output additional inference results in response to the additional prompts and can transmit the additional inference results to the AI ​​orchestrator (6350).

[0142] The AI ​​orchestrator (6350) can obtain an inference result or additional inference result received from the AI ​​model as a response result, and can output the obtained response result to an illustrator (6400).

[0143] The illustrator (6400) can output service information or perform driving control functions based on the response result output from the insightor (6300).

[0144] The illustrator (6400) may include a multimodal output encoder (6410), a visual interface (6420), an audio interface (6430), and a safety application (6440).

[0145] The multimodal output encoder (6410) can encode the response result output from the insightor (6300) and output the encoded response result data to the visual interface (6420) or audio interface (6430).

[0146] The visual interface (6420) or audio interface (6430) may be referred to as an output interface.

[0147] The visual interface (6420) can display response result data output from the multimodal output encoder (6410) or an image based on the response result data. The visual interface (6420) may include at least one of the plurality of displays (180a to 180c) of FIG. 4.

[0148] The visual interface (6420) can display an Augmented Reality (AR) image based on response result data or a Mixed Reality (MR) image based on response result data.

[0149] The audio interface (6430) can output response result data in the form of audio. The audio interface (6430) may be included in the audio output unit (185) of FIG. 4.

[0150] The safety application (6440) can perform Advanced Driver Assistance System (ADAS) control according to the response result received from the driver assistance control command or the insightor (6300), and can output a warning notification according to the warning notification output command.

[0151] The safety application (6440) may be included in the electronic control unit (770) of FIG. 4 or executed by the electronic control unit (770).

[0152] FIG. 6b is a drawing for explaining the configuration of an insightor according to another embodiment of the present disclosure.

[0153] In FIG. 6b, the multimodal context engine (6310) may include a context controller (6320).

[0154] Referring to FIG. 6b, the insightor (6300) may include a multimodal context engine (6310), an AI orchestrator (6350), and a multimodal LLM (6361).

[0155] The multimodal context engine (6310) may include a multimodal signal adapter (6311), a multimodal indexer (6312), a multimodal context buffer (6313), a multimodal context retriever (6314), a multimodal context descriptor (6315), a multimodal event monitor (6316), and a context controller (6320).

[0156] The multimodal signal adapter (6311) can generate pre-processed multimodal data by filtering, cleaning, synchronizing, and reformulating data received from various sensors or vehicle situation information received from a recognizer (6200).

[0157] At least one of the following can be input to the multimodal signal adapter (6311): audio data received through a microphone, front image data captured through a front camera, ADAS information obtained from the front image data or vehicle sensors, location data, IVI (In-Vehicle Infotainment) system data, IVI display information, DMS (Driver Monitoring System) information / IMS (Interior Monitoring System) information based on image data captured through an interior camera, and biometric data obtained from a biometric sensor.

[0158] The multimodal indexer (6312) can generate multimodal processing data by dividing preprocessed multimodal data into chunks and can index the multimodal processing data. The multimodal indexer (6312) can obtain an encoding vector or keyword representing the attribute (or meaning) of the multimodal processing data divided into chunks as an index.

[0159] The multimodal context buffer (6313) can store multimodal processing data and an index corresponding to the multimodal processing data.

[0160] The multimodal context retriever (6314) can search for multimodal processing data most relevant to the user query through an index from the multimodal context buffer (6313) and can select the searched multimodal processing data as a context candidate for the creation of multimodal context data.

[0161] The multimodal context descriptor (6315) can reconfigure multimodal processing data selected as a context candidate into multimodal context data having a prompt form that the multimodal LLM (6361) can interpret.

[0162] The multimodal event monitor (6316) can monitor whether an index matching the trigger condition is entered when a trigger condition for multimodal data is registered. If an index matching the trigger condition is entered, the multimodal event monitor (6316) can generate a trigger event to generate a proactive service query.

[0163] The context controller (6320) can control the overall operation of the multimodal context engine (6310). When the context controller (6320) receives a user query from the AI ​​orchestrator (6350), it can execute the process of generating multimodal context data. When the context controller (6320) receives a trigger event from the multimodal event monitor (6316), it can generate a preemptive service query.

[0164] The AI ​​orchestrator (6350) can generate a prompt by combining a user query and a multimodal context, and can transmit the generated prompt to a multimodal LLM (6361). The user query may be a query in the form of recognized text based on a voice command spoken by the user. The voice command may be received through a microphone, and the voice command may be converted into text through an Automatic Speech Recognition (ASR) process. The user query may be text converted through an Automatic Speech Recognition (ASR) process.

[0165] The multimodal LLM (6361) may be an example of an AI model included in the set of AI models (6360) of FIG. 6a. The multimodal LLM (6361) may output an inference result from a prompt received from the AI ​​orchestrator (6350) and may transmit the inference result to the AI ​​orchestrator (6350).

[0166] The inference result may include a result representing a response to the prompt.

[0167] For example, the inference result may include an API call result regarding whether an API call corresponding to a driving assistance function was successfully performed, and a feedback generation result regarding whether feedback corresponding to the API call was successfully generated.

[0168] FIG. 6c is a drawing for explaining the configuration of an AI orchestrator according to one embodiment of the present disclosure.

[0169] Referring to FIG. 6c, the AI ​​orchestrator (6350) may include a task arbitrator (6351), a prompt manager (6352), a sub-agent set (6353), a knowledge DB (6354), a workflow controller (6355), and a tool set / adapter set (6356).

[0170] The task arbitrator (6351) can determine one of the multiple sub-agents based on the voice recognition result and the multimodal context data output from the multimodal context engine (6310).

[0171] The prompt manager (6352) can generate a prompt for the operation of a determined sub-agent. The prompt manager (6352) can generate a prompt based on multimodal context data and user queries. The prompt manager (6352) can generate a prompt based on information about the functions that the determined sub-agent can perform, multimodal context data, the results of previously performed functions, conversation history, and user queries.

[0172] The sub-agent set (6353) may include multiple sub-agents. For example, the sub-agent set (6353) may include a navigation agent for navigation services, a vehicle function agent for providing vehicle functions, a telephony agent for automated telephone answering services, and a Q&A agent for providing response services to queries.

[0173] A sub-agent determined by the task arbitrator (6351) may call a cloud AI model or an on-device AI model assigned to it. The cloud model or on-device model may be a Large Language Model (LLM).

[0174] A sub-agent can call a cloud AI model or an on-device AI model assigned to it to obtain inference results corresponding to a prompt from the model.

[0175] Based on the obtained inference result, the sub-agent can call the knowledge DB (6354) to provide additional information, obtain additional information from the knowledge DB (6354), specify the name of the function to be executed and the parameter value of the function, and determine the feedback text to be provided to the user.

[0176] The sub-agent can transmit to the workflow controller (6355) a parsing result generated by parsing the inference result received from the model, which includes additional information called from the knowledge DB (6354), the name of the function to be executed, the parameter values ​​of the function, and a feedback phrase.

[0177] The knowledge DB (6354) can store additional information and information about functions. The information about functions may include the name of the function and the parameter values ​​of the function.

[0178] The workflow controller (6355) can generate control commands to perform the corresponding function based on the parsing results and can store the results of the conversation and function performed by the AI ​​model and sub-agent.

[0179] The tool set / adapter set (6356) can call an API corresponding to a control command generated by the workflow controller (6355). The tool set / adapter set (6356) can convert the control command into an execution command of an actual function or an API call command that the IVI system can understand and execute, and can execute the converted command.

[0180] FIG. 7 is a flowchart illustrating a method of operation of a device according to one embodiment of the present disclosure.

[0181] Additionally, although the embodiment of FIG. 7 is described as being performed by the processor (175) of the vehicle (200), it may also be performed by the processor (560) of the cockpit control module (CPG) or the server (900). The processor (175) may be provided in multiple numbers.

[0182] Referring to FIG. 7, the processor (175) of the vehicle (200) can obtain a voice command spoken by a user through a microphone (S701).

[0183] The user can utter a voice command while driving the vehicle (200), and the processor (175) can receive the voice command through the microphone.

[0184] The processor (175) can obtain a voice recognition result corresponding to a voice command and determine whether it is necessary to search for multimodal processing data based on the obtained voice recognition result.

[0185] The processor (175) can determine whether there is a need to search for previously stored multimodal processing data through the AI ​​orchestrator (6350).

[0186] In one embodiment, the AI ​​orchestrator (6350) may determine that search for multimodal processing data is necessary when ADAS information is required based on the voice recognition result.

[0187] In another embodiment, the AI ​​orchestrator (6350) may determine that when the voice recognition result requires context understanding, the search for multimodal processing data is necessary.

[0188] The AI ​​orchestrator (6350) can process voice recognition results when ADAS information is not required or when context recognition is not required. For example, if the voice recognition result is "<Give me directions home>", the AI ​​orchestrator (6350) can control the navigation to guide along a saved route.

[0189] The processor (175) of the vehicle (200) can search for multimodal processing data related to the acquired voice command (S703).

[0190] In one embodiment, multimodal processing data may be data generated based on multimodal data.

[0191] Multimodal data may include at least one of the following: audio data received through a microphone, front image data captured through a front camera, ADAS information, location data, IVI system data, IVI display information, DMS (Driver Monitoring System) information / IMS (Interior Monitoring System) information based on image data captured through an interior camera, and biometric data acquired from a biometric sensor.

[0192] ADAS information may include sensing information obtained from each of the camera (195), lidar (196), radar (197), and position sensor (198) of the vehicle (200). In particular, ADAS information may include information recognized in direct relation to safety during driving.

[0193] The processor (175) can generate multimodal processing data based on multimodal data and can store the generated multimodal processing data in a multimodal context buffer (6313) or memory (140) by matching it to an index.

[0194] The processor (175) can obtain a voice recognition result corresponding to a voice command and can search for multimodal processing data related to the voice recognition result among the multimodal processing data stored in the multimodal context buffer (6313) or memory (140).

[0195] The processor (175) compares the index matched to the multimodal processing data with the voice recognition result, and based on the comparison result, can search for multimodal processing data related to the voice recognition result.

[0196] The processor (175) can measure the similarity between the index matched to the multimodal processing data and the voice recognition result, and can obtain the multimodal processing data matched to the index that has a similarity greater than a certain similarity.

[0197] The processor (175) can obtain a text-based speech recognition result from a voice command through an ASR process. The speech recognition result may be referred to as a user query.

[0198] FIG. 8 is a flowchart illustrating the process of generating indexed multimodal processing data based on multimodal data according to one embodiment of the present disclosure.

[0199] The steps of Fig. 8 can be performed prior to step S701 of Fig. 7.

[0200] Referring to FIG. 8, the processor (175) of the vehicle (200) can acquire multimodal data (S801) and preprocess the acquired multimodal data (S803).

[0201] In one embodiment, the preprocessing process of multimodal data may include filtering, cleaning, synchronization, and reconfiguration.

[0202] The processor (175) can filter and refine multimodal data through a multimodal signal adapter (6311) and perform time synchronization processing on the filtered and refined multimodal data. The processor (175) can reconstruct the multimodal data through processes such as combining the multimodal data according to the time synchronization processing.

[0203] The processor (175) of the vehicle (200) can generate multimodal processed data by dividing the preprocessed multimodal data into chunks (S805).

[0204] In one embodiment, the chunk unit may be a detailed time unit. The detailed time unit may be 10 seconds, but this is merely an example. Multimodal processing data may be divided into chunk units.

[0205] The processor (175) of the vehicle (200) can generate an index corresponding to the generated multimodal processing data (S807), and can store the generated index by matching it to the multimodal processing data (S809).

[0206] In one embodiment, the processor (175) can obtain an index matching each multimodal processing data through the multimodal data indexer (6312). The multimodal data indexer (6312) can generate an index using an encoding vector or keyword representing the attributes of each multimodal processing data.

[0207] The processor (175) can match the generated index to the multimodal processing data and store it in the multimodal context buffer (6313). The multimodal context buffer (6313) may be included in the processor (175) or in the memory (140) of FIG. 4.

[0208] Multimodal processed data included in multimodal chunk data can have the same index.

[0209] Again, Figure 7 is explained.

[0210] The processor (175) of the vehicle (200) can generate a prompt to be input into an AI model based on the discovered multimodal processing data and voice commands (S705).

[0211] The processor (175) can generate multimodal context data based on multimodal processing data discovered through the context controller (6320). The multimodal context data may be data used to generate prompts to be input into an AI model.

[0212] The processor (175) can generate a prompt based on user queries obtained from multimodal context data and voice commands through the AI ​​orchestrator (6350).

[0213] The processor (175) of the vehicle (200) can obtain a response result for a prompt generated through an AI model (S707)

[0214] An AI model can be a multimodal LLM that simultaneously understands various forms of data, such as text, images, and voice, and outputs results.

[0215] The multimodal LLM may be provided in the memory (140) of the vehicle (200) or in the server (900).

[0216] In one embodiment, the processor (175) can input the generated prompt into a multimodal LLM provided in the vehicle (200) to obtain a response result.

[0217] In another embodiment, the processor (175) can transmit the generated prompt to a multimodal LLM provided in the server (900) to obtain a response result from the LLM provided in the server (900).

[0218] The processor (175) of the vehicle (200) can output the obtained response result (S709).

[0219] The processor (175) can output a response result through one or more of the visual interface (6420) or the audio interface (6430).

[0220] FIGS. 9a to 9f are drawings illustrating a process of outputting a response result based on a user's voice command and multimodal context data according to an embodiment of the present disclosure.

[0221] In particular, in the embodiments of FIGS. 9a to 9f, multimodal context data can be generated based on multimodal data including ADAS information.

[0222] Referring to FIG. 9a, the processor (175) of the vehicle (200) can obtain a first front image (901) captured through a front camera.

[0223] The processor (175) can recognize a speed limit sign (902) from the first front image (901), and the processor (175) can obtain a recognition result (910) from the speed limit sign (902) that the speed limit value is 90 km / h. The recognition result (910) can be included in ADAS information. Multimodal data may be data corresponding to a unit of time (e.g., 1 second).

[0224] The multimodal signal adapter (6311) can filter multiple multimodal data corresponding to multiple modals input at each unit time and can synchronize multiple multimodal data over time. A modal represents a type of data and can be any one of an image type, an audio type, or a text type.

[0225] After synchronization processing, the multimodal signal adapter (6311) can combine or reconfigure the multimodal data to generate preprocessed multimodal data (920) as illustrated in FIG. 9b. The preprocessed multimodal data (920) may include the sign type (speed_limit), content (value, unit), and timestamp (created_at).

[0226] The multimodal indexer (6312) can generate multimodal chunk data (930) including multimodal processing data (931, 932, 933) as illustrated in FIG. 9c by matching an index to each of the preprocessed multimodal data. The multimodal chunk data (930) may be data obtained by dividing the multimodal data into chunk units. The multimodal chunk data (930) may include a first multimodal processing data (931), a second multimodal processing data (932), and a third multimodal processing data (933).

[0227] The first multimodal processing data (931) may be a combination of multimodal data preprocessed at a first time point, the second multimodal processing data (932) may be a combination of multimodal data preprocessed at a second time point that is 1 second after the first time point, and the third multimodal processing data (933) may be a combination of multimodal data preprocessed at a third time point that is 1 second after the second time point. For example, the first multimodal processing data (931) may include preprocessed multimodal data (920).

[0228] The multimodal indexer (6312) can generate an index of an encoding vector (931a) or a keyword representing an attribute of the multimodal chunk data (930). For example, the keyword is<speed limit> It could be.

[0229] The multimodal indexer (6312) can match an encoding vector (931a) to each of the multimodal processed data (931, 932, 933) and store them in the multimodal context buffer (6313).

[0230] The multimodal indexer (6312) can store only the multimodal processing data from the previous T seconds relative to the current time point in the multimodal context buffer (6313) to prevent wasting memory capacity.

[0231] Referring to FIG. 9d, after the processor (175) acquires the first forward image (901), when N seconds have elapsed, the user speaks<What is the speed limit here now?> You can receive a voice command (950) that says [this].

[0232] The multimodal context retriever (6314) can search for multimodal processing data most relevant to the voice recognition result (speed limit inquiry of this section) of the voice command (950) received from the multimodal context buffer (6313).

[0233] The multimodal context retriever (6314) can search for the index most relevant to the voice recognition result (the speed limit inquiry of this section) and, based on the searched index, can obtain the latest multimodal processing data from a certain time interval prior to the time when the voice command (950) was received.

[0234] For example, the multimodal context retriever (6314) can select the third multimodal processed data (933), which is most relevant to the speed limit and is the most recent data, as the search result.

[0235] The multimodal context descriptor (6315) can generate multimodal context data (960) as illustrated in FIG. 9e based on the selected third multimodal processing data (933). The multimodal context descriptor (6315) can convert the third multimodal processing data (933) into multimodal context data (960) in the form of a prompt that can be interpreted by the multimodal LLM (6361).

[0236] Referring to FIG. 9f, the AI ​​orchestrator (6350) can generate a prompt by combining a voice recognition result (950) corresponding to a voice command (950) and multimodal context data (960), and can transmit the generated prompt to a multimodal LLM (6361).

[0237] A multimodal LLM (6361) can generate a response result (970) from an input prompt, and an AI orchestrator (6350) can receive the response result (970) from the multimodal LLM (6361). The response result (970) may include a first text (971) indicating that a speed limit sign (902) was detected N seconds ago, a second text (972) indicating the speed limit sign (902) and that the current speed limit is 90 km / h.

[0238] The AI ​​orchestrator (6350) can display the received response result (970) through the visual interface (6420).

[0239] In this way, according to an embodiment of the present disclosure, a function for continuously storing and searching multimodal processing data is provided, so that information about objects that have already passed or information about past situations can be efficiently provided.

[0240] FIGS. 10a to 10d are drawings illustrating a process of outputting a response result based on a user's voice command and multimodal context data according to another embodiment of the present disclosure.

[0241] In particular, in the embodiments of FIGS. 10a to 10d, multimodal context data may be generated based on multimodal data that does not include ADAS information. ADAS information may include information used to perform ADAS functions.

[0242] Referring to FIG. 10a, the processor (175) of the vehicle (200) can acquire a front image (1000) captured through a front camera.

[0243] The multimodal signal adapter (6311) can filter multiple multimodal data corresponding to multiple modals input at each unit time and can synchronize multiple multimodal data over time.

[0244] After synchronization processing, the multimodal signal adapter (6311) can combine or reconfigure the multimodal data to generate preprocessed multimodal data for each object.

[0245] The multimodal indexer (6312) can generate multimodal chunk data (1010, 1020, 1030) corresponding to each of the multiple objects as illustrated in FIG. 10b by matching an index to each of the preprocessed multimodal data, and can index the generated multimodal chunk data (1010, 1020, 1030) and store them in the multimodal context buffer (6313).

[0246] Each multimodal chunk data may contain information about each object. The information about each object may include text contained in the object, location information of a bounding box identifying the object, and timestamp information.

[0247] For example, multimodal chunk data (1010) may include information about a parking restriction sign (1001) included in a front image (1000). Multimodal chunk data (1010) may include a fourth multimodal processing data (1011) and a fifth multimodal processing data (1012).

[0248] The multimodal indexer (6312) can generate an index of an encoding vector or keyword (1011a) representing the attributes of each multimodal chunk of data. For example, the keyword (1011a) is<No stopping> It could be.

[0249] The multimodal indexer (6312) can match a keyword (1011a) to each of the fourth multimodal processing data (1011) and the fifth multimodal processing data (1012) and store them in the multimodal context buffer (6313).

[0250] The multimodal indexer (6312) can store only the multimodal processing data from the previous T seconds relative to the current time point in the multimodal context buffer (6313) to prevent wasting memory capacity.

[0251] The fourth multimodal processing data (1011) may include a keyword (No stopping), location information of a parking restriction sign (1001), and a time stamp of a first point in time, and the fifth multimodal processing data (1012) may include a keyword (No stopping), location information of a parking restriction sign (1001), and a time stamp of a second point in time.

[0252] Referring to FIG. 10c, the processor (175) acquires a forward image (1000) and then the user speaks<Is it possible to park here now?> It can receive a voice command (1040) that is

[0253] The multimodal context retriever (6314) can search for multimodal processing data most relevant to the voice recognition result (inquiry on parking availability) of the voice command (1040) received from the multimodal context buffer (6313).

[0254] The multimodal context retriever (6314) can search for multiple multimodal processed data related to a parking availability inquiry (selection of N-best instance).

[0255] The multimodal context retriever (6314) can search for an index (or keyword) that is most related to the voice recognition result (inquiry regarding parking availability) among multiple multimodal processed data, and based on the searched index, can obtain the latest multimodal processed data from a certain time interval prior to the time when the voice command (1040) was received.

[0256] For example, the multimodal context retriever (6314) can select the fifth multimodal processed data (1012), which is most relevant to the inquiry regarding parking availability and is the most recent data, as the search result.

[0257] The multimodal context descriptor (6315) can generate multimodal context data (1050) as illustrated in FIG. 10c based on the selected fifth multimodal processing data (1012). The multimodal context descriptor (6315) can convert the fifth multimodal processing data (1012) into multimodal context data (1050) in the form of a prompt that can be interpreted by the multimodal LLM (6361).

[0258] The multimodal context descriptor (6315) can generate multiple multimodal context data corresponding to the multiple multimodal processing data when multiple multimodal processing data are searched. The AI ​​orchestrator (6350) can obtain data selected through user input among the multiple multimodal context data as the final multimodal context data.

[0259] Referring to FIG. 10d, the AI ​​orchestrator (6350) can generate a prompt by combining a voice recognition result corresponding to a voice command (1040) and a multimodal context (1050), and can transmit the generated prompt to a multimodal LLM (6361).

[0260] A multimodal LLM (6361) can generate a response result (1060) from an input prompt, and an AI orchestrator (6350) can receive the response result (1060) from the multimodal LLM (6361). The response result (1060) may include a first text (1061) indicating that a parking restriction sign (1001) has been detected, and a second text (1063) indicating a response regarding the parking restriction sign (1001) and whether parking is available.

[0261] The AI ​​orchestrator (6350) can display the received response result (1060) through the visual interface (6420).

[0262] As such, according to an embodiment of the present disclosure, even if ADAS information is limited, a function to continuously store and search multimodal processing data is provided so that a response to a user query can be efficiently provided.

[0263] FIGS. 11a to 11d are drawings illustrating a process of outputting a response result based on a user's voice command and multimodal context data according to another embodiment of the present disclosure.

[0264] In particular, in the embodiments of FIGS. 11a to 11d, multimodal context data can be generated based on multimodal data including DMS information.

[0265] DMS information may include user eye-gaze information.

[0266] Referring to FIG. 11a, the processor (175) of the vehicle (200) can acquire a front image (1100) captured through a front camera.

[0267] The multimodal signal adapter (6311) can filter multiple multimodal data corresponding to multiple modals input at each unit time and can synchronize multiple multimodal data over time.

[0268] After synchronization processing, the multimodal signal adapter (6311) can combine or reconfigure the multimodal data to generate preprocessed multimodal data for each object.

[0269] Each preprocessed multimodal data may include information about each object. The information about each object may include text contained in the object, location information of a bounding box identifying the object, and timestamp information.

[0270] The multimodal indexer (6312) can generate multimodal chunk data (1110, 1120, 1130) corresponding to each of the multiple objects as illustrated in FIG. 11b by matching an index to each of the preprocessed multimodal data, and can index the generated multimodal chunk data (1110, 1120, 1130) and store them in the multimodal context buffer (6313).

[0271] Multimodal chunk data (1110) may include information about a yellow flag (1101) included in a front image (1100). Multimodal chunk data (1110) may include sixth multimodal processing data (1111) and seventh multimodal processing data (1112).

[0272] The multimodal indexer (6312) can generate an index of an encoding vector or keyword (1111a) representing the attributes of each multimodal chunk of data. For example, the keyword (1111a) is <fisney>It could be.

[0273] The multimodal indexer (6312) can match a keyword (1111a) to each of the sixth multimodal processing data (1111) and the seventh multimodal processing data (1112) and store them in the multimodal context buffer (6313).

[0274] The multimodal indexer (6312) can assign a tag to multimodal processing data if the user's gaze stays on a specific object for a certain period of time or longer based on the user's gaze information included in the DMS information. For example, the multimodal indexer (6312) can assign a tag "DMS" to the sixth multimodal processing data (1111) if the user's gaze stays on an object (yellow flag) corresponding to the multimodal chunk data (1110) for a certain period of time or longer.

[0275] The multimodal indexer (6312) can store only the multimodal processing data from the previous T seconds relative to the current time point in the multimodal context buffer (6313) to prevent wasting memory capacity.

[0276] The sixth multimodal processing data (1111) may include a keyword (Fisney), location information of a yellow flag (1101), and a time stamp of a first time point, and the seventh multimodal processing data (1112) may include a keyword (Fisney), location information of a yellow flag (1101), and a time stamp of a second time point.

[0277] Referring to FIG. 11c, the processor (175) acquires the forward image (1100) and then the user speaks<What is that yellow flag?> It can receive a voice command (1140) that says [this].

[0278] The multimodal context retriever (6314) can search for multimodal processing data most relevant to the voice recognition result (inquiry about a yellow flag) of the voice command (1140) received from the multimodal context buffer (6313).

[0279] The multimodal context retriever (6314) can search for the index most relevant to the voice recognition result (inquiry about a yellow flag) and, based on the searched index, can obtain the latest multimodal processing data from a certain time interval prior to the time when the voice command (1140) was received.

[0280] The multimodal context retriever (6314) can select the most relevant sixth multimodal processing data (1111) as a search result based on voice recognition results and DMS tags. The multimodal context retriever (6314) can extract the latest multimodal processing data related to the voice recognition results and select the final multimodal processing data based on at least one of the presence or absence of a DMS tag or the value of a DMS tag among the extracted multimodal processing data. A DMS tag is a label that classifies a driver's state or behavior into a specific category, and can be expressed as forward gaze, left gaze, right gaze, etc.

[0281] The value of the DMS tag may be a time value or a count corresponding to the DMS tag. For example, the value of the DMS tag may be either the time the user (driver) gazed at the object or the number of times the object was looked at.

[0282] The multimodal context retriever (6314) can give a high weight to multimodal processing data where a DMS tag exists, or give a high weight to multimodal processing data where the DMS tag value has a large value.

[0283] The multimodal context retriever (6314) can obtain the multimodal processing data with the largest value of the DMS tag among the extracted multimodal processing data as the final multimodal processing data.

[0284] The multimodal context retriever (6314) can give more weight to the multimodal processing data if the DMS tag exists in the multimodal processing data.

[0285] The multimodal context descriptor (6315) can generate multimodal context data (1150) as illustrated in FIG. 11c based on the selected sixth multimodal processing data (1111). The multimodal context descriptor (6315) can convert the sixth multimodal processing data (1111) into multimodal context data (1150) in the form of a prompt that can be interpreted by the multimodal LLM (6361).

[0286] When the multimodal context descriptor (6315) receives URL information related to the yellow flag (1101) from the AI ​​orchestrator (6350), it can generate multimodal context data by further considering the received URL information.

[0287] Referring to FIG. 11d, the AI ​​orchestrator (6350) can generate a prompt by combining a voice recognition result corresponding to a voice command (1140) and a multimodal context (1150), and can transmit the generated prompt to a multimodal LLM (6361).

[0288] A multimodal LLM (6361) can generate a response result (1160) from an input prompt, and an AI orchestrator (6350) can receive the response result (1160) from the multimodal LLM (6361). The response result (1160) may include a yellow flag (1101) and text (1161) representing a description of the yellow flag (1101).

[0289] The AI ​​orchestrator (6350) can display the received response result (1160) through the visual interface (6420).

[0290] As such, according to an embodiment of the present disclosure, a function for continuously storing and searching multimodal processing data using DMS information is provided, so that a response to a user query can be efficiently provided.

[0291] FIG. 12 is a flowchart illustrating a method of operation of a device according to another embodiment of the present disclosure.

[0292] The embodiment of FIG. 12 is described as being performed by the processor (175) of the vehicle (200), but it may also be performed by the processor (560) of the cockpit control module (CPG) or the server (900).

[0293] The embodiment of FIG. 12 is an example in which, when there is no voice command from the user in the embodiment of FIG. 7, a preemptive service query is generated and a response result is automatically output.

[0294] Referring to FIG. 12, the processor (175) of the vehicle (200) can register a trigger condition (S1201).

[0295] In one embodiment, the trigger condition may be a condition for generating a trigger event. For example, the trigger condition may be registered by a user request.

[0296] The context controller (6320) can register trigger conditions according to user requests. For example, the trigger conditions may include High Occupancy Vehicle (HOV) lane conditions or Carpool lane conditions.

[0297] The HOV lane condition may be a condition indicating that an HOV lane has been detected from a front image obtained through the front camera of the vehicle (200).

[0298] The carpool lane condition may be a condition indicating that a carpool lane has been detected from a front image obtained through the front camera of the vehicle (200).

[0299] The trigger condition may further include a multi-passenger condition indicating that multiple people are detected inside the vehicle (200) from an internal image obtained through an internal camera.

[0300] Step S1201 can be performed before Step S801 of FIG. 8, and Step S1203 and subsequent steps can be performed after S809.

[0301] The processor (175) of the vehicle (200) can monitor whether the trigger condition for generating a trigger event is satisfied (S1203).

[0302] The processor (175) can monitor whether a trigger condition is satisfied through a multimodal event monitor (6316). The multimodal event monitor (6316) can determine whether a trigger condition is satisfied based on multimodal data. The multimodal data may include one or more of a front image obtained from a front camera, text recognized from the front image, and an internal image obtained through an internal camera.

[0303] The processor (175) of the vehicle (200) can generate a trigger event when the trigger condition for generating the trigger event is satisfied as a result of monitoring (S1205), and can generate a preemptive service query according to the generated trigger event (S1207).

[0304] In one embodiment, the processor (175) may generate a trigger event through the multimodal event monitor (6316) when one of the conditions for an HOV lane condition or a carpool lane condition is satisfied. The trigger event may be an event indicating that an HOV lane or a carpool lane has been detected.

[0305] The processor (175) can generate a preemptive service query through the context controller (6320) as a trigger event is generated. The preemptive service query may be a service query that is available in relation to the trigger event. For example, the preemptive service query may be a query to check whether entry into a carpool lane is possible.

[0306] The processor (175) of the vehicle (200) can generate a prompt based on the generated preemptive service query and IMS information (S1209).

[0307] The AI ​​orchestrator (6350) can generate a prompt from the preemptive service query and IMS information received from the context controller (6320). The IMS information may include an internal image of the vehicle (200) obtained through the internal camera of the vehicle (200) at the time the trigger event is generated.

[0308] The processor (175) of the vehicle (200) can obtain a response result to a prompt through an AI model (S1211) and can output the obtained response result (S1213).

[0309] An AI model can be a multimodal LLM that simultaneously understands various forms of data, such as text, images, and voice, and outputs results.

[0310] The multimodal LLM may be provided in the memory (140) of the vehicle (200) or in the server (900).

[0311] In one embodiment, the processor (175) can input the generated prompt into a multimodal LLM provided in the vehicle (200) to obtain a response result.

[0312] In another embodiment, the processor (175) can transmit the generated prompt to a multimodal LLM provided in the server (900) to obtain a response result from the LLM provided in the server (900).

[0313] The processor (175) can output a response result through one or more of the visual interface (6420) or the audio interface (6430).

[0314] FIGS. 13a to 13c are drawings illustrating a process of generating a preemptive service query and outputting a response result when a trigger event occurs according to an embodiment of the present disclosure.

[0315] Referring to FIG. 13a, the processor (175) of the vehicle (200) can acquire a front image (1300) captured through a front camera.

[0316] The multimodal indexer (6312) can generate multimodal chunk data (1310, 1320, 1330) corresponding to each of the multiple objects as illustrated in FIG. 13b by matching an index to each of the preprocessed multimodal data, and can index the generated multimodal chunk data (1310, 1320, 1330) and store them in the multimodal context buffer (6313).

[0317] The multimodal chunk data (1310) may include information about the carpool lane sign (1301) included in the front image (1300). The multimodal chunk data (1310) may include the eighth multimodal processing data (1311).

[0318] The multimodal indexer (6312) can generate an index of an encoding vector or keyword (1311a) representing the attributes of the eighth multimodal processed data (1311). For example, the keyword (1311a) is <carppools>It could be.

[0319] The 8th multimodal processing data (1311) may include keywords (CARPPOOLS), location information of carpool lane signs (1301), and timestamps.

[0320] Referring to FIG. 13c, the context controller (6320) can generate a preemptive service query (1340) and IMS information (1350) for a service provision inquiry related to the trigger event when a trigger event occurs.

[0321] The AI ​​orchestrator (6350) can generate a prompt based on the preemptive service query (1340) and IMS information (1350), and can transmit the generated prompt to the multimodal LLM (6361).

[0322] The AI ​​orchestrator (6350) can obtain a response result (1360) corresponding to a prompt through a multimodal LLM (6361) and can display the obtained response result (1360).

[0323] The response result (1360) may include a first text (1361) indicating that the current time is within the usable section of the carpool lane, a carpool lane sign (1301), a second text (1363) indicating that the carpool lane is usable because multiple people are riding inside the vehicle, an image (1364) indicating information on the number of people allowed to carpool, and a seat occupancy icon (1365) indicating the number of people currently riding in the vehicle.

[0324] As such, according to an embodiment of the present disclosure, when a pre-registered trigger condition is satisfied, a preemptive service is automatically provided, thereby enabling the provision of a customized service to the driver.

[0325] An apparatus (200, 100) according to one embodiment of the present disclosure may include: an output interface; a memory (140) for storing multimodal processing data of a preset time unit generated based on multimodal data; and one or more processors (175) for acquiring a voice command, searching for multimodal processing data related to the acquired voice command from the memory, generating a prompt based on the searched multimodal processing data and the acquired voice command, acquiring a response result from the prompt through an artificial intelligence model, and outputting the acquired response result through the output interface.

[0326] One or more processors (175) may generate multimodal context data based on the searched multimodal processing data and generate the prompt based on the generated multimodal context data and the voice recognition result corresponding to the voice command.

[0327] The above one or more processors (175) can convert the searched multimodal processing data into data that the artificial intelligence model can interpret, thereby generating the multimodal context data.

[0328] The above one or more processors (175) can acquire one or more multimodal processing data related to a voice recognition result corresponding to the acquired voice command, assign weights to the one or more multimodal processing data, and select a final multimodal processing data according to the result of assigning weights.

[0329] The above weight may be assigned based on the presence or absence of a Driver Monitoring System (DMS) tag included in each multimodal processing data or at least one of the values ​​of said DMS tag.

[0330] The above one or more processors (175) may give a high weight to multimodal processing data in which the DMS tag exists, or give a high weight to multimodal processing data in which the value of the DMS tag has a large value.

[0331] The above one or more processors (175) can generate the multimodal processed data by filtering, refining, time-synchronizing, and reconfiguring the multimodal data.

[0332] One or more processors (175) can obtain an encoding vector or keyword representing the attributes of the multimodal processing data as an index, and match the obtained index to the multimodal processing data and store it in the memory.

[0333] The above one or more processors (175) can compare the index with the voice recognition result corresponding to the voice command and search the multimodal processing data according to the comparison result.

[0334] The above device may be a vehicle (200) or a CDC (Cockpit domain controller).

[0335] The functions of the elements disclosed in the present invention may be implemented using circuits or processing circuits comprising general-purpose processors, special-purpose processors, integrated circuits, ASICs (Application-Specific Integrated Circuits), conventional circuits, and / or combinations thereof. A processor may be defined as a processing circuit or circuit comprising transistors and other circuits.

[0336] In the present invention, circuits, units, or means may be hardware designed or programmed to perform a specified function. The hardware may be the hardware disclosed in the present invention or other known hardware programmed or configured to perform a specified function. Where the hardware is a processor that can be considered a type of circuit, the circuits, means, or units may be a combination of hardware and software, and the software may constitute the hardware and / or the processor.

[0337] The above-described disclosure can be implemented as computer-readable code on a medium on which a program is recorded. A computer-readable medium includes all types of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable media include a Hard Disk Drive (HDD), a Solid State Disk (SSD), a Silicon Disk Drive (SDD), ROM, RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc. Additionally, the computer may include a processor (175).

[0338] Although preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above. Various modifications are possible by those skilled in the art without departing from the essence of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical spirit or perspective of the present disclosure.< / carppools> < / fisney>

Claims

1. In the device, Output interface; A memory for storing multimodal processing data of preset time units generated based on multimodal data; and One or more processors comprising acquiring a voice command, searching for multimodal processing data related to the acquired voice command from the memory, generating a prompt based on the searched multimodal processing data and the acquired voice command, acquiring a response result from the prompt through an artificial intelligence model, and outputting the acquired response result through the output interface. device.

2. In Paragraph 1, The above one or more processors Generate multimodal context data based on the above-mentioned multimodal processing data, and generate the prompt based on the generated multimodal context data and a voice recognition result corresponding to the voice command. device.

3. In Paragraph 2, The above one or more processors Converting the aforementioned explored multimodal processing data into a form that the artificial intelligence model can interpret, thereby generating the multimodal context data. device.

4. In Paragraph 1, The above one or more processors Acquiring one or more multimodal processing data related to a voice recognition result corresponding to the acquired voice command, assigning weights to the one or more multimodal processing data, and selecting a final multimodal processing data according to the result of assigning the weights. device.

5. In Paragraph 4, The above weight is Assigned based on the presence or absence of a Driver Monitoring System (DMS) tag included in each multimodal processing data, or at least one of the values ​​of said DMS tag device.

6. In Paragraph 5, The above one or more processors Assigning a high weight to multimodal processing data where the above DMS tag exists, or assigning a high weight to multimodal processing data where the value of the above DMS tag has a large value. device.

7. In Paragraph 1, The above one or more processors Generating the multimodal processed data by filtering, refining, time-synchronizing, and reconstructing the above multimodal data device.

8. In Paragraph 7, The above one or more processors An encoding vector or keyword representing the attributes of the multimodal processing data is obtained as an index, and the obtained index is matched to the multimodal processing data and stored in the memory. device.

9. In Paragraph 8, The above one or more processors Comparing the above index with the voice recognition result corresponding to the above voice command, and searching the above multimodal processing data according to the comparison result device.

10. In Paragraph 1, The above device Vehicle or CDC (Cockpit domain controller) device.

11. In the method of operating the device, A step of storing multimodal processing data of a preset time unit generated based on multimodal data; Step of obtaining a voice command; A step of searching for multimodal processing data related to the above-mentioned acquired voice command; A step of generating a prompt based on the searched multimodal processing data and the acquired voice command; A step of obtaining a response result from the above prompt through an artificial intelligence model; and A step comprising outputting the above-mentioned obtained response result Method of operation of the device.

12. In Paragraph 11, The step of generating the above prompt is A step of generating multimodal context data based on the aforementioned explored multimodal processing data and The step of generating the prompt based on the generated multimodal context data and the voice recognition result corresponding to the voice command. Method of operation of the device.

13. In Paragraph 12, The step of generating the above multimodal context data The method includes the step of converting the aforementioned explored multimodal processing data into a form that the artificial intelligence model can interpret, thereby generating the multimodal context data. Method of operation of the device.

14. In Paragraph 11, The step of exploring the above multimodal processing data is A step of acquiring one or more multimodal processing data related to a voice recognition result corresponding to the above-mentioned acquired voice command, A step of assigning weights to one or more of the above multimodal processing data and A step of selecting final multimodal processing data based on the result of assigning the above weights. Method of operation of the device.

15. In Paragraph 14, The above weight is Assigned based on the presence or absence of a Driver Monitoring System (DMS) tag included in each multimodal processing data, or at least one of the values ​​of said DMS tag Method of operation of the device.

16. In Paragraph 15, The step of assigning the above weights A step of assigning a high weight to multimodal processing data where the above DMS tag exists, or a step of assigning a high weight to multimodal processing data where the value of the above DMS tag has a large value. Method of operation of the device.

17. In Paragraph 11, The method further includes the step of generating the multimodal processed data by filtering, refining, time-synchronizing, and reconstructing the multimodal data. Method of operation of the device.

18. In Paragraph 17, A step of obtaining an encoding vector or keyword representing the attributes of the multimodal processed data as an index; and The method further includes the step of matching the acquired index to the multimodal processing data and storing it in memory. Method of operation of the device.

19. In Paragraph 11, The step of exploring the above multimodal processing data is A step of comparing the above index with a voice recognition result corresponding to the above voice command and A step including the exploration of the multimodal processing data based on the comparison result Method of operation of the device.

20. A computer-readable non-volatile recording medium having a program for performing a method of operating a device, The above method of operation A step of storing multimodal processing data of a preset time unit generated based on multimodal data; Step of obtaining a voice command; A step of searching for multimodal processing data related to the above-mentioned acquired voice command; A step of generating a prompt based on the searched multimodal processing data and the acquired voice command; A step of obtaining a response result from the above prompt through an artificial intelligence model; and A step comprising outputting the above-mentioned obtained response result Non-volatile recording media.

Images