Apparatus, operation method of apparatus, and non-volatile computer-readable recording medium having recorded thereon program for performing operation method of apparatus

Abstract

An apparatus according to an embodiment of the present disclosure may comprise: an output interface; and one or more processors that generate context information by using at least one of sensing information, driver information, and driving information; extract, from the generated context information, a cause description indicating a cause of a vehicle event; generate a prompt on the basis of the extracted cause description; acquire driving feedback from the generated prompt through an artificial intelligence model; and output the acquired driving feedback through the output interface.

Description

A computer-readable non-volatile recording medium storing a device, a method of operating the device, and a program for performing the method of operating the device.

[0001] The present disclosure relates to an invention for providing personalized driving feedback.

[0002] A Driving Behavior Analysis System is a system that analyzes a driver's driving style, behavior, and habits based on data to help evaluate or improve safety, fuel efficiency, and maintenance needs. Utilizing various sensors and technologies, the system collects driving data and provides meaningful information or feedback based on this analysis.

[0003] However, existing driving habit analysis systems are rule-based, simple feedback systems, which have the problem of being difficult to provide personalized feedback.

[0004] In addition, existing driving habit analysis systems have difficulty reflecting various driving situations other than predefined rules and lack a response to exceptional situations.

[0005] Furthermore, the driving coaching results provided by existing driving habit analysis systems are presented in a simple text-based format, resulting in poor readability. Additionally, since they are provided in a fixed format rather than an efficient one for the driver, there is a problem with reduced effectiveness of the feedback.

[0006] The purpose of the present disclosure may be to provide personalized driving feedback using context and a generative artificial intelligence (AI) model.

[0007] The purpose of the present disclosure may be to provide driving feedback by reflecting various driving situations and exceptions.

[0008] The purpose of the present disclosure may be to provide driving feedback in a multimodal form, thereby providing driving feedback intuitively.

[0009] An apparatus according to one embodiment of the present disclosure may include an output interface; and one or more processors that generate context information using at least one of sensing information, driver information, or driving information, extract a cause description indicating the cause of a vehicle event from the generated context information, generate a prompt based on the extracted cause description, obtain driving feedback from the generated prompt through an artificial intelligence model, and output the obtained driving feedback through the output interface.

[0010] A method of operation of a device according to one embodiment of the present disclosure may include: generating context information using at least one of sensing information, driver information, or driving information; extracting a cause description indicating the cause of a vehicle event from the generated context information; generating a prompt based on the extracted cause description; obtaining driving feedback from the generated prompt through an artificial intelligence model; and outputting the obtained driving feedback.

[0011] A computer-readable non-volatile recording medium storing a program for performing a method of operation of a device according to one embodiment of the present disclosure, wherein the method of operation may include: generating context information using at least one of sensing information, driver information, or driving information; extracting a cause description indicating the cause of a vehicle event from the generated context information; generating a prompt based on the extracted cause description; obtaining driving feedback from the generated prompt through an artificial intelligence model; and outputting the obtained driving feedback.

[0012] According to various embodiments of the present disclosure, personalized feedback (regarding individual driving habits and external conditions) can be provided by utilizing driving history, driver information, current conditions, etc., rather than a simple rule-based method.

[0013] According to various embodiments of the present disclosure, feedback can be provided even for various driving situations and exceptions that are not predefined, so the accuracy of reflecting the actual driving environment can be improved.

[0014] According to various embodiments of the present disclosure, new driving situations or rules can be easily added, making system updates easy.

[0015] According to various embodiments of the present disclosure, the provision of an automated coaching system can facilitate efficient driver management for FMS companies.

[0016] According to various embodiments of the present disclosure, driving feedback is provided in a multimodal form, allowing the driver to intuitively understand the driving results.

[0017] Figure 1 is a drawing illustrating an example of the exterior and interior of a vehicle.

[0018] Figure 2 is a diagram illustrating the architecture of a signal processing system for vehicles.

[0019] FIG. 3a is a drawing illustrating an example of the arrangement of a vehicle display device inside a vehicle.

[0020] FIG. 3b is a drawing illustrating another example of the arrangement of a vehicle display device inside a vehicle.

[0021] Figure 4 is an example of an internal block diagram of the vehicle of Figure 1.

[0022] FIG. 5 is a block diagram illustrating the configuration of a server according to one embodiment of the present disclosure.

[0023] FIGS. 6a to 6c are drawings for explaining the configuration of a system according to an embodiment of the present disclosure.

[0024] FIG. 7 is a flowchart illustrating a method of operation of a device according to one embodiment of the present disclosure.

[0025] FIGS. 8A and FIGS. 8B are drawings illustrating the cause of a vehicle event and a cause description explaining the cause according to an embodiment of the present disclosure.

[0026] FIGS. 9a and 9b are drawings illustrating the process of obtaining driving feedback from a prompt through an AI model according to one embodiment of the present disclosure.

[0027] FIGS. 10a to 10c are drawings showing examples of driving feedback according to a prompt according to various embodiments of the present disclosure.

[0028] FIGS. 11a and FIGS. 11b are drawings illustrating an example of updating a prompt based on the evaluation result of driving feedback obtained according to an embodiment of the present disclosure, and outputting new driving feedback from the updated prompt.

[0029] FIGS. 12a and FIGS. 12b are drawings illustrating an example of generating a coaching video based on a driver's driving habits according to an embodiment of the present disclosure.

[0030] FIGS. 13a and FIGS. 13b are drawings illustrating an example of providing a driving-related survey based on a driver's driving habits according to an embodiment of the present disclosure.

[0031] FIGS. 14a and FIGS. 14b are drawings illustrating an example of providing driving feedback to a driver based on the driving habit history of a driver group according to an embodiment of the present disclosure.

[0032] FIG. 15 shows an example of providing a platform in which a driver can participate in a game format according to an embodiment of the present disclosure.

[0033] The present disclosure will be described in more detail below with reference to the drawings.

[0034] The suffixes "module" and "part" for components used in the following description are assigned solely for the ease of drafting this specification and do not inherently confer any particularly significant meaning or role. Accordingly, the terms "module" and "part" may be used interchangeably.

[0035] Figure 1 is a drawing illustrating an example of the exterior and interior of a vehicle.

[0036] Referring to the drawing, the vehicle (200) is operated by a plurality of wheels (103FR, 103FL, 103RL,...) that rotate by a power source, and a steering wheel (150) for controlling the direction of travel of the vehicle (200).

[0037] Meanwhile, the vehicle (200) may further be equipped with a camera (195), etc., for acquiring an image of the front of the vehicle.

[0038] Meanwhile, the vehicle (200) may be equipped with a plurality of displays (180a, 180b) for displaying images, information, etc. inside.

[0039] In FIG. 1, a cluster display (180a) and an AVN (Audio Video Navigation) display (180b) are exemplified as multiple displays (180a, 180b). Other displays such as a HUD (Head Up Display) are also possible.

[0040] Meanwhile, the AVN (Audio Video Navigation) display (180b) may also be named the Center Information Display.

[0041] Meanwhile, the vehicle (200) described in this specification may be a concept that includes all of the following: a vehicle equipped with an engine as a power source, a hybrid vehicle equipped with an engine and an electric motor as a power source, an electric vehicle equipped with an electric motor as a power source, etc.

[0042] Figure 2 is a diagram illustrating the architecture of a signal processing system for vehicles.

[0043] Referring to the drawing, the architecture (300a) of the vehicle signal processing system can correspond to a zone-based architecture.

[0044] Accordingly, sensor devices and processors inside the vehicle may be placed in each of the multiple zones (Z1 to Z4), and a signal processing device (170a) including a vehicle communication gateway (GWDa) may be placed in the central area of ​​the multiple zones (Z1 to Z4).

[0045] Meanwhile, the signal processing device (170a) may additionally include an autonomous driving control module (ACC), a cockpit control module (CPG), etc., in addition to the vehicle communication gateway (GWDa).

[0046] The vehicle communication gateway (GWDa) within the signal processing device (170a) may be a High Performance Computing (HPC) gateway.

[0047] That is, the signal processing device (170a) of FIG. 2 is an integrated HPC and can exchange data with an external communication module (not shown) or a processor (not shown) in a plurality of zones (Z1 to Z4).

[0048] FIG. 3a is a drawing illustrating an example of the arrangement of a vehicle display device inside a vehicle.

[0049] Referring to the drawing, the vehicle interior may be equipped with a cluster display (180a), an AVN (Audio Video Navigation) display (180b), a rear seat entertainment display (180c, 180d), a rearview mirror display (not shown), etc.

[0050] FIG. 3b is a drawing illustrating another example of the arrangement of a vehicle display device inside a vehicle.

[0051] A vehicle display device (100) according to an embodiment of the present disclosure may include a plurality of displays (180a to 180b) and a signal processing device (170) that performs signal processing for displaying images, information, etc. on the plurality of displays (180a to 180b) and outputs an image signal to at least one display (180a to 180b).

[0052] Among the plurality of displays (180a to 180b), the first display (180a) is a cluster display (180a) for displaying driving status, operation information, etc., and the second display (180b) may be an AVN (Audio Video Navigation) display (180b) for displaying vehicle operation information, navigation map, various entertainment information or video.

[0053] The signal processing device (170) has a processor (175) inside and can execute a first virtualization machine to a third virtualization machine (not shown) on a hypervisor (not shown) within the processor (175).

[0054] A second virtualization machine (not shown) operates for the first display (180a), and a third virtualization machine (not shown) can operate for the second display (180b).

[0055] Meanwhile, the first virtualization machine (not shown) within the processor (175) can be controlled to set up a shared memory (508) based on a hypervisor (505) for the same data transmission to the second virtualization machine (not shown) and the third virtualization machine (not shown). Accordingly, the same information or the same image can be synchronized and displayed on the first display (180a) and the second display (180b) within the vehicle.

[0056] Meanwhile, the first virtualization machine (not shown) within the processor (175) shares at least a portion of the data with the second virtualization machine (not shown) and the third virtualization machine (not shown) for data sharing processing. Accordingly, data can be shared and processed by multiple virtualization machines for multiple displays within the vehicle.

[0057] Meanwhile, the first virtualization machine (not shown) within the processor (175) can receive and process wheel speed sensor data of the vehicle and transmit the processed wheel speed sensor data to at least one of the second virtualization machine (not shown) or the third virtualization machine (not shown). Accordingly, the wheel speed sensor data of the vehicle can be shared with at least one virtualization machine, etc.

[0058] Meanwhile, the vehicle display device (100) according to the embodiment of the present disclosure may further include a rear seat entertainment display (180c) for displaying driving status information, simple navigation information, various entertainment information or images.

[0059] The signal processing device (170) can control the RSE display (180c) by running a fourth virtualization machine (not shown) in addition to the first to third virtualization machines (not shown) on a hypervisor (not shown) within the processor (175).

[0060] Accordingly, various displays (180a to 180c) can be controlled using a single signal processing device (170).

[0061] Meanwhile, some of the multiple displays (180a to 180c) operate under a Linux OS, and others can operate under a Web OS.

[0062] A signal processing device (170) according to an embodiment of the present disclosure can control displays (180a to 180c) operating under various operating systems (OS) to synchronize and display the same information or the same image.

[0063] Meanwhile, FIG. 3b illustrates that a vehicle speed indicator (212a) and a vehicle interior temperature indicator (213a) are displayed on a first display (180a), a home screen (222) including a plurality of applications, a vehicle speed indicator (212b), and a vehicle interior temperature indicator (213b) is displayed on a second display (180b), and a second home screen (222b) including a plurality of applications and a vehicle interior temperature indicator (213c) is displayed on a third display (180c).

[0064] Figure 4 is an example of an internal block diagram of the vehicle of Figure 1.

[0065] Referring to the drawings, a vehicle (200) according to an embodiment of the present disclosure may be equipped with a lamp drive unit (751), a steering drive unit (752), a brake drive unit (753), a power source drive unit (754), a suspension drive unit (756), an air conditioning drive unit (757), a window drive unit (758), a seat drive unit (761), and a signal processing device (170).

[0066] Meanwhile, the vehicle (200) may further be equipped with an ECU (770), a plurality of sensor devices (SN), and a plurality of communication modules (EMa~EMd).

[0067] Meanwhile, the vehicle (200) according to the embodiment of the present disclosure may further be equipped with a vehicle display device (100).

[0068] A vehicle display device (100) according to an embodiment of the present disclosure may include an input unit (110), a communication device (120) for communication with an external device, a plurality of communication modules (EMa~EMd) for internal communication, a memory (140), a signal processing device (170), a plurality of displays (180a~180c), an audio output unit (185), and a power supply unit (190).

[0069] Multiple communication modules (EMa~EMd) can be placed in each of the multiple zones (Z1~Z4) of FIG. 2, for example.

[0070] Meanwhile, the signal processing device (170) may have a communication switch (736b) inside for data communication with each communication module (EM1~EM4).

[0071] Each communication module (EM1~EM4) can perform data communication with a plurality of sensor devices (SN), ECU (770), or area signal processing device (170Z).

[0072] Meanwhile, a plurality of sensor devices (SN) may include a camera (195), lidar (196), radar (197), or position sensor (198).

[0073] The input unit (110) may be equipped with physical buttons, pads, etc. for button input, touch input, etc.

[0074] Meanwhile, the input unit (110) may be equipped with a microphone (not shown) for user voice input.

[0075] The communication device (120) can exchange data wirelessly with a mobile terminal (800) or a server (900).

[0076] In particular, the communication device (120) can wirelessly exchange data with the vehicle driver's mobile terminal. Various data communication methods are possible as wireless data communication methods, such as Bluetooth, WiFi, WiFi Direct, and APiX.

[0077] The communication device (120) can receive weather information, road traffic condition information, for example, TPEG (Transport Protocol Expert Group) information from a mobile terminal (800) or a server (900). To this end, the communication device (120) may be equipped with a mobile communication module (not shown).

[0078] Meanwhile, the communication device (120) can exchange data wirelessly with an adjacent vehicle.

[0079] For example, the communication device (120) can exchange vehicle messages wirelessly with an adjacent vehicle through V2X (Vehicle-to-everything) communication.

[0080] A plurality of communication modules (EM1~EM4) can receive sensor data, etc. from an ECU (770), a sensor device (SN), or a region signal processing device (170Z), and transmit the received sensor data to the signal processing device (170).

[0081] Here, the sensor data may include at least one of vehicle direction data, vehicle location data (GPS data), vehicle angle data, vehicle speed data, vehicle acceleration data, vehicle tilt data, vehicle forward / reverse data, battery data, fuel data, tire data, vehicle lamp data, vehicle interior temperature data, and vehicle interior humidity data.

[0082] Such sensor data can be obtained from a heading sensor, a yaw sensor, a gyro sensor, a position module, a vehicle forward / reverse sensor, a wheel sensor, a vehicle speed sensor, a vehicle body inclination sensor, a battery sensor, a fuel sensor, a tire sensor, a steering sensor based on steering wheel rotation, a vehicle interior temperature sensor, a vehicle interior humidity sensor, etc.

[0083] Meanwhile, the position module may include a GPS module or a position sensor (198) for receiving GPS information.

[0084] Meanwhile, at least one of the multiple communication modules (EM1 to EM4) can transmit location information data sensed from a GPS module or a location sensor (198) to a signal processing device (170).

[0085] Meanwhile, at least one of the plurality of communication modules (EM1 to EM4) can receive vehicle front image data, vehicle side image data, vehicle rear image data, and obstacle distance information around the vehicle from a camera (195), lidar (196), radar (197), etc., and transmit the received information to a signal processing device (170).

[0086] The memory (140) can store various data for the overall operation of the vehicle display device (100), such as a program for processing or controlling the signal processing device (170).

[0087] For example, memory (140) can store data regarding a hypervisor, a first virtualization machine to a third virtualization machine, for execution within a processor (175).

[0088] The audio output unit (185) converts an electrical signal from the signal processing device (170) into an audio signal and outputs it. To do this, a speaker or the like may be provided.

[0089] The power supply unit (190) can supply power necessary for the operation of each component under the control of the signal processing unit (170). In particular, the power supply unit (190) can receive power from a battery inside the vehicle, etc.

[0090] The signal processing device (170) controls the overall operation of each unit within the vehicle display device (100) or vehicle (200).

[0091] For example, the signal processing device (170) may include a processor (175) that performs signal processing for a vehicle display (180a, 180b).

[0092] The processor (175) can run a first virtualization machine to a third virtualization machine (not shown) on a hypervisor (not shown) within the processor (175).

[0093] Among the first to third virtual machines (not shown), the first virtual machine (not shown) may be named a Server Virtual Machine, and the second to third virtual machines (not shown) may be named a Guest Virtual Machine.

[0094] For example, a first virtualization machine (not shown) within a processor (175) can receive sensor data from a plurality of sensor devices, such as vehicle sensor data, location information data, camera image data, audio data, or touch input data, and process or modify it to output it.

[0095] In this way, by performing most of the data processing in the first virtualization machine (not shown), 1:N data sharing becomes possible.

[0096] As another example, the first virtualization machine (not shown) can directly receive and process CAN data, Ethernet data, audio data, radio data, USB data, and wireless communication data for the second virtualization machine to the third virtualization machine (not shown).

[0097] And, the first virtualization machine (not shown) can transmit the processed data to the second virtualization machine to the third virtualization machine (not shown).

[0098] Accordingly, among the first to third virtualization machines (not shown), only the first virtualization machine (not shown) receives sensor data, communication data, or external input data from a plurality of sensor devices and performs signal processing, thereby reducing the signal processing burden on other virtualization machines and enabling 1:N data communication, which enables synchronization when sharing data.

[0099] Meanwhile, the first virtualization machine (not shown) can control the sharing of the same data with the second virtualization machine (not shown) and the third virtualization machine (not shown) by writing data to the shared memory (508).

[0100] For example, the first virtualization machine (not shown) can record vehicle sensor data, the location information data, the camera image data, or the touch input data in a shared memory (508) and control the sharing of the same data with the second virtualization machine (not shown) and the third virtualization machine (not shown). Accordingly, data sharing in a 1:N manner becomes possible.

[0101] Ultimately, by performing most of the data processing on the first virtualization machine (not shown), 1:N data sharing becomes possible.

[0102] Meanwhile, the first virtualization machine (not shown) within the processor (175) can control the second virtualization machine (not shown) and the third virtualization machine (not shown) to set up a shared memory (508) based on the hypervisor (505) for the same data transmission.

[0103] Meanwhile, the signal processing device (170) can process various signals such as audio signals, video signals, and data signals. To this end, the signal processing device (170) can be implemented in the form of a System On Chip (SOC).

[0104] FIG. 5 is a block diagram illustrating the configuration of a server according to one embodiment of the present disclosure.

[0105] Referring to FIG. 5, the server (900) may refer to a device that trains an artificial neural network using a machine learning algorithm or uses a trained artificial neural network. Here, the server (900) may be composed of multiple servers to perform distributed processing and may be defined as a 5G network.

[0106] The server (900) may be included as part of the vehicle (200) and may perform at least some of the AI ​​processing together.

[0107] The server (900) may include a communication interface (510), memory (530), a learning processor (540), and a processor (560).

[0108] The communication interface (510) can transmit and receive data with the vehicle (200) or an external device.

[0109] The memory (530) may include a model storage unit (531). The model storage unit (531) may store a model (or artificial neural network, 531a) that is being learned or has been learned through the learning processor (540).

[0110] The learning processor (540) can train the artificial neural network (531a) using training data. The training model may be used while mounted on the server (900) of the artificial neural network, or it may be used while mounted on an external device such as a vehicle (200).

[0111] The learning model may be implemented in hardware, software, or a combination of hardware and software. If part or all of the learning model is implemented in software, one or more instructions constituting the learning model may be stored in memory (530).

[0112] The processor (560) can use a learning model to infer a result value for new input data and generate a response or control command based on the inferred result value.

[0113] FIGS. 6a to 6c are drawings for explaining the configuration of a system according to an embodiment of the present disclosure.

[0114] The system (6000) of Fig. 6a can be referred to as an IVEX (In Vehicle Experience) system.

[0115] Referring to FIG. 6a, the system (6000) may include an edge sensor group (6100), a recognizer (6200), an insightor (6300), and an illustrator (6400).

[0116] The edge sensor group (6100) may be included in the plurality of sensor devices (SN) of FIG. 2.

[0117] The recognizer (6200) and the insightor (6300) may be included in the signal processing device (170) of the vehicle display device (100) of FIG. 4. In particular, the recognizer (6200) and the insightor (6300) may be included in the processor (175) of the signal processing device (170).

[0118] In another embodiment, the recognizer (6200) and the insightor (6300) may be included in the cockpit control module (CPG) of FIG. 2.

[0119] An illustrator (6400) may be included in the vehicle display device (100) of FIG. 2 and FIG. 4.

[0120] The recognizer (6200) can recognize the vehicle situation based on a set of sensing data received from the edge sensor group (6100). The edge sensor group (6100) may include vehicle internal sensors (6110) and external sensors (6120).

[0121] The vehicle interior sensors (6110) are sensors placed inside the vehicle (200) and may include a front camera, lidar, radar, interior camera, vehicle interior temperature sensor, and vehicle interior humidity sensor.

[0122] The external sensors (6120) are sensors placed on the exterior of the vehicle (200) and may include an external camera, lidar, radar, heading sensor, yaw sensor, gyro sensor, position module, vehicle forward / reverse sensor, wheel sensor, vehicle speed sensor, vehicle body inclination sensor, battery sensor, fuel sensor, tire sensor, steering sensor based on steering wheel rotation, etc. The position module may include a GPS module or a position sensor (198) for receiving GPS information.

[0123] The sensing data set may include an internal vehicle data set and an external vehicle data set.

[0124] The vehicle interior data set may be a data set sensed by the vehicle interior sensors (6110). The vehicle interior data set may include at least one of vehicle interior temperature data, vehicle interior humidity data, battery data, fuel data, vehicle lamp data, tire data or vehicle interior image data, and audio data received through a microphone.

[0125] The vehicle external data set may be a data set sensed by external sensors (6120). The vehicle external data set may include at least one of vehicle location data (GPS), vehicle direction data, vehicle angle data, vehicle speed data, vehicle acceleration data, vehicle tilt data, whether the vehicle is moving forward or backward, front image data, or rear image data.

[0126] The recognizer (6200) can perform preprocessing and calibration on the sensing data set and can obtain vehicle internal information (6210) and vehicle external information (6220) based on the preprocessed and calibrated sensing data set.

[0127] The vehicle interior information (6210) may include a vehicle interior data set or information about the vehicle interior situation obtained from the vehicle interior data set.

[0128] The vehicle external information (6220) may include a vehicle external data set or information about the vehicle external situation obtained from the vehicle external data set.

[0129] The recognizer (6200) can recognize a vehicle situation based on at least one of vehicle internal information (6210) or vehicle external information (6220), and can generate vehicle situation information for the recognized vehicle situation.

[0130] Vehicle situation information may include at least one of vehicle external situation information indicating a situation regarding the exterior of the vehicle (200), vehicle internal situation information indicating a situation regarding the interior of the vehicle (200), or user situation information indicating a situation regarding a user inside the vehicle (200). User situation information may be included in the vehicle internal situation information.

[0131] User situation information may include information about the user's face identifier (Face ID), distraction, gaze, position, gesture, emotional state, whether they are drinking or taking drugs, drowsiness, and other actions inside the vehicle (200).

[0132] The recognizer (6200) can transmit the generated vehicle situation information to the insightor (6300).

[0133] The insightor (6300) can generate a response result based on vehicle situation information received from the recognizer (6200).

[0134] The insightor (6300) may include a multimodal context engine (6310), a context controller (6320), a safety assistance engine (6330), a user context engine (6340), an AI orchestrator (6350), and an AI model set (6360).

[0135] The multimodal context engine (6310) can generate multimodal context data based on vehicle situation information received from the recognizer (6200). The multimodal context data may include at least one of text data or image data describing the vehicle situation generated based on the vehicle situation information.

[0136] The context controller (6320) may be a component included in the multimodal context engine (6310) or provided separately from the multimodal context engine (6310).

[0137] When the context controller (6320) receives a user query from the AI ​​orchestrator (6350), it can execute a process of generating multimodal context data. The user query may be a speech recognition result corresponding to a voice command spoken by the user.

[0138] The safety assist engine (6330) can determine whether the current situation is a safe situation or a dangerous situation based on vehicle situation information received from the recognizer (6200).

[0139] If the safety assist engine (6330) determines that the current situation is a dangerous situation, it can transmit driver assistance control commands and warning notification output commands to the safety application (6440) of the illustrator (6400).

[0140] The user context engine (6340) can generate user context data based on user information. User information may be referred to as a user persona. User information may include at least one of the user's nationality, age, gender, occupation, personality, or psychological type (Myers-Briggs Type Indicator, MBTI).

[0141] The AI ​​orchestrator (6350) can generate a prompt based on at least one of multimodal context data or user context data. The AI ​​orchestrator (6350) can transmit the generated prompt to at least one of a plurality of AI models included in the AI ​​model set (6360).

[0142] The AI ​​model set (6360) may include multiple AI models. Each AI model may output an inference result in response to a prompt received from the AI ​​orchestrator (6350) and may transmit the inference result to the AI ​​orchestrator (6350).

[0143] The AI ​​orchestrator (6350) can generate additional prompts based on inference results and can transmit the additional prompts to at least one AI model in the set of AI models (6360). The AI ​​model that receives the additional prompts can output additional inference results in response to the additional prompts and can transmit the additional inference results to the AI ​​orchestrator (6350).

[0144] The AI ​​orchestrator (6350) can obtain an inference result or additional inference result received from the AI ​​model as a response result, and can output the obtained response result to an illustrator (6400).

[0145] The illustrator (6400) can output service information or perform driving control functions based on the response result output from the insightor (6300).

[0146] The illustrator (6400) may include a multimodal output encoder (6410), a visual interface (6420), an audio interface (6430), and a safety application (6440).

[0147] The multimodal output encoder (6410) can encode the response result output from the insightor (6300) and output the encoded response result data to the visual interface (6420) or audio interface (6430).

[0148] The visual interface (6420) or audio interface (6430) may be referred to as an output interface.

[0149] The visual interface (6420) can display response result data output from the multimodal output encoder (6410) or an image based on the response result data. The visual interface (6420) may include at least one of the plurality of displays (180a to 180c) of FIG. 4.

[0150] The visual interface (6420) can display an Augmented Reality (AR) image based on response result data or a Mixed Reality (MR) image based on response result data.

[0151] The audio interface (6430) can output response result data in the form of audio. The audio interface (6430) may be included in the audio output unit (185) of FIG. 4.

[0152] The safety application (6440) can perform Advanced Driver Assistance System (ADAS) control according to the response result received from the driver assistance control command or the insightor (6300), and can output a warning notification according to the warning notification output command.

[0153] The safety application (6440) may be included in the electronic control unit (770) of FIG. 4 or executed by the electronic control unit (770).

[0154] FIG. 6b is a drawing for explaining the configuration of an insightor according to another embodiment of the present disclosure.

[0155] In FIG. 6b, the multimodal context engine (6310) may include a context controller (6320).

[0156] Referring to FIG. 6b, the insightor (6300) may include a multimodal context engine (6310), an AI orchestrator (6350), and a multimodal LLM (6361).

[0157] The multimodal context engine (6310) may include a multimodal signal adapter (6311), a multimodal indexer (6312), a multimodal context buffer (6313), a multimodal context retriever (6314), a multimodal context descriptor (6315), a multimodal event monitor (6316), and a context controller (6320).

[0158] The multimodal signal adapter (6311) can generate pre-processed multimodal data by filtering, cleaning, synchronizing, and reformulating data received from various sensors or vehicle situation information received from a recognizer (6200).

[0159] At least one of the following can be input to the multimodal signal adapter (6311): audio data received through a microphone, front image data captured through a front camera, ADAS information obtained from the front image data or vehicle sensors, location data, IVI (In-Vehicle Infotainment) system data, IVI display information, DMS (Driver Monitoring System) information / IMS (Interior Monitoring System) information based on image data captured through an interior camera, and biometric data obtained from a biometric sensor.

[0160] The multimodal indexer (6312) can generate multimodal processing data by dividing preprocessed multimodal data into chunks and can index the multimodal processing data. The multimodal indexer (6312) can obtain an encoding vector or keyword representing the attribute (or meaning) of the multimodal processing data divided into chunks as an index.

[0161] The multimodal context buffer (6313) can store multimodal processing data and an index corresponding to the multimodal processing data.

[0162] The multimodal context retriever (6314) can search for multimodal processing data most relevant to the user query through an index from the multimodal context buffer (6313) and can select the searched multimodal processing data as a context candidate for the creation of multimodal context data.

[0163] The multimodal context descriptor (6315) can reconfigure multimodal processing data selected as a context candidate into multimodal context data having a prompt form that can be interpreted by the multimodal LLM (6361).

[0164] The multimodal event monitor (6316) can monitor whether an index matching the trigger condition is entered when a trigger condition for multimodal data is registered. If an index matching the trigger condition is entered, the multimodal event monitor (6316) can generate a trigger event to generate a proactive service query.

[0165] The context controller (6320) can control the overall operation of the multimodal context engine (6310). When the context controller (6320) receives a user query from the AI ​​orchestrator (6350), it can execute the process of generating multimodal context data. When the context controller (6320) receives a trigger event from the multimodal event monitor (6316), it can generate a preemptive service query.

[0166] The AI ​​orchestrator (6350) can generate a prompt by combining a user query and a multimodal context, and can transmit the generated prompt to a multimodal LLM (6361). The user query may be a query in the form of recognized text based on a voice command spoken by the user. The voice command may be received through a microphone, and the voice command may be converted into text through an Automatic Speech Recognition (ASR) process. The user query may be text converted through an Automatic Speech Recognition (ASR) process.

[0167] The multimodal LLM (6361) may be an example of an AI model included in the set of AI models (6360) of FIG. 6a. The multimodal LLM (6361) may output an inference result from a prompt received from the AI ​​orchestrator (6350) and may transmit the inference result to the AI ​​orchestrator (6350).

[0168] The inference result may include a result representing a response to the prompt.

[0169] For example, the inference result may include an API call result regarding whether an API call corresponding to a driving assistance function was successfully performed, and a feedback generation result regarding whether feedback corresponding to the API call was successfully generated.

[0170] FIG. 6c is a drawing for explaining the configuration of an AI orchestrator according to one embodiment of the present disclosure.

[0171] Referring to FIG. 6c, the AI ​​orchestrator (6350) may include a task arbitrator (6351), a prompt manager (6352), a sub-agent set (6353), a knowledge DB (6354), a workflow controller (6355), and a tool set / adapter set (6356).

[0172] The task arbitrator (6351) can determine one of the multiple sub-agents based on the voice recognition result and the multimodal context data output from the multimodal context engine (6310).

[0173] The prompt manager (6352) can generate a prompt for the operation of a determined sub-agent. The prompt manager (6352) can generate a prompt based on multimodal context data and user queries. The prompt manager (6352) can generate a prompt based on information about the functions that the determined sub-agent can perform, multimodal context data, the results of previously performed functions, conversation history, and user queries.

[0174] The sub-agent set (6353) may include multiple sub-agents. For example, the sub-agent set (6353) may include a navigation agent for navigation services, a vehicle function agent for providing vehicle functions, a telephony agent for automated telephone answering services, and a Q&A agent for providing response services to queries.

[0175] A sub-agent determined by the task arbitrator (6351) may call a cloud AI model or an on-device AI model assigned to it. The cloud model or on-device model may be a Large Language Model (LLM).

[0176] A sub-agent can call a cloud AI model or an on-device AI model assigned to it to obtain inference results corresponding to a prompt from the model.

[0177] Based on the obtained inference result, the sub-agent can call the knowledge DB (6354) to provide additional information, obtain additional information from the knowledge DB (6354), specify the name of the function to be executed and the parameter value of the function, and determine the feedback text to be provided to the user.

[0178] The sub-agent can transmit to the workflow controller (6355) a parsing result generated by parsing the inference result received from the model, which includes additional information called from the knowledge DB (6354), the name of the function to be executed, the parameter values ​​of the function, and a feedback phrase.

[0179] The knowledge DB (6354) can store additional information and information about functions. The information about functions may include the name of the function and the parameter values ​​of the function.

[0180] The workflow controller (6355) can generate control commands to perform the corresponding function based on the parsing results and can store the results of the conversation and function performed by the AI ​​model and sub-agent.

[0181] The tool set / adapter set (6356) can call an API corresponding to a control command generated by the workflow controller (6355). The tool set / adapter set (6356) can convert the control command into an execution command of an actual function or an API call command that the IVI system can understand and execute, and can execute the converted command.

[0182] FIG. 7 is a flowchart illustrating a method of operation of a device according to one embodiment of the present disclosure.

[0183] Additionally, although the embodiment of FIG. 7 is described as being performed by the processor (175) of the vehicle (200), it may also be performed by the processor (560) of the cockpit control module (CPG) or the server (900). The processor (175) may be provided in multiple numbers.

[0184] Referring to FIG. 7, the processor (175) of the vehicle (200) can generate context information using at least one of sensing information, driver information, or driving information (S701).

[0185] In one embodiment, the sensing information may include a vehicle interior data set included in a sensing data set received from an edge sensor group (6100).

[0186] The vehicle interior data set may be a data set sensed by the vehicle interior sensors (6110). The vehicle interior data set may include at least one of vehicle interior temperature data, vehicle interior humidity data, battery data, fuel data, vehicle lamp data, tire data or vehicle interior image data, and audio data received through a microphone.

[0187] In one embodiment, driver information may include one or more of the driver's age, gender, driving tendency, or driving experience.

[0188] In one embodiment, the driving information may include information related to the driving of the vehicle (200). The driving information may include a vehicle external data set included in a sensing data set received from an edge sensor group (6100).

[0189] The vehicle external data set may be a data set sensed by external sensors (6120). The vehicle external data set may include at least one of vehicle location data (GPS), vehicle direction data, vehicle angle data, vehicle speed data, vehicle acceleration data, vehicle tilt data, whether the vehicle is moving forward or backward, front image data, or rear image data.

[0190] The recognizer (6200) included in the processor (175) can recognize the vehicle situation based on at least one of sensing information, driver information, or driving information and generate context information.

[0191] Context information may be information representing the vehicle situation. Context information may include one or more of vehicle events, contexts that form the basis for analyzing the driver's driving habits, or user actions.

[0192] A vehicle event may be any one of a harsh brake or hard braking event, a harsh accelerating event, or a crash event, but this is merely an example.

[0193] User actions can be any one of calling, smoking, or drowsiness, but this is merely an example.

[0194] The processor (175) of the vehicle (200) can extract a cause description explaining the cause of the vehicle event from the generated context information (S703).

[0195] A vehicle event may be an event related to the driving of a vehicle (200). A vehicle event may be referred to as a vehicle driving event.

[0196] The processor (175) can extract the cause of the vehicle event and a cause description explaining the cause from the context information.

[0197] In one embodiment, the cause description may include a detailed description supporting the cause of the vehicle event.

[0198] The cause of a vehicle event may be any one of distracted driving, reckless driving, reckless driving by others, an obstacle on the road, or no reason.

[0199] The cause description may be detailed information explaining why the cause of a vehicle event was derived through a time-series analysis of context information. The cause description may include one or more causal factors explaining the cause of the vehicle event. One or more causal factors may have time-series characteristics.

[0200] For example, a cause description explaining the cause of a vehicle event called distracted driving may include the user's distracted actions such as calling, smoking, and drowsiness.

[0201] As another example, the cause description explaining the cause of the vehicle event known as reckless driving may include causal factors such as tailgating and sudden braking of the vehicle ahead (failure to maintain a safe distance).

[0202] Cause descriptions can be generated on an image frame basis or at regular time intervals. To this end, context information can also be obtained on an image frame basis or at regular intervals.

[0203] FIGS. 8A and FIGS. 8B are drawings illustrating the cause of a vehicle event and a cause description explaining the cause according to an embodiment of the present disclosure.

[0204] Referring to Fig. 8a, the cause of the vehicle event is expressed as Cause of events, and the cause description is expressed as Detailed Time-Series Description.

[0205] The cause of each vehicle event may be matched with a cause description that includes multiple causal elements.

[0206] For example, cause descriptions corresponding to the cause of a vehicle event called Obstruction on the road may include Pdestrain, Animal, Roadblock, Emergency Vehicles, and Parked Vehicles.

[0207] Referring to Fig. 8b, examples of cause descriptions corresponding to distracted driving and reckless driving, which are causes of the vehicle event called hard braking, are shown.

[0208] FIG. 8b is an example of a time-series cause description. The cause description in FIG. 8b describes a case where the driver made an emergency stop after seeing a red traffic light ahead, immediately after the vehicle ahead cut out, while the driver was on the phone from 3 seconds to 14 seconds due to failure to maintain a safe distance from the vehicle ahead.

[0209] That is, the cause description may include call actions, the vehicle ahead cutting out, ego-tailgating actions, and red traffic lights. The cause description may include causal elements from the driver's perspective (call actions) and causal elements from the road's perspective (cutting out, ego-tailgating actions, and red traffic lights).

[0210] The cause description of FIG. 8b may include the results of a time-series analysis of the cause of the vehicle event. Each of the cause elements may be an element that exists within a certain time period before and after the time the vehicle event occurred.

[0211] Again, Figure 7 is explained.

[0212] The processor (175) of the vehicle (200) can generate a prompt based on the extracted cause description (S705).

[0213] The processor (175) can generate a prompt through the AI ​​orchestrator (6350).

[0214] In one embodiment, the processor (175) can generate a prompt using the cause of the vehicle event and the extracted cause description.

[0215] In another embodiment, the processor (175) can generate a prompt using a vehicle event, the cause of the vehicle event, and an extracted cause description.

[0216] In another embodiment, the processor (175) may generate a prompt using the cause of the vehicle event, the extracted cause description, driver information, driving environment information, basic rules, and additional rules.

[0217] Driver information may include at least one of the driver's age, gender, driving experience, or driving tendencies.

[0218] Driving environment information may include at least one of sensing information or environmental data (weather, traffic conditions) related to the current driving of the vehicle.

[0219] The basic rule may be a rule generated based on driver information and driving information. The driving information may include one or more of the front image and the vehicle interior image.

[0220] For example, the basic rule may include a basic prompt such as <The driver is in their 00s, gender is 00, and driving experience is 00 years. Please generate feedback on this driver's driving data by referring to the original video and time analysis results>.

[0221] Additional rules can include content added beyond the basic prompts to generate granular and personalized driving feedback.

[0222] Additional rules may include at least one of analysis criteria, feedback type, feedback history, driver-tailored coaching, driver's secondary feedback, or tone.

[0223] The analysis criteria may be standards that are provided intensively in the driving feedback. For example, the analysis criteria may include at least one of compliance with legal speed limits, minimization of sudden acceleration / braking, or fuel efficiency optimization.

[0224] The feedback type is a type of driving feedback and may include at least one of a positive / negative feedback type, a feedback type for areas needing improvement, and a visual feedback type.

[0225] The feedback history may include the key points and dates of previously provided driving feedback.

[0226] Driver-customized coaching may include one or more driver-customized messages or prior knowledge generated based on feedback history.

[0227] The driver's secondary feedback may include the driver's secondary feedback history regarding previously provided driving feedback.

[0228] The tone may be the tone of a person or character preferred by the driver. If information regarding the tone is included in the prompt, driving feedback may be output in voice with that tone. The tone may be a friendly tone, a cool tone, or a professional tone.

[0229] The processor (175) of the vehicle (200) can obtain driving feedback from the prompt through an AI model (S707).

[0230] The processor (175) can input a prompt into the AI ​​model to obtain driving feedback output from the AI ​​model. The AI ​​model may be a large language model (LLM).

[0231] In one embodiment, driving feedback is feedback provided to the driver regarding vehicle events and may include at least one of vehicle control signals, text, images, or videos.

[0232] For example, driving feedback may include one or more of the driver's correct driving points, text indicating areas for improvement related to the driver's driving, or educational videos for driving improvement.

[0233] The vehicle control signal is a control signal to assist in driving the vehicle (200), and may be a signal transmitted to the ECU (770).

[0234] The AI ​​orchestrator (6350) included in the processor (175) can send a prompt to any one of the AI ​​models in the set of AI models (6360) and receive driving feedback from the AI ​​model.

[0235] FIGS. 9a and 9b are drawings illustrating the process of obtaining driving feedback from a prompt through an AI model according to one embodiment of the present disclosure.

[0236] The AI ​​model (9000) may include at least one generative AI model among a video generation model (901) that generates video from a prompt, a text generation model (902) that generates text from a prompt, an image generation model (903) that generates an image from a prompt, and an audio generation model (904) that generates audio from a prompt.

[0237] In FIG. 9a, the first prompt (911) may be a prompt generated based on the cause of the vehicle event, the cause description, and the basic rule.

[0238] The processor (175) can input a first prompt (911) into the AI ​​model (9000) to obtain first driving feedback (921) for the first prompt (910).

[0239] In FIG. 9b, the second prompt (912) may be a prompt generated based on the cause of the vehicle event, the cause description, driver information, basic rules, and additional rules.

[0240] The processor (175) can input a second prompt (912) into the AI ​​model (9000) to obtain second driving feedback (922) for the second prompt (912).

[0241] In this way, the form of the prompt can be modified and generated to suit the goal. The prompt may have a predefined form, or the elements constituting the prompt may change through learning.

[0242] Again, Figure 7 is explained.

[0243] The processor (175) of the vehicle (200) can output the acquired driving feedback (S709).

[0244] In one embodiment, the processor (175) can output driving feedback through either a visual interface (6420) or an audio interface (6430).

[0245] In another embodiment, the processor (175) can transmit driving feedback to a mobile terminal (800) via a communication device (120). The mobile terminal (800) may be a terminal such as a smartphone or smart pad used by the driver.

[0246] In one embodiment, the processor (175) may output driving feedback at the time when the operation of the vehicle (200) ends. The time when the operation ends may be the time when the engine of the vehicle (200) is turned off. After the engine is turned off, the processor (175) may display driving feedback on any one of the first to third displays (180a, 180b, 180c).

[0247] After the engine is turned off, the processor (175) can output driving feedback including a video corresponding to the driver's correct driving point or a video of the driver's improvement needs.

[0248] In another embodiment, the processor (175) can output driving feedback when the vehicle stops after the occurrence of a vehicle event.

[0249] In another embodiment, the processor (175) can output driving feedback after a certain amount of time has elapsed following the occurrence of a vehicle event.

[0250] In another embodiment, the processor (175) can output driving feedback at a time set by the driver.

[0251] In one embodiment, driving feedback may be provided in different forms for each driver. For example, driving feedback containing only text may be provided to the first driver, and driving feedback containing only video may be provided to the second driver.

[0252] For example, the processor (175) can induce a sense of caution by including a serious accident video in the driving feedback if the driver has a tendency to speed.

[0253] FIGS. 10a to 10c are drawings showing examples of driving feedback according to a prompt according to various embodiments of the present disclosure.

[0254] In FIG. 10a and FIG. 10b, it is assumed that the vehicle event is hard braking.

[0255] Referring to FIG. 10a, the prompt (1000) may include a front image (1001), an interior image (1002) of a vehicle (200), a first cause description (1003), a second cause description (1004), and a first text prompt (1005).

[0256] The front image (1001) may be an image of the front of the vehicle (200) taken before and after the point in time when hard braking occurs.

[0257] The indoor video (1002) may be a video of the interior of the vehicle (200) taken before and after the point in time when the vehicle (200) suddenly brakes.

[0258] The first cause description (1003) may be a graph that records the occurrence of causative factors for sudden braking in a time series before and after the time when sudden braking occurred.

[0259] The second cause description (1004) may be a text file that records the cause elements extracted from the first cause description (1003) in text form.

[0260] The first text prompt (1005) may be text containing parts of basic rules and additional rules. Parts of additional rules may include analysis criteria and feedback types.

[0261] The processor (175) can input a prompt (1000) into the AI ​​model (9000) and obtain a first driving feedback (1010) from the AI ​​model (9000).

[0262] The first driving feedback (1010) may include text feedback (1011) and video feedbacks (1012, 1013).

[0263] Text feedback (1011) may include one or more of the causes of vehicle events, causative factors, driving improvements, or driving guides.

[0264] The video feedbacks (1012, 1013) may be educational videos related to driving improvements or driving guides.

[0265] As such, according to an embodiment of the present disclosure, personalized feedback for each driver can be provided by utilizing driving history, driver information, current situation, etc., rather than a simple rule-based method.

[0266] In addition, according to the embodiments of the present disclosure, feedback can be provided for various driving situations and exceptions that are not predefined, so the accuracy of reflecting the actual driving environment can be improved.

[0267] Referring to FIG. 10b, the prompt (1030) may include a front image (1001), an interior image (1002) of the vehicle (200), a first cause description (1003), a second cause description (1004), and a second text prompt (1020).

[0268] The front image (1001) may be an image of the front of the vehicle (200) taken before and after the point in time when hard braking occurs.

[0269] The indoor video (1002) may be a video of the interior of the vehicle (200) taken before and after the point in time when the vehicle (200) suddenly brakes.

[0270] The first cause description (1003) may be a graph that records the occurrence of causative factors for sudden braking in a time series before and after the time when sudden braking occurred.

[0271] The second cause description (1004) may be a text file that records the cause elements extracted from the first cause description (1003) in text form.

[0272] The second text prompt (1020) may include a first subtext (1021), a second subtext (1022), and a third subtext (1023).

[0273] The first subtext (1021) may be text containing parts of basic rules and additional rules. Parts of additional rules may include analysis criteria and feedback types.

[0274] The second subtext (1022) may be text containing feedback history.

[0275] The third subtext (1023) may be text containing a driver-customized coaching message.

[0276] The processor (175) can input a prompt (1030) into the AI ​​model (9000) to obtain second driving feedback (1040) from the AI ​​model (9000).

[0277] The second driving feedback (1040) may include text feedback (1041) and video feedbacks (1042, 1043).

[0278] Text feedback (1041) may include a first sub-feedback (1041a) containing one or more of the causes of vehicle events, causal factors, driving improvements, or driving guides, a second sub-feedback (1041b) indicating the driver's driving tendencies, and a third sub-feedback (1042c) for driver-customized coaching.

[0279] The first video feedback (1042) may be an educational video related to driving improvements or driving guides. The second video feedback (1043) may be an educational video related to driver-customized coaching messages.

[0280] As such, according to an embodiment of the present disclosure, personalized feedback for each driver can be provided using feedback history, driver-customized coaching messages, driver information, current driving conditions, etc., rather than a simple rule-based method.

[0281] In addition, according to the embodiments of the present disclosure, feedback can be provided for various driving situations and exceptions that are not predefined, so the accuracy of reflecting the actual driving environment can be improved.

[0282] Referring to FIG. 10c, driving feedback (1050) provided when a vehicle event of sudden stopping occurs is illustrated.

[0283] The first driving video (1051) may be a video showing the actual driving of a driver, including the occurrence of a vehicle event. The second driving video (1052) may be a video showing best practice driving, which is the correct driving method in that situation.

[0284] The driving feedback (1050) may include a first driving video (1051) of the actual driving by the driver in a situation where a vehicle event of sudden braking occurs, and a second driving video (1052) showing a situation where a safe distance from the vehicle in front is maintained.

[0285] The driver can easily determine how to drive in the future through the first driving video (1051) and the second driving video (1052).

[0286] The first driving video (1051) may be a video showing a driver's driving habit (failure to maintain a safe distance) generated based on driving feedback, and the second driving video (1052) may be a video showing an exemplary driving case corresponding to the driving habit.

[0287] The first driving video (1051) and the second driving video (1052) can be provided in various views.

[0288] FIGS. 11a and FIGS. 11b are drawings illustrating an example of updating a prompt based on the evaluation result of driving feedback obtained according to an embodiment of the present disclosure, and outputting new driving feedback from the updated prompt.

[0289] Referring to FIG. 11a, the steps of FIG. 7 can be performed after step S709.

[0290] The processor (175) can obtain a feedback evaluation result for driving feedback (S1101).

[0291] The feedback evaluation result may be the result of analyzing one or more driving feedbacks. One or more driving feedbacks may constitute a feedback history. The feedback evaluation result may include at least one of the driver's driving tendencies (or driving habits), the need for driving coaching, the method of providing driver feedback, or the result value of driving feedback performance.

[0292] The processor (175) can extract the driver's driving habits based on data included in the driving feedback. For example, the processor (175) can determine the driver's driving habits as aggressive driving tendencies if the number of sudden braking incidents included in the driving feedback is a certain number of times or more. As another example, the processor (175) can determine the driver's driving habits as speeding tendencies if the number of speeding incidents included in the driving feedback is a certain number of times or more.

[0293] Driving tendencies can be any one of the following: basic traffic law compliance / non-compliance tendency, safe driving tendency, or dangerous driving tendency.

[0294] The need for driving coaching can indicate whether driving coaching is required in specific driving situations.

[0295] The method of providing driving feedback may be a method of providing driving feedback through text, audio, video, or any one of these.

[0296] The result value of the driving feedback may be a value representing the result of the driver controlling the vehicle (200) according to the driving feedback. For example, if the driving feedback includes a guide regarding caution regarding sudden acceleration, the result value may be a value representing the reduction in the intensity of the accelerator pedal input during the next drive.

[0297] The processor (175) can generate feedback evaluation results at regular intervals. The regular interval may be one week, one month, or one year, but this is merely an example.

[0298] The processor (175) can update the prompt based on the acquired feedback evaluation result (S1103).

[0299] For example, if the driver's driving tendency is a tendency to not comply with basic laws, the processor (175) can update the prompt by generating additional educational video footage or increasing the real-time feedback rate.

[0300] As another example, the processor (175) can update the prompt to generate motion feedback including text and video in certain driving situations.

[0301] As another example, the processor (175) may include an element in the prompt that indicates that the performance of rapid acceleration is improved when the input strength of the accelerator pedal is reduced as a result of the performance of rapid acceleration.

[0302] The processor (175) can obtain updated driving feedback from an updated prompt through an AI model (9000) (S1105).

[0303] The processor (175) can obtain updated driving feedback by inputting an updated prompt into the AI ​​model (9000).

[0304] In another embodiment, the processor (175) may obtain updated driving feedback by fine-tuning the AI ​​model (9000) based on the results of the feedback evaluation instead of updating the prompt.

[0305] The processor (175) can output updated driving feedback (S1107).

[0306] The processor (175) can output updated driving feedback through either the visual interface (6420) or the audio interface (6430).

[0307] In this way, according to an embodiment of the present disclosure, the prompt is updated based on the evaluation result of the driving feedback, so that optimized driving feedback can be provided to the driver.

[0308] The steps of FIG. 11b and the steps of FIG. 7 can be performed after step S709. FIG. 11b is a diagram illustrating the process of generating a driving coaching prompt and outputting driving coaching feedback when it is determined from the feedback evaluation results that driving coaching is needed.

[0309] The processor (175) can obtain feedback evaluation results for driving feedback (S1111).

[0310] The feedback evaluation result may be the result of analyzing one or more driving feedbacks. One or more driving feedbacks may constitute a feedback history.

[0311] The feedback evaluation results may include at least one of the driver's driving tendency (or driving habit), the need for driving coaching, the method of providing driver feedback, or the result value of driving feedback performance.

[0312] Driving tendencies can be any one of the following: basic traffic law compliance / non-compliance tendency, safe driving tendency, or dangerous driving tendency.

[0313] The need for driving coaching can indicate whether driving coaching is required in specific driving situations.

[0314] The method of providing driving feedback may be a method of providing driving feedback through text, audio, video, or any one of these.

[0315] The result value of the driving feedback may be a value representing the result of the driver controlling the vehicle (200) according to the driving feedback. For example, if the driving feedback includes a guide regarding caution regarding sudden acceleration, the result value may be a value representing the reduction in the intensity of the accelerator pedal input during the next drive.

[0316] The processor (175) can generate feedback evaluation results at regular intervals. The regular interval may be one week, one month, or one year, but this is merely an example.

[0317] The processor (175) can determine whether driving coaching is necessary based on the acquired feedback evaluation results (S1113).

[0318] In one embodiment, the processor (175) may determine that driving coaching is required if, based on the analysis of the feedback history, it is determined that there have been 3 instances of unprotected left-turn traffic violations.

[0319] In another embodiment, the processor (175) may determine that driving coaching is required if a specific driving situation is detected, even if no vehicle event occurs, based on the analysis of the feedback history. For example, if driving at 120 km / h is detected on Highway A, it may determine that driving coaching is required for driving coaching feedback to ensure compliance with the regular speed limit of 100 km / h.

[0320] If the processor (175) determines that driving coaching is needed, it can generate a driver-customized coaching message (S1115).

[0321] If the processor (175) determines that driving coaching is needed, it can generate driver-customized coaching messages required in specific driving situations.

[0322] For example, the processor (175) can generate a driver-customized coaching message such as <needs guidance for unprotected left turn>.

[0323] The processor (175) can generate a driving coaching prompt based on the generated driver-customized coaching message (S1117).

[0324] The processor (175) can obtain basic traffic laws related to driver-customized coaching messages and can generate a prompt to guide the basic traffic laws. For example, the processor (175) can generate a driving coaching prompt that says "Guide me on traffic laws before unprotected left turn."

[0325] The processor (175) can obtain driving coaching feedback from a driving coaching prompt through an AI model (S1119) and can output the obtained driving coaching feedback (S1121).

[0326] Driving coaching feedback may include one or more of text, audio, or video that teaches driving methods when making unprotected left turns.

[0327] In this way, according to an embodiment of the present disclosure, feedback for driving coaching is provided in accordance with the driver's driving feedback, and driving education information optimized for the driver can be provided.

[0328] FIGS. 12a and FIGS. 12b are drawings illustrating an example of generating a coaching video based on a driver's driving habits according to an embodiment of the present disclosure.

[0329] In FIG. 12a, the driving habit data set (1210) may include the driving habit history of each driver belonging to a group of men in their 30s. The driving habit history may include information on best cases representing desirable driving practices in specific driving situations and information on worst cases representing bad driving practices.

[0330] The processor (175) can extract best practice text describing the best practice of a driver with correct driving habits and worst practice text describing the worst practice of a driver with incorrect driving habits from the driving habit data set (1210).

[0331] The processor (175) inputs the best case text and the worst case text, respectively, into the AI ​​model (9000) to obtain the best coaching video corresponding to the best case text and the worst coaching video corresponding to the worst case text. The best coaching video and the worst coaching video can be included in the driving feedback.

[0332] As such, according to an embodiment of the present disclosure, best practices from a group similar to the driver are selected based on feedback history, thereby providing more customized and empathetic driving feedback to the driver.

[0333] Referring to FIG. 12b, the driving habit data set (1220) may include the driving habit history of each driver included in the exemplary driver group.

[0334] The processor (175) can extract best practice text describing the best practice from the driving habit data set (1220).

[0335] The processor (175) can input best practice text and current driving information into the AI ​​model (9000) to obtain driving feedback including safe driving information. The current driving information may include the current driving route and the driver's driving habits.

[0336] For example, assume that the exit of Highway A is a section where frequent accidents occur because the distance from the previous highway entrance is short. The processor (175) can retrieve driving records of drivers who can safely exit Highway A from the driving habit data set (1220).

[0337] The processor (175) can input driving records retrieved from the AI ​​model (9000) and the driver's current driving information to obtain driving feedback including driving routes and speeds used by exemplary drivers.

[0338] FIGS. 13a and FIGS. 13b are drawings illustrating an example of providing a driving-related survey based on a driver's driving habits according to an embodiment of the present disclosure.

[0339] The steps of Fig. 13a can be performed after step S709 of Fig. 7.

[0340] Referring to [the reference], the processor (175) can extract the driver's driving habits based on driving feedback (1301).

[0341] The processor (175) can extract the driver's driving habits based on the feedback evaluation results representing the analysis results of the driving feedback. This is replaced by the description of step S1101.

[0342] The processor (175) can generate a driving-related survey based on extracted driving habits (S1303) and output the generated driving-related survey (S1305).

[0343] The processor (175) can obtain a driving-related survey by inputting a survey prompt requesting a survey on extracted driving habits into the AI ​​model (9000).

[0344] The processor (175) can display the acquired driving-related survey through the visual interface (6420).

[0345] Driving-related surveys may derive priorities for driving habits requiring improvement by considering past driving information and accident history, and may include survey / quiz items based on those priorities.

[0346] Referring to Fig. 13b, a driving-related survey (1300) is illustrated.

[0347] A driving-related survey (1300) may include a first survey item (1310) and a second survey item (1320). The first survey item (1310) may be an item generated when the driver's driving habit is using a smartphone while driving. The second survey item (1320) may be an item generated when the driver's driving habit is a tendency to speed.

[0348] The processor (175) can receive the driver's response to each survey item and can improve driving feedback based on the received response.

[0349] FIGS. 14a and FIGS. 14b are drawings illustrating an example of providing driving feedback to a driver based on the driving habit history of a driver group according to an embodiment of the present disclosure.

[0350] The processor (175) can generate multiple driving habit histories corresponding to each of the multiple driver groups (S1401).

[0351] Driver groups can be classified according to vehicle type, age, or family drivers of the same vehicle. Referring to FIG. 14b, driver group A shows multiple driving habit histories according to vehicle type, driver group B shows multiple driving habit histories according to age, and driver group C shows multiple driving habit histories of family drivers of the same vehicle.

[0352] Each driver group may include multiple subgroups according to type, and each subgroup may have a driving habit history.

[0353] Driving habit history may include the driving tendencies and driving history of subgroups included in each driver group.

[0354] The processor (175) can extract the driving habit history of a group of drivers corresponding to driver information driving the vehicle (200) (S1403).

[0355] Driver information may include the driver's age, vehicle type, and information on vehicles jointly driven by family members.

[0356] The processor (175) can compare driver information with each driver group and extract one or more driving habit histories based on the comparison result.

[0357] For example, the processor (175) can extract lane change accident history if the driver is in their 40s and the number of lane change accidents in the driving habit history of the group of drivers in their 40s is greater than a certain number.

[0358] The processor (175) can generate driving feedback from the driving habit history extracted through the AI ​​model (S1405) and can output the generated driving feedback (S1407).

[0359] The processor (175) can obtain driving feedback by inputting a prompt to prevent lane change accidents into the AI ​​model (9000). The driving feedback may include one or more of video or text for maintaining a safe distance.

[0360] The video may be a short video with a short playback time.

[0361] Driving feedback can be included in the driver's feedback history and stored in memory (140).

[0362] If the rate of vehicle change accidents is reduced due to the driving feedback provided through the video, the processor (175) may include the video in the driving feedback provided in the future.

[0363] FIG. 15 shows an example of providing a platform in which a driver can participate in a game format according to an embodiment of the present disclosure.

[0364] Referring to FIG. 15, the review achievement (1500) by team (or fleet) is illustrated. The review achievement (1500) may include the result of accumulating the number of reviews for each team (or fleet) corresponding to a specific organization.

[0365] A 1:1 coaching system can be established by utilizing Fleet bulletin boards, etc., to select exemplary drivers and teach them how to drive safely.

[0366] If your review of poor driving behavior is not completed within a certain period, it may be made public on the entire board or subject to penalties in other aspects.

[0367] The team with the most reviews or the team that generates the most exemplary driving cases may be eligible for insurance premium discounts.

[0368] In addition, grades are assigned to each driver based on the number of reviews and driving scores, which can encourage individual review completion.

[0369] The processor (175) can generate a team feedback history (1510) by combining driving feedbacks and feedback evaluation results of the team members. The team feedback history may include feedbacks intended to induce group driving behavior among the team members.

[0370] The functions of the elements disclosed in the present invention may be implemented using circuits or processing circuits comprising general-purpose processors, special-purpose processors, integrated circuits, ASICs (Application-Specific Integrated Circuits), conventional circuits, and / or combinations thereof. A processor may be defined as a processing circuit or circuit comprising transistors and other circuits.

[0371] In the present invention, circuits, units, or means may be hardware designed or programmed to perform a specified function. The hardware may be the hardware disclosed in the present invention or other known hardware programmed or configured to perform a specified function. Where the hardware is a processor that can be considered a type of circuit, the circuits, means, or units may be a combination of hardware and software, and the software may constitute the hardware and / or the processor.

[0372] The above-described disclosure can be implemented as computer-readable code on a medium on which a program is recorded. A computer-readable medium includes all types of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable media include a Hard Disk Drive (HDD), a Solid State Disk (SSD), a Silicon Disk Drive (SDD), ROM, RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc. Additionally, the computer may include a processor (175).

[0373] Although preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above. Various modifications are possible by those skilled in the art without departing from the essence of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical spirit or perspective of the present disclosure.

Claims

1. In the device, Output interface; and One or more processors comprising: generating context information using at least one of sensing information, driver information, or driving information; extracting a cause description indicating the cause of a vehicle event from the generated context information; generating a prompt based on the extracted cause description; obtaining driving feedback from the generated prompt through an artificial intelligence model; and outputting the obtained driving feedback through an output interface. device.

2. In Paragraph 1, The above one or more processors Generating the prompt by further considering basic rules based on the above driver information and the above driving information device.

3. In Paragraph 2, The above one or more processors Generate the above prompt by further considering additional rules to provide personalized driving feedback, and The above additional rule is including at least one of analysis criteria, feedback type, feedback history, driver-tailored coaching, driver's secondary feedback, or tone device.

4. In Paragraph 1, The above cause description is including one or more time-series causal elements supporting the cause of the above vehicle event device.

5. In Paragraph 1, The above one or more processors Obtaining a feedback evaluation result which is the result of analyzing multiple driving feedbacks, updating the prompt based on the obtained feedback evaluation result, obtaining updated driving feedback from the updated prompt through the artificial intelligence model, and outputting the updated driving feedback through the output interface. device.

6. In Paragraph 5, The above feedback evaluation results including at least one of the driver's driving tendencies, the need for driving coaching, the method of providing driver feedback, or the result value of the driving feedback performance. device.

7. In Paragraph 1, The above driving feedback is including at least one of vehicle control signals, text, images, or videos device.

8. In Paragraph 1, The above driving feedback is Includes the occurrence of the above vehicle event, and includes a first driving video representing the driver's actual driving and a second driving video representing best practice driving. device.

9. In Paragraph 1, The above one or more processors Outputting the driving feedback at the time the vehicle's engine is turned off or at the time the vehicle stops after the occurrence of the vehicle event. device.

10. In Paragraph 1, The above device Vehicle or CDC (Cockpit domain controller) device.

11. In the method of operating the device, A step of generating context information using at least one of sensing information, driver information, or driving information; A step of extracting a cause description indicating the cause of a vehicle event from the generated context information; A step of generating a prompt based on the above-mentioned extracted cause description; A step of obtaining driving feedback from the generated prompt through an artificial intelligence model; and A step comprising outputting the above-mentioned driving feedback Method of operation of the device.

12. In Paragraph 11, The step of generating the above prompt is A step comprising generating the prompt by further considering basic rules based on the driver information and the driving information. Method of operation of the device.

13. In Paragraph 12, The step of generating the above prompt is The method includes the step of generating the above prompt by further considering additional rules for providing personalized driving feedback, The above additional rule is including at least one of analysis criteria, feedback type, feedback history, driver-tailored coaching, driver's secondary feedback, or tone Method of operation of the device.

14. In Paragraph 11, The above cause description is including one or more time-series causal elements supporting the cause of the above vehicle event Method of operation of the device.

15. In Paragraph 11, A step of obtaining a feedback evaluation result, which is the result of analyzing multiple driving feedbacks; A step of updating the prompt based on the feedback evaluation result obtained above; A step of obtaining updated driving feedback from the updated prompt through the artificial intelligence model; and Further including the step of outputting the above-mentioned updated driving feedback Method of operation of the device.

16. In Paragraph 15, The above feedback evaluation results including at least one of the driver's driving tendencies, the need for driving coaching, the method of providing driver feedback, or the result value of the driving feedback performance. Method of operation of the device.

17. In Paragraph 11, The above driving feedback is including at least one of vehicle control signals, text, images, or videos Method of operation of the device.

18. In Paragraph 11, The above driving feedback is Includes the occurrence of the above vehicle event, and includes a first driving video representing the driver's actual driving and a second driving video representing best practice driving. Method of operation of the device.

19. In Paragraph 11, The step of outputting the above driving feedback A step comprising outputting the driving feedback at the time when the vehicle's engine is turned off or at the time when the vehicle has stopped after the occurrence of the vehicle event. Method of operation of the device.

20. A computer-readable non-volatile recording medium having a program for performing a method of operating a device, The above method of operation A step of generating context information using at least one of sensing information, driver information, or driving information; A step of extracting a cause description indicating the cause of a vehicle event from the generated context information; A step of generating a prompt based on the above-mentioned extracted cause description; A step of obtaining driving feedback from the generated prompt through an artificial intelligence model; and A step comprising outputting the above-mentioned driving feedback Non-volatile recording media.

Images