Signal processing device and vehicle control device including same

Abstract

According to an embodiment of the present disclosure, a signal processing device and a vehicle control device including same comprise a processor for receiving camera data outside a vehicle, camera data inside the vehicle, and sensor data in the vehicle, wherein the processor: detects an object on the basis of the camera data outside the vehicle; monitors a state of an occupant including a driver on the basis of the camera data inside the vehicle; on the basis of the monitored state information of the occupant, information on the detected object, the sensor data, and driving pattern information of the driver stored in a database, extracts multimodal context data corresponding to a driving context related to vehicle driving; calculates a risk level of the vehicle driving on the basis of the extracted multimodal context data; and outputs a vehicle control signal on the basis of the calculated risk level. Accordingly, a situation inside the vehicle can be determined quickly and accurately so as to stably control the vehicle.

Description

Signal processing device, and vehicle control device having the same

[0001] The present disclosure relates to a signal processing device and a vehicle control device equipped with the same, and more specifically, to a signal processing device capable of rapidly and accurately determining the situation inside a vehicle and performing stable vehicle control, and a vehicle control device equipped with the same.

[0002] A vehicle is a device that moves the user in the desired direction. A typical example is an automobile.

[0003] Meanwhile, for the convenience of users, a vehicle control device is installed inside the vehicle.

[0004] The vehicle control unit includes a signal processing unit and can perform signal processing based on sensor data from various internal sensor devices.

[0005] The prior art U.S. Patent No. 7719431 discloses a system for detecting and responding to drowsy driving, which determines the drowsy driving state of a vehicle based on a determination of whether there is a lane violation, and outputs a driver alarm signal, etc. based thereon.

[0006] However, according to prior art, there is a problem in that real-time monitoring of the vehicle interior cannot be performed and the situation inside the vehicle cannot be accurately recognized. Consequently, a problem arises where driver alarm signals unrelated to the situation inside the vehicle are output.

[0007] The problem that the present disclosure aims to solve is to provide a signal processing device capable of rapidly and accurately determining the situation inside a vehicle and performing stable vehicle control, and a vehicle control device equipped with the same.

[0008] Another problem that the present disclosure aims to solve is to provide a signal processing device capable of performing stable vehicle control by rapidly and accurately determining the situation within a vehicle based on multimodal context data, and a vehicle control device equipped with the same.

[0009] To solve the above technical problem, a signal processing device according to one embodiment of the present disclosure and a vehicle control device equipped therewith include a processor that receives camera data outside the vehicle, camera data inside the vehicle, and sensor data inside the vehicle. The processor performs object detection based on camera data outside the vehicle, monitors the state of a passenger including a driver based on camera data inside the vehicle, extracts multimodal context data corresponding to a driving context related to vehicle driving based on monitored passenger state information, detected object information, sensor data, and driver's driving pattern information stored in a database, calculates a risk level of vehicle driving based on an inference result or response result obtained based on the extracted multimodal context data, and outputs a vehicle control signal based on the calculated risk level.

[0010] Meanwhile, the processor can extract multimodal context data based on monitored occupant status information, detected object information, sensor data, and driver's driving pattern information, obtain an inference result or response result based on the extracted multimodal context data, calculate a risk level of vehicle driving based on the obtained inference result or response result, and output a vehicle control signal based on the calculated risk level.

[0011] Meanwhile, the processor can classify the risk level of vehicle driving into levels 1 through 4.

[0012] Meanwhile, the processor can control authority adjustment, safety extension, and sensitivity adjustment for vehicle control based on the risk level of vehicle driving.

[0013] Meanwhile, the processor calculates authority adjustment data for vehicle control, safety extension data for safety extension, and sensitivity adjustment data for sensitivity adjustment based on the risk level of vehicle driving, and can generate a vehicle control signal based on the authority adjustment data, safety extension data, and sensitivity adjustment data.

[0014] Meanwhile, the processor can perform learning based on the calculated risk level and occupant profile information, and output a vehicle control signal, a warning message, or a guide message based on the result data of the learning.

[0015] Meanwhile, the processor can compute driving ability evaluation data and emergency level evaluation data based on the driver's context data and driver's characteristic data, and compute the risk level of vehicle driving based on the driving ability evaluation data and emergency level evaluation data.

[0016] Meanwhile, the processor can calculate driving ability evaluation data and emergency level evaluation data based on the driver's context data, driver's characteristic data, passenger's context data, and passenger's characteristic data, and calculate the risk level of vehicle driving based on the driving ability evaluation data and emergency level evaluation data.

[0017] Meanwhile, the processor can classify the risk level of vehicle driving into multiple levels based on vehicle internal context data, vehicle external context data, and occupant profile information.

[0018] Meanwhile, the processor can classify the risk level of vehicle driving into multiple levels based on vehicle internal context data, vehicle external context data, occupant characteristic information, occupant behavior information, and driver's driving ability information.

[0019] Meanwhile, the processor classifies the risk level of vehicle driving into multiple levels, and when the risk level of vehicle driving is at the first level, controls the authority adjustment, safety extension, and sensitivity adjustment for vehicle control to remain unchanged, and when the risk level of vehicle driving is at the second level, which is higher than the first level, switches the authority adjustment and safety extension for vehicle control to a standby state and can increase the sensitivity of the sensitivity adjustment above the first level.

[0020] Meanwhile, if the risk level of vehicle driving is at a third level higher than the second level, the processor may ignore or supplement the operation input within the authority adjustment for vehicle control, increase the threshold value within the safety extension, or raise the sensitivity of the sensitivity adjustment above the second level.

[0021] Meanwhile, when the risk level of vehicle driving is at its highest level, the processor can control steering and acceleration to adjust authority for vehicle control, set the threshold within the safety extension to the maximum value, and set the sensitivity of the sensitivity adjustment to the maximum value.

[0022] Meanwhile, the processor can calculate the risk level of vehicle driving as a first level when the driver's driving ability information among the monitored passenger status information indicates that the driver can operate the vehicle, calculate the risk level of vehicle driving as a second level higher than the first level when the driver's driving ability information indicates that the driver can operate the vehicle but is in a distracted state, calculate the risk level of vehicle driving as a third level higher than the second level when the driver's driving ability information indicates that the driver can operate the vehicle but is in a delayed reaction state, and calculate the risk level of vehicle driving as a fourth level higher than the third level when the driver's driving ability information indicates that the driver cannot operate the vehicle.

[0023] Meanwhile, the processor may calculate the risk level of vehicle driving as a first level if the occupant characteristic information does not include habit behavior information or is not personalized habit behavior information, calculate the risk level of vehicle driving as a second level higher than the first level if the occupant characteristic information includes habit behavior information that interferes with driving, calculate the risk level of vehicle driving as a third level higher than the second level if the occupant characteristic information includes repetitive dangerous behavior and medical history or accident history, and calculate the risk level of vehicle driving as a fourth level higher than the third level if the occupant characteristic information includes risk information based on medical history.

[0024] Meanwhile, the processor may calculate the risk level of vehicle driving to a first level when the passenger's behavior information includes stable state information, calculate the risk level of vehicle driving to a second level higher than the first level when the passenger's behavior information includes voice information at a level below a threshold, calculate the risk level of vehicle driving to a third level higher than the second level when the passenger's behavior information includes repeated voice information at a level above a threshold, and calculate the risk level of vehicle driving to a fourth level higher than the third level when the passenger's behavior information includes actions that interfere with the driver's driving operation.

[0025] Meanwhile, the processor can generate a prompt for calculating the risk level of vehicle driving based on multimodal context data, calculate the risk level of vehicle driving based on an inference result or response result obtained based on the prompt, and output a vehicle control signal based on the calculated risk level.

[0026] A signal processing device according to one embodiment of the present disclosure and a vehicle control device equipped therewith further comprises a neural processor that executes inference or a response based on a prompt; and the processor may generate a prompt for calculating a risk level of vehicle driving based on multimodal context data, calculate a risk level of vehicle driving based on an inference result or a response result executed by the neural processor based on the prompt, and output a vehicle control signal based on the calculated risk level.

[0027] Meanwhile, the processor executes a hypervisor and executes a driving control virtualization machine on the hypervisor, and the driving control virtualization machine receives camera data outside the vehicle, camera data inside the vehicle, and sensor data inside the vehicle through shared memory within the hypervisor, and can execute a driving control service or a driving control application based on the camera data outside the vehicle, camera data inside the vehicle, sensor data, and driver's driving pattern information stored in a database.

[0028] A signal processing device according to one embodiment of the present disclosure and a vehicle control device equipped therewith include a processor that receives camera data outside the vehicle, camera data inside the vehicle, and sensor data inside the vehicle. The processor performs object detection based on camera data outside the vehicle, monitors the state of a passenger including a driver based on camera data inside the vehicle, extracts multimodal context data corresponding to a driving context related to vehicle driving based on monitored passenger state information, detected object information, sensor data, and driver's driving pattern information stored in a database, calculates a risk level of vehicle driving based on the extracted multimodal context data, and outputs a vehicle control signal based on the calculated risk level. Accordingly, it is possible to perform stable vehicle control by quickly and accurately determining the situation inside the vehicle. In particular, it is possible to perform stable vehicle control by quickly and accurately determining the situation inside the vehicle based on multimodal context data.

[0029] Meanwhile, the processor can extract multimodal context data based on monitored occupant status information, detected object information, sensor data, and driver's driving pattern information, obtain an inference result or response result based on the extracted multimodal context data, calculate a risk level of vehicle driving based on the obtained inference result or response result, and output a vehicle control signal based on the calculated risk level. Accordingly, it becomes possible to rapidly and accurately determine the situation inside the vehicle and stably perform vehicle control.

[0030] Meanwhile, the processor can classify the risk level of vehicle driving into levels 1 through 4. Accordingly, it is possible to quickly and accurately determine the situation inside the vehicle and perform stable vehicle control.

[0031] Meanwhile, the processor can control authority adjustment, safety extension, and sensitivity adjustment for vehicle control based on the risk level of vehicle driving. Accordingly, it becomes possible to perform stable vehicle control by quickly and accurately determining the situation inside the vehicle.

[0032] Meanwhile, the processor computes authority adjustment data for vehicle control, safety extension data for safety extension, and sensitivity adjustment data for sensitivity adjustment based on the risk level of vehicle driving, and can generate a vehicle control signal based on the authority adjustment data, safety extension data, and sensitivity adjustment data. Accordingly, it becomes possible to rapidly and accurately determine the situation within the vehicle and stably perform vehicle control.

[0033] Meanwhile, the processor performs learning based on the calculated risk level and occupant profile information, and outputs vehicle control signals, warning messages, or guide messages based on the resulting learning data. Accordingly, it becomes possible to quickly and accurately determine the situation inside the vehicle and stably perform vehicle control.

[0034] Meanwhile, the processor can compute driving capability evaluation data and emergency level evaluation data based on the driver's context data and driver characteristic data, and compute the risk level of vehicle driving based on the driving capability evaluation data and emergency level evaluation data. Accordingly, it becomes possible to quickly and accurately determine the situation inside the vehicle and perform stable vehicle control.

[0035] Meanwhile, the processor can compute driving capability evaluation data and emergency level evaluation data based on the driver's context data, driver's characteristic data, passenger's context data, and passenger's characteristic data, and compute the risk level of vehicle driving based on the driving capability evaluation data and emergency level evaluation data. Accordingly, it becomes possible to quickly and accurately determine the situation inside the vehicle and perform stable vehicle control.

[0036] Meanwhile, the processor can classify the risk level of vehicle driving into multiple levels based on vehicle internal context data, vehicle external context data, and occupant profile information. Accordingly, it becomes possible to quickly and accurately determine the situation inside the vehicle and perform stable vehicle control.

[0037] Meanwhile, the processor can classify the risk level of vehicle driving into multiple levels based on vehicle internal context data, vehicle external context data, occupant characteristic information, occupant behavior information, and driver's driving ability information. Accordingly, it becomes possible to perform stable vehicle control by quickly and accurately determining the situation inside the vehicle.

[0038] Meanwhile, the processor classifies the risk level of vehicle driving into multiple levels. When the risk level of vehicle driving is at Level 1, it controls the authority adjustment, safety extension, and sensitivity adjustment for vehicle control to remain unchanged. When the risk level of vehicle driving is at Level 2, which is higher than Level 1, it switches the authority adjustment and safety extension for vehicle control to a standby state and can increase the sensitivity of the sensitivity adjustment above Level 1. Accordingly, it becomes possible to quickly and accurately determine the situation inside the vehicle and perform stable vehicle control.

[0039] Meanwhile, if the risk level of vehicle driving is at a third level higher than the second level, the processor may ignore or supplement the operation input within the authority adjustment for vehicle control, increase the threshold value within the safety extension, or raise the sensitivity of the sensitivity adjustment above the second level. Accordingly, it becomes possible to quickly and accurately determine the situation inside the vehicle and perform stable vehicle control.

[0040] Meanwhile, when the risk level of vehicle driving is at its highest level, the processor controls steering and acceleration to adjust authority for vehicle control, sets the threshold within the safety extension to its maximum value, and sets the sensitivity of the sensitivity adjustment to its maximum value. Accordingly, it becomes possible to quickly and accurately determine the situation inside the vehicle and perform stable vehicle control.

[0041] Meanwhile, the processor can calculate the risk level of vehicle driving as a first level if the driver's driving ability information among the monitored occupant status information indicates that the driver is capable of operating the vehicle; calculate the risk level of vehicle driving as a second level higher than the first level if the driver's driving ability information indicates that the driver is capable of operating the vehicle but is in a distracted state; calculate the risk level of vehicle driving as a third level higher than the second level if the driver's driving ability information indicates that the driver is unable to operate the vehicle; and calculate the risk level of vehicle driving as a fourth level higher than the third level if the driver's driving ability information indicates that the driver is unable to operate the vehicle. Accordingly, it becomes possible to quickly and accurately determine the situation inside the vehicle and perform stable vehicle control.

[0042] Meanwhile, the processor can calculate the risk level of vehicle driving as a first level if the occupant characteristic information does not include habitual behavior information or is not personalized habitual behavior information; calculate the risk level of vehicle driving as a second level higher than the first level if the occupant characteristic information includes habitual behavior information that interferes with driving; calculate the risk level of vehicle driving as a third level higher than the second level if the occupant characteristic information includes repetitive dangerous behaviors and medical history or accident history; and calculate the risk level of vehicle driving as a fourth level higher than the third level if the occupant characteristic information includes risk information based on medical history. Accordingly, it becomes possible to quickly and accurately determine the situation inside the vehicle and perform stable vehicle control.

[0043] Meanwhile, the processor can calculate the risk level of vehicle driving to a first level if the passenger's behavioral information includes stable state information, calculate the risk level of vehicle driving to a second level higher than the first level if the passenger's behavioral information includes voice information at a level below a threshold, calculate the risk level of vehicle driving to a third level higher than the second level if the passenger's behavioral information includes repeated voice information at a level above the threshold, and calculate the risk level of vehicle driving to a fourth level higher than the third level if the passenger's behavioral information includes actions that interfere with the driver's driving operation. Accordingly, it is possible to quickly and accurately determine the situation inside the vehicle and perform stable vehicle control.

[0044] Meanwhile, the processor can generate a prompt for calculating the risk level of vehicle driving based on multimodal context data, calculate the risk level of vehicle driving based on the inference result or response result obtained based on the prompt, and output a vehicle control signal based on the calculated risk level. Accordingly, it becomes possible to rapidly and accurately determine the situation inside the vehicle and stably perform vehicle control.

[0045] A signal processing device according to one embodiment of the present disclosure and a vehicle control device equipped therewith further include a neural processor that executes inference or a response based on a prompt; wherein the processor generates a prompt for calculating a risk level of vehicle driving based on multimodal context data, calculates a risk level of vehicle driving based on an inference result or a response result executed by the neural processor based on the prompt, and outputs a vehicle control signal based on the calculated risk level. Accordingly, it is possible to rapidly and accurately determine the situation inside the vehicle and stably perform vehicle control.

[0046] Meanwhile, the processor executes a hypervisor and runs a driving control virtualization machine on the hypervisor. The driving control virtualization machine receives camera data from outside the vehicle, camera data from inside the vehicle, and sensor data from inside the vehicle through shared memory within the hypervisor. Based on the camera data from outside the vehicle, camera data from inside the vehicle, sensor data, and driver's driving pattern information stored in a database, the driving control service or driving control application can be executed. Accordingly, the situation inside the vehicle can be determined quickly and accurately, enabling stable vehicle control.

[0047] Figure 1 is a drawing illustrating an example of the exterior and interior of a vehicle.

[0048] Figure 2 is a diagram illustrating an example of the architecture of a vehicle control device.

[0049] FIG. 3a is a drawing illustrating an example of the arrangement of displays inside a vehicle.

[0050] Figure 3b is a drawing illustrating another example of the arrangement of displays inside a vehicle.

[0051] FIG. 4 is an example of an internal block diagram of a vehicle control device according to an embodiment of the present disclosure.

[0052] FIG. 5 is an example of a configuration diagram of a signal processing device according to an embodiment of the present disclosure.

[0053] FIG. 6 is an example of a block diagram of a vehicle control device according to an embodiment of the present disclosure.

[0054] FIG. 7 is an example of a block diagram of a signal processing device according to an embodiment of the present disclosure.

[0055] FIGS. 8a to 8c are drawings referenced in the description of FIG. 7.

[0056] FIG. 9 is another example of a block diagram of a signal processing device according to an embodiment of the present disclosure.

[0057] FIGS. 10 to 15 are drawings referenced in the description of FIG. 9.

[0058] FIG. 16 is an example of a flowchart illustrating a method of operation of a signal processing device according to an embodiment of the present disclosure.

[0059] FIG. 17 is another example of a flowchart illustrating a method of operation of a signal processing device according to an embodiment of the present disclosure.

[0060] FIG. 18 is another example of a block diagram of a signal processing device according to an embodiment of the present disclosure.

[0061] The present disclosure will be described in more detail below with reference to the drawings.

[0062] The suffixes "module" and "part" for components used in the following description are assigned solely for the ease of drafting this specification and do not inherently confer any particularly significant meaning or role. Accordingly, the terms "module" and "part" may be used interchangeably.

[0063] Figure 1 is a drawing illustrating an example of the exterior and interior of a vehicle.

[0064] Referring to the drawing, the vehicle (200) is operated by a plurality of wheels (103FR, 103FL, 103RL,...) that rotate by a power source, and a steering wheel (150) for controlling the direction of travel of the vehicle (200).

[0065] Meanwhile, the vehicle (200) may further be equipped with a camera (195), etc., for acquiring an image of the front of the vehicle.

[0066] Meanwhile, the vehicle (200) may be equipped with a plurality of displays (180a, 180b) for displaying images, information, etc. inside.

[0067] In FIG. 1, a cluster display (180a) and an IVI (In-Vehicle Infotainment) display (180b) are exemplified as multiple displays (180a, 180b). In addition, a HUD (Head Up Display) and the like are also possible.

[0068] Meanwhile, the IVI (In-Vehicle Infotainment) display (180b) may also be named a Center Information Display or an AVN (Audio Video Navigation) display.

[0069] Meanwhile, the vehicle (200) described in this specification may be a concept that includes all of the following: a vehicle equipped with an engine as a power source, a hybrid vehicle equipped with an engine and an electric motor as a power source, an electric vehicle equipped with an electric motor as a power source, etc.

[0070] Figure 2 is a diagram illustrating an example of the architecture of a vehicle control device.

[0071] Referring to the drawing, the architecture (300a) of the vehicle control device can correspond to a zone-based architecture.

[0072] Accordingly, sensor devices and processors inside the vehicle may be placed in each of the multiple zones (Z1 to Z4), and a signal processing device (170a) including a gateway (GWDa) may be placed in the central area of ​​the multiple zones (Z1 to Z4).

[0073] Meanwhile, the signal processing device (170a) may additionally include an autonomous driving control module (ACC), a cockpit control module (CPG), etc., in addition to the gateway (GWDa).

[0074] The gateway (GWDa) within the signal processing device (170a) may be a High Performance Computing (HPC) gateway.

[0075] That is, the signal processing device (170a) of FIG. 2 is an integrated HPC and can exchange data with an external communication module (not shown) or a processor (not shown) in a plurality of zones (Z1 to Z4).

[0076] FIG. 3a is a drawing illustrating an example of the arrangement of displays inside a vehicle.

[0077] Referring to the drawing, the vehicle interior may be equipped with a cluster display (180a), an IVI (In-Vehicle Infotainment) display (180b), a rear seat entertainment display (180c, 180d), a rearview mirror display (not shown), etc.

[0078] Meanwhile, in addition to the display, an interior camera (195i) may be installed inside the vehicle.

[0079] Figure 3b is a drawing illustrating another example of the arrangement of displays inside a vehicle.

[0080] A vehicle control device (100) according to an embodiment of the present disclosure may include a plurality of displays (180a to 180b) and a signal processing device (170) that performs signal processing for displaying images, information, etc. on the plurality of displays (180a to 180b) and outputs an image signal to at least one display (180a to 180b).

[0081] Among the plurality of displays (180a to 180b), the first display (180a) is a cluster display (180a) for displaying driving status, operation information, etc., and the second display (180b) may be an IVI (In-Vehicle Infotainment) display (180b) for displaying vehicle operation information, navigation map, various entertainment information or video.

[0082] The signal processing device (170) has a processor (175) inside and can execute a first virtualization machine to a third virtualization machine (not shown) on a hypervisor (not shown) within the processor (175).

[0083] A second virtualization machine (not shown) operates for the first display (180a), and a third virtualization machine (not shown) can operate for the second display (180b).

[0084] Meanwhile, the first virtualization machine (not shown) within the processor (175) can be controlled to set up a shared memory (508) based on a hypervisor (505) for the same data transmission to the second virtualization machine (not shown) and the third virtualization machine (not shown). Accordingly, the same information or the same image can be synchronized and displayed on the first display (180a) and the second display (180b) within the vehicle.

[0085] Meanwhile, the first virtualization machine (not shown) within the processor (175) shares at least a portion of the data with the second virtualization machine (not shown) and the third virtualization machine (not shown) for data sharing processing. Accordingly, data can be shared and processed by multiple virtualization machines for multiple displays within the vehicle.

[0086] Meanwhile, the first virtualization machine (not shown) within the processor (175) can receive and process wheel speed sensor data of the vehicle and transmit the processed wheel speed sensor data to at least one of the second virtualization machine (not shown) or the third virtualization machine (not shown). Accordingly, the wheel speed sensor data of the vehicle can be shared with at least one virtualization machine, etc.

[0087] Meanwhile, the vehicle control device (100) according to the embodiment of the present disclosure may further include a rear seat entertainment display (180c) for displaying driving status information, simple navigation information, various entertainment information or images.

[0088] The signal processing device (170) can control the RSE display (180c) by running a fourth virtualization machine (not shown) in addition to the first to third virtualization machines (not shown) on a hypervisor (not shown) within the processor (175).

[0089] Accordingly, various displays (180a to 180c) can be controlled using a single signal processing device (170).

[0090] Meanwhile, some of the multiple displays (180a to 180c) operate under a Linux OS, and others can operate under a Web OS.

[0091] A signal processing device (170) according to an embodiment of the present disclosure can control displays (180a to 180c) operating under various operating systems (OS) to synchronize and display the same information or the same image.

[0092] Meanwhile, FIG. 3b illustrates that a vehicle speed indicator (212a) and a vehicle interior temperature indicator (213a) are displayed on a first display (180a), a home screen (222) including a plurality of applications, a vehicle speed indicator (212b), and a vehicle interior temperature indicator (213b) is displayed on a second display (180b), and a second home screen (222b) including a plurality of applications and a vehicle interior temperature indicator (213c) is displayed on a third display (180c).

[0093] FIG. 4 is an example of an internal block diagram of a vehicle control device according to an embodiment of the present disclosure.

[0094] Referring to the drawings, a vehicle control device (100) according to an embodiment of the present disclosure may include an input device (110), a communication unit (120) for communication with an external device, a plurality of communication modules (EMa~EMd) for internal communication, a memory (140), a signal processing device (170), a plurality of displays (180a~180c), an audio output unit (185), and a power supply unit (190).

[0095] Multiple communication modules (EMa~EMd) can be placed in each of the multiple zones (Z1~Z4) of FIG. 2, for example.

[0096] Meanwhile, the signal processing device (170) may have a communication switch (736b) inside for data communication with each communication module (EM1~EM4).

[0097] Each communication module (EM1~EM4) can perform data communication with a plurality of sensor devices (SN), ECU (770), or area signal processing device (170Z).

[0098] Meanwhile, a plurality of sensor devices (SN) may include a camera (195), lidar (196), radar (197), or position sensor (198).

[0099] The input device (110) may be equipped with physical buttons, pads, etc. for button input, touch input, etc.

[0100] Meanwhile, the input device (110) may be equipped with a microphone (112) for user voice input.

[0101] The communication unit (120) can exchange data wirelessly with a mobile terminal (600) or a server (400).

[0102] In particular, the communication unit (120) can wirelessly exchange data with the vehicle driver's mobile terminal. Various data communication methods are possible as wireless data communication methods, such as Bluetooth, WiFi, WiFi Direct, and APiX.

[0103] The communication unit (120) can receive weather information, road traffic condition information, for example, TPEG (Transport Protocol Expert Group) information from a mobile terminal (600) or a server (400). To this end, the communication unit (120) may be equipped with a mobile communication module (not shown).

[0104] A plurality of communication modules (EM1~EM4) can receive sensor data, etc. from an ECU (770), a sensor device (SN), or a region signal processing device (170Z), and transmit the received sensor data to the signal processing device (170).

[0105] Here, the sensor data may include at least one of vehicle direction data, vehicle location data (GPS data), vehicle angle data, vehicle speed data, vehicle acceleration data, vehicle tilt data, vehicle forward / reverse data, battery data, fuel data, tire data, vehicle lamp data, vehicle interior temperature data, and vehicle interior humidity data.

[0106] Such sensor data can be obtained from a heading sensor, a yaw sensor, a gyro sensor, a position module, a vehicle forward / reverse sensor, a wheel sensor, a vehicle speed sensor, a vehicle body inclination sensor, a battery sensor, a fuel sensor, a tire sensor, a steering sensor based on steering wheel rotation, a vehicle interior temperature sensor, a vehicle interior humidity sensor, etc.

[0107] Meanwhile, the position module may include a GPS module or a position sensor (198) for receiving GPS information.

[0108] Meanwhile, at least one of the multiple communication modules (EM1 to EM4) can transmit location information data sensed from a GPS module or a location sensor (198) to a signal processing device (170).

[0109] Meanwhile, at least one of the plurality of communication modules (EM1 to EM4) can receive vehicle front image data, vehicle side image data, vehicle rear image data, and obstacle distance information around the vehicle from a camera (195), lidar (196), radar (197), etc., and transmit the received information to a signal processing device (170).

[0110] The memory (140) can store various data for the overall operation of the vehicle control device (100), such as a program for processing or controlling the signal processing device (170).

[0111] For example, memory (140) can store data regarding a hypervisor, a first virtualization machine to a third virtualization machine, for execution within a processor (175).

[0112] The audio output unit (185) converts an electrical signal from the signal processing device (170) into an audio signal and outputs it. To do this, a speaker or the like may be provided.

[0113] The power supply unit (190) can supply power necessary for the operation of each component under the control of the signal processing unit (170). In particular, the power supply unit (190) can receive power from a battery inside the vehicle, etc.

[0114] The signal processing unit (170) controls the overall operation of each unit within the vehicle control unit (100).

[0115] For example, the signal processing device (170) may include a processor (175) that performs signal processing for a vehicle display (180a, 180b).

[0116] The processor (175) can run a first virtualization machine to a third virtualization machine (not shown) on a hypervisor (505 in FIG. 5) within the processor (175).

[0117] Among the first to third virtual machines (not shown), the first virtual machine (not shown) may be named a Server Virtual Machine, and the second to third virtual machines (not shown) may be named a Guest Virtual Machine.

[0118] For example, a first virtualization machine (not shown) within a processor (175) can receive sensor data from a plurality of sensor devices, such as vehicle sensor data, location information data, camera image data, audio data, or touch input data, and process or modify it to output it.

[0119] In this way, by performing most of the data processing in the first virtualization machine (not shown), 1:N data sharing becomes possible.

[0120] As another example, the first virtualization machine (not shown) can directly receive and process CAN data, Ethernet data, audio data, radio data, USB data, and wireless communication data for the second virtualization machine to the third virtualization machine (not shown).

[0121] And, the first virtualization machine (not shown) can transmit the processed data to the second virtualization machine to the third virtualization machine (not shown).

[0122] Accordingly, among the first to third virtualization machines (not shown), only the first virtualization machine (not shown) receives sensor data, communication data, or external input data from a plurality of sensor devices and performs signal processing, thereby reducing the signal processing burden on other virtualization machines and enabling 1:N data communication, which enables synchronization when sharing data.

[0123] Meanwhile, the first virtualization machine (not shown) can control the sharing of the same data with the second virtualization machine (not shown) and the third virtualization machine (not shown) by writing data to the shared memory (508 in FIG. 5).

[0124] For example, the first virtualization machine (not shown) can record vehicle sensor data, the location information data, the camera image data, or the touch input data in a shared memory (508) and control the sharing of the same data with the second virtualization machine (not shown) and the third virtualization machine (not shown). Accordingly, data sharing in a 1:N manner becomes possible.

[0125] Ultimately, by performing most of the data processing on the first virtualization machine (not shown), 1:N data sharing becomes possible.

[0126] Meanwhile, the first virtualization machine (not shown) within the processor (175) can control the second virtualization machine (not shown) and the third virtualization machine (not shown) to set up a shared memory (508) based on the hypervisor (505) for the same data transmission.

[0127] Meanwhile, the signal processing device (170) can process various signals such as audio signals, video signals, and data signals. To this end, the signal processing device (170) can be implemented in the form of a System On Chip (SOC).

[0128] Meanwhile, the signal processing device (170) in the display device (100) of FIG. 4 may be the same as the signal processing device (170) of the vehicle control device of FIG. 5 and below.

[0129] FIG. 5 is an example of a configuration diagram of a signal processing device according to an embodiment of the present disclosure.

[0130] Referring to the drawings, a signal processing device (170) according to an embodiment of the present disclosure includes a processor (175).

[0131] Meanwhile, the signal processing device (170) may be named as an HPC (High Performance Computing) signal processing device.

[0132] Meanwhile, the signal processing device (170) according to an embodiment of the present disclosure may further include a neural processor (179).

[0133] Meanwhile, the neural processor (179) may also be referred to as an on-device-based learning processor.

[0134] Meanwhile, the processor (175) in the signal processing device (170) can execute the hypervisor (505) and execute the first to third virtualization machines (510 to 530) on the hypervisor (505).

[0135] The first virtualization machine (510) may be a gateway virtualization machine corresponding to the gateway (GWDa) of FIG. 2.

[0136] The second virtualization machine (520) may be a driving control virtualization machine corresponding to the autonomous driving control module (ACC) of FIG. 2.

[0137] The driving control virtualization machine (520) at this time can control the vehicle driving assistance (ADAS) or the autonomous driving (AD).

[0138] The third virtualization machine (530) may be a display virtualization machine corresponding to the cockpit control module (CPG) of FIG. 2 or the display (180a, 180b, 180c) of FIG. 3.

[0139] For example, the third virtualization machine (530) may include a cluster virtualization machine (530b) for a cluster display (180a) and an IVI virtualization machine (530b) for an IVI display (180b).

[0140] Meanwhile, the third virtualization machine (530) may further include a HUD virtualization machine (not shown) for a HUD display (180c).

[0141] Meanwhile, the processor (175) can share data between each virtualization machine (510~530) through shared memory (505) within the hypervisor (505).

[0142] Meanwhile, the processor (175) can share data with a plurality of sensor devices (SN) or ECUs (770) or area signal processing devices (170Z) through shared memory (505) within the hypervisor (505).

[0143] Meanwhile, the processor (175) can exchange data with an external server (400) or mobile terminal (600) through the communication device (120) of FIG. 4.

[0144] Meanwhile, the data received by the processor (175) within the signal processing device (170) may include camera data or sensor data.

[0145] For example, sensor data within the vehicle may include at least one of vehicle wheel speed data, vehicle direction data, vehicle location data (GPS data), vehicle angle data, vehicle speed data, vehicle acceleration data, vehicle tilt data, vehicle forward / reverse data, battery data, fuel data, tire data, vehicle lamp data, vehicle interior temperature data, vehicle interior humidity data, vehicle external radar data, and vehicle external lidar data.

[0146] Meanwhile, camera data may include external vehicle camera data and internal vehicle camera data.

[0147] Meanwhile, the processor (175) within the signal processing device (170) can execute multiple virtualization machines (510 to 530) based on safety standards.

[0148] Meanwhile, the processor (175) in the signal processing unit (170a) can execute the hypervisor (505) and, on the hypervisor (505), execute the first to third virtualization machines (510 to 530) according to the automotive safety integrity level (Automotive SIL and ASIL).

[0149] For example, the first virtualization machine (510) may be a virtualization machine corresponding to ASIL C or ASIL D, in which the sum of Severity, Exposure, and Controllability in the Automotive Safety Integrity Level (ASIL) is 9 or 10.

[0150] Meanwhile, ASIL D can correspond to the grade requiring the highest safety level.

[0151] The first virtualization machine (510) can run a safety operating system (not shown) and an application (not shown) on the safety operating system.

[0152] Meanwhile, the first virtualization machine (510) may run a container runtime (not shown) and a container runtime (not shown) on a safety operating system.

[0153] Meanwhile, unlike the drawing, the first virtualization machine (510) may also be executed through a separate processor core instead of the processor (175).

[0154] Meanwhile, the second virtualization machine (520) may be a virtualization machine corresponding to ASIL A or ASIL B, in which the sum of Severity, Exposure, and Controllability in the Automotive Safety Integrity Level (ASIL) is 7 or 8.

[0155] Meanwhile, the second virtualization machine (520) can run an operating system (not shown), a container runtime (not shown) on the operating system (not shown), and a container (not shown) on the container runtime.

[0156] Alternatively, the second virtualization machine (520) can run an operating system (not shown) and an application on the operating system (not shown).

[0157] Meanwhile, the third virtualization machine (530) may be a virtualization machine corresponding to Quality Management (QM), which is the lowest safety level and non-mandatory grade in the Automotive Safety Integrity Level (ASIL).

[0158] Meanwhile, the third virtualization machine (530) can run an operating system (not shown), a container runtime (not shown) on the operating system (not shown), and a container (not shown) on the container runtime.

[0159] Alternatively, the third virtualization machine (530) can run an operating system (not shown) and an application on the operating system (not shown).

[0160] FIG. 6 is an example of a block diagram of a vehicle control device according to an embodiment of the present disclosure.

[0161] Referring to the drawings, a vehicle control device (900) according to an embodiment of the present disclosure comprises a signal processing device (170).

[0162] A vehicle control device (900) according to an embodiment of the present disclosure may further include at least one display.

[0163] In the drawing, at least one display is exemplified as a cluster display (180a) and an IVI display (180b).

[0164] Meanwhile, the vehicle control device (900) may further include a plurality of area signal processing devices (170Z1 to 170Z4).

[0165] The signal processing device (170) at this time is a high-performance centralized signal processing and control device having a plurality of CPUs (175), GPUs (178), NPUs (179), etc., and can be named as a High Performance Computing (HPC) signal processing device or a central signal processing device.

[0166] Multiple area signal processing devices (170Z1~170Z4) and signal processing device (170) are connected by wired cables (CB1~CB4).

[0167] Meanwhile, multiple area signal processing devices (170Z1~170Z4) can be connected to each other by wired cables (CBa~CBd).

[0168] The wired cable (CBa~CBd) at this time may include a CAN communication cable, an Ethernet communication cable, or a PCI Express cable.

[0169] Meanwhile, the signal processing device (170) according to the embodiment of the present disclosure may have at least one processor (175, 178, 177) and a large-capacity storage device (925).

[0170] For example, a signal processing device (170) according to an embodiment of the present disclosure may include a central processor (175, 177), a graphics processor (178), and a neural processor (179).

[0171] Meanwhile, sensor data can be transmitted from at least one of the multiple area signal processing devices (170Z1 to 170Z4) to the signal processing device (170). In particular, the sensor data can be stored in a storage device (925) within the signal processing device (170).

[0172] The sensor data at this time may include at least one of camera data, lidar data, radar data, vehicle direction data, vehicle position data (GPS data), vehicle angle data, vehicle speed data, vehicle acceleration data, vehicle tilt data, vehicle forward / reverse data, battery data, fuel data, tire data, vehicle lamp data, vehicle interior temperature data, and vehicle interior humidity data.

[0173] In the drawing, camera data from a camera (195a) and lidar data from a lidar sensor (196) are input to a first area signal processing device (170Z1), and the camera data and lidar data are transmitted to a signal processing device (170) via a second area signal processing device (170Z2) and a third area signal processing device (170Z3), etc.

[0174] Meanwhile, since the data reading or writing speed to the storage device (925) is faster than the network speed when sensor data is transmitted from at least one of the multiple area signal processing devices (170Z1~170Z4) to the signal processing device (170), it is desirable to perform multipath routing so that network bottlenecks do not occur.

[0175] To this end, the signal processing device (170) according to an embodiment of the present disclosure can perform multipath routing based on a Software Defined Network (SDN). Accordingly, a stable network environment can be secured when reading or writing data of the storage device (925). Furthermore, since data can be transmitted to the storage device (925) using multiple paths, data can be transmitted by dynamically changing the network configuration.

[0176] Data communication between a plurality of area signal processing devices (170Z1~170Z4) and a signal processing device (170) within a vehicle control device (900) according to an embodiment of the present disclosure is preferably Peripheral Component Interconnect Express communication or Ethernet communication for high-bandwidth, low-latency communication.

[0177] FIG. 7 is an example of a block diagram of a signal processing device according to an embodiment of the present disclosure.

[0178] Referring to the drawings, the signal processing system (700) according to an embodiment of the present disclosure may be referred to as an IVEX (In Vehicle Experience) system.

[0179] Meanwhile, the signal processing system (700) according to an embodiment of the present disclosure may include an edge sensor group (SN), a signal processing device (170) in a vehicle, and a server (400).

[0180] Meanwhile, the signal processing device (170) according to an embodiment of the present disclosure includes a recognizer (720), an insightor (750), and an illustrator (780).

[0181] Meanwhile, the edge sensor group (SN) can correspond to a plurality of sensor devices (SN) of FIG. 4.

[0182] Meanwhile, the recognizer (720), the insightor (750), and the illustrator (780) may be included in the signal processing device (170) of the vehicle display device (100) of FIG. 4.

[0183] In particular, the recognizer (720), the insightor (750), and the illustrator (780) may be included in the processor (175) of the signal processing device (170).

[0184] Meanwhile, the edge sensor group (SN) may include vehicle interior sensors (710) and vehicle exterior sensors (715).

[0185] Meanwhile, the edge sensor group (SN) may further include a data receiving unit (718) for receiving vehicle internal data or vehicle external data. The data receiving unit (718) may correspond to the communication device (120) of FIG. 2.

[0186] Meanwhile, the vehicle interior sensors (710) are sensors placed inside the vehicle (200) and may include a front camera (195), lidar (196), radar (197), interior camera (195i), vehicle interior temperature sensor, or vehicle interior humidity sensor.

[0187] The vehicle external sensors (715) are sensors positioned on the exterior of the vehicle (200) and may include an external camera, lidar (196), radar (197), heading sensor, yaw sensor, gyro sensor, position module, vehicle forward / reverse sensor, wheel sensor, vehicle speed sensor, vehicle body inclination sensor, battery sensor, fuel sensor, tire sensor, or steering sensor based on steering wheel rotation. The position module may include a GPS module or a position sensor (198) for receiving GPS information.

[0188] Sensing data may include vehicle interior sensing data and vehicle exterior sensing data.

[0189] The vehicle interior sensing data may be data sensed by the vehicle interior sensors (710).

[0190] Vehicle interior sensing data may include at least one of vehicle interior temperature data, vehicle interior humidity data, battery data, fuel data, vehicle lamp data, tire data, or vehicle interior camera data, or audio data received through a microphone.

[0191] The vehicle external sensing data may be data sensed by the vehicle external sensors (715).

[0192] Vehicle external sensing data may include at least one of vehicle location data (GPS), vehicle direction data, vehicle angle data, vehicle speed data, vehicle acceleration data, vehicle tilt data, whether the vehicle is moving forward or backward, front camera data, or rear camera data.

[0193] Meanwhile, the recognizer (720) can recognize the vehicle situation based on sensing data received from the edge sensor group (SN). In this regard, the recognizer (720) may be referred to as a situation recognition unit.

[0194] The vehicle situation at this time may include external vehicle conditions and internal vehicle conditions.

[0195] Meanwhile, the recognizer (720) can recognize the vehicle situation based on the vehicle internal sensing data or the vehicle internal sensing data, and can generate vehicle situation information regarding the recognized vehicle situation.

[0196] For example, the vehicle situation information (740) may include at least one of the external situation information of the vehicle and the situation information of the vehicle's occupants.

[0197] As another example, the vehicle situation information (740) may include at least one of the vehicle's external situation information, the vehicle's internal situation information, and the vehicle's occupant situation information.

[0198] Meanwhile, the recognizer (720) can recognize external situation information of the vehicle based on external sensing data from the edge sensor group (SN), recognize internal situation information of the vehicle based on internal sensing data of the vehicle, or recognize situation information of the vehicle's occupant based on internal sensing data of the vehicle.

[0199] Meanwhile, the recognizer (720) can output external situation information of the vehicle based on internal vehicle sensing data or external vehicle sensing data, output internal situation information of the vehicle based on internal vehicle sensing data or external vehicle sensing data, or output situation information of the vehicle's occupant based on internal vehicle sensing data or external vehicle sensing data.

[0200] Meanwhile, the recognizer (720) can perform preprocessing and calibration of the vehicle interior sensing data or the vehicle interior sensing data, and can obtain vehicle situation information based on the preprocessed and calibrated sensing data.

[0201] For example, the internal recognition unit (725) within the recognition unit (720) can perform preprocessing and calibration (721) on the vehicle internal sensing data, and can obtain internal situation information of the vehicle based on the preprocessed and calibrated vehicle internal sensing data.

[0202] Specifically, the internal recognizer (725) can perform preprocessing and calibration (721) on the vehicle interior sensing data, perform primitive detection (722) or sensor fusion (723), and based on this, obtain information about the vehicle's interior situation.

[0203] For example, an external recognizer (730) within the recognizer (720) can perform preprocessing and calibration (721) on the external vehicle sensing data, and can obtain external situation information of the vehicle based on the preprocessed and calibrated external vehicle sensing data.

[0204] Specifically, the external recognizer (730) can perform preprocessing and calibration (731) on external vehicle sensing data, execute a perception engine (732), perform sensor fusion (733), or perform segmentation (734), and based on this, obtain external situation information of the vehicle.

[0205] Meanwhile, the passenger recognition device (730) within the recognition device (720) can perform facial recognition (741) of the passenger based on the vehicle interior sensing data, recognize distraction (742), recognize gaze, position, gesture (743), recognize emotion (744), recognize whether the passenger is drinking or taking medication (746), recognize drowsiness (747), or recognize information (748) about other actions, and obtain situational information of the vehicle's passenger based thereon.

[0206] Meanwhile, the situational information of the occupant may include at least one of the following: a face identifier (Face ID) of the occupant inside the vehicle (200), distraction information, gaze information, position information, gesture information, emotion information, information on whether alcohol or drugs have been taken, drowsiness information, or other information regarding actions.

[0207] Meanwhile, the recognizer (720) can transmit the generated vehicle situation information to the insightor (750).

[0208] Meanwhile, the insightor (750) can generate an inference result or a response result based on the situation information of the vehicle received from the recognizer (720).

[0209] Meanwhile, the insightor (750) can generate an inference result or a response result based on the vehicle situation information received from the recognizer (720), and can generate or control a service to be executed based on the inference result or the response result.

[0210] Accordingly, the insightor (750) may be named a response result generation unit or a service generation unit.

[0211] Meanwhile, the insightor (750) may include a multimodal context engine (755), a context controller (754), a safety assistance engine (757), a user characteristic engine (758), an AI orchestrator (760), and an AI model set (765).

[0212] Meanwhile, the multimodal context engine (755) can generate multimodal context data based on the vehicle situation information received from the recognizer (720).

[0213] For example, multimodal context data may include at least one of text data or image data describing a vehicle situation generated based on vehicle situation information.

[0214] Meanwhile, the context controller (754) may be a component included in the multimodal context engine (755) or provided separately from the multimodal context engine (755).

[0215] Meanwhile, the context controller (754) can execute the process of generating multimodal context data when it receives a passenger query from the AI ​​orchestrator (760).

[0216] For example, the passenger curie may be a voice recognition result corresponding to a voice command spoken by the passenger.

[0217] Meanwhile, the safety assist engine (757) can determine whether the current situation is a safe situation or a dangerous situation based on the vehicle situation information received from the recognizer (720).

[0218] Meanwhile, the safety assist engine (757) can transmit a driver assistance control command or a warning notification output command to the safety application (790) of the illustrator (780) if the current situation is determined to be a dangerous situation.

[0219] Meanwhile, the user characteristic engine (758) can generate user context data or passenger context data based on user information or passenger information.

[0220] Meanwhile, user information or passenger information may be referred to as user persona or passenger persona.

[0221] Meanwhile, user information or passenger information may include at least one of the user or passenger's nationality, age, gender, occupation, personality, or psychological type (Myers-Briggs Type Indicator, MBTI).

[0222] Meanwhile, the AI ​​orchestrator (760) can generate a prompt based on at least one of multimodal context data or passenger context data.

[0223] Meanwhile, the AI ​​orchestrator (760) can send the generated prompt to the AI ​​model set (765).

[0224] Meanwhile, the AI ​​model set (765) may include at least one AI model.

[0225] For example, the AI ​​model set (765) may include an on-device AI model (767).

[0226] Meanwhile, the on-device AI model (767) may include at least one AI model. For example, the on-device AI model (767) may include a Small Language Model (LLM).

[0227] Meanwhile, the AI ​​model set (765) may include an interface (766) for data exchange with the AI ​​model (769) in the server (400).

[0228] Meanwhile, the AI ​​model (769) within the server (400) may include at least one AI model. For example, the AI ​​model (769) within the server (400) may include a Large Language Model (LLM).

[0229] That is, the on-device AI model (767) may have a smaller capacity or size than the AI ​​model (769) in the server (400).

[0230] Meanwhile, the AI ​​model set (765) can output an inference result in response to a prompt received from the AI ​​orchestrator (760) and can transmit the inference result to the AI ​​orchestrator (760).

[0231] Meanwhile, the AI ​​orchestrator (760) can generate additional prompts based on the inference results and send the additional prompts to the AI ​​model set (765).

[0232] Meanwhile, the AI ​​model set (765) that receives the additional prompt can output an additional inference result in response to the additional prompt and can transmit the additional inference result to the AI ​​orchestrator (760).

[0233] The AI ​​orchestrator (760) can obtain an inference result or additional inference result received from the AI ​​model set (765) as a response result, and can output the obtained response result to the illustrator (780).

[0234] Meanwhile, the illustrator (780) can execute a service, output service information, execute an application, or output application information based on the response result output from the insightor (750).

[0235] For example, the illustrator (780) can execute a navigation service, execute a vehicle driving assistance control service, execute an autonomous driving service, or execute a display-related service based on the response result output from the insightor (750).

[0236] As another example, the illustrator (780) can run a navigation application, run a vehicle driving assistance control application, run an autonomous driving application, or run a display-related application based on the response result output from the insightor (750).

[0237] Meanwhile, the illustrator (780) may include a multimodal output encoder (781), a visual interface (782), an audio interface (785), and a safety application (790).

[0238] Meanwhile, the multimodal output encoder (781) can encode the response result output from the insightor (750) and output the encoded response result data to the visual interface (782) or audio interface (785).

[0239] Meanwhile, the visual interface (782) or audio interface (785) may be referred to as an output interface.

[0240] Meanwhile, the visual interface (782) can output response result data output from the multimodal output encoder (781).

[0241] Accordingly, at least one of the plurality of displays (180a to 180c) of FIG. 4 can display an image based on response result data from the visual interface (782).

[0242] Meanwhile, the visual interface (782) can output augmented reality (AR) video based on response result data or mixed reality (MR) video data based on response result data.

[0243] Meanwhile, the audio interface (785) can output response result data in the form of audio.

[0244] Accordingly, the audio output unit (185) of FIG. 4 can output a sound corresponding to the response result data from the audio interface (785).

[0245] Meanwhile, the safety application (790) can perform Advanced Driver Assistance System (ADAS) control based on the driver assistance control command or the response result received from the insightor (750).

[0246] For example, the safety application (790) can output warning notification data according to a warning notification output command.

[0247] In response to this, at least one of the electronic control unit (770) of FIG. 4 or a plurality of displays (180a to 180c) can output warning notification data.

[0248] FIGS. 8a to 8c are drawings referenced in the description of FIG. 7.

[0249] First, FIG. 8a is a drawing referenced in the description of the insight of FIG. 7.

[0250] Referring to the drawing, the insightor (750) may include a multimodal context engine (755), an AI orchestrator (760), and a multimodal LLM (767).

[0251] Meanwhile, the multimodal context engine (755) may include a multimodal signal adapter (751), a multimodal indexer (812), a multimodal context buffer (752), a multimodal context retriever (814), a multimodal context descriptor (815), a multimodal event monitor (811), and a context controller (754).

[0252] Unlike Fig. 7, the multimodal context engine (755) may include a context controller (754).

[0253] Meanwhile, the multimodal signal adapter (751) can generate pre-processed multimodal data by filtering, cleaning, synchronizing, and reformulating data received from various sensors or vehicle situation information received from the recognizer (720).

[0254] Meanwhile, the multimodal signal adapter (751) can receive at least one of audio data received through a microphone, front image data captured through a front camera, ADAS information obtained from the front image data or vehicle sensor, location data, IVI (In-Vehicle Infotainment) system data, IVI display information, DMS (Driver Monitoring System) information or IMS (Interior Monitoring System) information based on image data captured through an interior camera (195i), and biometric data obtained from a biometric sensor.

[0255] Meanwhile, the multimodal indexer (812) can generate multimodal processing data by dividing the preprocessed multimodal data into chunks and can index the multimodal processing data.

[0256] Meanwhile, the multimodal indexer (812) can obtain an encoding vector or keyword representing the attribute (or meaning) of the multimodal processed data divided into chunks as an index.

[0257] Meanwhile, the multimodal context buffer (752) can store multimodal processing data and an index corresponding to the multimodal processing data.

[0258] Meanwhile, the multimodal context retriever (814) can search for multimodal processing data most related to the passenger query through an index from the multimodal context buffer (752), and can select the searched multimodal processing data as a context candidate for the creation of multimodal context data.

[0259] Meanwhile, the multimodal context descriptor (815) can reconfigure multimodal processing data selected as a context candidate into multimodal context data having a prompt form that the multimodal LLM (767) can interpret.

[0260] Meanwhile, the multimodal event monitor (811) can monitor whether an index matching the trigger condition is entered when a trigger condition for multimodal data is registered. If an index matching the trigger condition is entered, the multimodal event monitor (811) can generate a trigger event to generate a proactive service query.

[0261] Meanwhile, the context controller (754) can control the overall operation of the multimodal context engine (755).

[0262] Meanwhile, the context controller (754) can execute the process of generating multimodal context data when it receives a passenger query from the AI ​​orchestrator (760).

[0263] Meanwhile, the context controller (754) can generate a preemptive service query when it receives a trigger event from the multimodal event monitor (811).

[0264] Meanwhile, the AI ​​orchestrator (760) can generate a prompt by combining the passenger query and the multimodal context, and can send the generated prompt to the multimodal LLM (767).

[0265] Meanwhile, the passenger curry may be a query in the form of recognized text based on a voice command spoken by the passenger. The voice command may be received through a microphone, and the voice command may be converted into text through an Automatic Speech Recognition (ASR) process.

[0266] Meanwhile, passenger curry may be text converted through an Automatic Speech Recognition (ASR) process.

[0267] Meanwhile, multimodal LLM (767) may be an example of an AI model included in the set of AI models (765) of FIG. 6a.

[0268] Meanwhile, the multimodal LLM (767) can output an inference result from a prompt received from the AI ​​orchestrator (760) and can transmit the inference result to the AI ​​orchestrator (760).

[0269] Meanwhile, the inference result may include a result representing a response to the prompt.

[0270] For example, the inference result may include an API call result regarding whether an API call corresponding to a driving assistance function was successfully performed, and a feedback generation result regarding whether feedback corresponding to the API call was successfully generated.

[0271] Next, Fig. 8b is a drawing referenced in the description of the AI ​​orchestrator of Fig. 7.

[0272] Referring to the drawing, the AI ​​orchestrator (760) may include a task arbitrator (771), a prompt manager (763), a sub-agent set (761), a knowledge database (762), a workflow controller (772), and a tool set / adapter set (764).

[0273] Meanwhile, the task arbitrator (771) can determine one of the multiple sub-agents based on the voice recognition result and the multimodal context data output from the multimodal context engine (755).

[0274] Meanwhile, the prompt manager (763) can generate a prompt for the operation of the determined sub-agent. The prompt manager (763) can generate a prompt based on multimodal context data and passenger queries.

[0275] Meanwhile, the prompt manager (763) can generate a prompt based on information about the functions that the determined sub-agent can perform, multimodal context data, the results of previously performed functions, conversation history and passenger query.

[0276] Meanwhile, the sub-agent set (761) may include multiple sub-agents.

[0277] For example, the sub-agent set (761) may include a navigation agent for navigation services, a vehicle function agent for providing vehicle functions, a telephony agent for automated telephone answering services, and a Q&A agent for providing response services to queries.

[0278] Meanwhile, the sub-agent determined by the task arbitrator (771) can call the cloud AI model (769) or on-device AI model (767) assigned to it.

[0279] Meanwhile, the cloud AI model (769) or on-device AI model (767) may be a Large Language Model (LLM).

[0280] Meanwhile, a sub-agent within a sub-agent set (761) can call a cloud AI model (769) or an on-device AI model (767) assigned to it to obtain an inference result corresponding to a prompt from the model.

[0281] Meanwhile, a sub-agent in the sub-agent set (761) can call the knowledge database (762) to provide additional information based on the acquired inference result, obtain additional information from the knowledge database (762), specify the name of the function to be executed and the parameter value of the function, and determine the feedback phrase to be provided to the user.

[0282] Meanwhile, a sub-agent in the sub-agent set (761) can transmit to the workflow controller (772) a parsing result including additional information called from the knowledge database (762), the name of the function to be executed, the parameter value of the function, and a feedback phrase, which is generated by parsing the inference result received from the model.

[0283] Meanwhile, the knowledge database (762) can store additional information and information about functions. The information about functions may include the name of the function and the parameter values ​​of the function.

[0284] Meanwhile, the workflow controller (772) can generate control commands to perform the corresponding function based on the parsing results and can store the results of the conversation and function performed by the AI ​​model and sub-agent.

[0285] Meanwhile, the tool set / adapter set (764) can call an API corresponding to a control command generated by the workflow controller (772). The tool set / adapter set (764) can convert the control command into an execution command of an actual function or an API call command that the IVI system can understand and execute, and can execute the converted command.

[0286] Fig. 8c is an example of an internal block diagram of the server of Fig. 7.

[0287] Referring to the drawing, the server (400) may represent a device that trains an artificial neural network using a machine learning algorithm or uses a trained artificial neural network.

[0288] Here, the server (400) may be composed of multiple servers to perform distributed processing. Alternatively, the server (400) may be defined as a 5G network.

[0289] The server (400) may be included as part of the vehicle (200) and may perform at least some of the AI ​​processing together.

[0290] The server (400) may include a communication interface (410), memory (430), a learning processor (440), and a processor (470).

[0291] The communication interface (410) can transmit and receive data with the vehicle (200) or an external device.

[0292] The memory (430) may include a model storage unit (431). The model storage unit (431) may store a model (or artificial neural network, 531a) that is being learned or has been learned through the learning processor (440).

[0293] The learning processor (440) can train the artificial neural network (431a) using the training data.

[0294] For example, the learning processor (440) can execute a learning model. The learning model may include a cloud AI model (769) such as FIG. 7.

[0295] The learning model may be used while mounted on the server (400) of the artificial neural network, or may be used while mounted on an external device such as a vehicle (200).

[0296] The learning model may be implemented in hardware, software, or a combination of hardware and software. If part or all of the learning model is implemented in software, one or more instructions constituting the learning model may be stored in memory (430).

[0297] The processor (470) can infer a result value for new input data using a learning model and generate a response or control command based on the inferred result value.

[0298] Meanwhile, the processor (470) can control the learning processor (440) to execute the cloud AI model (769) based on requests from the signal processing device (170) in the vehicle (200), etc.

[0299] FIG. 9 is another example of a block diagram of a signal processing device according to an embodiment of the present disclosure.

[0300] Referring to the drawings, a signal processing device (170) according to an embodiment of the present disclosure includes a processor (175) that receives camera data outside the vehicle, camera data inside the vehicle, and sensor data inside the vehicle.

[0301] Meanwhile, the processor (175) according to an embodiment of the present disclosure performs object detection based on camera data outside the vehicle and monitors the state of a passenger, including a driver, based on camera data inside the vehicle.

[0302] Meanwhile, the processor (175) according to an embodiment of the present disclosure extracts multimodal context data corresponding to a driving context related to vehicle driving based on monitored passenger status information, detected object information, sensor data, and driver's driving pattern information stored in a database.

[0303] Meanwhile, the processor (175) according to an embodiment of the present disclosure calculates a risk level of vehicle driving based on the extracted multimodal context data and outputs a vehicle control signal based on the calculated risk level.

[0304] Accordingly, it becomes possible to perform stable vehicle control by rapidly and accurately determining the situation within the vehicle. In particular, stable vehicle control is possible by rapidly and accurately determining the situation within the vehicle based on multimodal context data.

[0305] Meanwhile, the processor (175) according to an embodiment of the present disclosure can obtain an inference result or a response result based on the extracted multimodal context data, calculate a risk level of vehicle driving based on the obtained inference result or response result, and output a vehicle control signal based on the calculated risk level. Accordingly, the situation inside the vehicle can be determined quickly and accurately to perform stable vehicle control.

[0306] Meanwhile, the processor (175) in the signal processing device (170) according to an embodiment of the present disclosure may include a recognizer (720), an insightor (750), and an illustrator (780).

[0307] Meanwhile, the recognizer (720) performs object detection based on camera data outside the vehicle and monitors the status of passengers, including the driver, based on camera data inside the vehicle.

[0308] Meanwhile, the recognizer (720) can recognize the vehicle situation based on sensing data received from the edge sensor group (SN). In this regard, the recognizer (720) may be referred to as a situation recognition unit. The vehicle situation at this time may include the vehicle external situation and the vehicle internal situation.

[0309] Meanwhile, the edge sensor group (SN) can correspond to a plurality of sensor devices (SN) of FIG. 4.

[0310] Meanwhile, the edge sensor group (SN) may include a microphone (112) which is an example of an input device (110), a vehicle interior camera (195i), a vehicle front camera (195), and a position sensor (198) that receives GPS data, etc.

[0311] Among these, the microphone (112) and the vehicle interior camera (195i) may be included in the vehicle interior sensors (710) of FIG. 7.

[0312] Meanwhile, the vehicle front camera (195) or position sensor (198) may be included in the vehicle external sensors (715).

[0313] Meanwhile, the recognizer (720) can recognize the situation inside the vehicle based on the vehicle interior sensing data (725) or the vehicle interior situation information (740a), and can generate the vehicle's situation information based on the recognized vehicle interior situation information.

[0314] The vehicle interior sensing data (725) at this time may be body sensing data of an occupant, including a driver or a passenger. For example, the body sensing data may include heart rate sensing data, respiration sensing data, etc.

[0315] Meanwhile, the vehicle interior situation information (740a) may include motion information and voice information of a passenger, including a driver or co-passenger, from the vehicle interior camera (195i) and microphone (112).

[0316] Meanwhile, the vehicle situation information (740 in FIG. 7) may include external vehicle situation information in addition to internal vehicle situation information.

[0317] Meanwhile, the vehicle's internal situation information may include occupant situation information. For example, the vehicle's internal situation information may include driver situation information and passenger situation information.

[0318] Meanwhile, the recognizer (720) can recognize external situation information of the vehicle based on external sensing data from the edge sensor group (SN), recognize internal situation information of the vehicle based on internal sensing data of the vehicle, or recognize situation information of the vehicle's occupant based on internal sensing data of the vehicle.

[0319] Meanwhile, the recognizer (720) can output external situation information of the vehicle based on internal vehicle sensing data or external vehicle sensing data, output internal situation information of the vehicle based on internal vehicle sensing data or external vehicle sensing data, or output situation information of the vehicle's occupant based on internal vehicle sensing data or external vehicle sensing data.

[0320] Meanwhile, the recognizer (720) can perform vehicle sensing data or preprocessing and calibration of vehicle sensing data, and can obtain vehicle situation information based on the preprocessed and calibrated sensing data.

[0321] Meanwhile, vehicle sensing data may include vehicle interior sensing data and vehicle exterior sensing data.

[0322] Meanwhile, vehicle situation information may include internal vehicle situation information and external vehicle situation information.

[0323] For example, the recognizer (720) can perform preprocessing and calibration of the vehicle interior sensing data, and can obtain information on the vehicle's interior situation based on the preprocessed and calibrated vehicle interior sensing data.

[0324] For example, the recognizer (720) can perform preprocessing and calibration on the vehicle external sensing data, and can obtain external situation information of the vehicle based on the preprocessed and calibrated vehicle external sensing data.

[0325] Meanwhile, the recognition device (720) can perform facial identification of the occupant based on the vehicle interior sensing data, recognize distraction, recognize gaze, position, gesture, recognize emotion, recognize whether alcohol or drugs have been taken, recognize drowsiness, or recognize information (748) about other actions, and obtain situational information of the vehicle occupant based thereon.

[0326] Meanwhile, the situational information of the occupant may include at least one of the following: a face identifier (Face ID) of the occupant inside the vehicle (200), distraction information, gaze information, position information, gesture information, emotion information, information on whether alcohol or drugs have been taken, drowsiness information, or other information regarding actions.

[0327] Meanwhile, the recognizer (720) can transmit the generated vehicle situation information to the insightor (750).

[0328] Meanwhile, the recognizer (720) receives camera data from outside the vehicle from the vehicle's external camera (195) and performs object detection based on the camera data from outside the vehicle.

[0329] For example, the recognizer (720) can receive vehicle front image data from the vehicle's external camera (195), perform segmentation on the vehicle front image data, and detect multiple objects (735).

[0330] Meanwhile, the recognizer (720) can receive location information from the location sensor (198 in FIG. 4) and transmit the location information to the insightor (750).

[0331] Meanwhile, the insightor (750) extracts multimodal context data corresponding to a driving context related to vehicle driving based on monitored passenger status information, detected object information, sensor data, and driver's driving pattern information stored in a database.

[0332] Meanwhile, the insightor (750) calculates the risk level of vehicle driving based on the extracted multimodal context data.

[0333] Meanwhile, the insightor (750) can obtain an inference result or a response result based on the extracted multimodal context data, and calculate the risk level of vehicle driving based on the obtained inference result or response result.

[0334] Meanwhile, the insightor (750) can generate an inference result or a response result based on the situation information of the vehicle received from the recognizer (720).

[0335] Meanwhile, the insightor (750) can generate an inference result or a response result based on the vehicle situation information received from the recognizer (720), and can generate or control a service to be executed based on the inference result or the response result.

[0336] Accordingly, the insightor (750) may be named a response result generation unit or a service generation unit.

[0337] Meanwhile, the insightor (750) may include a multimodal context engine (755) and a safety assist engine (757).

[0338] Meanwhile, the multimodal context engine (755) can generate multimodal context data based on the vehicle situation information received from the recognizer (720).

[0339] Specifically, the multimodal context engine (755) can generate multimodal context data corresponding to a driving context related to vehicle driving based on monitored passenger status information, detected object information, sensor data, and driver's driving pattern information stored in a database.

[0340] For example, multimodal context data may include at least one of text data, image data, or audio data describing a vehicle situation generated based on vehicle situation information.

[0341] Meanwhile, the multimodal context engine (755) can generate multimodal context data based on real-time context data (905) and feature data (907).

[0342] Meanwhile, real-time context data (905) may include monitored passenger status information, detected object information, and in-vehicle sensor data.

[0343] For example, real-time context data (905) may include monitored driver status information, detected object information, and in-vehicle sensor data.

[0344] As another example, real-time context data (905) may include monitored driver status information, monitored passenger status information, detected object information, and in-vehicle sensor data.

[0345] Meanwhile, the characteristic data (907) may include the characteristic data of the occupant.

[0346] For example, characteristic data (907) may include characteristic data of the driver.

[0347] As another example, characteristic data (907) may include characteristic data of the driver and characteristic data of the passenger.

[0348] Meanwhile, the safety assist engine (757) may include an on-device AI model (767), a safety extension module (910), an authority adjustment module (915), and a sensitivity adjustment module (920).

[0349] Meanwhile, the safety assist engine (757) can calculate the risk level of vehicle driving based on the extracted multimodal context data.

[0350] Meanwhile, the safety assist engine (757) can obtain an inference result or a response result based on the extracted multimodal context data, and calculate the risk level of vehicle driving based on the obtained inference result or response result.

[0351] Specifically, the on-device AI model (767) can perform inference or response based on the extracted multimodal context data and output the inference result or response result.

[0352] And, the safety assist engine (757) can calculate the risk level of vehicle driving based on the inference result or response result obtained from the on-device AI model (767).

[0353] Meanwhile, the safety assist engine (757) can classify the risk level of vehicle driving into multiple levels.

[0354] For example, the safety assist engine (757) can classify the risk level of vehicle driving into levels 1 through 4.

[0355] Meanwhile, the safety assist engine (757) can calculate the risk level of vehicle driving as the first level when the driver's driving ability information among the monitored passenger status information indicates that the driver can operate the vehicle.

[0356] Meanwhile, the safety assist engine (757) can calculate the risk level of vehicle driving to a second level higher than the first level when the driver's driving ability information indicates that the driver is capable of operating the vehicle or is distracted.

[0357] Meanwhile, the safety assist engine (757) can calculate the risk level of vehicle driving to a third level higher than the second level when the driver's driving ability information indicates that the driver is capable of driving or that the reaction is delayed.

[0358] Meanwhile, the safety assist engine (757) can calculate the risk level of vehicle driving as a fourth level higher than the third level when the driver's driving ability information indicates that the driver cannot operate the vehicle. Accordingly, it is possible to quickly and accurately determine the situation inside the vehicle and stably perform vehicle control.

[0359] Meanwhile, the safety assist engine (757) can calculate the risk level of vehicle driving as a first level if the occupant characteristic information does not include habit behavior information or is not personalized habit behavior information.

[0360] Meanwhile, the safety assist engine (757) can calculate the risk level of vehicle driving as a second level higher than the first level when the occupant characteristic information includes habitual behavior information that interferes with driving.

[0361] Meanwhile, the safety assist engine (757) can calculate the risk level of vehicle driving as a third level higher than the second level if the occupant characteristic information includes repeated dangerous behaviors and medical history or accident history.

[0362] Meanwhile, the safety assist engine (757) can calculate the risk level of vehicle driving as a fourth level higher than the third level when the occupant characteristic information includes risk information based on medical history. Accordingly, it is possible to quickly and accurately determine the situation inside the vehicle and stably perform vehicle control.

[0363] Meanwhile, the safety assist engine (757) can calculate the risk level of vehicle driving as a first level when the passenger's behavior information includes stable state information.

[0364] Meanwhile, the safety assist engine (757) can calculate the risk level of vehicle driving as a second level higher than the first level when the passenger's behavior information includes voice information at a level below the threshold.

[0365] Meanwhile, the safety assist engine (757) can calculate the risk level of vehicle driving as a third level higher than the second level when the passenger's behavior information includes voice information at a level higher than the repeated threshold.

[0366] Meanwhile, the safety assist engine (757) can calculate the risk level of vehicle driving as a fourth level higher than the third level if the passenger's behavior information includes actions that interfere with the driver's driving operation. Accordingly, it is possible to quickly and accurately determine the situation inside the vehicle and stably perform vehicle control.

[0367] Meanwhile, the safety assist engine (757) can control the authority adjustment, safety extension, and sensitivity adjustment for vehicle control based on the risk level of vehicle driving.

[0368] Meanwhile, the authority adjustment module (915) can calculate authority adjustment data for vehicle control based on the risk level of vehicle driving.

[0369] Meanwhile, the authority adjustment module (915) can control the authority adjustment for vehicle control to remain unchanged when the risk level of vehicle driving is at the first level.

[0370] Meanwhile, the authority adjustment module (915) can switch the authority adjustment for vehicle control to a standby state when the risk level of vehicle driving is a second level higher than the first level.

[0371] Meanwhile, the authority adjustment module (915) can ignore or supplement the operation input within the authority adjustment for vehicle control when the risk level of vehicle driving is a third level higher than the second level.

[0372] Meanwhile, the authority adjustment module (915) can control steering and acceleration for authority adjustment for vehicle control when the risk level of vehicle driving is at the highest level, level 4.

[0373] Meanwhile, the safety extension module (910) can calculate safety extension data for safety extension based on the risk level of vehicle driving.

[0374] Meanwhile, the safety extension module (910) can control the safety extension to remain as is when the risk level of vehicle driving is at the first level.

[0375] Meanwhile, the safety extension module (910) can switch the safety extension for vehicle control to a standby state when the risk level of vehicle driving is at a second level higher than the first level.

[0376] Meanwhile, the safety extension module (910) can increase the threshold value within the safety extension for vehicle control when the risk level of vehicle driving is a third level higher than the second level.

[0377] Meanwhile, the safety extension module (910) can set the threshold value within the safety extension to the maximum value when the risk level of vehicle driving is the highest level, which is the fourth level.

[0378] Meanwhile, the sensitivity adjustment module (920) can calculate sensitivity adjustment data for sensitivity adjustment based on the risk level of vehicle driving.

[0379] Meanwhile, the sensitivity adjustment module (920) can control the sensitivity adjustment for vehicle control to remain unchanged when the risk level of vehicle driving is at the first level.

[0380] Meanwhile, the sensitivity adjustment module (920) can switch the sensitivity adjustment for vehicle control to a standby state or an initial state when the risk level of vehicle driving is a second level higher than the first level.

[0381] Meanwhile, the sensitivity adjustment module (920) can increase the sensitivity of the sensitivity adjustment above the second level when the risk level of vehicle driving is a third level higher than the second level.

[0382] Meanwhile, the sensitivity adjustment module (920) can set the sensitivity of the sensitivity adjustment to the maximum value when the risk level of vehicle driving is the highest level, which is the fourth level.

[0383] Meanwhile, the illustrator (780) can execute a service, output service information, execute an application, or output application information based on the inference result or response result output from the insightor (750).

[0384] For example, the illustrator (780) can execute an autonomous driving service or a driving assistance service based on the response result output from the insightor (750).

[0385] Meanwhile, the illustrator (780) outputs a vehicle control signal based on the risk level calculated by the insightor (750). Accordingly, the situation inside the vehicle can be determined quickly and accurately, enabling stable vehicle control. In particular, the situation inside the vehicle can be determined quickly and accurately based on multimodal context data, enabling stable vehicle control.

[0386] Meanwhile, the illustrator (780) can generate a vehicle control signal based on authority adjustment data, safety extension data, and sensitivity adjustment data from the insightor (750).

[0387] For example, the illustrator (780) can generate vehicle control signals for controlling Lane Centering Systems (LCS), Adaptive Cruise Control (ACC), Lane Keeping Assist (LKA), Autonomous Emergency Braking (AEB), or Emergency Stop System (ESS), based on authority adjustment data, safety extension data, and sensitivity adjustment data from the insightor (750). Accordingly, the situation inside the vehicle can be determined quickly and accurately to perform stable vehicle control.

[0388] Meanwhile, the illustrator (780) can output a vehicle control signal, a warning message, or a guide message based on the calculated risk level and occupant profile information. Accordingly, the situation inside the vehicle can be determined quickly and accurately, and stable vehicle control can be performed.

[0389] FIGS. 10 to 15 are drawings referenced in the description of FIG. 9.

[0390] First, FIG. 10 is an example of a block diagram of a signal processing system according to an embodiment of the present disclosure.

[0391] Referring to the drawings, the signal processing system according to an embodiment of the present disclosure may be referred to as an IVEX (In Vehicle Experience) system or a vehicle control device.

[0392] Meanwhile, the signal processing system (900) according to an embodiment of the present disclosure includes an edge sensor group (SN) and a signal processing device (170) in a vehicle.

[0393] Meanwhile, the signal processing system according to the embodiment of the present disclosure may further include a server (400).

[0394] Meanwhile, the signal processing device (170) according to an embodiment of the present disclosure includes a recognizer (720), an insightor (750), and an illustrator (780).

[0395] Meanwhile, the edge sensor group (SN) can correspond to a plurality of sensor devices (SN) of FIG. 4.

[0396] Meanwhile, the edge sensor group (SN) may include an internal vehicle sensor (710) and an external vehicle sensor (715).

[0397] The vehicle interior sensor (710) may include a microphone (112 in FIG. 4), a vehicle interior camera (195i), etc.

[0398] The vehicle external sensor (715) may include a vehicle front camera (195), a position sensor (198), etc.

[0399] Meanwhile, the recognizer (720) can recognize the vehicle situation based on sensing data received from the edge sensor group (SN). In this regard, the recognizer (720) may be referred to as a situation recognition unit.

[0400] The vehicle situation at this time may include external vehicle conditions and internal vehicle conditions.

[0401] Meanwhile, the vehicle interior recognition unit (725) within the recognition unit (720) can recognize the vehicle interior situation based on vehicle interior sensing data from the vehicle interior sensor (710).

[0402] For example, the vehicle interior recognition device (725) within the recognition device (720) can monitor the condition of the occupants, including the driver, based on camera data inside the vehicle.

[0403] Meanwhile, the vehicle external recognition unit (730) within the recognition unit (720) can recognize the external vehicle situation based on the external vehicle sensing data from the external vehicle sensor (715).

[0404] For example, the vehicle external recognizer (730) within the recognizer (720) can perform object detection based on camera data from outside the vehicle.

[0405] Meanwhile, the integrated scene understandr (935) within the recognizer (720) can recognize vehicle situation information including an integrated scene inside and outside the vehicle based on vehicle inside situation information from the vehicle inside recognizer (725) and vehicle outside situation information from the vehicle outside recognizer (730).

[0406] Meanwhile, vehicle situation information may include vehicle interior situation information, vehicle exterior situation information, and vehicle occupant situation information.

[0407] Meanwhile, the integrated scene comprehension unit (935) within the recognition unit (720) can recognize vehicle situation information that further includes warning signs of the vehicle occupant.

[0408] Meanwhile, the insightor (750) can generate an inference result or a response result based on the situation information of the vehicle received from the recognizer (720).

[0409] Meanwhile, the insightor (750) can generate an inference result or a response result based on the vehicle situation information received from the recognizer (720), and can generate or control a service to be executed based on the inference result or the response result.

[0410] Accordingly, the insightor (750) may be named a response result generation unit or a service generation unit.

[0411] Meanwhile, the insightor (750) may include a multimodal context engine (755), a risk level determination unit (940), and a personalization module (945).

[0412] Meanwhile, the multimodal context engine (755) can generate multimodal context data based on the vehicle situation information received from the recognizer (720).

[0413] For example, multimodal context data may include at least one of text data, image data, or audio data describing a vehicle situation generated based on vehicle situation information.

[0414] Meanwhile, the multimodal context engine (755) may include a database (752) that stores driver's driving pattern information.

[0415] At this time, the driver's driving pattern information may include risk response information or risk response pattern information.

[0416] Meanwhile, the multimodal context engine (755) may include a database (752) that stores driver's driving pattern information and passenger's pattern information.

[0417] Meanwhile, the multimodal context engine (755) can extract multimodal context data corresponding to a driving context related to vehicle driving based on monitored passenger status information, detected object information, sensor data, and driver's driving pattern information stored in a database.

[0418] Meanwhile, the multimodal context engine (755) can transmit individual driving pattern information collected to the server (400).

[0419] Meanwhile, the risk level determination unit (940) can calculate the risk level of vehicle driving based on the extracted multimodal context data.

[0420] Meanwhile, the risk level determination unit (940) can obtain an inference result or a response result based on the extracted multimodal context data, and calculate the risk level of vehicle driving based on the obtained inference result or response result.

[0421] For example, the risk level determination unit (940) can classify the risk level of vehicle driving into levels 1 through 4.

[0422] Meanwhile, the risk level determination unit (940) can calculate driving ability evaluation data and emergency level evaluation data based on the driver's context data and driver's characteristic data.

[0423] Meanwhile, the risk level determination unit (940) can calculate driving ability evaluation data and emergency level evaluation data based on the driver's context data, the driver's characteristic data, the passenger's context data, and the passenger's characteristic data.

[0424] Meanwhile, the risk level determination unit (940) can calculate driving ability evaluation data and emergency level evaluation data based on the driver's context data and driver's characteristic data, and calculate the risk level of vehicle driving based on the driving ability evaluation data and emergency level evaluation data.

[0425] Meanwhile, the risk level determination unit (940) can calculate driving ability evaluation data and emergency level evaluation data based on the driver's context data, driver's characteristic data, passenger's context data, and passenger's characteristic data, and calculate the risk level of vehicle driving based on the driving ability evaluation data and emergency level evaluation data.

[0426] Meanwhile, the risk level determination unit (940) can classify the risk level of vehicle driving into multiple levels based on vehicle internal context data, vehicle external context data, and occupant profile information.

[0427] Meanwhile, the risk level determination unit (940) can classify the risk level of vehicle driving into multiple levels based on vehicle internal context data, vehicle external context data, occupant characteristic information, occupant behavior information, and driver's driving ability information.

[0428] Meanwhile, the risk level determination unit (940) can transmit risk level information of vehicle driving to the personalization module (945).

[0429] Meanwhile, the personalization module (945) can receive individual profile information from the server (400).

[0430] Meanwhile, the learning model (767) within the personalization module (945) can perform learning based on the calculated risk level information and occupant profile information, and output the result data of the learning.

[0431] Meanwhile, the personalization module (945) can compute authority adjustment data for vehicle control, safety extension data for safety extension, and sensitivity adjustment data for sensitivity adjustment based on the result data of the learning.

[0432] Meanwhile, the illustrator (780) can execute a service, output service information, execute an application, or output application information based on the inference result or response result output from the insightor (750).

[0433] Meanwhile, the illustrator (780) can output a vehicle control signal based on the risk level calculated by the insightor (750).

[0434] Meanwhile, the illustrator (780) can output a vehicle control signal based on the authority adjustment data for vehicle control, the safety extension data for safety extension, and the sensitivity adjustment data for sensitivity adjustment calculated in the insightor (750).

[0435] Meanwhile, the illustrator (780) may include a multimodal output encoder (781) including a modality manager (795), a line suggestion application (931), a warning application (932), a system control (930), etc.

[0436] Meanwhile, the multimodal output encoder (781) can encode the inference result or response result output from the insightor (750).

[0437] Meanwhile, the first proposal application (931), the warning application (932), and the system control (930) can each be executed based on encoded inference result data or response result data from the multimodal output encoder (781).

[0438] Meanwhile, the multimodal output encoder (781) can encode data corresponding to the calculated risk level.

[0439] Meanwhile, the first proposal application (931), the warning application (932), and the system control (930) can each be executed based on data corresponding to the calculated risk level from the multimodal output encoder (781).

[0440] For example, the system control (930) can output a vehicle control signal based on data corresponding to the calculated risk level.

[0441] Meanwhile, the pre-proposal application (931) can generate and output a guide message related to a pre-proposal for vehicle control.

[0442] For example, the pre-proposal application (931) can output a guide message for vehicle speed reduction, a guide message for lane change, etc.

[0443] Meanwhile, the pre-proposal application (931) can output a guide message related to a pre-proposal for vehicle control as Augmented Reality (AR) or video data (783) or Mixed Reality (MR) video data (784).

[0444] Meanwhile, the pre-proposal application (931) can output a guide message related to a pre-proposal for vehicle control as audio, such as voice.

[0445] Meanwhile, the warning application (932) can output a warning message audio, visual, or physically.

[0446] Meanwhile, the multimodal output encoder (781) can encode the result data of the learning from the personalization module (945).

[0447] Meanwhile, the first proposal application (931), the warning application (932), and the system control (930) can each be executed based on the result data of learning from the multimodal output encoder (781).

[0448] For example, the pre-proposal application (931) can output a guide message related to a pre-proposal for vehicle control based on the result data of the learning.

[0449] Meanwhile, the warning application (932) can output a warning message based on the result data of the learning.

[0450] Meanwhile, the system control (930) can output a vehicle control signal based on the result data of the learning.

[0451] Meanwhile, the multimodal output encoder (781) can encode authority adjustment data for vehicle control, safety extension data for safety extension, and sensitivity adjustment data for sensitivity adjustment.

[0452] Meanwhile, the first proposal application (931), the warning application (932), and the system control (930) can be executed based on the authority adjustment data for vehicle control, the safety extension data for safety extension, and the sensitivity adjustment data for sensitivity adjustment from the multimodal output encoder (781), respectively.

[0453] Meanwhile, the pre-proposal application (931) can output a guide message based on authority adjustment data for vehicle control, safety extension data for safety extension, and sensitivity adjustment data for sensitivity adjustment.

[0454] Meanwhile, the warning application (932) can output a warning message based on authority adjustment data for vehicle control, safety extension data for safety extension, and sensitivity adjustment data for sensitivity adjustment.

[0455] Meanwhile, the system control (930) can output a vehicle control signal based on authority adjustment data for vehicle control, safety extension data for safety extension, and sensitivity adjustment data for sensitivity adjustment.

[0456] FIG. 11a is a drawing referenced in the description of the operation of a processor according to one embodiment of the present disclosure.

[0457] Referring to the drawings, a processor (175) in a signal processing device (170) according to one embodiment of the present disclosure may include a risk level determination unit (940) and a level corresponding control module (950).

[0458] The processor (175) receives vehicle internal data from vehicle internal sensors (710), receives vehicle external data from vehicle external sensors (715), and can receive data around the vehicle through the data receiving unit (718).

[0459] Meanwhile, the processor (175) performs object detection based on camera data outside the vehicle and monitors the status of passengers, including the driver, based on camera data inside the vehicle.

[0460] Meanwhile, the processor (175) extracts multimodal context data corresponding to a driving context related to vehicle driving based on monitored passenger status information, detected object information, sensor data, and driver's driving pattern information stored in a database.

[0461] Meanwhile, the risk level determination unit (940) within the processor (175) calculates the risk level of vehicle driving based on the extracted multimodal context data.

[0462] Meanwhile, the level-corresponding control module (950) within the processor (175) outputs a vehicle control signal based on the calculated risk level. Accordingly, the situation inside the vehicle can be determined quickly and accurately, and stable vehicle control can be performed.

[0463] Meanwhile, the risk level determination unit (940) within the processor (175) can obtain an inference result or a response result based on the extracted multimodal context data, and calculate the risk level of vehicle driving based on the obtained inference result or response result.

[0464] That is, the processor (175) can generate a prompt for calculating the risk level of vehicle driving based on multimodal context data, calculate the risk level of vehicle driving based on the inference result or response result obtained based on the prompt, and output a vehicle control signal based on the calculated risk level.

[0465] Meanwhile, a signal processing device (170) according to one embodiment of the present disclosure may further include a neural processor (179) that executes inference or a response based on a prompt.

[0466] Meanwhile, the processor (175) can generate a prompt for calculating the risk level of vehicle driving based on multimodal context data, calculate the risk level of vehicle driving based on the inference result or response result executed by the neural processor (179) based on the prompt, and output a vehicle control signal based on the calculated risk level. Accordingly, the situation inside the vehicle can be determined quickly and accurately to perform stable vehicle control.

[0467] Meanwhile, the processor (175) can run a hypervisor (505 in FIG. 5) and run a driving control virtualization machine (520 in FIG. 5) on the hypervisor (505).

[0468] Meanwhile, the driving control virtualization machine (520) receives camera data outside the vehicle, camera data inside the vehicle, and sensor data inside the vehicle through the shared memory (508) within the hypervisor (505), and can execute a driving control service or a driving control application based on the camera data outside the vehicle, camera data inside the vehicle, and sensor data inside the vehicle. Accordingly, the situation inside the vehicle can be determined quickly and accurately, and stable vehicle control can be performed.

[0469] Meanwhile, the risk level determination unit (940) within the processor (175) can classify the risk level of vehicle driving into multiple levels based on vehicle internal context data, vehicle external context data, and occupant profile information.

[0470] Meanwhile, the processor (175) can classify the risk level of vehicle driving into multiple levels based on vehicle internal context data, vehicle external context data, occupant characteristic information, occupant behavior information, and driver's driving ability information.

[0471] Meanwhile, the risk level determination unit (940) within the processor (175) can classify the risk level of vehicle driving into multiple levels.

[0472] Specifically, the risk level determination unit (940) within the processor (175) can classify the risk level of vehicle driving into levels 1 through 4.

[0473] For example, the risk level determination unit (940) within the processor (175) can calculate the risk level of vehicle driving as a first level if the driver's driving ability information among the monitored passenger status information indicates that the driver can operate the vehicle.

[0474] As another example, the risk level determination unit (940) within the processor (175) can calculate the risk level of vehicle driving as a second level higher than the first level when the driver's driving ability information indicates that the driver is capable of operating the vehicle or is in a distracted state.

[0475] As another example, the risk level determination unit (940) within the processor (175) can calculate the risk level of vehicle driving as a third level higher than the second level when the driver's driving ability information indicates that the driver is capable of driving or is in a delayed response state.

[0476] As another example, the risk level determination unit (940) within the processor (175) can calculate the risk level of vehicle driving as a fourth level higher than the third level when the driver's driving ability information indicates that the driver cannot operate the vehicle. Accordingly, the situation inside the vehicle can be determined quickly and accurately, thereby enabling stable vehicle control.

[0477] Meanwhile, the risk level determination unit (940) within the processor (175) can calculate the risk level of vehicle driving as a first level if the occupant characteristic information does not include habit behavior information or is not personalized habit behavior information.

[0478] Meanwhile, the risk level determination unit (940) within the processor (175) can calculate the risk level of vehicle driving as a second level higher than the first level when the occupant characteristic information includes habitual behavior information that interferes with driving.

[0479] For example, the risk level determination unit (940) within the processor (175) can calculate the risk level of vehicle driving as a second level higher than the first level if the occupant characteristic information includes habitual behavior information that interferes with driving.

[0480] Meanwhile, the risk level determination unit (940) within the processor (175) can calculate the risk level of vehicle driving as a third level higher than the second level if the occupant characteristic information includes repeated dangerous behaviors and medical history or accident history.

[0481] For example, the risk level determination unit (940) within the processor (175) can calculate the risk level of vehicle driving as a third level higher than the second level if the occupant characteristic information includes repeated dangerous behaviors and medical history or accident history that interfere with driving.

[0482] Meanwhile, the risk level determination unit (940) within the processor (175) can calculate the risk level of vehicle driving as a fourth level higher than the third level when the occupant characteristic information includes risk information based on medical history. Accordingly, the situation inside the vehicle can be determined quickly and accurately, enabling stable vehicle control.

[0483] Meanwhile, the risk level determination unit (940) within the processor (175) can calculate the risk level of vehicle driving as a first level when the passenger's behavior information includes stable state information.

[0484] Meanwhile, the risk level determination unit (940) within the processor (175) can calculate the risk level of vehicle driving as a second level higher than the first level when the passenger's behavior information includes voice information at a level below the threshold.

[0485] For example, the risk level determination unit (940) within the processor (175) can calculate the risk level of vehicle driving as a second level higher than the first level when the passenger's behavior information includes voice stimuli that distract concentration on driving, such as voice information at a level below a reference value.

[0486] Meanwhile, the risk level determination unit (940) within the processor (175) can calculate the risk level of vehicle driving as a third level higher than the second level when the passenger's behavior information includes voice information at a level higher than a repeated threshold.

[0487] For example, the risk level determination unit (940) within the processor (175) can calculate the risk level of vehicle driving as a third level higher than the second level if the passenger's behavior information includes voice stimuli that interfere with driving, such as voice information at a level higher than a threshold.

[0488] Meanwhile, the risk level determination unit (940) within the processor (175) can calculate the risk level of vehicle driving as a fourth level higher than the third level if the passenger's behavior information includes actions that interfere with the driver's driving operation. Accordingly, the situation inside the vehicle can be determined quickly and accurately, enabling stable vehicle control.

[0489] Meanwhile, the risk level determination unit (940) within the processor (175) can calculate driving ability evaluation data and emergency level evaluation data based on the driver's context data and driver's characteristic data, and calculate the risk level of vehicle driving based on the driving ability evaluation data and emergency level evaluation data.

[0490] Meanwhile, the risk level determination unit (940) within the processor (175) can calculate driving ability evaluation data and emergency level evaluation data based on the driver's context data, the driver's characteristic data, the passenger's context data, and the passenger's characteristic data.

[0491] Meanwhile, the level-corresponding control module (950) within the processor (175) can control authority adjustment, safety extension, and sensitivity adjustment for vehicle control based on the risk level of vehicle driving.

[0492] Meanwhile, the level-corresponding control module (950) within the processor (175) can calculate authority adjustment data for vehicle control, safety extension data for safety extension, and sensitivity adjustment data for sensitivity adjustment based on the risk level of vehicle driving.

[0493] Meanwhile, the level-corresponding control module (950) within the processor (175) can generate a vehicle control signal based on authority adjustment data, safety extension data, and sensitivity adjustment data. Accordingly, the situation inside the vehicle can be determined quickly and accurately, and stable vehicle control can be performed.

[0494] Meanwhile, the level-corresponding control module (950) within the processor (175) can control the authority adjustment, safety extension, and sensitivity adjustment for vehicle control to remain unchanged when the risk level of vehicle driving is at the first level.

[0495] Meanwhile, the level-corresponding control module (950) within the processor (175) can switch the authority adjustment and safety expansion for vehicle control to a standby state and increase the sensitivity of the sensitivity adjustment above the first level when the risk level of vehicle driving is at a second level higher than the first level. Accordingly, it is possible to quickly and accurately determine the situation inside the vehicle and stably perform vehicle control.

[0496] Meanwhile, the level-corresponding control module (950) within the processor (175) can ignore or supplement the operation input within the authority adjustment for vehicle control, increase the threshold value within the safety extension, or raise the sensitivity of the sensitivity adjustment above the second level when the risk level of vehicle driving is at a third level higher than the second level. Accordingly, the situation inside the vehicle can be determined quickly and accurately, and stable vehicle control can be performed.

[0497] Meanwhile, the level-corresponding control module (950) within the processor (175) can control steering and acceleration to adjust authority for vehicle control when the risk level of vehicle driving is at the highest level (Level 4), set the threshold value within the safety extension to the maximum value, and set the sensitivity of the sensitivity adjustment to the maximum value. Accordingly, the situation inside the vehicle can be determined quickly and accurately to perform stable vehicle control.

[0498] Meanwhile, the processor (175) can perform learning based on the calculated risk level and occupant profile information, and output a vehicle control signal, a warning message, or a guide message based on the result data of the learning. Accordingly, the situation inside the vehicle can be determined quickly and accurately to perform stable vehicle control.

[0499] Figure 11b is a drawing referenced in the description of the integrated scene analyzer of Figure 10.

[0500] Referring to the drawing, the integrated scene understandr (935) may include a vehicle interior context understandr (960) and a vehicle exterior context understandr (970).

[0501] The vehicle internal context understandr (960) may include a driver state information extractor (962) and a driver behavior information extractor (964).

[0502] For example, the driver state information extractor (962) can extract driver state information such as body-skeleton information, hands-on information, distraction information, behavior information, and health information through camera data from an internal camera (195i) and sensing data from a bio-sensor device.

[0503] Meanwhile, the driver behavior information extractor (964) can extract driver behavior information such as spilling, child crying, driver disturbance, and heart attack based on various information or data from the driver state information extractor (962).

[0504] The vehicle external context understandr (970) may include a setting external information extractor (972) and a real-time external information extractor (974).

[0505] Meanwhile, the external information extractor (972) may be named the Operational Design Domain (ODD).

[0506] The external information extractor (972) can extract surrounding environment information including a vehicle or a vulnerable road user (VRU), extract road context data, or extract highway context data based on communication data from the data receiver (718) and external vehicle sensing data from the external vehicle sensors (715).

[0507] The real-time external information extractor (974) can extract real-time external information such as traffic jam information, crowded vulnerable road user (Crowded VRU) information, and uncongested highway information based on information or data from the set external information extractor (972).

[0508] Meanwhile, the risk level determination unit (940) may include a vehicle interior risk determination unit (952) and a driving risk determination unit (954).

[0509] The vehicle interior risk determination unit (952) can calculate the risk level inside the vehicle based on vehicle interior context data from the vehicle interior context understandr (960). Meanwhile, the risk level inside the vehicle can be transmitted to the safety assist engine (757).

[0510] Meanwhile, the driving risk determination unit (954) can calculate the risk level outside the vehicle based on the vehicle external context data from the vehicle external context understandr (970). Meanwhile, the risk level outside the vehicle can be transmitted to the safety assist engine (757).

[0511] Meanwhile, the safety assist engine (757) may include the level corresponding control module (950) of FIG. 11a.

[0512] Meanwhile, the authority adjustment module (915) within the safety assistance engine (757) can adjust driving authority based on the risk level of driving the vehicle, including the risk level inside the vehicle and the risk level of driving the vehicle.

[0513] For example, the authority adjustment module (915) within the safety assist engine (757) can partially restrict or supplement the vehicle operation authority when the driver's ability to operate in a dangerous situation is below a standard level.

[0514] As another example, the authority adjustment module (915) within the safety assist engine (757) can control the driver's ability to operate in a dangerous situation to return to its original state by reducing the intervention of the driving authority by the system when the driver's ability to operate returns to a level exceeding the threshold. Accordingly, the situation inside the vehicle can be determined quickly and accurately, and stable vehicle control can be performed.

[0515] Meanwhile, the safety extension module (910) within the safety assistance engine (757) can perform safety extension control based on the risk level of vehicle driving, including the risk level inside the vehicle and the risk level of vehicle driving.

[0516] For example, the safety extension module (910) within the safety assist engine (757) can activate the system or adjust functional limitations regarding the allowable operating range in response to the risk level. Accordingly, the situation inside the vehicle can be determined quickly and accurately, enabling stable vehicle control.

[0517] Meanwhile, the sensitivity adjustment module (920) within the safety assist engine (757) can perform sensitivity adjustment based on the risk level of the vehicle, including the risk level inside the vehicle and the risk level of the vehicle driving.

[0518] For example, the sensitivity adjustment module (920) within the safety assist engine (757) can control the sensitivity to be increased when the risk level exceeds a set value, and control the sensitivity to return to its original state when the risk level returns to below the set value. Accordingly, the situation inside the vehicle can be determined quickly and accurately, and stable vehicle control can be performed.

[0519] FIG. 12 is a diagram illustrating a vehicle driving assistance function corresponding to a driving state.

[0520] Referring to the drawing, the operating state can be classified into normal mode, first emergency mode, second emergency mode, and emergency release mode.

[0521] Meanwhile, Normal Mode or Emergency Release Mode can correspond to Level 1 of the risk levels of vehicle driving.

[0522] Meanwhile, the first emergency mode can respond to the second or third level of the risk level of vehicle driving.

[0523] Meanwhile, the second emergency mode can respond to Level 4, which is the highest level among the risk levels of vehicle driving.

[0524] Meanwhile, in the case of the first emergency mode, the safety extension module (910) within the processor (175) controls the lane centering system (LCS) and the adaptive cruise control (ACC) to gradually increase the maximum allowable lateral acceleration or gradually increase the operating limit speed, and the emergency braking system (ESS) can be kept in a ready state.

[0525] Meanwhile, in the case of the first emergency mode, the authority adjustment module (915) within the processor (175) can control the lane centering system (LCS) and lane keeping assist (LKA) to gradually increase the steering torque, or control the adaptive cruise control (ACC) to gradually increase the steering torque, and keep the emergency braking system (ESS) in a ready state.

[0526] Meanwhile, in the case of the first emergency mode, the sensitivity adjustment module (920) within the processor (175) can control the lane centering system (LCS) and lane keeping assist (LKA) to gradually lower the lane width limit, or control the adaptive cruise control (ACC) to gradually increase the distance to the vehicle ahead, and the emergency braking system (ESS) can be kept in a ready state.

[0527] Meanwhile, in the case of the second emergency mode, the safety extension module (910) within the processor (175) can control the lane centering system (LCS) and the adaptive cruise control (ACC) to increase the maximum allowable lateral acceleration or increase the operating speed limit, and can control the emergency braking system (ESS) to operate immediately. Accordingly, the situation inside the vehicle can be determined quickly and accurately to perform stable vehicle control.

[0528] Meanwhile, in the case of the second emergency mode, the authority adjustment module (915) within the processor (175) controls the Lane Centering System (LCS) and Lane Keeping Assist (LKA) to increase the steering override torque or to maintain the lane even in the event of a turn signal malfunction, controls the Adaptive Cruise Control (ACC) to increase the accelerator pedal override torque or to respond to a dangerous object ahead even in the event of a brake pedal malfunction or to prevent an Overtaking Function (OTF) even in the event of a turn signal malfunction, and controls the Emergency Braking System (ESS) so that the Emergency Braking System operation is performed even if the brake pedal, accelerator pedal, or steering override torque increases. Accordingly, the situation inside the vehicle can be determined quickly and accurately, and stable vehicle control can be performed.

[0529] Meanwhile, in the case of the second emergency mode, the sensitivity adjustment module (920) within the processor (175) can control the Lane Centering System (LCS) to increase the target distance level for object tracking or to decrease the lane width limit, control the Lane Keeping Assist (LKA) to increase the distance between the vehicle and the lane, control the Adaptive Cruise Control (ACC) to change the Time to Collision (TTC) or change the ACC Early Warning, or to decrease the target speed depending on the curve or road type, and control the Automatic Emergency Braking (AEB) to change the Time to Collision (TTC) or change the ACC Early Warning. Accordingly, the situation inside the vehicle can be determined quickly and accurately, and stable vehicle control can be performed.

[0530] Meanwhile, in the case of emergency release mode, the safety extension module (910) within the processor (175) controls the lane centering system (LCS) and adaptive cruise control (ACC) to gradually restore or reduce the maximum allowable lateral acceleration or gradually restore or reduce the operating limit speed, and the emergency braking system (ESS) can be immediately released.

[0531] Meanwhile, in the case of emergency release mode, the authority adjustment module (915) within the processor (175) can control the lane centering system (LCS), lane keeping assist (LKA), and adaptive cruise control (ACC) to gradually restore or reduce the brake pedal, accelerator pedal, or steering override torque, or restore the turn signal dependency function, and the emergency braking system (ESS) can be immediately released.

[0532] Meanwhile, in the case of emergency release mode, the sensitivity adjustment module (920) within the processor (175) can control the lane centering system (LCS) and lane keeping assist (LKA) to gradually restore or reduce the target distance, lane width limit, parameters, etc., and can control the adaptive cruise control (ACC) to gradually restore or reduce the Time to Collision (TTC).

[0533] FIG. 13a is a drawing referenced in the description of the vehicle's operation in normal mode of FIG. 12.

[0534] Referring to the drawing, the vehicle (200) can operate in normal mode at points Pma, Pmb, and Pmc.

[0535] Meanwhile, the processor (175) can disable the lane centering system (LCS) and adaptive cruise control (ACC) in normal mode.

[0536] For example, in normal mode, when the lane centering system (LCS) is deactivated, the direction of travel of the vehicle (200) is changed based on the driver's steering wheel operation input.

[0537] As another example, in normal mode, when the lane centering system (LCS) is deactivated, the turn signal is turned on or off based on the driver's turn signal operation input.

[0538] In the drawing, based on the driver's steering wheel operation input in normal mode, the vehicle (200) is shown passing through the center line of the lane between the lanes, such as at point Pmb.

[0539] Meanwhile, in Normal mode, if the driver's steering input corresponds to a leftward movement input, the vehicle proceeds to the left of the lane rather than the center line, as at the Pmc point.

[0540] Meanwhile, in Normal mode, if the driver's steering input corresponds to a centering input, the vehicle will continue to deviate to the left of the lane rather than the center line, as seen at the Pmd point.

[0541] FIG. 13b is a drawing referenced in the description of the operation of the vehicle in the second emergency mode of FIG. 12.

[0542] Referring to the drawing, the vehicle (200) can operate in normal mode at point Pna, and then operate in a second emergency mode at points Pnb, Pnc, and Pnd.

[0543] For example, the processor (175) can disable the lane centering system (LCS) and adaptive cruise control (ACC) in normal mode.

[0544] Meanwhile, after the Pna point, if the risk level of vehicle driving changes, for example, if an emergency situation occurs, the processor (175) can switch to a second emergency mode.

[0545] Accordingly, the processor (175) can activate the lane centering system (LCS) and adaptive cruise control (ACC) in the second emergency mode.

[0546] At this time, the processor (175) can control the lane centering system (LCS) and adaptive cruise control (ACC) to remain active and not deactivated despite the driver's steering wheel operation input or the driver's turn signal operation input.

[0547] For example, the processor (175) can ignore the input of the accelerator pedal based on the lane centering system (LCS) and adaptive cruise control (ACC) when there is an unintended vehicle operation input at the Pnc point in the second emergency mode, for example, when there is an input of the accelerator pedal.

[0548] Accordingly, as at point Pnd, the vehicle (200) can maintain its lane so that it passes the center line, while maintaining the distance from the vehicle ahead.

[0549] As another example, the processor (175) can ignore the driver's steering wheel operation input in the second emergency mode, based on the lane centering system (LCS) and adaptive cruise control (ACC). Accordingly, the vehicle (200) can maintain its lane so as to pass the center line.

[0550] As another example, the processor (175) can ignore the driver's turn signal operation input in the second emergency mode based on the lane centering system (LCS) and adaptive cruise control (ACC). Accordingly, the vehicle (200) can maintain the lane so as to pass the center line.

[0551] FIG. 13c is a drawing illustrating an example of various behaviors of a passenger inside a vehicle.

[0552] Referring to the drawing, the processor (175) can monitor various behaviors of the occupants in the vehicle and calculate the risk level of the vehicle's driving based on the monitored behavior information of the occupants.

[0553] Meanwhile, the processor (175) monitors various behaviors of the occupants in the vehicle and, based on the monitored behavior information of the occupants, can classify them into use cases and non-use cases.

[0554] Meanwhile, the processor (175) may not output a vehicle control signal when the behavior information of the occupant in the vehicle corresponds to a use case.

[0555] Meanwhile, the processor (175) can disable the Lane Centering System (LCS), Lane Keeping Assist (LKA), Adaptive Cruise Control (ACC), or Emergency Braking System (ESS) when the behavior information of the occupant in the vehicle corresponds to a use case.

[0556] Meanwhile, the processor (175) may not output a vehicle control signal when the behavior information of the occupant in the vehicle corresponds to a use case.

[0557] Meanwhile, the processor (175) can activate at least one of the lane centering system (LCS), lane keeping assist (LKA), adaptive cruise control (ACC), or emergency braking system (ESS) when the behavior information of the occupant in the vehicle corresponds to a non-use case.

[0558] Meanwhile, the processor (175) can monitor various behaviors of the occupants in the vehicle and, based on the monitored behavior information of the occupants, classify the risk level of the vehicle driving into levels 1 through 4.

[0559] At this time, the first level can correspond to a non-use case, and the second to fourth levels can correspond to a use case.

[0560] That is, the processor (175) monitors various behaviors of the occupants in the vehicle and, based on the monitored behavior information of the occupants, can distinguish between non-use cases and use cases.

[0561] Meanwhile, the processor (175) can monitor various behaviors of the occupants in the vehicle and, based on the monitored behavior information of the occupants, control the execution of any one of a normal mode, a first emergency mode, a second emergency mode, or an emergency release mode.

[0562] At this time, the normal mode and the emergency release mode can correspond to non-use cases, and the first emergency mode and the second emergency mode can correspond to use cases.

[0563] For example, the processor (175) can be classified as a non-use case in the case of smoking, talking, laughing, singing, navigation, object catching, calling, and phone usage.

[0564] Meanwhile, the processor (175) can classify the case of drinking a warm beverage during drinking as a non-use case.

[0565] Meanwhile, the processor (175) can classify the case as a use case when visual distraction occurs due to liquid spread or movement while drinking.

[0566] Meanwhile, the processor (175) can classify the case of eating warm food as a non-use case.

[0567] Meanwhile, the processor (175) can classify the case as a use case when visual distraction occurs due to food dispersion or movement during eating.

[0568] Meanwhile, the processor (175) can classify the case where eye scratching occurs as a use case.

[0569] Meanwhile, the processor (175) can classify cases where sneezing occurs as non-use cases.

[0570] Meanwhile, the processor (175) can classify the case of visual obstruction caused by sneezing (Sneezing while not looking forward) as a use case.

[0571] Meanwhile, the processor (175) can classify cases where yawning occurs as non-use cases.

[0572] Meanwhile, the processor (175) can classify the case of yawning while closing eyes as a use case.

[0573] FIG. 13d is a drawing illustrating different examples of various behaviors of occupants inside a vehicle.

[0574] Referring to the drawing, the processor (175) can be classified into non-use cases such as object searching, grooming, object dropping, and object placing.

[0575] Meanwhile, the processor (175) can classify cases of emergency and driver disturbance as use cases.

[0576] Meanwhile, in the case of the use case of FIG. 13c to FIG. 13d, the processor (175) can activate at least one of the lane centering system (LCS), lane keeping assist (LKA), adaptive cruise control (ACC), or emergency braking system (ESS).

[0577] Meanwhile, in the case of the non-use cases of FIGS. 13c to 13d, the processor (175) may not activate the lane centering system (LCS), lane keeping assist (LKA), adaptive cruise control (ACC), or emergency braking system (ESS), or may not operate them even if the conditions are met.

[0578] FIG. 14 is a drawing referenced in the description of multiple driving safety levels.

[0579] Referring to the drawing, the risk level determination unit (940) within the processor (175) can calculate the risk level of vehicle driving based on vehicle internal context data (941), vehicle external context data (942), and personal profile information (945).

[0580] Meanwhile, the risk level determination unit (940) within the processor (175) can classify the risk level of vehicle driving into a first level to a fourth level (951, 952, 953, 954).

[0581] Meanwhile, the level corresponding control module (950) can control the performance of corresponding operations based on the first to fourth levels (951, 952, 953, 954).

[0582] Meanwhile, the level corresponding control module (950) can control that, after performing an operation corresponding to the first level (951), if the risk level increases, an operation corresponding to the second level (952) is performed, and after performing an operation corresponding to the second level (952), if the risk level increases, an operation corresponding to the third level (953) is performed, and after performing an operation corresponding to the third level (953), if the risk level increases, an operation corresponding to the fourth level (954) is performed.

[0583] Meanwhile, the level-corresponding control module (950) can control the execution of an operation corresponding to the first level (951) when the risk level decreases after the operation corresponding to the second level (952) is performed.

[0584] Meanwhile, the level-corresponding control module (950) can control the execution of an operation corresponding to the first level (951) or the second level (952) when the risk level decreases after the operation corresponding to the third level (953) is performed.

[0585] Meanwhile, the level-corresponding control module (950) can control the execution of an operation corresponding to the first level (951) when the risk level decreases after the operation corresponding to the fourth level (954) is performed.

[0586] FIG. 15 is a drawing referenced in the description of the operation of the level-corresponding control module of FIG. 14.

[0587] Referring to the drawing, the processor (175) can classify the risk level of vehicle driving into multiple levels based on vehicle internal context data, vehicle external context data, personal profile information, passenger behavior information, and driver driving ability information.

[0588] For example, the processor (175) can classify the risk level of vehicle driving into levels 1 through 4.

[0589] At this time, Level 1 indicates a normal state, Level 2 indicates a need for caution, Level 3 indicates imminent danger, and Level 4 may indicate a stage requiring immediate intervention.

[0590] Meanwhile, Levels 2 and 3 can correspond to the prodromal symptom stage.

[0591] For example, the processor (175) can determine the risk level of vehicle driving to be the first level when the vehicle internal behavior is normal based on the vehicle internal context data and the vehicle driving environment is stable based on the vehicle external context data.

[0592] Meanwhile, the processor (175) can determine the risk level of vehicle driving to be a second level when a single sign appears based on the vehicle internal context data or the vehicle external context data.

[0593] Meanwhile, the processor (175) can determine the risk level of vehicle driving to be a third level when a complex sign appears based on vehicle internal context data or vehicle external context data.

[0594] Meanwhile, the processor (175) can determine the risk level of vehicle driving to be a fourth level when a dangerous situation occurs due to vehicle internal context data or vehicle external context data.

[0595] For example, the processor (175) can determine the risk level of vehicle driving as the first level if there is no repetitive behavior and the habit is stable based on personal profile information.

[0596] Meanwhile, the processor (175) can determine the risk level of vehicle driving to be a second level if habitual behavior exists based on personal profile information.

[0597] Meanwhile, the processor (175) can determine the risk level of vehicle driving to be a third level if there is repeated risk behavior based on personal profile information and there is a history of past personal medical history or accidents.

[0598] Meanwhile, the processor (175) can determine the risk level of vehicle driving to be a fourth level if it is confident of the risk based on the behavioral results and medical history based on personal profile information.

[0599] For example, the processor (175) can determine the risk level of vehicle driving as a first level when the passenger's behavior is in a stable state (e.g., wearing a seatbelt) based on passenger behavior information.

[0600] Meanwhile, the processor (175) can determine the risk level of driving the vehicle to be a second level if there is a mild voice stimulus (e.g., talking) based on passenger behavior information.

[0601] Meanwhile, the processor (175) can determine the risk level of driving the vehicle to a third level if the driver's attention is distracted due to shouting or physical interference based on passenger behavior information.

[0602] Meanwhile, the processor (175) can determine the risk level of vehicle driving to be a fourth level if there is strong interference or panic inducement or interference with the driver's hand operation based on passenger behavior information.

[0603] For example, the processor (175) can determine the risk level of vehicle driving to be a first level if the driver is capable of operating the vehicle based on driver driving ability information.

[0604] Meanwhile, the processor (175) can determine the risk level of vehicle driving to be a second level based on driver driving ability information, when the driver is able to operate the vehicle but is slightly distracted.

[0605] Meanwhile, the processor (175) can determine the risk level of vehicle driving to be a third level based on driver driving ability information, if the driver is able to operate but there is a delay in reaction or increased stress.

[0606] Meanwhile, the processor (175) can determine the risk level of vehicle driving to be the fourth level if the driver is unable to operate the vehicle, is unresponsive, or is in a panic state based on driver driving ability information.

[0607] Meanwhile, the processor (175) can determine the risk level of the vehicle driving as the first level when driving while looking ahead, when there is no abnormality in the vital signs, or when stopping next to the U-turn line.

[0608] Meanwhile, the processor (175) can determine the risk level of driving the vehicle to be a second level if the vehicle is driven with one hand while holding a drink, the heart rate increases, or the vehicle stops after crossing the U-turn line.

[0609] Meanwhile, the processor (175) can determine the risk level of the vehicle driving to be the third level if there is a history of accidents such as sudden braking while driving with one hand holding a drink, or if there is a history of heart rate increase, irregular breathing, or past cardiac arrest, or if there is a stop beyond the U-turn line and an oncoming motorcycle.

[0610] Meanwhile, the processor (175) can determine the risk level of driving the vehicle to be the fourth level if, while startled by spilling a drink, the gaze is directed downward rather than forward and the hand is removed from the steering wheel, or if there is cardiac arrest and steering, or if there is no response or misoperation of the pedal, or if the vehicle stops beyond the U-turn line and an oncoming vehicle is coming.

[0611] Meanwhile, when the risk level of vehicle driving is the first level, the authority adjustment module (915), safety extension module (910), and sensitivity adjustment module (920) within the processor (175) can all remain in a non-intervention state.

[0612] Meanwhile, when the risk level of vehicle driving is at level 2, the authority adjustment module (915) within the processor (175) can learn and monitor operation patterns in a standby state, the safety extension module (910) can prepare to relax functional restrictions in a standby state, and the sensitivity adjustment module (920) can perform simple visual or auditory warnings by enhancing sensitivity.

[0613] Meanwhile, when the risk level of vehicle driving is at the third level, the authority adjustment module (915) within the processor (175) may ignore or begin to correct some misoperations such as blinking errors, the safety extension module (910) may partially relax functional restrictions or gradually increase the threshold, and the sensitivity adjustment module (920) may further enhance sensitivity, perform multiple warnings, perform adaptive warnings based on personalization, actively perform warnings in case of behavioral risk, or mitigate warnings in case of health risk.

[0614] Meanwhile, when the risk level of vehicle driving is at level 4, the authority adjustment module (915) within the processor (175) can immediately intervene to perform steering control or acceleration control, the safety extension module (910) can completely release functional restrictions or set the threshold to the maximum value, and the sensitivity adjustment module (920) can maintain heightened sensitivity or suppress warnings to prevent confusion. Accordingly, the situation inside the vehicle can be determined quickly and accurately to perform stable vehicle control.

[0615] FIG. 16 is an example of a flowchart illustrating a method of operation of a signal processing device according to an embodiment of the present disclosure.

[0616] Referring to the drawings, the processor (175) in the signal processing device (170) according to an embodiment of the present disclosure can obtain context data of the driver inside the vehicle based on the vehicle's internal sensor device or internal camera (195i), etc. (S1402).

[0617] Meanwhile, the processor (175) can obtain driver profile information or driver characteristic data based on the database (762) (S1403).

[0618] Meanwhile, the processor (175) can obtain context data of the occupant based on the vehicle's internal sensor device or internal camera (195i), etc. (S1404).

[0619] Meanwhile, the processor (175) can obtain vehicle external context data based on vehicle external sensor devices, etc. (S1405).

[0620] Meanwhile, the processor (175) can compute driving ability evaluation data based on the driver's context data, driver profile information or driver's characteristic data, passenger's context data, and vehicle external context data (S1410).

[0621] Next, the processor (175) can calculate emergency level evaluation data based on driving ability evaluation data (S1420).

[0622] Alternatively, unlike the drawing, the processor (175) can calculate emergency level evaluation data related to the safety level of vehicle driving based on the driver's context data, driver profile information or driver characteristic data, passenger's context data, and vehicle external context data.

[0623] Meanwhile, the processor (175) determines whether the emergency level evaluation data related to the safety level of vehicle driving is at the first level (S1422), and if so, can control it to remain as is without separate interference (S1420).

[0624] If not applicable in step 1422 (S1422), the processor (175) determines whether the emergency level evaluation data related to the safety level of vehicle driving is at the second level (S1424), and if applicable, performs a safety extension corresponding to the second level (S1431), performs an authority adjustment corresponding to the second level (S1432), and performs a sensitivity adjustment corresponding to the second level (S1433).

[0625] If not applicable in step 1442 (S1424), the processor (175) determines whether the emergency level evaluation data related to the safety level of vehicle driving is at the third level (S1426), and if applicable, performs a safety extension corresponding to the third level (S1434), performs an authority adjustment corresponding to the third level (S1435), and performs a sensitivity adjustment corresponding to the third level (S1436).

[0626] If not applicable in step 1426 (S1426), the processor (175) determines whether the emergency level evaluation data related to the safety level of vehicle driving is at level 4 (S1428), and if applicable, performs a safety extension corresponding to level 4 (S1437), performs an authority adjustment corresponding to level 4 (S1438), and performs a sensitivity adjustment corresponding to level 4 (S1439).

[0627] Accordingly, it becomes possible to perform stable vehicle control by rapidly and accurately determining the situation within the vehicle. In particular, stable vehicle control is possible by rapidly and accurately determining the situation within the vehicle based on multimodal context data.

[0628] FIG. 17 is another example of a flowchart illustrating a method of operation of a signal processing device according to an embodiment of the present disclosure.

[0629] Referring to the drawings, the processor (175) in the signal processing device (170) according to an embodiment of the present disclosure can obtain context data of the driver inside the vehicle based on the vehicle's internal sensor device or internal camera (195i), etc. (S1402).

[0630] Meanwhile, the processor (175) can obtain driver profile information or driver characteristic data based on the database (762) (S1403).

[0631] Meanwhile, the processor (175) can obtain context data of the occupant based on the vehicle's internal sensor device or internal camera (195i), etc. (S1404).

[0632] Meanwhile, the processor (175) can obtain vehicle external context data based on vehicle external sensor devices, etc. (S1405).

[0633] Meanwhile, the processor (175) can compute driving ability evaluation data based on the driver's context data, driver profile information or driver's characteristic data, passenger's context data, and vehicle external context data (S1410).

[0634] Next, the processor (175) can calculate emergency level evaluation data based on driving ability evaluation data (S1420).

[0635] Alternatively, unlike the drawing, the processor (175) can calculate emergency level evaluation data related to the safety level of vehicle driving based on the driver's context data, driver profile information or driver characteristic data, passenger's context data, and vehicle external context data.

[0636] Meanwhile, the processor (175) determines whether the emergency level evaluation data related to the safety level of vehicle driving is at the first level (S1422), and if so, can perform safety extension, authority adjustment, and sensitivity adjustment corresponding to the first level (S1510).

[0637] If it does not apply in step 1422 (S1422), the processor (175) determines whether the emergency level evaluation data related to the safety level of vehicle driving is at the second level (S1424), and if it is, can perform safety extension, authority adjustment, and sensitivity adjustment corresponding to the second level (S1520).

[0638] If it does not apply in step 1442 (S1424), the processor (175) determines whether the emergency level evaluation data related to the safety level of vehicle driving is at the third level (S1426), and if it is, can perform safety extension, authority adjustment, and sensitivity adjustment corresponding to the third level (S1530).

[0639] If it does not apply in step 1426 (S1426), the processor (175) determines whether the emergency level evaluation data related to the safety level of vehicle driving is at the fourth level (S1428), and if it is, can perform safety extension, authority adjustment, and sensitivity adjustment corresponding to the fourth level (S1540).

[0640] Accordingly, it becomes possible to perform stable vehicle control by rapidly and accurately determining the situation within the vehicle. In particular, stable vehicle control is possible by rapidly and accurately determining the situation within the vehicle based on multimodal context data.

[0641] FIG. 18 is another example of a block diagram of a signal processing device according to an embodiment of the present disclosure.

[0642] Referring to the drawings, a signal processing device (170mm) according to an embodiment of the present disclosure includes a processor (175) that receives camera data outside the vehicle, camera data inside the vehicle, and sensor data inside the vehicle.

[0643] Meanwhile, the processor (175) according to an embodiment of the present disclosure includes a recognizer (720), an insightor (750), and an illustrator (780).

[0644] The internal recognizer (725) within the recognizer (720) can recognize gaze, position, gesture, vital signs, facial expression, facial identity, or voice.

[0645] The external recognizer (730) within the recognizer (720) can recognize road participants, traffic signs, roads, or scenes.

[0646] Meanwhile, the recognizer (720) can recognize vehicle interior state information such as distraction, impaired cognitive function, drowsiness, behavior, or health based on data from the internal recognizer (725).

[0647] Meanwhile, the multimodal context engine (755) within the insightor (750) can generate real-time context data (905) based on vehicle internal state information, external recognition data by the external recognizer (730), and vehicle and external data (718).

[0648] Meanwhile, the multimodal context engine (755) within the insightor (750) can generate multimodal context data based on real-time context data (905) and feature data (907).

[0649] Meanwhile, the AI ​​orchestrator (760) within the insightor (750) can generate a prompt for vehicle driving control based on multimodal context data from the multimodal context engine (755), profile information from the knowledge device (762), etc.

[0650] In particular, the prompt manager (763) within the AI ​​orchestrator (760) can generate a prompt for vehicle driving control based on multimodal context data from the multimodal context engine (755), profile information from the knowledge device (762), etc.

[0651] And, the generated prompt can be transmitted to an AI model set (765) or a safety assistance engine (757).

[0652] Unlike Fig. 7, the insightor (750) of Fig. 18 may not include an AI model set (765) and a safety assistance engine (757).

[0653] That is, according to FIG. 18, an AI model set (765) and a safety assistance engine (757) may be provided separately from the insightor (750) in the processor (175).

[0654] The AI ​​model set (765) may include an interface (766) for data exchange between an on-device AI model (767) and an AI model (769) within the server (400).

[0655] The AI ​​model set (765) can perform inference or response based on a prompt received from the insightor (750) and output an inference result or a response result.

[0656] Meanwhile, the safety assist engine (757) can determine whether the current situation is a safe situation or a dangerous situation based on the prompt received from the insightor (750).

[0657] For example, the safety assist engine (757) can calculate the risk level of vehicle driving based on a prompt received from the insightor (750).

[0658] As another example, the safety assist engine (757) can calculate the risk level of vehicle driving based on multimodal context data, profile information, etc. received from the insightor (750).

[0659] Meanwhile, the illustrator (780) may include a multimodal output encoder (781), a visual interface (782) including AR (783) and MR (784), and an audio / display / haptic interface (786).

[0660] Meanwhile, unlike Fig. 7, the safety application (790) within the illustrator (780) is not included and can be provided or executed separately within the processor (175).

[0661] Meanwhile, the illustrator (780) outputs a vehicle control signal based on the calculated risk level. Accordingly, the situation inside the vehicle can be determined quickly and accurately, and stable vehicle control can be performed.

[0662] Meanwhile, the illustrator (780) can output a vehicle control signal, a warning message, or a guide message based on the calculated risk level. Accordingly, the situation inside the vehicle can be determined quickly and accurately, and stable vehicle control can be performed.

[0663] Although preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above. Various modifications are possible by those skilled in the art without departing from the essence of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical spirit or perspective of the present disclosure.

Claims

1. A processor that receives camera data from outside the vehicle, camera data from inside the vehicle, and sensor data inside the vehicle; and The above processor is, Based on the camera data outside the vehicle mentioned above, object detection is performed, and Based on camera data inside the vehicle, the condition of passengers including the driver is monitored, and Based on the monitored occupant status information, the detected object information, the sensor data, and the driver's driving pattern information stored in the database, multimodal context data corresponding to a driving context related to vehicle driving is extracted, and A signal processing device that calculates the risk level of the vehicle driving based on the extracted multimodal context data and outputs a vehicle control signal based on the calculated risk level.

2. In Paragraph 1, The above processor is, Based on the monitored occupant status information, the detected object information, the sensor data, and the driver's driving pattern information stored in the database, multimodal context data corresponding to a driving context related to vehicle driving is extracted, and A signal processing device that obtains an inference result or a response result based on the extracted multimodal context data, calculates a risk level of the vehicle driving based on the obtained inference result or response result, and outputs a vehicle control signal based on the calculated risk level.

3. In Paragraph 1, The above processor is, A signal processing device that classifies the risk level of the above-mentioned vehicle driving into levels 1 through 4.

4. In Paragraph 1, The above processor is, A signal processing device that controls authority adjustment, safety extension, and sensitivity adjustment for vehicle control based on the risk level of the vehicle driving.

5. In Paragraph 1, The above processor is, Based on the risk level of the vehicle driving described above, authority adjustment data for vehicle control, safety extension data for safety extension, and sensitivity adjustment data for sensitivity adjustment are calculated, A signal processing device that generates the vehicle control signal based on the above authority adjustment data, safety extension data, and sensitivity adjustment data.

6. In Paragraph 1, The above processor is, A signal processing device that performs learning based on the above-calculated risk level and occupant profile information, and outputs the vehicle control signal, outputs a warning message, or outputs a guide message based on the result data of the learning.

7. In Paragraph 1, The above processor is, Based on the context data of the above driver and the characteristic data of the above driver, driving ability evaluation data and emergency level evaluation data are calculated, and A signal processing device that calculates the risk level of the vehicle driving based on the above driving ability evaluation data and emergency level evaluation data.

8. In Paragraph 1, The above processor is, Based on the driver's context data, the driver's characteristic data, the passenger's context data, and the passenger's characteristic data, driving ability evaluation data and emergency level evaluation data are calculated, and A signal processing device that calculates the risk level of the vehicle driving based on the above driving ability evaluation data and emergency level evaluation data.

9. In Paragraph 1, The above processor is, A signal processing device that classifies the risk level of the vehicle driving into multiple levels based on vehicle internal context data, vehicle external context data, and occupant profile information.

10. In Paragraph 1, The above processor is, A signal processing device that classifies the risk level of the vehicle's driving into a plurality of levels based on vehicle internal context data, vehicle external context data, occupant characteristic information, occupant behavior information, and the driver's driving ability information.

11. In Paragraph 1, The above processor is, The risk level of the above vehicle driving is classified into multiple levels, and When the risk level of the above vehicle driving is Level 1, control is maintained so that the authority adjustment, safety extension, and sensitivity adjustment for vehicle control remain as they are, and A signal processing device that, when the risk level of the vehicle driving is a second level higher than the first level, switches the authority adjustment and safety extension for vehicle control to a standby state and increases the sensitivity of the sensitivity adjustment above the first level.

12. In Paragraph 11, The above processor is, A signal processing device that, when the risk level of the vehicle driving is a third level higher than the second level, ignores or supplements the operation input within the authority adjustment for vehicle control, increases the threshold value within the safety extension, or raises the sensitivity of the sensitivity adjustment above the second level.

13. In Paragraph 11, The above processor is, A signal processing device that, when the risk level of the vehicle driving is at its highest level, controls steering and acceleration for the authority adjustment for vehicle control, sets the threshold value within the safety extension to the maximum value, and sets the sensitivity of the sensitivity adjustment to the maximum value.

14. In Paragraph 1, The above processor is, If the driver's driving ability information among the monitored passenger status information indicates that the driver can operate the vehicle, the risk level of the vehicle driving is calculated as a first level, and If the driving ability information of the above driver indicates that the driver is capable of operating the vehicle or is in a distracted state, the risk level of the vehicle driving is calculated as a second level higher than the first level, and If the driving ability information of the above driver indicates that the driver is capable of operation or is in a reaction delay state, the risk level of the vehicle driving is calculated as a third level higher than the second level, and A signal processing device that calculates the risk level of the vehicle driving as a fourth level higher than the third level when the driver's driving ability information indicates that the driver cannot operate the vehicle.

15. In Paragraph 1, The above processor is, If the occupant characteristic information does not include habit behavior information or is not personalized habit behavior information, the risk level of the vehicle driving is calculated as the first level, and If the above passenger characteristic information includes habitual behavior information that interferes with driving, the risk level of the vehicle driving is calculated as a second level higher than the first level, and If the above occupant characteristic information includes recurring risky behaviors and medical history or accident history, the risk level of the vehicle driving is calculated as a third level higher than the second level, and A signal processing device that calculates the risk level of the vehicle driving as a fourth level higher than the third level when the above occupant characteristic information includes risk information based on the above medical history.

16. In Paragraph 1, The above processor is, If the passenger's behavior information includes stable state information, the risk level of the vehicle driving is calculated as a first level, and If the behavioral information of the above-mentioned passenger includes voice information at a level below a threshold, the risk level of the vehicle driving is calculated as a second level higher than the first level, and If the behavioral information of the passenger includes voice information at a level higher than the repeated threshold, the risk level of the vehicle driving is calculated as a third level higher than the second level, and A signal processing device that calculates the risk level of the vehicle driving as a fourth level higher than the third level when the behavioral information of the passenger includes actions that interfere with the driving operation of the driver.

17. In Paragraph 1, The above processor is, Based on the above multimodal context data, a prompt is generated for calculating the risk level of vehicle driving, and A signal processing device that calculates the risk level of the vehicle driving based on the inference result or response result obtained based on the above prompt, and outputs a vehicle control signal based on the calculated risk level.

18. In Paragraph 1, Further comprising a neural processor that executes inference or a response based on the above prompt, The above processor is, Based on the above multimodal context data, a prompt is generated for calculating the risk level of vehicle driving, and A signal processing device that calculates a risk level of vehicle driving based on an inference result or response result executed in the neural processor based on the above prompt, and outputs a vehicle control signal based on the calculated risk level.

19. In Paragraph 1, The above processor is, Run a hypervisor, and on the hypervisor, run a driving control virtualization machine, and The above driving control virtualization machine is, Through shared memory within the hypervisor, camera data from outside the vehicle, camera data from inside the vehicle, and sensor data from inside the vehicle are received, and A signal processing device that executes a driving control service or a driving control application based on camera data outside the vehicle, camera data inside the vehicle, sensor data, and driving pattern information of the driver stored in the database.

20. A vehicle control device comprising a signal processing device according to any one of claims 1 to 19.

Images