Signal processing device and vehicle control device including same

Abstract

According to an embodiment of the present disclosure, a signal processing device and a vehicle control device including same comprise a processor for receiving out-of-vehicle camera data, in-vehicle camera data, and in-vehicle sensor data, wherein the processor: performs object detection on the basis of the out-of-vehicle camera data; monitors a state of an occupant including a driver on the basis of the in-vehicle camera data; extracts multimodal context data corresponding to a driving context related to vehicle driving, on the basis of information on the monitored state of the occupant, information on the detected object, the sensor data, and the driver's driving pattern information stored in a database; and on the basis of the extracted multimodal context data, sets a mode for vehicle driving assistance control or autonomous driving. Accordingly, it is possible to adaptively perform vehicle control by rapidly and accurately determining situations inside and outside the vehicle.

Description

Signal processing device, and vehicle control device having the same

[0001] The present disclosure relates to a signal processing device and a vehicle control device equipped with the same, and more specifically, to a signal processing device capable of rapidly and accurately determining the situation inside and outside the vehicle and adaptively performing vehicle control, and a vehicle control device equipped with the same.

[0002] A vehicle is a device that moves the user in the desired direction. A typical example is an automobile.

[0003] Meanwhile, for the convenience of users, a vehicle control device is installed inside the vehicle.

[0004] The vehicle control unit includes a signal processing unit and can perform signal processing based on sensor data from various internal sensor devices.

[0005] The prior art U.S. Patent No. 7719431 discloses a system for detecting and responding to drowsy driving, which determines the drowsy driving state of a vehicle based on a determination of whether there is a lane violation, and outputs a driver alarm signal, etc. based thereon.

[0006] However, according to prior art, there is a problem in that real-time monitoring of the vehicle interior cannot be performed and the situation inside the vehicle cannot be accurately recognized. Consequently, a problem arises where driver alarm signals unrelated to the situation inside the vehicle are output.

[0007] The problem that the present disclosure aims to solve is to provide a signal processing device capable of rapidly and accurately determining the situation inside and outside the vehicle and adaptively performing vehicle control, and a vehicle control device equipped with the same.

[0008] Another problem that the present disclosure aims to solve is to provide a signal processing device capable of rapidly and accurately determining the situation inside and outside the vehicle based on multimodal context data and adaptively performing vehicle control, and a vehicle control device equipped with the same.

[0009] To solve the above technical problem, a signal processing device according to one embodiment of the present disclosure and a vehicle control device equipped therewith include a processor that receives camera data outside the vehicle, camera data inside the vehicle, and sensor data inside the vehicle. The processor performs object detection based on camera data outside the vehicle, monitors the state of a passenger including a driver based on camera data inside the vehicle, extracts multimodal context data corresponding to a driving context related to vehicle driving based on monitored passenger state information, detected object information, sensor data, and driver's driving pattern information stored in a database, and sets a mode for vehicle driving assistance control or autonomous driving based on the extracted multimodal context data.

[0010] Meanwhile, the processor can control the vehicle driving assistance control or autonomous driving to operate in an off mode, on mode, or safety mode based on the extracted multimodal context data.

[0011] Meanwhile, the processor calculates the risk level of vehicle driving based on the extracted multimodal context data, and based on the calculated risk level, can set the vehicle driving assistance control or autonomous driving to one of the off mode, on mode, or safety mode.

[0012] Meanwhile, the processor can set a mode for vehicle driving assistance control or autonomous driving based on the extracted multimodal context data, and control authority adjustment, safety extension, or sensitivity adjustment for vehicle control based on the set mode.

[0013] Meanwhile, the processor can determine whether it is a danger precursor stage or a danger situation stage based on the extracted multimodal context data, and control the operation mode of vehicle driving assistance control or autonomous driving to be variable according to the danger precursor stage or danger situation stage.

[0014] Meanwhile, the processor can calculate the risk level of vehicle driving based on the extracted multimodal context data and output a vehicle control signal based on the calculated risk level.

[0015] Meanwhile, the processor can classify the risk level of vehicle driving into levels 1 through 4.

[0016] Meanwhile, the processor can calculate the risk level of vehicle driving based on the extracted multimodal context data, and control authority adjustment, safety extension, or sensitivity adjustment for vehicle control based on the calculated risk level.

[0017] Meanwhile, the processor calculates a risk level of vehicle driving based on the extracted multimodal context data, calculates authority adjustment data for vehicle control, safety extension data for safety extension, and sensitivity adjustment data for sensitivity adjustment based on the calculated risk level, and can generate a vehicle control signal based on the authority adjustment data, safety extension data, and sensitivity adjustment data.

[0018] Meanwhile, the processor can calculate the risk level of vehicle driving based on the extracted multimodal context data, perform learning based on the calculated risk level and occupant profile information, and output a vehicle control signal, a warning message, or a guide message based on the result data of the learning.

[0019] Meanwhile, the processor can compute driving ability evaluation data and emergency level evaluation data based on the driver's context data and driver's characteristic data, and compute the risk level of vehicle driving based on the driving ability evaluation data and emergency level evaluation data.

[0020] Meanwhile, the processor can calculate driving ability evaluation data and emergency level evaluation data based on the driver's context data, driver's characteristic data, passenger's context data, and passenger's characteristic data, and calculate the risk level of vehicle driving based on the driving ability evaluation data and emergency level evaluation data.

[0021] Meanwhile, the processor can classify the risk level of vehicle driving into multiple levels based on vehicle internal context data, vehicle external context data, occupant characteristic information, occupant behavior information, and driver's driving ability information.

[0022] Meanwhile, the processor calculates the risk level of vehicle driving based on the extracted multimodal context data, and based on the calculated risk level, if the risk level of vehicle driving is at the first level, controls the authority adjustment, safety extension, and sensitivity adjustment for vehicle control to remain as they are, and if the risk level of vehicle driving is at the second level which is higher than the first level, switches the authority adjustment and safety extension for vehicle control to a standby state and can increase the sensitivity of the sensitivity adjustment above the first level.

[0023] Meanwhile, if the risk level of vehicle driving is at a third level higher than the second level, the processor may ignore or supplement the operation input within the authority adjustment for vehicle control, increase the threshold value within the safety extension, or raise the sensitivity of the sensitivity adjustment above the second level.

[0024] Meanwhile, when the risk level of vehicle driving is at its highest level, the processor can control steering and acceleration to adjust authority for vehicle control, set the threshold within the safety extension to the maximum value, and set the sensitivity of the sensitivity adjustment to the maximum value.

[0025] Meanwhile, the processor calculates a risk level of vehicle driving based on the extracted multimodal context data, and calculates the risk level of vehicle driving as a first level when the driver's driving ability information among the monitored passenger status information indicates that the driver can operate the vehicle, calculates the risk level of vehicle driving as a second level higher than the first level when the driver's driving ability information indicates that the driver can operate the vehicle but is in a distracted state, calculates the risk level of vehicle driving as a third level higher than the second level when the driver's driving ability information indicates that the driver cannot operate the vehicle, and calculates the risk level of vehicle driving as a fourth level higher than the third level when the driver's driving ability information indicates that the driver cannot operate the vehicle.

[0026] Meanwhile, the processor calculates the risk level of vehicle driving based on the extracted multimodal context data, and calculates the risk level of vehicle driving as a first level if the occupant characteristic information does not include habit behavior information or is not personalized habit behavior information, calculates the risk level of vehicle driving as a second level higher than the first level if the occupant characteristic information includes habit behavior information that interferes with driving, calculates the risk level of vehicle driving as a third level higher than the second level if the occupant characteristic information includes repetitive risk behavior and medical history or accident history, and calculates the risk level of vehicle driving as a fourth level higher than the third level if the occupant characteristic information includes risk information based on medical history.

[0027] Meanwhile, the processor calculates a risk level of vehicle driving based on extracted multimodal context data, and if the passenger's behavior information includes stable state information, calculates the risk level of vehicle driving as a first level; if the passenger's behavior information includes voice information at a level below a threshold, calculates the risk level of vehicle driving as a second level higher than the first level; if the passenger's behavior information includes repeated voice information at a level above a threshold, calculates the risk level of vehicle driving as a third level higher than the second level; and if the passenger's behavior information includes actions that interfere with the driver's driving operation, calculates the risk level of vehicle driving as a fourth level higher than the third level.

[0028] A signal processing device according to one embodiment of the present disclosure and a vehicle control device equipped therewith further include a neural processor that executes inference or response based on a prompt, and the processor generates a prompt for calculating a risk level of vehicle driving based on multimodal context data, obtains an inference result or response result executed by the neural processor based on the prompt, calculates a risk level of vehicle driving based on the obtained inference result or response result, and outputs a vehicle control signal based on the calculated risk level.

[0029] Meanwhile, the processor executes a hypervisor and executes a driving control virtualization machine on the hypervisor, and the driving control virtualization machine receives camera data outside the vehicle, camera data inside the vehicle, and sensor data inside the vehicle through shared memory within the hypervisor, and can execute a driving control service or a driving control application based on the camera data outside the vehicle, camera data inside the vehicle, sensor data, and driver's driving pattern information stored in a database.

[0030] A signal processing device according to one embodiment of the present disclosure and a vehicle control device equipped therewith include a processor that receives camera data from outside the vehicle, camera data from inside the vehicle, and sensor data inside the vehicle. The processor performs object detection based on camera data from outside the vehicle, monitors the state of a passenger including a driver based on camera data from inside the vehicle, extracts multimodal context data corresponding to a driving context related to vehicle driving based on monitored passenger state information, detected object information, sensor data, and driver's driving pattern information stored in a database, and sets a mode for vehicle driving assistance control or autonomous driving based on the extracted multimodal context data. Accordingly, it is possible to rapidly and accurately determine the situation inside and outside the vehicle and perform adaptive vehicle control. In particular, it is possible to rapidly and accurately determine the situation inside and outside the vehicle based on multimodal context data and perform adaptive vehicle control.

[0031] Meanwhile, the processor can be controlled to operate in off, on, or safety modes for vehicle driving assistance control or autonomous driving, based on the extracted multimodal context data. Accordingly, it becomes possible to adaptively control the vehicle by rapidly and accurately determining the situation inside and outside the vehicle.

[0032] Meanwhile, the processor calculates the risk level of vehicle driving based on the extracted multimodal context data, and based on the calculated risk level, can set the vehicle driving assistance control or autonomous driving to one of the off mode, on mode, or safety mode. Accordingly, it becomes possible to adaptively control the vehicle by quickly and accurately determining the situation inside and outside the vehicle.

[0033] Meanwhile, the processor can set a mode for vehicle driving assistance control or autonomous driving based on the extracted multimodal context data, and control authority adjustment, safety extension, or sensitivity adjustment for vehicle control based on the set mode. Accordingly, it becomes possible to adaptively perform vehicle control by rapidly and accurately determining the situation inside and outside the vehicle.

[0034] Meanwhile, the processor determines whether it is a danger precursor stage or a danger situation stage based on the extracted multimodal context data, and controls the operation mode of vehicle driving assistance control or autonomous driving to vary according to the danger precursor stage or danger situation stage. Accordingly, it becomes possible to quickly and accurately determine the situation inside and outside the vehicle and perform adaptive vehicle control.

[0035] Meanwhile, the processor can calculate the risk level of vehicle driving based on the extracted multimodal context data and output a vehicle control signal based on the calculated risk level. Accordingly, it becomes possible to quickly and accurately determine the situation inside and outside the vehicle and perform adaptive vehicle control.

[0036] Meanwhile, the processor can classify the risk level of vehicle driving into levels 1 through 4. Accordingly, it is possible to quickly and accurately determine the situation inside and outside the vehicle and perform adaptive vehicle control.

[0037] Meanwhile, the processor calculates the risk level of vehicle driving based on the extracted multimodal context data, and controls authority adjustment, safety extension, or sensitivity adjustment for vehicle control based on the calculated risk level. Accordingly, it becomes possible to adaptively perform vehicle control by rapidly and accurately determining the situation inside and outside the vehicle.

[0038] Meanwhile, the processor calculates a risk level of vehicle driving based on extracted multimodal context data, calculates authority adjustment data for vehicle control, safety extension data for safety extension, and sensitivity adjustment data for sensitivity adjustment based on the calculated risk level, and can generate a vehicle control signal based on the authority adjustment data, safety extension data, and sensitivity adjustment data. Accordingly, it becomes possible to rapidly and accurately determine the situation inside and outside the vehicle and adaptively perform vehicle control.

[0039] Meanwhile, the processor calculates the risk level of vehicle driving based on the extracted multimodal context data, performs learning based on the calculated risk level and occupant profile information, and outputs a vehicle control signal, a warning message, or a guide message based on the result data of the learning. Accordingly, it becomes possible to quickly and accurately determine the situation inside and outside the vehicle and perform adaptive vehicle control.

[0040] Meanwhile, the processor can compute driving capability evaluation data and emergency level evaluation data based on the driver's context data and driver characteristic data, and compute the risk level of vehicle driving based on the driving capability evaluation data and emergency level evaluation data. Accordingly, it becomes possible to rapidly and accurately determine the situation inside and outside the vehicle and perform adaptive vehicle control.

[0041] Meanwhile, the processor can compute driving capability evaluation data and emergency level evaluation data based on the driver's context data, driver's characteristic data, passenger's context data, and passenger's characteristic data, and compute the risk level of vehicle driving based on the driving capability evaluation data and emergency level evaluation data. Accordingly, it becomes possible to rapidly and accurately determine the situation inside and outside the vehicle and perform adaptive vehicle control.

[0042] Meanwhile, the processor can classify the risk level of vehicle driving into multiple levels based on vehicle internal context data, vehicle external context data, occupant characteristic information, occupant behavior information, and driver driving ability information. Accordingly, it becomes possible to rapidly and accurately determine the situation inside and outside the vehicle and perform adaptive vehicle control.

[0043] Meanwhile, the processor calculates the risk level of vehicle driving based on the extracted multimodal context data. Based on the calculated risk level, if the risk level of vehicle driving is at Level 1, it controls the authority adjustment, safety extension, and sensitivity adjustment for vehicle control to remain unchanged. If the risk level of vehicle driving is at Level 2, which is higher than Level 1, it switches the authority adjustment and safety extension for vehicle control to a standby state and increases the sensitivity of the sensitivity adjustment above Level 1. Accordingly, it becomes possible to quickly and accurately determine the situation inside and outside the vehicle and perform adaptive vehicle control.

[0044] Meanwhile, if the risk level of vehicle driving is at a third level higher than the second level, the processor may ignore or supplement the operation input within the authority adjustment for vehicle control, increase the threshold value within the safety extension, or raise the sensitivity of the sensitivity adjustment above the second level. Accordingly, it becomes possible to quickly and accurately determine the situation inside and outside the vehicle and perform adaptive vehicle control.

[0045] Meanwhile, when the risk level of vehicle driving is at its highest level, the processor can control steering and acceleration to adjust authority for vehicle control, set the threshold within the safety extension to the maximum value, and set the sensitivity of the sensitivity adjustment to the maximum value. Accordingly, it becomes possible to quickly and accurately determine the situation inside and outside the vehicle and perform adaptive vehicle control.

[0046] Meanwhile, the processor calculates a risk level of vehicle driving based on extracted multimodal context data. If the driver's driving ability information among the monitored occupant status information indicates that the driver is capable of operating the vehicle, the risk level of vehicle driving is calculated as a first level; if the driver's driving ability information indicates that the driver is capable of operating the vehicle but is in a distracted state, the risk level of vehicle driving is calculated as a second level higher than the first level; if the driver's driving ability information indicates that the driver is capable of operating the vehicle but is in a delayed reaction state, the risk level of vehicle driving is calculated as a third level higher than the second level; and if the driver's driving ability information indicates that the driver is unable to operate the vehicle, the risk level of vehicle driving is calculated as a fourth level higher than the third level. Accordingly, it becomes possible to quickly and accurately determine the situation inside and outside the vehicle and perform adaptive vehicle control.

[0047] Meanwhile, the processor calculates the risk level of vehicle driving based on the extracted multimodal context data. If the occupant characteristic information does not include habitual behavior information or is not personalized habitual behavior information, the risk level of vehicle driving is calculated as a first level. If the occupant characteristic information includes habitual behavior information that interferes with driving, the risk level of vehicle driving is calculated as a second level higher than the first level. If the occupant characteristic information includes repetitive risky behaviors and medical history or accident history, the risk level of vehicle driving is calculated as a third level higher than the second level. If the occupant characteristic information includes risk information based on medical history, the risk level of vehicle driving is calculated as a fourth level higher than the third level. Accordingly, it becomes possible to quickly and accurately determine the situation inside and outside the vehicle and perform adaptive vehicle control.

[0048] Meanwhile, the processor calculates a risk level of vehicle driving based on extracted multimodal context data, and if the passenger's behavioral information includes stable state information, calculates the risk level of vehicle driving as a first level; if the passenger's behavioral information includes voice information at a level below a threshold, calculates the risk level of vehicle driving as a second level higher than the first level; if the passenger's behavioral information includes repeated voice information at a level above a threshold, calculates the risk level of vehicle driving as a third level higher than the second level; and if the passenger's behavioral information includes actions that interfere with the driver's driving operation, calculates the risk level of vehicle driving as a fourth level higher than the third level. Accordingly, it becomes possible to quickly and accurately determine the situation inside and outside the vehicle and perform adaptive vehicle control.

[0049] A signal processing device according to one embodiment of the present disclosure and a vehicle control device equipped therewith further include a neural processor that executes inference or response based on a prompt. The processor generates a prompt for calculating a risk level of vehicle driving based on multimodal context data, obtains an inference result or response result executed by the neural processor based on the prompt, calculates a risk level of vehicle driving based on the obtained inference result or response result, and outputs a vehicle control signal based on the calculated risk level. Accordingly, it is possible to rapidly and accurately determine the situation inside and outside the vehicle and adaptively perform vehicle control.

[0050] Meanwhile, the processor executes a hypervisor and executes a driving control virtualization machine on the hypervisor. The driving control virtualization machine receives camera data from outside the vehicle, camera data from inside the vehicle, and sensor data from inside the vehicle through shared memory within the hypervisor. Based on the camera data from outside the vehicle, camera data from inside the vehicle, sensor data, and driver's driving pattern information stored in a database, the driving control service or driving control application can be executed. Accordingly, it becomes possible to quickly and accurately determine the situation inside and outside the vehicle and adaptively perform vehicle control.

[0051] Figure 1 is a drawing illustrating an example of the exterior and interior of a vehicle.

[0052] Figure 2 is a diagram illustrating an example of the architecture of a vehicle control device.

[0053] FIG. 3a is a drawing illustrating an example of the arrangement of displays inside a vehicle.

[0054] Figure 3b is a drawing illustrating another example of the arrangement of displays inside a vehicle.

[0055] FIG. 4 is an example of an internal block diagram of a vehicle control device according to an embodiment of the present disclosure.

[0056] FIG. 5 is an example of a configuration diagram of a signal processing device according to an embodiment of the present disclosure.

[0057] FIG. 6 is an example of a block diagram of a vehicle control device according to an embodiment of the present disclosure.

[0058] FIG. 7 is an example of a block diagram of a signal processing device according to an embodiment of the present disclosure.

[0059] FIGS. 8a to 8c are drawings referenced in the description of FIG. 7.

[0060] FIG. 9 is another example of a block diagram of a signal processing device according to an embodiment of the present disclosure.

[0061] FIGS. 10 to 15 are drawings referenced in the description of FIG. 9.

[0062] FIG. 16 is an example of a flowchart illustrating a method of operation of a signal processing device according to an embodiment of the present disclosure.

[0063] FIG. 17 is another example of a flowchart illustrating a method of operation of a signal processing device according to an embodiment of the present disclosure.

[0064] FIG. 18 is another example of a block diagram of a signal processing device according to an embodiment of the present disclosure.

[0065] The present disclosure will be described in more detail below with reference to the drawings.

[0066] The suffixes "module" and "part" for components used in the following description are assigned solely for the ease of drafting this specification and do not inherently confer any particularly significant meaning or role. Accordingly, the terms "module" and "part" may be used interchangeably.

[0067] Figure 1 is a drawing illustrating an example of the exterior and interior of a vehicle.

[0068] Referring to the drawing, the vehicle (200) is operated by a plurality of wheels (103FR, 103FL, 103RL,...) that rotate by a power source, and a steering wheel (150) for controlling the direction of travel of the vehicle (200).

[0069] Meanwhile, the vehicle (200) may further be equipped with a camera (195), etc., for acquiring an image of the front of the vehicle.

[0070] Meanwhile, the vehicle (200) may be equipped with a plurality of displays (180a, 180b) for displaying images, information, etc. inside.

[0071] In FIG. 1, a cluster display (180a) and an IVI (In-Vehicle Infotainment) display (180b) are exemplified as multiple displays (180a, 180b). In addition, a HUD (Head Up Display) and the like are also possible.

[0072] Meanwhile, the IVI (In-Vehicle Infotainment) display (180b) may also be named a Center Information Display or an AVN (Audio Video Navigation) display.

[0073] Meanwhile, the vehicle (200) described in this specification may be a concept that includes all of the following: a vehicle equipped with an engine as a power source, a hybrid vehicle equipped with an engine and an electric motor as a power source, an electric vehicle equipped with an electric motor as a power source, etc.

[0074] Figure 2 is a diagram illustrating an example of the architecture of a vehicle control device.

[0075] Referring to the drawing, the architecture (300a) of the vehicle control device can correspond to a zone-based architecture.

[0076] Accordingly, sensor devices and processors inside the vehicle may be placed in each of the multiple zones (Z1 to Z4), and a signal processing device (170a) including a gateway (GWDa) may be placed in the central area of ​​the multiple zones (Z1 to Z4).

[0077] Meanwhile, the signal processing device (170a) may additionally include an autonomous driving control module (ACC), a cockpit control module (CPG), etc., in addition to the gateway (GWDa).

[0078] The gateway (GWDa) within the signal processing device (170a) may be a High Performance Computing (HPC) gateway.

[0079] That is, the signal processing device (170a) of FIG. 2 is an integrated HPC and can exchange data with an external communication module (not shown) or a processor (not shown) in a plurality of zones (Z1 to Z4).

[0080] FIG. 3a is a drawing illustrating an example of the arrangement of displays inside a vehicle.

[0081] Referring to the drawing, the vehicle interior may be equipped with a cluster display (180a), an IVI (In-Vehicle Infotainment) display (180b), a rear seat entertainment display (180c, 180d), a rearview mirror display (not shown), etc.

[0082] Meanwhile, in addition to the display, an interior camera (195i) may be installed inside the vehicle.

[0083] Figure 3b is a drawing illustrating another example of the arrangement of displays inside a vehicle.

[0084] A vehicle control device (100) according to an embodiment of the present disclosure may include a plurality of displays (180a to 180b) and a signal processing device (170) that performs signal processing for displaying images, information, etc. on the plurality of displays (180a to 180b) and outputs an image signal to at least one display (180a to 180b).

[0085] Among the plurality of displays (180a to 180b), the first display (180a) is a cluster display (180a) for displaying driving status, operation information, etc., and the second display (180b) may be an IVI (In-Vehicle Infotainment) display (180b) for displaying vehicle operation information, navigation map, various entertainment information or video.

[0086] The signal processing device (170) has a processor (175) inside and can execute a first virtualization machine to a third virtualization machine (not shown) on a hypervisor (not shown) within the processor (175).

[0087] A second virtualization machine (not shown) operates for the first display (180a), and a third virtualization machine (not shown) can operate for the second display (180b).

[0088] Meanwhile, the first virtualization machine (not shown) within the processor (175) can be controlled to set up a shared memory (508) based on a hypervisor (505) for the same data transmission to the second virtualization machine (not shown) and the third virtualization machine (not shown). Accordingly, the same information or the same image can be synchronized and displayed on the first display (180a) and the second display (180b) within the vehicle.

[0089] Meanwhile, the first virtualization machine (not shown) within the processor (175) shares at least a portion of the data with the second virtualization machine (not shown) and the third virtualization machine (not shown) for data sharing processing. Accordingly, data can be shared and processed by multiple virtualization machines for multiple displays within the vehicle.

[0090] Meanwhile, the first virtualization machine (not shown) within the processor (175) can receive and process wheel speed sensor data of the vehicle and transmit the processed wheel speed sensor data to at least one of the second virtualization machine (not shown) or the third virtualization machine (not shown). Accordingly, the wheel speed sensor data of the vehicle can be shared with at least one virtualization machine, etc.

[0091] Meanwhile, the vehicle control device (100) according to the embodiment of the present disclosure may further include a rear seat entertainment display (180c) for displaying driving status information, simple navigation information, various entertainment information or images.

[0092] The signal processing device (170) can control the RSE display (180c) by running a fourth virtualization machine (not shown) in addition to the first to third virtualization machines (not shown) on a hypervisor (not shown) within the processor (175).

[0093] Accordingly, various displays (180a to 180c) can be controlled using a single signal processing device (170).

[0094] Meanwhile, some of the multiple displays (180a to 180c) operate under a Linux OS, and others can operate under a Web OS.

[0095] A signal processing device (170) according to an embodiment of the present disclosure can control displays (180a to 180c) operating under various operating systems (OS) to synchronize and display the same information or the same image.

[0096] Meanwhile, FIG. 3b illustrates that a vehicle speed indicator (212a) and a vehicle interior temperature indicator (213a) are displayed on a first display (180a), a home screen (222) including a plurality of applications, a vehicle speed indicator (212b), and a vehicle interior temperature indicator (213b) is displayed on a second display (180b), and a second home screen (222b) including a plurality of applications and a vehicle interior temperature indicator (213c) is displayed on a third display (180c).

[0097] FIG. 4 is an example of an internal block diagram of a vehicle control device according to an embodiment of the present disclosure.

[0098] Referring to the drawings, a vehicle control device (100) according to an embodiment of the present disclosure may include an input device (110), a communication unit (120) for communication with an external device, a plurality of communication modules (EMa~EMd) for internal communication, a memory (140), a signal processing device (170), a plurality of displays (180a~180c), an audio output unit (185), and a power supply unit (190).

[0099] Multiple communication modules (EMa~EMd) can be placed in each of the multiple zones (Z1~Z4) of FIG. 2, for example.

[0100] Meanwhile, the signal processing device (170) may have a communication switch (736b) inside for data communication with each communication module (EM1~EM4).

[0101] Each communication module (EM1~EM4) can perform data communication with a plurality of sensor devices (SN), ECU (770), or area signal processing device (170Z).

[0102] Meanwhile, a plurality of sensor devices (SN) may include a camera (195), lidar (196), radar (197), or position sensor (198).

[0103] The input device (110) may be equipped with physical buttons, pads, etc. for button input, touch input, etc.

[0104] Meanwhile, the input device (110) may be equipped with a microphone (112) for user voice input.

[0105] The communication unit (120) can exchange data wirelessly with a mobile terminal (600) or a server (400).

[0106] In particular, the communication unit (120) can wirelessly exchange data with the vehicle driver's mobile terminal. Various data communication methods are possible as wireless data communication methods, such as Bluetooth, WiFi, WiFi Direct, and APiX.

[0107] The communication unit (120) can receive weather information, road traffic condition information, for example, TPEG (Transport Protocol Expert Group) information from a mobile terminal (600) or a server (400). To this end, the communication unit (120) may be equipped with a mobile communication module (not shown).

[0108] A plurality of communication modules (EM1~EM4) can receive sensor data, etc. from an ECU (770), a sensor device (SN), or a region signal processing device (170Z), and transmit the received sensor data to the signal processing device (170).

[0109] Here, the sensor data may include at least one of vehicle direction data, vehicle location data (GPS data), vehicle angle data, vehicle speed data, vehicle acceleration data, vehicle tilt data, vehicle forward / reverse data, battery data, fuel data, tire data, vehicle lamp data, vehicle interior temperature data, and vehicle interior humidity data.

[0110] Such sensor data can be obtained from a heading sensor, a yaw sensor, a gyro sensor, a position module, a vehicle forward / reverse sensor, a wheel sensor, a vehicle speed sensor, a vehicle body inclination sensor, a battery sensor, a fuel sensor, a tire sensor, a steering sensor based on steering wheel rotation, a vehicle interior temperature sensor, a vehicle interior humidity sensor, etc.

[0111] Meanwhile, the position module may include a GPS module or a position sensor (198) for receiving GPS information.

[0112] Meanwhile, at least one of the multiple communication modules (EM1 to EM4) can transmit location information data sensed from a GPS module or a location sensor (198) to a signal processing device (170).

[0113] Meanwhile, at least one of the plurality of communication modules (EM1 to EM4) can receive vehicle front image data, vehicle side image data, vehicle rear image data, and obstacle distance information around the vehicle from a camera (195), lidar (196), radar (197), etc., and transmit the received information to a signal processing device (170).

[0114] The memory (140) can store various data for the overall operation of the vehicle control device (100), such as a program for processing or controlling the signal processing device (170).

[0115] For example, memory (140) can store data regarding a hypervisor, a first virtualization machine to a third virtualization machine, for execution within a processor (175).

[0116] The audio output unit (185) converts an electrical signal from the signal processing device (170) into an audio signal and outputs it. To do this, a speaker or the like may be provided.

[0117] The power supply unit (190) can supply power necessary for the operation of each component under the control of the signal processing unit (170). In particular, the power supply unit (190) can receive power from a battery inside the vehicle, etc.

[0118] The signal processing unit (170) controls the overall operation of each unit within the vehicle control unit (100).

[0119] For example, the signal processing device (170) may include a processor (175) that performs signal processing for a vehicle display (180a, 180b).

[0120] The processor (175) can run a first virtualization machine to a third virtualization machine (not shown) on a hypervisor (505 in FIG. 5) within the processor (175).

[0121] Among the first to third virtual machines (not shown), the first virtual machine (not shown) may be named a Server Virtual Machine, and the second to third virtual machines (not shown) may be named a Guest Virtual Machine.

[0122] For example, a first virtualization machine (not shown) within a processor (175) can receive sensor data from a plurality of sensor devices, such as vehicle sensor data, location information data, camera image data, audio data, or touch input data, and process or modify it to output it.

[0123] In this way, by performing most of the data processing in the first virtualization machine (not shown), 1:N data sharing becomes possible.

[0124] As another example, the first virtualization machine (not shown) can directly receive and process CAN data, Ethernet data, audio data, radio data, USB data, and wireless communication data for the second virtualization machine to the third virtualization machine (not shown).

[0125] And, the first virtualization machine (not shown) can transmit the processed data to the second virtualization machine to the third virtualization machine (not shown).

[0126] Accordingly, among the first to third virtualization machines (not shown), only the first virtualization machine (not shown) receives sensor data, communication data, or external input data from a plurality of sensor devices and performs signal processing, thereby reducing the signal processing burden on other virtualization machines and enabling 1:N data communication, which enables synchronization when sharing data.

[0127] Meanwhile, the first virtualization machine (not shown) can control the sharing of the same data with the second virtualization machine (not shown) and the third virtualization machine (not shown) by writing data to the shared memory (508 in FIG. 5).

[0128] For example, the first virtualization machine (not shown) can record vehicle sensor data, the location information data, the camera image data, or the touch input data in a shared memory (508) and control the sharing of the same data with the second virtualization machine (not shown) and the third virtualization machine (not shown). Accordingly, data sharing in a 1:N manner becomes possible.

[0129] Ultimately, by performing most of the data processing on the first virtualization machine (not shown), 1:N data sharing becomes possible.

[0130] Meanwhile, the first virtualization machine (not shown) within the processor (175) can control the second virtualization machine (not shown) and the third virtualization machine (not shown) to set up a shared memory (508) based on the hypervisor (505) for the same data transmission.

[0131] Meanwhile, the signal processing device (170) can process various signals such as audio signals, video signals, and data signals. To this end, the signal processing device (170) can be implemented in the form of a System On Chip (SOC).

[0132] Meanwhile, the signal processing device (170) in the display device (100) of FIG. 4 may be the same as the signal processing device (170) of the vehicle control device of FIG. 5 and below.

[0133] FIG. 5 is an example of a configuration diagram of a signal processing device according to an embodiment of the present disclosure.

[0134] Referring to the drawings, a signal processing device (170) according to an embodiment of the present disclosure includes a processor (175).

[0135] Meanwhile, the signal processing device (170) may be named as an HPC (High Performance Computing) signal processing device.

[0136] Meanwhile, the signal processing device (170) according to an embodiment of the present disclosure may further include a neural processor (179).

[0137] Meanwhile, the neural processor (179) may also be referred to as an on-device-based learning processor.

[0138] Meanwhile, the processor (175) in the signal processing device (170) can execute the hypervisor (505) and execute the first to third virtualization machines (510 to 530) on the hypervisor (505).

[0139] The first virtualization machine (510) may be a gateway virtualization machine corresponding to the gateway (GWDa) of FIG. 2.

[0140] The second virtualization machine (520) may be a driving control virtualization machine corresponding to the autonomous driving control module (ACC) of FIG. 2.

[0141] The driving control virtualization machine (520) at this time can control the vehicle driving assistance (ADAS) or the autonomous driving (AD).

[0142] The third virtualization machine (530) may be a display virtualization machine corresponding to the cockpit control module (CPG) of FIG. 2 or the display (180a, 180b, 180c) of FIG. 3.

[0143] For example, the third virtualization machine (530) may include a cluster virtualization machine (530b) for a cluster display (180a) and an IVI virtualization machine (530b) for an IVI display (180b).

[0144] Meanwhile, the third virtualization machine (530) may further include a HUD virtualization machine (not shown) for a HUD display (180c).

[0145] Meanwhile, the processor (175) can share data between each virtualization machine (510~530) through shared memory (505) within the hypervisor (505).

[0146] Meanwhile, the processor (175) can share data with a plurality of sensor devices (SN) or ECUs (770) or area signal processing devices (170Z) through shared memory (505) within the hypervisor (505).

[0147] Meanwhile, the processor (175) can exchange data with an external server (400) or mobile terminal (600) through the communication device (120) of FIG. 4.

[0148] Meanwhile, the data received by the processor (175) within the signal processing device (170) may include camera data or sensor data.

[0149] For example, sensor data within the vehicle may include at least one of vehicle wheel speed data, vehicle direction data, vehicle location data (GPS data), vehicle angle data, vehicle speed data, vehicle acceleration data, vehicle tilt data, vehicle forward / reverse data, battery data, fuel data, tire data, vehicle lamp data, vehicle interior temperature data, vehicle interior humidity data, vehicle external radar data, and vehicle external lidar data.

[0150] Meanwhile, camera data may include external vehicle camera data and internal vehicle camera data.

[0151] Meanwhile, the processor (175) within the signal processing device (170) can execute multiple virtualization machines (510 to 530) based on safety standards.

[0152] Meanwhile, the processor (175) in the signal processing unit (170a) can execute the hypervisor (505) and, on the hypervisor (505), execute the first to third virtualization machines (510 to 530) according to the automotive safety integrity level (Automotive SIL and ASIL).

[0153] For example, the first virtualization machine (510) may be a virtualization machine corresponding to ASIL C or ASIL D, in which the sum of Severity, Exposure, and Controllability in the Automotive Safety Integrity Level (ASIL) is 9 or 10.

[0154] Meanwhile, ASIL D can correspond to the grade requiring the highest safety level.

[0155] The first virtualization machine (510) can run a safety operating system (not shown) and an application (not shown) on the safety operating system.

[0156] Meanwhile, the first virtualization machine (510) may run a container runtime (not shown) and a container runtime (not shown) on a safety operating system.

[0157] Meanwhile, unlike the drawing, the first virtualization machine (510) may also be executed through a separate processor core instead of the processor (175).

[0158] Meanwhile, the second virtualization machine (520) may be a virtualization machine corresponding to ASIL A or ASIL B, in which the sum of Severity, Exposure, and Controllability in the Automotive Safety Integrity Level (ASIL) is 7 or 8.

[0159] Meanwhile, the second virtualization machine (520) can run an operating system (not shown), a container runtime (not shown) on the operating system (not shown), and a container (not shown) on the container runtime.

[0160] Alternatively, the second virtualization machine (520) can run an operating system (not shown) and an application on the operating system (not shown).

[0161] Meanwhile, the third virtualization machine (530) may be a virtualization machine corresponding to Quality Management (QM), which is the lowest safety level and non-mandatory grade in the Automotive Safety Integrity Level (ASIL).

[0162] Meanwhile, the third virtualization machine (530) can run an operating system (not shown), a container runtime (not shown) on the operating system (not shown), and a container (not shown) on the container runtime.

[0163] Alternatively, the third virtualization machine (530) can run an operating system (not shown) and an application on the operating system (not shown).

[0164] FIG. 6 is an example of a block diagram of a vehicle control device according to an embodiment of the present disclosure.

[0165] Referring to the drawings, a vehicle control device (900) according to an embodiment of the present disclosure comprises a signal processing device (170).

[0166] A vehicle control device (900) according to an embodiment of the present disclosure may further include at least one display.

[0167] In the drawing, at least one display is exemplified as a cluster display (180a) and an IVI display (180b).

[0168] Meanwhile, the vehicle control device (900) may further include a plurality of area signal processing devices (170Z1 to 170Z4).

[0169] The signal processing device (170) at this time is a high-performance centralized signal processing and control device having a plurality of CPUs (175), GPUs (178), NPUs (179), etc., and can be named as a High Performance Computing (HPC) signal processing device or a central signal processing device.

[0170] Multiple area signal processing devices (170Z1~170Z4) and signal processing device (170) are connected by wired cables (CB1~CB4).

[0171] Meanwhile, multiple area signal processing devices (170Z1~170Z4) can be connected to each other by wired cables (CBa~CBd).

[0172] The wired cable (CBa~CBd) at this time may include a CAN communication cable, an Ethernet communication cable, or a PCI Express cable.

[0173] Meanwhile, the signal processing device (170) according to the embodiment of the present disclosure may have at least one processor (175, 178, 177) and a large-capacity storage device (925).

[0174] For example, a signal processing device (170) according to an embodiment of the present disclosure may include a central processor (175, 177), a graphics processor (178), and a neural processor (179).

[0175] Meanwhile, sensor data can be transmitted from at least one of the multiple area signal processing devices (170Z1 to 170Z4) to the signal processing device (170). In particular, the sensor data can be stored in a storage device (925) within the signal processing device (170).

[0176] The sensor data at this time may include at least one of camera data, lidar data, radar data, vehicle direction data, vehicle position data (GPS data), vehicle angle data, vehicle speed data, vehicle acceleration data, vehicle tilt data, vehicle forward / reverse data, battery data, fuel data, tire data, vehicle lamp data, vehicle interior temperature data, and vehicle interior humidity data.

[0177] In the drawing, camera data from a camera (195a) and lidar data from a lidar sensor (196) are input to a first area signal processing device (170Z1), and the camera data and lidar data are transmitted to a signal processing device (170) via a second area signal processing device (170Z2) and a third area signal processing device (170Z3), etc.

[0178] Meanwhile, since the data reading or writing speed to the storage device (925) is faster than the network speed when sensor data is transmitted from at least one of the multiple area signal processing devices (170Z1~170Z4) to the signal processing device (170), it is desirable to perform multipath routing so that network bottlenecks do not occur.

[0179] To this end, the signal processing device (170) according to an embodiment of the present disclosure can perform multipath routing based on a Software Defined Network (SDN). Accordingly, a stable network environment can be secured when reading or writing data of the storage device (925). Furthermore, since data can be transmitted to the storage device (925) using multiple paths, data can be transmitted by dynamically changing the network configuration.

[0180] Data communication between a plurality of area signal processing devices (170Z1~170Z4) and a signal processing device (170) within a vehicle control device (900) according to an embodiment of the present disclosure is preferably Peripheral Component Interconnect Express communication or Ethernet communication for high-bandwidth, low-latency communication.

[0181] FIG. 7 is an example of a block diagram of a signal processing device according to an embodiment of the present disclosure.

[0182] Referring to the drawings, the signal processing system (700) according to an embodiment of the present disclosure may be referred to as an IVEX (In Vehicle Experience) system.

[0183] Meanwhile, the signal processing system (700) according to an embodiment of the present disclosure may include an edge sensor group (SN), a signal processing device (170) in a vehicle, and a server (400).

[0184] Meanwhile, the signal processing device (170) according to an embodiment of the present disclosure includes a recognizer (720), an insightor (750), and an illustrator (780).

[0185] Meanwhile, the edge sensor group (SN) can correspond to a plurality of sensor devices (SN) of FIG. 4.

[0186] Meanwhile, the recognizer (720), the insightor (750), and the illustrator (780) may be included in the signal processing device (170) of the vehicle display device (100) of FIG. 4.

[0187] In particular, the recognizer (720), the insightor (750), and the illustrator (780) may be included in the processor (175) of the signal processing device (170).

[0188] Meanwhile, the edge sensor group (SN) may include vehicle interior sensors (710) and vehicle exterior sensors (715).

[0189] Meanwhile, the edge sensor group (SN) may further include a data receiving unit (718) for receiving vehicle internal data or vehicle external data. The data receiving unit (718) may correspond to the communication device (120) of FIG. 2.

[0190] Meanwhile, the vehicle interior sensors (710) are sensors placed inside the vehicle (200) and may include a front camera (195), lidar (196), radar (197), interior camera (195i), vehicle interior temperature sensor, or vehicle interior humidity sensor.

[0191] The vehicle external sensors (715) are sensors positioned on the exterior of the vehicle (200) and may include an external camera, lidar (196), radar (197), heading sensor, yaw sensor, gyro sensor, position module, vehicle forward / reverse sensor, wheel sensor, vehicle speed sensor, vehicle body inclination sensor, battery sensor, fuel sensor, tire sensor, or steering sensor based on steering wheel rotation. The position module may include a GPS module or a position sensor (198) for receiving GPS information.

[0192] Sensing data may include vehicle interior sensing data and vehicle exterior sensing data.

[0193] The vehicle interior sensing data may be data sensed by the vehicle interior sensors (710).

[0194] Vehicle interior sensing data may include at least one of vehicle interior temperature data, vehicle interior humidity data, battery data, fuel data, vehicle lamp data, tire data, or vehicle interior camera data, or audio data received through a microphone.

[0195] The vehicle external sensing data may be data sensed by the vehicle external sensors (715).

[0196] Vehicle external sensing data may include at least one of vehicle location data (GPS), vehicle direction data, vehicle angle data, vehicle speed data, vehicle acceleration data, vehicle tilt data, whether the vehicle is moving forward or backward, front camera data, or rear camera data.

[0197] Meanwhile, the recognizer (720) can recognize the vehicle situation based on sensing data received from the edge sensor group (SN). In this regard, the recognizer (720) may be referred to as a situation recognition unit.

[0198] The vehicle situation at this time may include external vehicle conditions and internal vehicle conditions.

[0199] Meanwhile, the recognizer (720) can recognize the vehicle situation based on the vehicle internal sensing data or the vehicle internal sensing data, and can generate vehicle situation information regarding the recognized vehicle situation.

[0200] For example, the vehicle situation information (740) may include at least one of the external situation information of the vehicle and the situation information of the vehicle's occupants.

[0201] As another example, the vehicle situation information (740) may include at least one of the vehicle's external situation information, the vehicle's internal situation information, and the vehicle's occupant situation information.

[0202] Meanwhile, the recognizer (720) can recognize external situation information of the vehicle based on external sensing data from the edge sensor group (SN), recognize internal situation information of the vehicle based on internal sensing data of the vehicle, or recognize situation information of the vehicle's occupant based on internal sensing data of the vehicle.

[0203] Meanwhile, the recognizer (720) can output external situation information of the vehicle based on internal vehicle sensing data or external vehicle sensing data, output internal situation information of the vehicle based on internal vehicle sensing data or external vehicle sensing data, or output situation information of the vehicle's occupant based on internal vehicle sensing data or external vehicle sensing data.

[0204] Meanwhile, the recognizer (720) can perform preprocessing and calibration of the vehicle interior sensing data or the vehicle interior sensing data, and can obtain vehicle situation information based on the preprocessed and calibrated sensing data.

[0205] For example, the internal recognition unit (725) within the recognition unit (720) can perform preprocessing and calibration (721) on the vehicle internal sensing data, and can obtain internal situation information of the vehicle based on the preprocessed and calibrated vehicle internal sensing data.

[0206] Specifically, the internal recognizer (725) can perform preprocessing and calibration (721) on the vehicle interior sensing data, perform primitive detection (722) or sensor fusion (723), and based on this, obtain information about the vehicle's interior situation.

[0207] For example, an external recognizer (730) within the recognizer (720) can perform preprocessing and calibration (721) on the external vehicle sensing data, and can obtain external situation information of the vehicle based on the preprocessed and calibrated external vehicle sensing data.

[0208] Specifically, the external recognizer (730) can perform preprocessing and calibration (731) on external vehicle sensing data, execute a perception engine (732), perform sensor fusion (733), or perform segmentation (734), and based on this, obtain external situation information of the vehicle.

[0209] Meanwhile, the passenger recognition device (730) within the recognition device (720) can perform passenger face identification (741), recognize distraction (742), recognize gaze, position, gesture (743), recognize emotion (744), recognize whether alcohol or medication is consumed (746), recognize drowsiness (747), or recognize information (748) regarding other actions based on the vehicle interior sensing data, and can obtain situational information of the vehicle's passenger based thereon.

[0210] Meanwhile, the situational information of the occupant may include at least one of the following: a face ID of the occupant inside the vehicle (200), distraction information, gaze information, position information, gesture information, emotion information, information on whether alcohol or medication has been taken, drowsiness information, or other information regarding actions.

[0211] Meanwhile, the recognizer (720) can transmit the generated vehicle situation information to the insightor (750).

[0212] Meanwhile, the insightor (750) can generate an inference result or a response result based on the situation information of the vehicle received from the recognizer (720).

[0213] Meanwhile, the insightor (750) can generate an inference result or a response result based on the vehicle situation information received from the recognizer (720), and can generate or control a service to be executed based on the inference result or the response result.

[0214] Accordingly, the insightor (750) may be named a response result generation unit or a service generation unit.

[0215] Meanwhile, the insightor (750) may include a multimodal context engine (755), a context controller (754), a safety assistance engine (757), a user characteristic engine (758), an AI orchestrator (760), and an AI model set (765).

[0216] Meanwhile, the multimodal context engine (755) can generate multimodal context data based on the vehicle situation information received from the recognizer (720).

[0217] For example, multimodal context data may include at least one of text data or image data describing a vehicle situation generated based on vehicle situation information.

[0218] Meanwhile, the context controller (754) may be a component included in the multimodal context engine (755) or provided separately from the multimodal context engine (755).

[0219] Meanwhile, the context controller (754) can execute the process of generating multimodal context data when it receives a passenger query from the AI ​​orchestrator (760).

[0220] For example, the passenger curie may be a voice recognition result corresponding to a voice command spoken by the passenger.

[0221] Meanwhile, the user characteristic engine (758) can generate user context data or passenger context data based on user information or passenger information.

[0222] Meanwhile, user information or passenger information may be referred to as user persona or passenger persona.

[0223] Meanwhile, user information or passenger information may include at least one of the user or passenger's nationality, age, gender, occupation, personality, or psychological type (Myers-Briggs Type Indicator, MBTI).

[0224] Meanwhile, the AI ​​orchestrator (760) can generate a prompt based on at least one of multimodal context data or passenger context data.

[0225] Meanwhile, the AI ​​orchestrator (760) can send the generated prompt to the AI ​​model set (765).

[0226] Meanwhile, the AI ​​model set (765) may include at least one AI model.

[0227] For example, the AI ​​model set (765) may include an on-device AI model (767).

[0228] Meanwhile, the on-device AI model (767) may include at least one AI model. For example, the on-device AI model (767) may include a Small Language Model (LLM).

[0229] Meanwhile, the AI ​​model set (765) may include an interface (766) for data exchange with the AI ​​model (769) in the server (400).

[0230] Meanwhile, the AI ​​model (769) within the server (400) may include at least one AI model. For example, the AI ​​model (769) within the server (400) may include a Large Language Model (LLM).

[0231] That is, the on-device AI model (767) may have a smaller capacity or size than the AI ​​model (769) in the server (400).

[0232] Meanwhile, the AI ​​model set (765) can output an inference result in response to a prompt received from the AI ​​orchestrator (760) and can transmit the inference result to the AI ​​orchestrator (760).

[0233] Meanwhile, the AI ​​orchestrator (760) can generate additional prompts based on the inference results and send the additional prompts to the AI ​​model set (765).

[0234] Meanwhile, the AI ​​model set (765) that receives the additional prompt can output an additional inference result in response to the additional prompt and can transmit the additional inference result to the AI ​​orchestrator (760).

[0235] The AI ​​orchestrator (760) can obtain an inference result or additional inference result received from the AI ​​model set (765) as a response result, and can output the obtained response result to the illustrator (780).

[0236] Meanwhile, the safety assist engine (757) can determine whether the current situation is a safe situation or a dangerous situation based on the vehicle situation information received from the recognizer (720).

[0237] Meanwhile, the safety assist engine (757) can transmit a driver assistance control command or a warning notification output command to the safety application (790) of the illustrator (780) if the current situation is determined to be a dangerous situation.

[0238] Meanwhile, the safety assistance engine (757) can set a mode for vehicle driving assistance control or autonomous driving based on multimodal context data or based on passenger context data.

[0239] For example, the safety assistance engine (757) can distinguish the mode of vehicle driving assistance control or autonomous driving into any one of off mode, on mode, or safety mode based on multimodal context data or based on occupant context data.

[0240] Meanwhile, the safety assistance engine (757) calculates the risk level of vehicle driving based on multimodal context data or based on passenger context data, and based on the calculated risk level, can be set to one of the off mode, on mode, or safety mode of vehicle driving assistance control or autonomous driving.

[0241] Meanwhile, the safety assistance engine (757) can set a mode for vehicle driving assistance control or autonomous driving based on multimodal context data or based on occupant context data, and control authority adjustment, safety extension, or sensitivity adjustment for vehicle control based on the set mode.

[0242] Meanwhile, the safety assistance engine (757) can determine whether it is a danger warning stage or a danger situation stage based on multimodal context data or based on passenger context data, and control the operation mode of the vehicle driving assistance control or autonomous driving to be variable according to the danger warning stage or danger situation stage.

[0243] For example, the safety assistance engine (757) can control the vehicle driving assistance control or autonomous driving operation mode to remain off mode or on mode when the vehicle driving situation information corresponds to a danger warning stage based on multimodal context data or based on occupant context data.

[0244] As another example, the safety assistance engine (757) can control the vehicle driving assistance control or autonomous driving operation mode to operate in safety mode by disabling the off mode or on mode when the vehicle driving situation information corresponds to a dangerous situation stage based on multimodal context data or based on occupant context data. Accordingly, it is possible to quickly and accurately determine the situation inside and outside the vehicle and adaptively perform vehicle control.

[0245] Meanwhile, the safety assist engine (757) may be included within the safety area (756) as shown in the drawing, but is not limited thereto, and may also be provided separately within the insightor (750) outside the safety area (756).

[0246] Meanwhile, the illustrator (780) can execute a service, output service information, execute an application, or output application information based on the response result output from the insightor (750).

[0247] For example, the illustrator (780) can execute a navigation service, execute a vehicle driving assistance control service, execute an autonomous driving service, or execute a display-related service based on the response result output from the insightor (750).

[0248] As another example, the illustrator (780) can run a navigation application, run a vehicle driving assistance control application, run an autonomous driving application, or run a display-related application based on the response result output from the insightor (750).

[0249] Meanwhile, the illustrator (780) may include a multimodal output encoder (781), a visual interface (782), an audio interface (785), and a safety application (790).

[0250] Meanwhile, the multimodal output encoder (781) can encode the response result output from the insightor (750) and output the encoded response result data to the visual interface (782) or audio interface (785).

[0251] Meanwhile, the visual interface (782) or audio interface (785) may be referred to as an output interface.

[0252] Meanwhile, the visual interface (782) can output response result data output from the multimodal output encoder (781).

[0253] Accordingly, at least one of the plurality of displays (180a to 180c) of FIG. 4 can display an image based on response result data from the visual interface (782).

[0254] Meanwhile, the visual interface (782) can output augmented reality (AR) video based on response result data or mixed reality (MR) video data based on response result data.

[0255] Meanwhile, the audio interface (785) can output response result data in the form of audio.

[0256] Accordingly, the audio output unit (185) of FIG. 4 can output a sound corresponding to the response result data from the audio interface (785).

[0257] Meanwhile, the safety application (790) can perform Advanced Driver Assistance System (ADAS) control based on the driver assistance control command or the response result received from the insightor (750).

[0258] For example, the safety application (790) can output warning notification data according to a warning notification output command.

[0259] In response to this, at least one of the electronic control unit (770) of FIG. 4 or a plurality of displays (180a to 180c) can output warning notification data.

[0260] FIGS. 8a to 8c are drawings referenced in the description of FIG. 7.

[0261] First, FIG. 8a is a drawing referenced in the description of the insight of FIG. 7.

[0262] Referring to the drawing, the insightor (750) may include a multimodal context engine (755), an AI orchestrator (760), and a multimodal LLM (767).

[0263] Meanwhile, the multimodal context engine (755) may include a multimodal signal adapter (751), a multimodal indexer (812), a multimodal context buffer (752), a multimodal context retriever (814), a multimodal context descriptor (815), a multimodal event monitor (811), and a context controller (754).

[0264] Unlike Fig. 7, the multimodal context engine (755) may include a context controller (754).

[0265] Meanwhile, the multimodal signal adapter (751) can generate pre-processed multimodal data by filtering, cleaning, synchronizing, and reformulating data received from various sensors or vehicle situation information received from the recognizer (720).

[0266] Meanwhile, the multimodal signal adapter (751) can receive at least one of audio data received through a microphone, front image data captured through a front camera, ADAS information obtained from the front image data or vehicle sensor, location data, IVI (In-Vehicle Infotainment) system data, IVI display information, DMS (Driver Monitoring System) information or IMS (Interior Monitoring System) information based on image data captured through an interior camera (195i), and biometric data obtained from a biometric sensor.

[0267] Meanwhile, the multimodal indexer (812) can generate multimodal processing data by dividing the preprocessed multimodal data into chunks and can index the multimodal processing data.

[0268] Meanwhile, the multimodal indexer (812) can obtain an encoding vector or keyword representing the attribute (or meaning) of the multimodal processed data divided into chunks as an index.

[0269] Meanwhile, the multimodal context buffer (752) can store multimodal processing data and an index corresponding to the multimodal processing data.

[0270] Meanwhile, the multimodal context retriever (814) can search for multimodal processing data most related to the passenger query through an index from the multimodal context buffer (752), and can select the searched multimodal processing data as a context candidate for the creation of multimodal context data.

[0271] Meanwhile, the multimodal context descriptor (815) can reconfigure multimodal processing data selected as a context candidate into multimodal context data having a prompt form that the multimodal LLM (767) can interpret.

[0272] Meanwhile, the multimodal event monitor (811) can monitor whether an index matching the trigger condition is entered when a trigger condition for multimodal data is registered. If an index matching the trigger condition is entered, the multimodal event monitor (811) can generate a trigger event to generate a proactive service query.

[0273] Meanwhile, the context controller (754) can control the overall operation of the multimodal context engine (755).

[0274] Meanwhile, the context controller (754) can execute the process of generating multimodal context data when it receives a passenger query from the AI ​​orchestrator (760).

[0275] Meanwhile, the context controller (754) can generate a preemptive service query when it receives a trigger event from the multimodal event monitor (811).

[0276] Meanwhile, the AI ​​orchestrator (760) can generate a prompt by combining the passenger query and the multimodal context, and can send the generated prompt to the multimodal LLM (767).

[0277] Meanwhile, the passenger curry may be a query in the form of recognized text based on a voice command spoken by the passenger. The voice command may be received through a microphone, and the voice command may be converted into text through an Automatic Speech Recognition (ASR) process.

[0278] Meanwhile, passenger curry may be text converted through an Automatic Speech Recognition (ASR) process.

[0279] Meanwhile, multimodal LLM (767) may be an example of an AI model included in the set of AI models (765) of FIG. 6a.

[0280] Meanwhile, the multimodal LLM (767) can output an inference result from a prompt received from the AI ​​orchestrator (760) and can transmit the inference result to the AI ​​orchestrator (760).

[0281] Meanwhile, the inference result may include a result representing a response to the prompt.

[0282] For example, the inference result may include an API call result regarding whether an API call corresponding to a driving assistance function was successfully performed, and a feedback generation result regarding whether feedback corresponding to the API call was successfully generated.

[0283] Next, Fig. 8b is a drawing referenced in the description of the AI ​​orchestrator of Fig. 7.

[0284] Referring to the drawing, the AI ​​orchestrator (760) may include a task arbitrator (771), a prompt manager (763), a sub-agent set (761), a knowledge database (762), a workflow controller (772), and a tool set / adapter set (764).

[0285] Meanwhile, the task arbitrator (771) can determine one of the multiple sub-agents based on the voice recognition result and the multimodal context data output from the multimodal context engine (755).

[0286] Meanwhile, the prompt manager (763) can generate a prompt for the operation of the determined sub-agent. The prompt manager (763) can generate a prompt based on multimodal context data and passenger queries.

[0287] Meanwhile, the prompt manager (763) can generate a prompt based on information about the functions that the determined sub-agent can perform, multimodal context data, the results of previously performed functions, conversation history and passenger query.

[0288] Meanwhile, the sub-agent set (761) may include multiple sub-agents.

[0289] For example, the sub-agent set (761) may include a navigation agent for navigation services, a vehicle function agent for providing vehicle functions, a telephony agent for automated telephone answering services, and a Q&A agent for providing response services to queries.

[0290] Meanwhile, the sub-agent determined by the task arbitrator (771) can call the cloud AI model (769) or on-device AI model (767) assigned to it.

[0291] Meanwhile, the cloud AI model (769) or on-device AI model (767) may be a Large Language Model (LLM).

[0292] Meanwhile, a sub-agent within a sub-agent set (761) can call a cloud AI model (769) or an on-device AI model (767) assigned to it to obtain an inference result corresponding to a prompt from the model.

[0293] Meanwhile, a sub-agent in the sub-agent set (761) can call the knowledge database (762) to provide additional information based on the obtained inference result, obtain additional information from the knowledge database (762), specify the name of the function to be executed and the parameter value of the function, and determine the feedback phrase to be provided to the user.

[0294] Meanwhile, a sub-agent in the sub-agent set (761) can transmit to the workflow controller (772) a parsing result including additional information called from the knowledge database (762), the name of the function to be executed, the parameter value of the function, and a feedback phrase, which is generated by parsing the inference result received from the model.

[0295] Meanwhile, the knowledge database (762) can store additional information and information about functions. The information about functions may include the name of the function and the parameter values ​​of the function.

[0296] Meanwhile, the workflow controller (772) can generate control commands to perform the corresponding function based on the parsing results and can store the results of the conversation and function performed by the AI ​​model and sub-agent.

[0297] Meanwhile, the tool set / adapter set (764) can call an API corresponding to a control command generated by the workflow controller (772). The tool set / adapter set (764) can convert the control command into an execution command of an actual function or an API call command that the IVI system can understand and execute, and can execute the converted command.

[0298] Fig. 8c is an example of an internal block diagram of the server of Fig. 7.

[0299] Referring to the drawing, the server (400) may represent a device that trains an artificial neural network using a machine learning algorithm or uses a trained artificial neural network.

[0300] Here, the server (400) may be composed of multiple servers to perform distributed processing. Alternatively, the server (400) may be defined as a 5G network.

[0301] The server (400) may be included as part of the vehicle (200) and may perform at least some of the AI ​​processing together.

[0302] The server (400) may include a communication interface (410), memory (430), a learning processor (440), and a processor (470).

[0303] The communication interface (410) can transmit and receive data with the vehicle (200) or an external device.

[0304] The memory (430) may include a model storage unit (431). The model storage unit (431) may store a model (or artificial neural network, 531a) that is being learned or has been learned through the learning processor (440).

[0305] The learning processor (440) can train the artificial neural network (431a) using the training data.

[0306] For example, the learning processor (440) can execute a learning model. The learning model may include a cloud AI model (769) such as FIG. 7.

[0307] The learning model may be used while mounted on the server (400) of the artificial neural network, or may be used while mounted on an external device such as a vehicle (200).

[0308] The learning model may be implemented in hardware, software, or a combination of hardware and software. If part or all of the learning model is implemented in software, one or more instructions constituting the learning model may be stored in memory (430).

[0309] The processor (470) can infer a result value for new input data using a learning model and generate a response or control command based on the inferred result value.

[0310] Meanwhile, the processor (470) can control the learning processor (440) to execute the cloud AI model (769) based on requests from the signal processing device (170) in the vehicle (200), etc.

[0311] FIG. 9 is another example of a block diagram of a signal processing device according to an embodiment of the present disclosure.

[0312] Referring to the drawings, a signal processing device (170) according to an embodiment of the present disclosure includes a processor (175) that receives camera data outside the vehicle, camera data inside the vehicle, and sensor data inside the vehicle.

[0313] Meanwhile, the processor (175) according to an embodiment of the present disclosure performs object detection based on camera data outside the vehicle and monitors the state of a passenger, including a driver, based on camera data inside the vehicle.

[0314] Meanwhile, the processor (175) according to an embodiment of the present disclosure extracts multimodal context data corresponding to a driving context related to vehicle driving based on monitored passenger status information, detected object information, sensor data, and driver's driving pattern information stored in a database.

[0315] Meanwhile, the processor (175) according to an embodiment of the present disclosure sets a mode for vehicle driving assistance control or autonomous driving based on the extracted multimodal context data.

[0316] Accordingly, it becomes possible to perform adaptive vehicle control by rapidly and accurately determining the situation inside and outside the vehicle. In particular, it enables adaptive vehicle control by rapidly and accurately determining the situation inside and outside the vehicle based on multimodal context data.

[0317] Meanwhile, the processor (175) according to the embodiment of the present disclosure can be controlled to operate in an off mode, on mode, or safety mode of vehicle driving assistance control or autonomous driving based on multimodal context data. Accordingly, it is possible to quickly and accurately determine the situation inside and outside the vehicle and adaptively perform vehicle control.

[0318] Meanwhile, the processor (175) according to the embodiment of the present disclosure calculates a risk level of vehicle driving based on multimodal context data and, based on the calculated risk level, can set the vehicle driving assistance control or autonomous driving to one of an off mode, an on mode, or a safety mode. Accordingly, it is possible to quickly and accurately determine the situation inside and outside the vehicle and adaptively perform vehicle control.

[0319] Meanwhile, the processor (175) according to an embodiment of the present disclosure can set a mode for vehicle driving assistance control or autonomous driving based on multimodal context data, and control authority adjustment, safety extension, or sensitivity adjustment for vehicle control based on the set mode. Accordingly, it is possible to quickly and accurately determine the situation inside and outside the vehicle and adaptively perform vehicle control.

[0320] Meanwhile, the processor (175) according to an embodiment of the present disclosure can determine whether it is a danger warning stage or a danger situation stage based on multimodal context data, and control the operation mode of vehicle driving assistance control or autonomous driving to vary according to the danger warning stage or danger situation stage. Accordingly, it is possible to quickly and accurately determine the situation inside and outside the vehicle and adaptively perform vehicle control.

[0321] Meanwhile, the processor (175) according to an embodiment of the present disclosure can obtain an inference result or a response result based on extracted multimodal context data, calculate a risk level of vehicle driving based on the obtained inference result or response result, and output a vehicle control signal based on the calculated risk level. Accordingly, it is possible to quickly and accurately determine the situation inside and outside the vehicle and adaptively perform vehicle control.

[0322] Meanwhile, the processor (175) in the signal processing device (170) according to an embodiment of the present disclosure may include a recognizer (720), an insightor (750), and an illustrator (780).

[0323] Meanwhile, the recognizer (720) performs object detection based on camera data outside the vehicle and monitors the status of passengers, including the driver, based on camera data inside the vehicle.

[0324] Meanwhile, the recognizer (720) can recognize the vehicle situation based on sensing data received from the edge sensor group (SN). In this regard, the recognizer (720) may be referred to as a situation recognition unit. The vehicle situation at this time may include the vehicle external situation and the vehicle internal situation.

[0325] Meanwhile, the edge sensor group (SN) can correspond to a plurality of sensor devices (SN) of FIG. 4.

[0326] Meanwhile, the edge sensor group (SN) may include a microphone (112) which is an example of an input device (110), a vehicle interior camera (195i), a vehicle front camera (195), and a position sensor (198) that receives GPS data, etc.

[0327] Among these, the microphone (112) and the vehicle interior camera (195i) may be included in the vehicle interior sensors (710) of FIG. 7.

[0328] Meanwhile, the vehicle front camera (195) or position sensor (198) may be included in the vehicle external sensors (715).

[0329] Meanwhile, the recognizer (720) can recognize the situation inside the vehicle based on the vehicle interior sensing data (725) or the vehicle interior situation information (740a), and can generate the vehicle's situation information based on the recognized vehicle interior situation information.

[0330] The vehicle interior sensing data (725) at this time may be body sensing data of an occupant, including a driver or a passenger. For example, the body sensing data may include heart rate sensing data, respiration sensing data, etc.

[0331] Meanwhile, the vehicle interior situation information (740a) may include motion information and voice information of a passenger, including a driver or co-passenger, from the vehicle interior camera (195i) and microphone (112).

[0332] Meanwhile, the vehicle situation information (740 in FIG. 7) may include external vehicle situation information in addition to internal vehicle situation information.

[0333] Meanwhile, the vehicle's internal situation information may include occupant situation information. For example, the vehicle's internal situation information may include driver situation information and passenger situation information.

[0334] Meanwhile, the recognizer (720) can recognize external situation information of the vehicle based on external sensing data from the edge sensor group (SN), recognize internal situation information of the vehicle based on internal sensing data of the vehicle, or recognize situation information of the vehicle's occupant based on internal sensing data of the vehicle.

[0335] Meanwhile, the recognizer (720) can output external situation information of the vehicle based on internal vehicle sensing data or external vehicle sensing data, output internal situation information of the vehicle based on internal vehicle sensing data or external vehicle sensing data, or output situation information of the vehicle's occupant based on internal vehicle sensing data or external vehicle sensing data.

[0336] Meanwhile, the recognizer (720) can perform vehicle sensing data or preprocessing and calibration of vehicle sensing data, and can obtain vehicle situation information based on the preprocessed and calibrated sensing data.

[0337] Meanwhile, vehicle sensing data may include vehicle interior sensing data and vehicle exterior sensing data.

[0338] Meanwhile, vehicle situation information may include internal vehicle situation information and external vehicle situation information.

[0339] For example, the recognizer (720) can perform preprocessing and calibration of the vehicle interior sensing data, and can obtain information on the vehicle's interior situation based on the preprocessed and calibrated vehicle interior sensing data.

[0340] For example, the recognizer (720) can perform preprocessing and calibration on the vehicle external sensing data, and can obtain external situation information of the vehicle based on the preprocessed and calibrated vehicle external sensing data.

[0341] Meanwhile, the recognition device (720) can perform facial identification of the occupant based on the vehicle interior sensing data, recognize distraction, recognize gaze, position, gesture, recognize emotion, recognize whether alcohol or medication has been taken, recognize drowsiness, or recognize information (748) about other actions, and obtain situational information of the vehicle occupant based thereon.

[0342] Meanwhile, the situational information of the occupant may include at least one of the following: a face ID of the occupant inside the vehicle (200), distraction information, gaze information, position information, gesture information, emotion information, information on whether alcohol or medication has been taken, drowsiness information, or other information regarding actions.

[0343] Meanwhile, the recognizer (720) can transmit the generated vehicle situation information to the insightor (750).

[0344] Meanwhile, the recognizer (720) receives camera data from outside the vehicle from the vehicle's external camera (195) and performs object detection based on the camera data from outside the vehicle.

[0345] For example, the recognizer (720) can receive vehicle front image data from the vehicle's external camera (195), perform segmentation on the vehicle front image data, and detect multiple objects (735).

[0346] Meanwhile, the recognizer (720) can receive location information from the location sensor (198 in FIG. 4) and transmit the location information to the insightor (750).

[0347] Meanwhile, the insightor (750) extracts multimodal context data corresponding to a driving context related to vehicle driving based on monitored passenger status information, detected object information, sensor data, and driver's driving pattern information stored in a database.

[0348] Meanwhile, the insightor (750) generates a prompt for vehicle driving control based on the extracted multimodal context data.

[0349] Meanwhile, the insightor (750) sets a mode for vehicle driving assistance control or autonomous driving based on multimodal context data.

[0350] Meanwhile, the insightor (750) can be controlled to operate in an off mode, on mode, or safety mode of vehicle driving assistance control or autonomous driving based on multimodal context data.

[0351] For example, the insightor (750) can calculate a risk level of vehicle driving based on an inference result or response result received from an AI orchestrator (760 in FIG. 7) or an AI model set (765 in FIG. 7), and based on the calculated risk level, set the vehicle driving assistance control or autonomous driving to an off mode, on mode, or safety mode.

[0352] Meanwhile, the insightor (750) can calculate the risk level of vehicle driving based on the extracted multimodal context data.

[0353] Meanwhile, the insightor (750) can calculate the risk level of vehicle driving based on multimodal context data.

[0354] Meanwhile, the insightor (750) can generate an inference result or a response result based on the situation information of the vehicle received from the recognizer (720).

[0355] Meanwhile, the insightor (750) can generate an inference result or a response result based on the vehicle situation information received from the recognizer (720), and can generate or control a service to be executed based on the inference result or the response result.

[0356] Accordingly, the insightor (750) may be named a response result generation unit or a service generation unit.

[0357] Meanwhile, the insightor (750) may include a multimodal context engine (755) and a safety assistance engine (757).

[0358] Meanwhile, the multimodal context engine (755) can generate multimodal context data based on the vehicle situation information received from the recognizer (720).

[0359] Specifically, the multimodal context engine (755) can generate multimodal context data corresponding to a driving context related to vehicle driving based on monitored passenger status information, detected object information, sensor data, and driver's driving pattern information stored in a database.

[0360] For example, multimodal context data may include at least one of text data, image data, or audio data describing a vehicle situation generated based on vehicle situation information.

[0361] Meanwhile, the multimodal context engine (755) can generate multimodal context data based on real-time context data (905) and feature data (907).

[0362] Meanwhile, real-time context data (905) may include monitored passenger status information, detected object information, and in-vehicle sensor data.

[0363] For example, real-time context data (905) may include monitored driver status information, detected object information, and in-vehicle sensor data.

[0364] As another example, real-time context data (905) may include monitored driver status information, monitored passenger status information, detected object information, and in-vehicle sensor data.

[0365] Meanwhile, the characteristic data (907) may include the characteristic data of the occupant.

[0366] For example, characteristic data (907) may include characteristic data of the driver.

[0367] As another example, characteristic data (907) may include characteristic data of the driver and characteristic data of the passenger.

[0368] Meanwhile, the safety assistance engine (757) may include an on-device AI model (767), a safety extension module (910), an authority adjustment module (915), and a sensitivity adjustment module (920).

[0369] Meanwhile, the safety assistance engine (757) can set a mode for vehicle driving assistance control or autonomous driving based on the extracted multimodal context data.

[0370] Meanwhile, the safety assistance engine (757) can set a mode for vehicle driving assistance control or autonomous driving based on multimodal context data or passenger context data.

[0371] For example, the safety assistance engine (757) can distinguish the mode of vehicle driving assistance control or autonomous driving as an off mode, an on mode, or a safety mode based on multimodal context data or occupant context data.

[0372] Meanwhile, the safety assistance engine (757) calculates the risk level of vehicle driving based on multimodal context data or occupant context data, and based on the calculated risk level, can be set to one of the off mode, on mode, or safety mode of vehicle driving assistance control or autonomous driving.

[0373] Meanwhile, the safety assistance engine (757) can set a mode for vehicle driving assistance control or autonomous driving based on multimodal context data or occupant context data, and control authority adjustment, safety extension, or sensitivity adjustment for vehicle control based on the set mode.

[0374] Meanwhile, the safety assistance engine (757) can determine whether it is a danger warning stage or a danger situation stage based on multimodal context data or passenger context data, and control the operation mode of the vehicle driving assistance control or autonomous driving to be variable according to the danger warning stage or danger situation stage.

[0375] For example, the safety assistance engine (757) can control the vehicle driving assistance control or autonomous driving operation mode to remain off mode or on mode when the vehicle driving situation information corresponds to a danger warning stage based on multimodal context data or occupant context data.

[0376] As another example, the safety assistance engine (757) can control the vehicle driving assistance control or autonomous driving operation mode to operate in safety mode by disabling the off mode or on mode when the vehicle driving situation information corresponds to a dangerous situation stage based on multimodal context data or occupant context data. Accordingly, it is possible to quickly and accurately determine the situation inside and outside the vehicle and adaptively perform vehicle control.

[0377] Meanwhile, the on-device AI model (767) can output an inference result or response result corresponding to the prompt.

[0378] Meanwhile, the safety assistance engine (757) can set a mode for vehicle driving assistance control or autonomous driving based on the inference result or response result received from the on-device AI model (767).

[0379] For example, the safety assistance engine (757) can distinguish the mode of vehicle driving assistance control or autonomous driving as an off mode, an on mode, or a safety mode based on the inference result or response result received from the on-device AI model (767).

[0380] Meanwhile, the safety assistance engine (757) calculates the risk level of vehicle driving based on the inference result or response result received from the on-device AI model (767), and based on the calculated risk level, can set the vehicle driving assistance control or autonomous driving to an off mode, on mode, or safety mode.

[0381] Meanwhile, the safety assistance engine (757) can set a mode for vehicle driving assistance control or autonomous driving based on the inference result or response result received from the on-device AI model (767), and control authority adjustment, safety extension, or sensitivity adjustment for vehicle control based on the set mode.

[0382] Meanwhile, the safety assistance engine (757) can determine whether it is a danger warning stage or a danger situation stage based on the inference result or response result received from the on-device AI model (767), and control the operation mode of the vehicle driving assistance control or autonomous driving to be variable according to the danger warning stage or danger situation stage.

[0383] For example, the safety assistance engine (757) can control the vehicle driving assistance control or autonomous driving operation mode to remain off or on based on the inference result or response result received from the on-device AI model (767), if the vehicle driving situation information corresponds to a danger warning stage.

[0384] As another example, the safety assistance engine (757) can control the vehicle driving assistance control or autonomous driving operation mode to operate in safety mode by disabling the off mode or on mode based on inference results or response results received from the on-device AI model (767), if the vehicle driving situation information corresponds to a dangerous situation stage. Accordingly, it is possible to quickly and accurately determine the situation inside and outside the vehicle and adaptively perform vehicle control.

[0385] Meanwhile, the safety assist engine (757) can calculate the risk level of vehicle driving based on the extracted multimodal context data.

[0386] Meanwhile, the safety assist engine (757) can obtain an inference result or a response result based on the extracted multimodal context data, and calculate the risk level of vehicle driving based on the obtained inference result or response result.

[0387] Specifically, the on-device AI model (767) can perform inference or response based on the extracted multimodal context data and output the inference result or response result.

[0388] And, the safety assistance engine (757) can calculate the risk level of vehicle driving based on the inference result or response result obtained from the on-device AI model (767).

[0389] Meanwhile, the safety assist engine (757) can classify the risk level of vehicle driving into multiple levels.

[0390] For example, the safety assist engine (757) can classify the risk level of vehicle driving into levels 1 through 4.

[0391] Meanwhile, the safety assist engine (757) can calculate the risk level of vehicle driving as the first level when the driver's driving ability information among the monitored passenger status information indicates that the driver can operate the vehicle.

[0392] Meanwhile, the safety assist engine (757) can calculate the risk level of vehicle driving to a second level higher than the first level when the driver's driving ability information indicates that the driver is capable of operating the vehicle or is distracted.

[0393] Meanwhile, the safety assist engine (757) can calculate the risk level of vehicle driving to a third level higher than the second level when the driver's driving ability information indicates that the driver can operate the vehicle or that the reaction is delayed.

[0394] Meanwhile, the safety assist engine (757) can calculate the risk level of vehicle driving to a fourth level higher than the third level when the driver's driving ability information indicates that the driver cannot operate the vehicle. Accordingly, it is possible to quickly and accurately determine the situation inside and outside the vehicle and adaptively perform vehicle control.

[0395] Meanwhile, the safety assist engine (757) can calculate the risk level of vehicle driving as the first level if the occupant characteristic information does not include habit behavior information or is not personalized habit behavior information.

[0396] Meanwhile, the safety assist engine (757) can calculate the risk level of vehicle driving as a second level higher than the first level when the occupant characteristic information includes habitual behavior information that interferes with driving.

[0397] Meanwhile, the safety assist engine (757) can calculate the risk level of vehicle driving as a third level higher than the second level if the occupant characteristic information includes repeated dangerous behaviors and medical history or accident history.

[0398] Meanwhile, the safety assist engine (757) can calculate the risk level of vehicle driving as a fourth level higher than the third level when the occupant characteristic information includes risk information based on medical history. Accordingly, it is possible to quickly and accurately determine the situation inside and outside the vehicle and adaptively perform vehicle control.

[0399] Meanwhile, the safety assist engine (757) can calculate the risk level of vehicle driving as a first level when the passenger's behavior information includes stable state information.

[0400] Meanwhile, the safety assist engine (757) can calculate the risk level of vehicle driving as a second level higher than the first level when the passenger's behavior information includes voice information at a level below the threshold.

[0401] Meanwhile, the safety assist engine (757) can calculate the risk level of vehicle driving as a third level higher than the second level when the passenger's behavior information includes voice information at a level higher than the repeated threshold.

[0402] Meanwhile, the safety assist engine (757) can calculate the risk level of vehicle driving as a fourth level higher than the third level if the passenger's behavior information includes actions that interfere with the driver's driving operation. Accordingly, it is possible to quickly and accurately determine the situation inside and outside the vehicle and adaptively perform vehicle control.

[0403] Meanwhile, the safety assist engine (757) can control the authority adjustment, safety extension, or sensitivity adjustment for vehicle control based on the risk level of vehicle driving.

[0404] Meanwhile, the authority adjustment module (915) can calculate authority adjustment data for vehicle control based on the risk level of vehicle driving.

[0405] Meanwhile, the authority adjustment module (915) can control the authority adjustment for vehicle control to remain unchanged when the risk level of vehicle driving is at the first level.

[0406] Meanwhile, the authority adjustment module (915) can switch the authority adjustment for vehicle control to a standby state when the risk level of vehicle driving is a second level higher than the first level.

[0407] Meanwhile, the authority adjustment module (915) can ignore or supplement the operation input within the authority adjustment for vehicle control when the risk level of vehicle driving is a third level higher than the second level.

[0408] Meanwhile, the authority adjustment module (915) can control steering and acceleration for authority adjustment for vehicle control when the risk level of vehicle driving is at the highest level, level 4.

[0409] Meanwhile, the safety extension module (910) can calculate safety extension data for safety extension based on the risk level of vehicle driving.

[0410] Meanwhile, the safety extension module (910) can control the safety extension to remain as is when the risk level of vehicle driving is at the first level.

[0411] Meanwhile, the safety extension module (910) can switch the safety extension for vehicle control to a standby state when the risk level of vehicle driving is at a second level higher than the first level.

[0412] Meanwhile, the safety extension module (910) can increase the threshold value within the safety extension for vehicle control when the risk level of vehicle driving is a third level higher than the second level.

[0413] Meanwhile, the safety extension module (910) can set the threshold value within the safety extension to the maximum value when the risk level of vehicle driving is the highest level, which is the fourth level.

[0414] Meanwhile, the sensitivity adjustment module (920) can calculate sensitivity adjustment data for sensitivity adjustment based on the risk level of vehicle driving.

[0415] Meanwhile, the sensitivity adjustment module (920) can control the sensitivity adjustment for vehicle control to remain unchanged when the risk level of vehicle driving is at the first level.

[0416] Meanwhile, the sensitivity adjustment module (920) can switch the sensitivity adjustment for vehicle control to a standby state or an initial state when the risk level of vehicle driving is a second level higher than the first level.

[0417] Meanwhile, the sensitivity adjustment module (920) can increase the sensitivity of the sensitivity adjustment above the second level when the risk level of vehicle driving is a third level higher than the second level.

[0418] Meanwhile, the sensitivity adjustment module (920) can set the sensitivity of the sensitivity adjustment to the maximum value when the risk level of vehicle driving is the highest level, which is the fourth level.

[0419] Meanwhile, the illustrator (780) can execute a service, output service information, execute an application, or output application information based on the inference result or response result output from the insightor (750).

[0420] For example, the illustrator (780) can execute an autonomous driving service or a driving assistance service based on the response result output from the insightor (750).

[0421] Meanwhile, the illustrator (780) outputs a vehicle control signal based on the risk level calculated by the insightor (750). Accordingly, the situation inside and outside the vehicle can be determined quickly and accurately to perform adaptive vehicle control. In particular, the situation inside and outside the vehicle can be determined quickly and accurately based on multimodal context data to perform adaptive vehicle control.

[0422] Meanwhile, the illustrator (780) can generate a vehicle control signal based on authority adjustment data, safety extension data, and sensitivity adjustment data from the insightor (750).

[0423] For example, the illustrator (780) can generate vehicle control signals for controlling Lane Centering Systems (LCS), Adaptive Cruise Control (ACC), Lane Keeping Assist (LKA), Autonomous Emergency Braking (AEB), or Emergency Stop System (ESS), based on authority adjustment data, safety extension data, and sensitivity adjustment data from the insightor (750). Accordingly, it is possible to quickly and accurately determine the situation inside and outside the vehicle and adaptively perform vehicle control.

[0424] Meanwhile, the illustrator (780) can output a vehicle control signal, a warning message, or a guide message based on the calculated risk level and occupant profile information. Accordingly, it is possible to quickly and accurately determine the situation inside and outside the vehicle and adaptively control the vehicle.

[0425] FIGS. 10 to 15 are drawings referenced in the description of FIG. 9.

[0426] First, FIG. 10 is an example of a block diagram of a signal processing system according to an embodiment of the present disclosure.

[0427] Referring to the drawings, the signal processing system according to an embodiment of the present disclosure may be referred to as an IVEX (In Vehicle Experience) system or a vehicle control device.

[0428] Meanwhile, the signal processing system (900) according to an embodiment of the present disclosure includes an edge sensor group (SN) and a signal processing device (170) in a vehicle.

[0429] Meanwhile, the signal processing system according to the embodiment of the present disclosure may further include a server (400).

[0430] Meanwhile, the signal processing device (170) according to an embodiment of the present disclosure includes a recognizer (720), an insightor (750), and an illustrator (780).

[0431] Meanwhile, the edge sensor group (SN) can correspond to a plurality of sensor devices (SN) of FIG. 4.

[0432] Meanwhile, the edge sensor group (SN) may include an internal vehicle sensor (710) and an external vehicle sensor (715).

[0433] The vehicle interior sensor (710) may include a microphone (112 in FIG. 4), a vehicle interior camera (195i), etc.

[0434] The vehicle external sensor (715) may include a vehicle front camera (195), a position sensor (198), etc.

[0435] Meanwhile, the recognizer (720) can recognize the vehicle situation based on sensing data received from the edge sensor group (SN). In this regard, the recognizer (720) may be referred to as a situation recognition unit.

[0436] The vehicle situation at this time may include external vehicle conditions and internal vehicle conditions.

[0437] Meanwhile, the vehicle interior recognition unit (725) within the recognition unit (720) can recognize the vehicle interior situation based on vehicle interior sensing data from the vehicle interior sensor (710).

[0438] For example, the vehicle interior recognition device (725) within the recognition device (720) can monitor the condition of the occupants, including the driver, based on camera data inside the vehicle.

[0439] Meanwhile, the vehicle external recognition unit (730) within the recognition unit (720) can recognize the external vehicle situation based on the external vehicle sensing data from the external vehicle sensor (715).

[0440] For example, the vehicle external recognizer (730) within the recognizer (720) can perform object detection based on camera data from outside the vehicle.

[0441] Meanwhile, the integrated scene understandr (935) within the recognizer (720) can recognize vehicle situation information including an integrated scene inside and outside the vehicle based on vehicle inside situation information from the vehicle inside recognizer (725) and vehicle outside situation information from the vehicle outside recognizer (730).

[0442] Meanwhile, vehicle situation information may include vehicle interior situation information, vehicle exterior situation information, and vehicle occupant situation information.

[0443] Meanwhile, the integrated scene comprehension unit (935) within the recognition unit (720) can recognize vehicle situation information that further includes warning signs of the vehicle occupant.

[0444] Meanwhile, the insightor (750) can generate an inference result or a response result based on the situation information of the vehicle received from the recognizer (720).

[0445] Meanwhile, the insightor (750) can generate an inference result or a response result based on the vehicle situation information received from the recognizer (720), and can generate or control a service to be executed based on the inference result or the response result.

[0446] Accordingly, the insightor (750) may be named a response result generation unit or a service generation unit.

[0447] Meanwhile, the insightor (750) may include a multimodal context engine (755), a risk level determination unit (940), and a personalization module (945).

[0448] Meanwhile, the multimodal context engine (755) can generate multimodal context data based on the vehicle situation information received from the recognizer (720).

[0449] For example, multimodal context data may include at least one of text data, image data, or audio data describing a vehicle situation generated based on vehicle situation information.

[0450] Meanwhile, the multimodal context engine (755) may include a database (752) that stores driver's driving pattern information.

[0451] At this time, the driver's driving pattern information may include risk response information or risk response pattern information.

[0452] Meanwhile, the multimodal context engine (755) may include a database (752) that stores driver's driving pattern information and passenger's pattern information.

[0453] Meanwhile, the multimodal context engine (755) can extract multimodal context data corresponding to a driving context related to vehicle driving based on monitored passenger status information, detected object information, sensor data, and driver's driving pattern information stored in a database.

[0454] Meanwhile, the multimodal context engine (755) can transmit individual driving pattern information collected to the server (400).

[0455] Meanwhile, the risk level determination unit (940) can calculate the risk level of vehicle driving based on the extracted multimodal context data.

[0456] Meanwhile, the risk level determination unit (940) can obtain an inference result or a response result based on the extracted multimodal context data, and calculate the risk level of vehicle driving based on the obtained inference result or response result.

[0457] For example, the risk level determination unit (940) can classify the risk level of vehicle driving into levels 1 through 4.

[0458] Meanwhile, the risk level determination unit (940) can calculate driving ability evaluation data and emergency level evaluation data based on the driver's context data and driver's characteristic data.

[0459] Meanwhile, the risk level determination unit (940) can calculate driving ability evaluation data and emergency level evaluation data based on the driver's context data, the driver's characteristic data, the passenger's context data, and the passenger's characteristic data.

[0460] Meanwhile, the risk level determination unit (940) can calculate driving ability evaluation data and emergency level evaluation data based on the driver's context data and driver's characteristic data, and calculate the risk level of vehicle driving based on the driving ability evaluation data and emergency level evaluation data.

[0461] Meanwhile, the risk level determination unit (940) can calculate driving ability evaluation data and emergency level evaluation data based on the driver's context data, driver's characteristic data, passenger's context data, and passenger's characteristic data, and calculate the risk level of vehicle driving based on the driving ability evaluation data and emergency level evaluation data.

[0462] Meanwhile, the risk level determination unit (940) can classify the risk level of vehicle driving into multiple levels based on vehicle internal context data, vehicle external context data, and occupant profile information.

[0463] Meanwhile, the risk level determination unit (940) can classify the risk level of vehicle driving into multiple levels based on vehicle internal context data, vehicle external context data, occupant characteristic information, occupant behavior information, and driver's driving ability information.

[0464] Meanwhile, the risk level determination unit (940) can transmit risk level information of vehicle driving to the personalization module (945).

[0465] Meanwhile, the personalization module (945) can receive individual profile information from the server (400).

[0466] Meanwhile, the learning model (767) within the personalization module (945) can perform learning based on the calculated risk level information and occupant profile information, and output the result data of the learning.

[0467] Meanwhile, the personalization module (945) can compute authority adjustment data for vehicle control, safety extension data for safety extension, and sensitivity adjustment data for sensitivity adjustment based on the result data of the learning.

[0468] Meanwhile, the insightor (750) can set a mode for vehicle driving assistance control or autonomous driving based on the risk level information of vehicle driving determined by the risk level determination unit (940).

[0469] For example, the insightor (750) can classify the vehicle driving assistance control or autonomous driving mode into one of the off mode, on mode, or safety mode based on the risk level of vehicle driving from the risk level determination unit (940).

[0470] Specifically, the insightor (750) can set the vehicle driving assistance control or autonomous driving mode to off mode when the risk level of vehicle driving from the risk level determination unit (940) is the first level.

[0471] Meanwhile, the insightor (750) can set the vehicle driving assistance control or autonomous driving mode to ON mode when the risk level of the vehicle driving from the risk level determination unit (940) is the second level or the third level.

[0472] Meanwhile, the insightor (750) can set the vehicle driving assistance control or autonomous driving mode to safety mode when the risk level of the vehicle driving from the risk level determination unit (940) is the fourth level.

[0473] Unlike the above, the insightor (750) can control the vehicle driving assistance control or autonomous driving mode to remain in off mode or on mode when the risk level of vehicle driving from the risk level determination unit (940) is at a first to third level.

[0474] Meanwhile, the insightor (750) can control the vehicle driving assistance control or autonomous driving mode to end the on or off mode when the risk level of the vehicle driving from the risk level determination unit (940) is the fourth level, and to enter the safety mode.

[0475] Meanwhile, the insightor (750) can control the authority adjustment, safety extension, or sensitivity adjustment for vehicle control based on the set mode.

[0476] Meanwhile, the insightor (750) determines whether it is a danger warning stage or a danger situation stage based on the danger level of the vehicle driving from the danger level judgment unit (940), and can control the operation mode of the vehicle driving assistance control or autonomous driving to be variable according to the danger warning stage or danger situation stage.

[0477] For example, the insightor (750) can determine a danger warning stage when the risk level of vehicle driving is at a second or third level, and can control the on mode of vehicle driving assistance control or autonomous driving mode in response to the danger warning stage.

[0478] Specifically, the insightor (750) can control the safety extension module (910), the authority adjustment module (915), and the sensitivity adjustment module (920) in response to the danger precursor stage.

[0479] As another example, the insightor (750) can determine the danger situation stage when the danger level of the vehicle driving is at level 4, and control the vehicle driving assistance control or autonomous driving mode to operate in safety mode in response to the danger situation stage.

[0480] Specifically, the insightor (750) can control the safety expansion module (910), the authority adjustment module (915), and the sensitivity adjustment module (920) in response to the risk situation stage.

[0481] Meanwhile, the illustrator (780) can execute a service, output service information, execute an application, or output application information based on the inference result or response result output from the insightor (750).

[0482] Meanwhile, the illustrator (780) can output a vehicle control signal based on the risk level calculated by the insightor (750).

[0483] Meanwhile, the illustrator (780) can output a vehicle control signal based on the authority adjustment data for vehicle control, the safety extension data for safety extension, and the sensitivity adjustment data for sensitivity adjustment calculated in the insightor (750).

[0484] Meanwhile, the illustrator (780) may include a multimodal output encoder (781) including a modality manager (795), a line suggestion application (931), a warning application (932), a system control (930), etc.

[0485] Meanwhile, the multimodal output encoder (781) can encode the inference result or response result output from the insightor (750).

[0486] Meanwhile, the first proposal application (931), the warning application (932), and the system control (930) can each be executed based on encoded inference result data or response result data from the multimodal output encoder (781).

[0487] Meanwhile, the multimodal output encoder (781) can encode data corresponding to the calculated risk level.

[0488] Meanwhile, the first proposal application (931), the warning application (932), and the system control (930) can each be executed based on data corresponding to the calculated risk level from the multimodal output encoder (781).

[0489] For example, the system control (930) can output a vehicle control signal based on data corresponding to the calculated risk level.

[0490] Meanwhile, the pre-proposal application (931) can generate and output a guide message related to a pre-proposal for vehicle control.

[0491] For example, the pre-proposal application (931) can output a guide message for vehicle speed reduction, a guide message for lane change, etc.

[0492] Meanwhile, the pre-proposal application (931) can output a guide message related to a pre-proposal for vehicle control as Augmented Reality (AR) or video data (783) or Mixed Reality (MR) video data (784).

[0493] Meanwhile, the pre-proposal application (931) can output a guide message related to a pre-proposal for vehicle control as audio, such as voice.

[0494] Meanwhile, the warning application (932) can output a warning message audio, visual, or physically.

[0495] Meanwhile, the multimodal output encoder (781) can encode the result data of the learning from the personalization module (945).

[0496] Meanwhile, the first proposal application (931), the warning application (932), and the system control (930) can each be executed based on the result data of learning from the multimodal output encoder (781).

[0497] For example, the pre-proposal application (931) can output a guide message related to a pre-proposal for vehicle control based on the result data of the learning.

[0498] Meanwhile, the warning application (932) can output a warning message based on the result data of the learning.

[0499] Meanwhile, the system control (930) can output a vehicle control signal based on the result data of the learning.

[0500] Meanwhile, the multimodal output encoder (781) can encode authority adjustment data for vehicle control, safety extension data for safety extension, and sensitivity adjustment data for sensitivity adjustment.

[0501] Meanwhile, the first proposal application (931), the warning application (932), and the system control (930) can be executed based on the authority adjustment data for vehicle control, the safety extension data for safety extension, and the sensitivity adjustment data for sensitivity adjustment from the multimodal output encoder (781), respectively.

[0502] Meanwhile, the pre-proposal application (931) can output a guide message based on authority adjustment data for vehicle control, safety extension data for safety extension, and sensitivity adjustment data for sensitivity adjustment.

[0503] Meanwhile, the warning application (932) can output a warning message based on authority adjustment data for vehicle control, safety extension data for safety extension, and sensitivity adjustment data for sensitivity adjustment.

[0504] Meanwhile, the system control (930) can output a vehicle control signal based on authority adjustment data for vehicle control, safety extension data for safety extension, and sensitivity adjustment data for sensitivity adjustment.

[0505] FIG. 11a is a drawing referenced in the description of the operation of a processor according to one embodiment of the present disclosure.

[0506] Referring to the drawings, a processor (175) in a signal processing device (170) according to one embodiment of the present disclosure may include a risk level determination unit (940) and a level corresponding control module (950).

[0507] The processor (175) receives vehicle internal data from vehicle internal sensors (710), receives vehicle external data from vehicle external sensors (715), and can receive data around the vehicle through the data receiving unit (718).

[0508] Meanwhile, the processor (175) performs object detection based on camera data outside the vehicle and monitors the status of passengers, including the driver, based on camera data inside the vehicle.

[0509] Meanwhile, the processor (175) extracts multimodal context data corresponding to a driving context related to vehicle driving based on monitored passenger status information, detected object information, sensor data, and driver's driving pattern information stored in a database.

[0510] Meanwhile, the risk level determination unit (940) within the processor (175) calculates the risk level of vehicle driving based on the extracted multimodal context data.

[0511] Meanwhile, the level-corresponding control module (950) within the processor (175) outputs a vehicle control signal based on the calculated risk level. Accordingly, the situation inside and outside the vehicle can be determined quickly and accurately, and adaptive vehicle control can be performed.

[0512] Meanwhile, the risk level determination unit (940) within the processor (175) can obtain an inference result or a response result based on the extracted multimodal context data, and calculate the risk level of vehicle driving based on the obtained inference result or response result.

[0513] That is, the processor (175) can generate a prompt for calculating the risk level of vehicle driving based on multimodal context data, calculate the risk level of vehicle driving based on the inference result or response result obtained based on the prompt, and output a vehicle control signal based on the calculated risk level.

[0514] Meanwhile, a signal processing device (170) according to one embodiment of the present disclosure may further include a neural processor (179) that executes inference or a response based on a prompt.

[0515] Meanwhile, the processor (175) can generate a prompt for calculating the risk level of vehicle driving based on multimodal context data, calculate the risk level of vehicle driving based on the inference result or response result executed by the neural processor (179) based on the prompt, and output a vehicle control signal based on the calculated risk level. Accordingly, the situation inside and outside the vehicle can be determined quickly and accurately to perform adaptive vehicle control.

[0516] Meanwhile, the risk level determination unit (940) within the signal processing unit (170) is included in the processor (175), as in FIG. 10 or FIG. 11a, and can classify the risk level of vehicle driving into a plurality of levels based on vehicle internal context data, vehicle external context data, and occupant profile information.

[0517] Meanwhile, the risk level determination unit (940) within the signal processing device (170) may be a concept including a processor (175) and a neural processor (179), unlike FIG. 10 or FIG. 11a.

[0518] Meanwhile, the risk level determination unit (940) within the signal processing unit (170) can operate as a hybrid type in the processor (175) and the neural processor (179), respectively.

[0519] For example, the processor (175) can calculate the risk level of vehicle driving for the event based on vehicle internal context data, vehicle external context data, and occupant profile information when an event occurs that requires urgent and immediate judgment, i.e., when the judgment completion period must be within the first set period.

[0520] In particular, the processor (175) can execute rule-based logic when urgent and immediate judgment is required to calculate the risk level of vehicle driving for the event.

[0521] Meanwhile, the processor (175) can control the neural processor (179) to calculate the risk level of vehicle driving when urgent and immediate judgment is not required.

[0522] For example, the processor (175) can control the neural processor (179) to calculate the risk level of vehicle driving when the situation is complex and various contexts must be considered together, that is, when an event occurs where the judgment completion period may exceed the first set period.

[0523] Specifically, the processor (175) can transmit vehicle internal context data, vehicle external context data, and passenger profile information to the neural processor (179) when an event occurs in which the situation is complex and various contexts must be considered together.

[0524] Meanwhile, the neural processor (179) can calculate the risk level of vehicle driving for the event based on the received vehicle internal context data, vehicle external context data, and occupant profile information.

[0525] For example, the neural processor (179) can calculate the risk level of vehicle driving for the event using a safety large-scale language model (LLM), etc.

[0526] Based on this hybrid method of risk level calculation, immediate or accurate risk level calculation becomes possible.

[0527] Meanwhile, the processor (175) can run a hypervisor (505 in FIG. 5) and run a driving control virtualization machine (520 in FIG. 5) on the hypervisor (505).

[0528] Meanwhile, the driving control virtualization machine (520) receives camera data outside the vehicle, camera data inside the vehicle, and sensor data inside the vehicle through the shared memory (508) within the hypervisor (505), and can execute a driving control service or a driving control application based on the camera data outside the vehicle, camera data inside the vehicle, and sensor data inside the vehicle. Accordingly, it is possible to quickly and accurately determine the situation inside and outside the vehicle and adaptively perform vehicle control.

[0529] Meanwhile, the risk level determination unit (940) within the processor (175) can classify the risk level of vehicle driving into multiple levels based on vehicle internal context data, vehicle external context data, and occupant profile information.

[0530] Meanwhile, the processor (175) can classify the risk level of vehicle driving into multiple levels based on vehicle internal context data, vehicle external context data, occupant characteristic information, occupant behavior information, and driver's driving ability information.

[0531] Meanwhile, the risk level determination unit (940) within the processor (175) can classify the risk level of vehicle driving into multiple levels.

[0532] Specifically, the risk level determination unit (940) within the processor (175) can classify the risk level of vehicle driving into levels 1 through 4.

[0533] For example, the risk level determination unit (940) within the processor (175) can calculate the risk level of vehicle driving as a first level if the driver's driving ability information among the monitored passenger status information indicates that the driver can operate the vehicle.

[0534] As another example, the risk level determination unit (940) within the processor (175) can calculate the risk level of vehicle driving as a second level higher than the first level when the driver's driving ability information indicates that the driver is capable of operating the vehicle or is in a distracted state.

[0535] As another example, the risk level determination unit (940) within the processor (175) can calculate the risk level of vehicle driving as a third level higher than the second level when the driver's driving ability information indicates that the driver is capable of driving or is in a delayed response state.

[0536] As another example, the risk level determination unit (940) within the processor (175) can calculate the risk level of vehicle driving as a fourth level higher than the third level when the driver's driving ability information indicates that the driver cannot operate the vehicle. Accordingly, the situation inside and outside the vehicle can be determined quickly and accurately to enable adaptive vehicle control.

[0537] Meanwhile, the risk level determination unit (940) within the processor (175) can calculate the risk level of vehicle driving as a first level if the occupant characteristic information does not include habit behavior information or is not personalized habit behavior information.

[0538] Meanwhile, the risk level determination unit (940) within the processor (175) can calculate the risk level of vehicle driving as a second level higher than the first level when the occupant characteristic information includes habitual behavior information that interferes with driving.

[0539] For example, the risk level determination unit (940) within the processor (175) can calculate the risk level of vehicle driving as a second level higher than the first level if the occupant characteristic information includes habitual behavior information that interferes with driving.

[0540] Meanwhile, the risk level determination unit (940) within the processor (175) can calculate the risk level of vehicle driving as a third level higher than the second level if the occupant characteristic information includes repeated dangerous behaviors and medical history or accident history.

[0541] For example, the risk level determination unit (940) within the processor (175) can calculate the risk level of vehicle driving as a third level higher than the second level if the occupant characteristic information includes repeated dangerous behaviors and medical history or accident history that interfere with driving.

[0542] Meanwhile, the risk level determination unit (940) within the processor (175) can calculate the risk level of vehicle driving as a fourth level higher than the third level when the occupant characteristic information includes risk information based on medical history. Accordingly, it is possible to quickly and accurately determine the situation inside and outside the vehicle and adaptively perform vehicle control.

[0543] Meanwhile, the risk level determination unit (940) within the processor (175) can calculate the risk level of vehicle driving as a first level when the passenger's behavior information includes stable state information.

[0544] Meanwhile, the risk level determination unit (940) within the processor (175) can calculate the risk level of vehicle driving as a second level higher than the first level when the passenger's behavior information includes voice information at a level below the threshold.

[0545] For example, the risk level determination unit (940) within the processor (175) can calculate the risk level of vehicle driving as a second level higher than the first level when the passenger's behavior information includes voice stimuli that distract concentration on driving, such as voice information at a level below a reference value.

[0546] Meanwhile, the risk level determination unit (940) within the processor (175) can calculate the risk level of vehicle driving as a third level higher than the second level when the passenger's behavior information includes voice information at a level higher than a repeated threshold.

[0547] For example, the risk level determination unit (940) within the processor (175) can calculate the risk level of vehicle driving as a third level higher than the second level if the passenger's behavior information includes voice stimuli that interfere with driving, such as voice information at a level higher than a threshold.

[0548] Meanwhile, the risk level determination unit (940) within the processor (175) can calculate the risk level of vehicle driving as a fourth level higher than the third level if the passenger's behavior information includes actions that interfere with the driver's driving operation. Accordingly, the situation inside and outside the vehicle can be determined quickly and accurately to enable adaptive vehicle control.

[0549] Meanwhile, the risk level determination unit (940) within the processor (175) can calculate driving ability evaluation data and emergency level evaluation data based on the driver's context data and driver's characteristic data, and calculate the risk level of vehicle driving based on the driving ability evaluation data and emergency level evaluation data.

[0550] Meanwhile, the risk level determination unit (940) within the processor (175) can calculate driving ability evaluation data and emergency level evaluation data based on the driver's context data, the driver's characteristic data, the passenger's context data, and the passenger's characteristic data.

[0551] Meanwhile, the level-corresponding control module (950) within the processor (175) can control authority adjustment, safety extension, or sensitivity adjustment for vehicle control based on the risk level of vehicle driving.

[0552] Meanwhile, the level-corresponding control module (950) within the processor (175) can calculate authority adjustment data for vehicle control, safety extension data for safety extension, and sensitivity adjustment data for sensitivity adjustment based on the risk level of vehicle driving.

[0553] Meanwhile, the level-corresponding control module (950) within the processor (175) can generate a vehicle control signal based on authority adjustment data, safety extension data, and sensitivity adjustment data. Accordingly, it is possible to quickly and accurately determine the situation inside and outside the vehicle and adaptively perform vehicle control.

[0554] Meanwhile, the level-corresponding control module (950) within the processor (175) can control the authority adjustment, safety extension, and sensitivity adjustment for vehicle control to remain unchanged when the risk level of vehicle driving is at the first level.

[0555] Meanwhile, the level-corresponding control module (950) within the processor (175) can switch the authority adjustment and safety expansion for vehicle control to a standby state and increase the sensitivity of the sensitivity adjustment above the first level when the risk level of vehicle driving is at a second level higher than the first level. Accordingly, it is possible to quickly and accurately determine the situation inside and outside the vehicle and adaptively perform vehicle control.

[0556] Meanwhile, the level-corresponding control module (950) within the processor (175) can ignore or supplement the operation input within the authority adjustment for vehicle control, increase the threshold value within the safety extension, or raise the sensitivity of the sensitivity adjustment above the second level when the risk level of vehicle driving is at a third level higher than the second level. Accordingly, it is possible to quickly and accurately determine the situation inside and outside the vehicle and adaptively perform vehicle control.

[0557] Meanwhile, the level-corresponding control module (950) within the processor (175) can control steering and acceleration for authority adjustment for vehicle control when the risk level of vehicle driving is at the highest level (Level 4), set the threshold value within the safety extension to the maximum value, and set the sensitivity of the sensitivity adjustment to the maximum value. Accordingly, it is possible to quickly and accurately determine the situation inside and outside the vehicle and adaptively perform vehicle control.

[0558] Meanwhile, the processor (175) can perform learning based on the calculated risk level and occupant profile information, and output a vehicle control signal, a warning message, or a guide message based on the result data of the learning. Accordingly, it is possible to quickly and accurately determine the situation inside and outside the vehicle and adaptively perform vehicle control.

[0559] Figure 11b is a drawing referenced in the description of the integrated scene analyzer of Figure 10.

[0560] Referring to the drawing, the integrated scene understandr (935) may include a vehicle interior context understandr (960) and a vehicle exterior context understandr (970).

[0561] The vehicle internal context understandr (960) may include a driver state information extractor (962) and a driver behavior information extractor (964).

[0562] For example, the driver state information extractor (962) can extract driver state information such as body-skeleton information, hands-on information, distraction information, behavior information, and health information through camera data from an internal camera (195i) and sensing data from a bio-sensor device.

[0563] Meanwhile, the driver behavior information extractor (964) can extract driver behavior information such as spilling, child crying, driver disturbance, and heart attack based on various information or data from the driver state information extractor (962).

[0564] The vehicle external context understandr (970) may include a setting external information extractor (972) and a real-time external information extractor (974).

[0565] Meanwhile, the external information extractor (972) may be named the Operational Design Domain (ODD).

[0566] The external information extractor (972) can extract surrounding environment information including a vehicle or a vulnerable road user (VRU), extract road context data, or extract highway context data based on communication data from the data receiver (718) and external vehicle sensing data from the external vehicle sensors (715).

[0567] The real-time external information extractor (974) can extract real-time external information such as traffic jam information, crowded vulnerable road user (Crowded VRU) information, and uncongested highway information based on information or data from the set external information extractor (972).

[0568] Meanwhile, the risk level determination unit (940) may include a vehicle interior risk determination unit (952) and a driving risk determination unit (954).

[0569] The vehicle interior risk determination unit (952) can calculate the risk level inside the vehicle based on vehicle interior context data from the vehicle interior context understandr (960). Meanwhile, the risk level inside the vehicle can be transmitted to the safety assist engine (757).

[0570] Meanwhile, the driving risk determination unit (954) can calculate the risk level outside the vehicle based on the vehicle external context data from the vehicle external context understandr (970). Meanwhile, the risk level outside the vehicle can be transmitted to the safety assist engine (757).

[0571] Meanwhile, the safety assist engine (757) may include the level corresponding control module (950) of FIG. 11a.

[0572] Meanwhile, the authority adjustment module (915) within the safety assistance engine (757) can adjust driving authority based on the risk level of driving the vehicle, including the risk level inside the vehicle and the risk level of driving the vehicle.

[0573] For example, the authority adjustment module (915) within the safety assist engine (757) can partially restrict or supplement the vehicle operation authority when the driver's ability to operate in a dangerous situation is below a standard level.

[0574] As another example, the authority adjustment module (915) within the safety assist engine (757) can control the driver's ability to operate in a dangerous situation to return to its original state by reducing the intervention of the driving authority by the system when the driver's ability to operate returns to a level exceeding the threshold. Accordingly, the situation inside and outside the vehicle can be determined quickly and accurately to perform adaptive vehicle control.

[0575] Meanwhile, the safety extension module (910) within the safety assistance engine (757) can perform safety extension control based on the risk level of vehicle driving, including the risk level inside the vehicle and the risk level of vehicle driving.

[0576] For example, the safety extension module (910) within the safety assist engine (757) can activate the system or adjust functional limitations regarding the allowable operating range in response to the risk level. Accordingly, it is possible to quickly and accurately determine the situation inside and outside the vehicle and perform adaptive vehicle control.

[0577] Meanwhile, the sensitivity adjustment module (920) within the safety assist engine (757) can perform sensitivity adjustment based on the risk level of the vehicle, including the risk level inside the vehicle and the risk level of the vehicle driving.

[0578] For example, the sensitivity adjustment module (920) within the safety assist engine (757) can control the sensitivity to be increased when the risk level exceeds a set value, and to be returned to its original state when the risk level returns to below the set value. Accordingly, the situation inside and outside the vehicle can be determined quickly and accurately to perform adaptive vehicle control.

[0579] FIG. 11c is a drawing referenced in the description of the operation of the personalization module of FIG. 10.

[0580] Referring to the drawing, the personalization module (945) can receive database data for initial profile generation from the user characteristic engine (758).

[0581] Meanwhile, the personalization module (945) can generate initial profile information based on the received database data.

[0582] At this time, the database data may include at least one of the user or passenger's nationality, age, gender, occupation, personality, or psychological type (Myers-Briggs Type Indicator, MBTI).

[0583] For example, the personalization module (945) can classify personal tendencies by category based on at least one of the nationality, age, gender, occupation, personality, or psychological type (Myers-Briggs Type Indicator, MBTI) of the user or passenger, and generate initial profile information based on the classified personal tendencies information.

[0584] Meanwhile, the personalization module (945) can perform mapping suitable for the vehicle domain based on initial profile information.

[0585] Meanwhile, the user characteristic engine (758) may be provided within the insightor (750) as in FIG. 7. Alternatively, the user characteristic engine (758) may be provided within the server (400) as in FIG. 7.

[0586] FIG. 11d is a diagram referenced in the description of the operation of a mode for vehicle driving assistance control or autonomous driving.

[0587] Referring to the drawings, a signal processing system (1600) according to an embodiment of the present disclosure includes an edge sensor group (SNa), a database (1610), and a signal processing device (170), as an IVEX (In Vehicle Experience) system or vehicle control device.

[0588] The edge sensor group (SNa) may be equipped with a plurality of sensors for vehicle driving assistance control or autonomous driving.

[0589] Meanwhile, the edge sensor group (SNa) may include an internal vehicle sensor (710) and an external vehicle sensor (715).

[0590] For example, the vehicle interior sensor (710) may include a microphone (112 in FIG. 4), a vehicle interior camera (195i), a biosensor (not shown) attached to the steering wheel (150), etc.

[0591] The vehicle external sensor (715) may include a vehicle front camera (195), lidar (196), radar (197), or position sensor (198), etc.

[0592] Meanwhile, the database (1610) can provide vehicle driving record data, driver medical history record data, disease symptom data, driver accident history data, driving habit data, or driving environment data stored internally in the signal processing device (170).

[0593] Meanwhile, the signal processing device (170) may include a processor (175).

[0594] The recognizer (725) in the processor (175) receives vehicle interior sensing data from the vehicle interior sensor (710) and can detect the occupant's face identification (Face ID, 741), gaze, head position, body skeleton, emotion, voice, heart beat, or object.

[0595] The recognizer (725) in the processor (175) can receive external vehicle sensing data from the external vehicle sensor (715) and detect a lane, road border, vehicle, traffic sign, or traffic light.

[0596] Meanwhile, the recognizer (725) in the processor (175) can calculate or extract driver behavior information, emergency situation information, or risk level of vehicle driving based on vehicle internal sensing data from the vehicle internal sensor (710), vehicle external sensing data from the vehicle external sensor (715), or vehicle internal driving record data, driver medical history record data, disease symptom data, driver accident history data, driving habit data, or driving environment data from the database (1610).

[0597] Meanwhile, the driving system control unit (1620) within the processor (175) can be controlled to operate in an off mode, on mode, or safety mode for vehicle driving assistance control or autonomous driving.

[0598] For example, the driving system control unit (1620) within the processor (175) can be controlled to operate in any one of the off mode, on mode, or safety mode of vehicle driving assistance control or autonomous driving based on an input signal.

[0599] Meanwhile, the action recognition unit (1730) within the processor (175) can perform driver action recognition based on an accelerator pedal input signal, a brake operation input signal, a turn signal input signal, or a steering wheel input signal.

[0600] Meanwhile, the processor (175) can generate a normal driving state as context data for each individual based on in-vehicle driving record data, driving habit data, or driving environment data, and detect signs of abnormal behavior based on this.

[0601] For example, the behavior recognition unit (1730) within the processor (175) can detect signs of abnormal behavior of the driver based on driver medical history data, disease symptom data, or driver accident history data, and accelerator pedal input signal, brake operation input signal, turn signal input signal, or steering wheel input signal, etc.

[0602] Meanwhile, the processor (175) can vary the maximum speed, maximum lateral acceleration, limit distance, minimum lane width, etc. when detecting signs of abnormal behavior.

[0603] For example, the system range setting unit (1635) within the processor (175) can vary the maximum speed, maximum lateral acceleration, limit distance, minimum lane width, etc. when abnormal behavior signs are detected.

[0604] Meanwhile, the driving operation unit (1640) within the processor (175) can set the mode for vehicle driving assistance control or autonomous driving to safety mode when abnormal behavior signs are detected, and output a vehicle control signal according to the safety mode.

[0605] For example, the driving operation unit (1640) within the processor (175) may, when operating in safety mode, limit override, temporarily relax the allowable range of system operation during vehicle driving assistance control or autonomous driving, or strengthen the system control sensitivity during vehicle driving assistance control or autonomous driving.

[0606] Meanwhile, the driving operation unit (1640) within the processor (175), when operating in safety mode, controls the lane centering system (LCS) and adaptive cruise control (ACC) to increase the maximum allowable lateral acceleration or increase the operating speed limit, and controls the emergency braking system (ESS) to operate immediately. Accordingly, it is possible to quickly and accurately determine the situation inside and outside the vehicle and adaptively control the vehicle.

[0607] Meanwhile, the driving operation unit (1640) within the processor (175), when operating in safety mode, controls the lane centering system (LCS) and lane keeping assist (LKA) to increase the steering override torque or to maintain the lane even in the event of a turn signal malfunction, controls the adaptive cruise control (ACC) to increase the accelerator pedal override torque or to respond to a dangerous object ahead even in the event of a brake pedal malfunction or to prevent an OTF (Overtaking Function) even in the event of a turn signal malfunction, and controls the emergency braking system (ESS) so that the emergency braking system operation is performed even if the brake pedal, accelerator pedal, or steering override torque increases. Accordingly, it is possible to quickly and accurately determine the situation inside and outside the vehicle and adaptively perform vehicle control.

[0608] Meanwhile, the driving operation unit (1640) within the processor (175), when operating in safety mode, can control the Lane Centering System (LCS) to increase the target distance level for object tracking or decrease the lane width limit, control the Lane Keeping Assist (LKA) to increase the distance between the vehicle and the lane, control the Adaptive Cruise Control (ACC) to change the Time to Collision (TTC) or change the ACC Early Warning, or decrease the target speed according to curves or road types, and control the Automatic Emergency Braking (AEB) to change the Time to Collision (TTC) or change the ACC Early Warning. Accordingly, it is possible to quickly and accurately determine the situation inside and outside the vehicle and adaptively perform vehicle control.

[0609] Meanwhile, the driving operation unit (1640) within the processor (175), when operating in safety mode, can perform steering control or acceleration control, completely remove functional restrictions on vehicle driving assistance control or autonomous driving, set a threshold value to the maximum, maintain heightened sensitivity, or suppress warnings to prevent confusion. Accordingly, it is possible to quickly and accurately determine the situation inside and outside the vehicle and adaptively perform vehicle control.

[0610] Meanwhile, the driving operation unit (1640) within the processor (175) can control the lane centering system (LCS) and the adaptive cruise control (ACC) when operating in the vehicle driving assistance control or autonomous driving on mode, thereby gradually increasing the allowable maximum lateral acceleration or gradually increasing the operating limit speed, and can keep the emergency braking system (ESS) in a ready state.

[0611] Meanwhile, the driving operation unit (1640) within the processor (175) can control the lane centering system (LCS) and lane keeping assist (LKA) to gradually increase the steering torque when operating in the vehicle driving assistance control or autonomous driving on mode, or control the adaptive cruise control (ACC) to gradually increase the steering torque, and the emergency braking system (ESS) can be kept in a ready state.

[0612] Meanwhile, the driving operation unit (1640) within the processor (175), when operating in the vehicle driving assistance control or autonomous driving on mode, can control the lane centering system (LCS) and lane keeping assistance (LKA) to gradually lower the lane width limit, or control the adaptive cruise control (ACC) to gradually increase the distance to the vehicle ahead, and can keep the emergency braking system (ESS) in a ready state.

[0613] Meanwhile, the driving operation unit (1640) within the processor (175) can, when operating in the ON mode of vehicle driving assistance control or autonomous driving, ignore or start correcting some errors such as blinking mistakes, partially relieve functional limitations of vehicle driving assistance control or autonomous driving, gradually increase the threshold, increase sensitivity further, perform multiple warnings, perform adaptive warnings based on personalization, actively perform warnings in case of behavioral risk, or relieve warnings in case of health risk.

[0614] FIG. 12 is a diagram illustrating a vehicle driving assistance function corresponding to a driving state.

[0615] Referring to the drawing, the operating state can be classified into normal mode, first emergency mode, second emergency mode, and emergency release mode.

[0616] Meanwhile, Normal Mode or Emergency Release Mode can correspond to Level 1 of the risk levels of vehicle driving.

[0617] Meanwhile, the first emergency mode can respond to the second or third level of the risk level of vehicle driving.

[0618] Meanwhile, the second emergency mode can respond to Level 4, which is the highest level among the risk levels of vehicle driving.

[0619] Meanwhile, in the case of the first emergency mode, the safety extension module (910) within the processor (175) controls the lane centering system (LCS) and the adaptive cruise control (ACC) to gradually increase the maximum allowable lateral acceleration or gradually increase the operating limit speed, and the emergency braking system (ESS) can be kept in a ready state.

[0620] Meanwhile, in the case of the first emergency mode, the authority adjustment module (915) within the processor (175) can control the lane centering system (LCS) and lane keeping assist (LKA) to gradually increase the steering torque, or control the adaptive cruise control (ACC) to gradually increase the steering torque, and keep the emergency braking system (ESS) in a ready state.

[0621] Meanwhile, in the case of the first emergency mode, the sensitivity adjustment module (920) within the processor (175) can control the lane centering system (LCS) and lane keeping assist (LKA) to gradually lower the lane width limit, or control the adaptive cruise control (ACC) to gradually increase the distance to the vehicle ahead, and the emergency braking system (ESS) can be kept in a ready state.

[0622] Meanwhile, in the case of the second emergency mode, the safety extension module (910) within the processor (175) can control the lane centering system (LCS) and the adaptive cruise control (ACC) to increase the maximum allowable lateral acceleration or increase the operating speed limit, and can control the emergency braking system (ESS) to operate immediately. Accordingly, it is possible to quickly and accurately determine the situation inside and outside the vehicle and adaptively control the vehicle.

[0623] Meanwhile, in the case of the second emergency mode, the authority adjustment module (915) within the processor (175) controls the Lane Centering System (LCS) and Lane Keeping Assist (LKA) to increase the steering override torque or to maintain the lane even in the event of a turn signal malfunction, controls the Adaptive Cruise Control (ACC) to increase the accelerator pedal override torque or to respond to a hazard ahead even in the event of a brake pedal malfunction or to prevent an Overtaking Function (OTF) even in the event of a turn signal malfunction, and controls the Emergency Braking System (ESS) so that the Emergency Braking System operation is performed even if the brake pedal, accelerator pedal, or steering override torque increases. Accordingly, the vehicle control can be performed adaptively by quickly and accurately determining the situation inside and outside the vehicle.

[0624] Meanwhile, in the case of the second emergency mode, the sensitivity adjustment module (920) within the processor (175) can control the Lane Centering System (LCS) to increase the target distance level for object tracking or to decrease the lane width limit, control the Lane Keeping Assist (LKA) to increase the distance between the vehicle and the lane, control the Adaptive Cruise Control (ACC) to change the Time to Collision (TTC) or change the ACC Early Warning, or to decrease the target speed depending on the curve or road type, and control the Automatic Emergency Braking (AEB) to change the Time to Collision (TTC) or change the ACC Early Warning. Accordingly, it is possible to quickly and accurately determine the situation inside and outside the vehicle and adaptively control the vehicle.

[0625] Meanwhile, in the case of emergency release mode, the safety extension module (910) within the processor (175) controls the lane centering system (LCS) and adaptive cruise control (ACC) to gradually restore or reduce the maximum allowable lateral acceleration or gradually restore or reduce the operating limit speed, and the emergency braking system (ESS) can be immediately released.

[0626] Meanwhile, in the case of emergency release mode, the authority adjustment module (915) within the processor (175) can control the lane centering system (LCS), lane keeping assist (LKA), and adaptive cruise control (ACC) to gradually restore or reduce the brake pedal, accelerator pedal, or steering override torque, or restore the turn signal dependency function, and the emergency braking system (ESS) can be immediately released.

[0627] Meanwhile, in the case of emergency release mode, the sensitivity adjustment module (920) within the processor (175) can control the lane centering system (LCS) and lane keeping assist (LKA) to gradually restore or reduce the target distance, lane width limit, parameters, etc., and can control the adaptive cruise control (ACC) to gradually restore or reduce the Time to Collision (TTC).

[0628] FIG. 13a is a drawing referenced in the description of the vehicle's operation in normal mode of FIG. 12.

[0629] Referring to the drawing, the vehicle (200) can operate in normal mode at points Pma, Pmb, and Pmc.

[0630] Meanwhile, the processor (175) can disable the lane centering system (LCS) and adaptive cruise control (ACC) in normal mode.

[0631] For example, in normal mode, when the lane centering system (LCS) is deactivated, the direction of travel of the vehicle (200) is changed based on the driver's steering wheel operation input.

[0632] As another example, in normal mode, when the lane centering system (LCS) is deactivated, the turn signal is turned on or off based on the driver's turn signal operation input.

[0633] In the drawing, based on the driver's steering wheel operation input in normal mode, the vehicle (200) is shown passing through the center line of the lane between the lanes, such as at point Pmb.

[0634] Meanwhile, in Normal mode, if the driver's steering input corresponds to a leftward movement input, the vehicle proceeds to the left of the lane rather than the center line, as at the Pmc point.

[0635] Meanwhile, in Normal mode, if the driver's steering input corresponds to a centering input, the vehicle will continue to deviate to the left of the lane rather than the center line, as seen at the Pmd point.

[0636] FIG. 13b is a drawing referenced in the description of the operation of the vehicle in the second emergency mode of FIG. 12.

[0637] Referring to the drawing, the vehicle (200) can operate in normal mode at point Pna, and then operate in a second emergency mode at points Pnb, Pnc, and Pnd.

[0638] For example, the processor (175) can disable the lane centering system (LCS) and adaptive cruise control (ACC) in normal mode.

[0639] Meanwhile, after the Pna point, if the risk level of vehicle driving changes, for example, if an emergency situation occurs, the processor (175) can switch to a second emergency mode.

[0640] Accordingly, the processor (175) can activate the lane centering system (LCS) and adaptive cruise control (ACC) in the second emergency mode.

[0641] At this time, the processor (175) can control the lane centering system (LCS) and adaptive cruise control (ACC) to remain active and not deactivated despite the driver's steering wheel operation input or the driver's turn signal operation input.

[0642] For example, the processor (175) can ignore the input of the accelerator pedal based on the lane centering system (LCS) and adaptive cruise control (ACC) when there is an unintended vehicle operation input at the Pnc point in the second emergency mode, for example, when there is an input of the accelerator pedal.

[0643] Accordingly, as at point Pnd, the vehicle (200) can maintain its lane so that it passes the center line, while maintaining the distance from the vehicle ahead.

[0644] As another example, the processor (175) can ignore the driver's steering wheel operation input in the second emergency mode, based on the lane centering system (LCS) and adaptive cruise control (ACC). Accordingly, the vehicle (200) can maintain its lane so as to pass the center line.

[0645] As another example, the processor (175) can ignore the driver's turn signal operation input in the second emergency mode based on the lane centering system (LCS) and adaptive cruise control (ACC). Accordingly, the vehicle (200) can maintain the lane so as to pass the center line.

[0646] FIG. 13c is a drawing illustrating an example of various behaviors of a passenger inside a vehicle.

[0647] Referring to the drawing, the processor (175) can monitor various behaviors of the occupants in the vehicle and calculate the risk level of the vehicle's driving based on the monitored behavior information of the occupants.

[0648] Meanwhile, the processor (175) monitors various behaviors of the occupants in the vehicle and, based on the monitored behavior information of the occupants, can classify them into use cases and non-use cases.

[0649] Meanwhile, the processor (175) may not output a vehicle control signal if the behavior information of the occupant in the vehicle corresponds to a non-use case.

[0650] Meanwhile, the processor (175) can disable the Lane Centering System (LCS), Lane Keeping Assist (LKA), Adaptive Cruise Control (ACC), or Emergency Braking System (ESS) when the behavior information of the occupant in the vehicle corresponds to a non-use case.

[0651] Meanwhile, the processor (175) can activate at least one of the lane centering system (LCS), lane keeping assist (LKA), adaptive cruise control (ACC), or emergency braking system (ESS) when the behavior information of the occupant in the vehicle corresponds to a use case.

[0652] Meanwhile, the processor (175) can monitor various behaviors of the occupants in the vehicle and, based on the monitored behavior information of the occupants, classify the risk level of the vehicle driving into levels 1 through 4.

[0653] At this time, the first level can correspond to a non-use case, and the second to fourth levels can correspond to a use case.

[0654] That is, the processor (175) monitors various behaviors of the occupants in the vehicle and, based on the monitored behavior information of the occupants, can distinguish between non-use cases and use cases.

[0655] Meanwhile, the processor (175) can monitor various behaviors of the occupants in the vehicle and, based on the monitored behavior information of the occupants, control the execution of any one of a normal mode, a first emergency mode, a second emergency mode, or an emergency release mode.

[0656] At this time, the normal mode and the emergency release mode can correspond to non-use cases, and the first emergency mode and the second emergency mode can correspond to use cases.

[0657] For example, the processor (175) can be classified as a non-use case in the case of smoking, talking, laughing, singing, navigation, object catching, calling, and phone usage.

[0658] Meanwhile, the processor (175) can classify the case of drinking a warm beverage during drinking as a non-use case.

[0659] Meanwhile, the processor (175) can classify the case where visual distraction occurs due to liquid spread or movement while drinking as a use case.

[0660] Meanwhile, the processor (175) can classify the case of eating warm food as a non-use case.

[0661] Meanwhile, the processor (175) can classify the case where visual distraction occurs due to food dispersion or movement during eating as a use case.

[0662] Meanwhile, the processor (175) can classify the case where eye scratching occurs as a use case.

[0663] Meanwhile, the processor (175) can classify cases where sneezing occurs as non-use cases.

[0664] Meanwhile, the processor (175) can classify the case of visual obstruction caused by sneezing (Sneezing while not looking forward) as a use case.

[0665] Meanwhile, the processor (175) can classify cases where yawning occurs as non-use cases.

[0666] Meanwhile, the processor (175) can classify the case of yawning while closing eyes as a use case.

[0667] FIG. 13d is a drawing illustrating different examples of various behaviors of occupants inside a vehicle.

[0668] Referring to the drawing, the processor (175) can be classified into non-use cases such as object searching, grooming, object dropping, and object placing.

[0669] Meanwhile, the processor (175) can classify cases of emergency and driver disturbance as use cases.

[0670] Meanwhile, in the case of the use case of FIG. 13c to FIG. 13d, the processor (175) can activate at least one of the lane centering system (LCS), lane keeping assist (LKA), adaptive cruise control (ACC), or emergency braking system (ESS).

[0671] Meanwhile, in the case of the non-use cases of FIGS. 13c to 13d, the processor (175) may not activate the lane centering system (LCS), lane keeping assist (LKA), adaptive cruise control (ACC), or emergency braking system (ESS), or may not operate them even if the conditions are met.

[0672] FIG. 14 is a drawing referenced in the description of multiple driving safety levels.

[0673] Referring to the drawing, the risk level determination unit (940) within the processor (175) can calculate the risk level of vehicle driving based on vehicle internal context data (941), vehicle external context data (942), and personal profile information (945).

[0674] Meanwhile, the risk level determination unit (940) within the processor (175) can classify the risk level of vehicle driving into a first level to a fourth level (951, 952, 953, 954).

[0675] Meanwhile, the level corresponding control module (950) can control the performance of corresponding operations based on the first to fourth levels (951, 952, 953, 954).

[0676] Meanwhile, the level-corresponding control module (950) can control that, after performing an operation corresponding to the first level (951), if the risk level increases, an operation corresponding to the second level (952) is performed, and after performing an operation corresponding to the second level (952), if the risk level increases, an operation corresponding to the third level (953) is performed, and after performing an operation corresponding to the third level (953), if the risk level increases, an operation corresponding to the fourth level (954) is performed.

[0677] Meanwhile, the level-corresponding control module (950) can control the execution of an operation corresponding to the first level (951) when the risk level decreases after the operation corresponding to the second level (952) is performed.

[0678] Meanwhile, the level-corresponding control module (950) can control the execution of an operation corresponding to the first level (951) or the second level (952) when the risk level decreases after the operation corresponding to the third level (953) is performed.

[0679] Meanwhile, the level-corresponding control module (950) can control the execution of an operation corresponding to the first level (951) when the risk level decreases after the operation corresponding to the fourth level (954) is performed.

[0680] FIG. 15 is a drawing referenced in the description of the operation of the level-corresponding control module of FIG. 14.

[0681] Referring to the drawing, the processor (175) can classify the risk level of vehicle driving into multiple levels based on vehicle internal context data, vehicle external context data, personal profile information, passenger behavior information, and driver driving ability information.

[0682] For example, the processor (175) can classify the risk level of vehicle driving into levels 1 through 4.

[0683] At this time, Level 1 indicates a normal state, Level 2 indicates a need for caution, Level 3 indicates imminent danger, and Level 4 may indicate a stage requiring immediate intervention.

[0684] Meanwhile, Levels 2 and 3 can correspond to the prodromal symptom stage.

[0685] For example, the processor (175) can determine the risk level of vehicle driving as a first level when the vehicle internal behavior is normal based on the vehicle internal context data and the vehicle driving environment is stable based on the vehicle external context data.

[0686] Meanwhile, the processor (175) can determine the risk level of vehicle driving to be a second level when a single sign appears based on the vehicle internal context data or the vehicle external context data.

[0687] Meanwhile, the processor (175) can determine the risk level of vehicle driving to be a third level when a complex sign appears based on vehicle internal context data or vehicle external context data.

[0688] Meanwhile, the processor (175) can determine the risk level of vehicle driving to be a fourth level when a dangerous situation occurs due to vehicle internal context data or vehicle external context data.

[0689] For example, the processor (175) can determine the risk level of vehicle driving as the first level if the habit is stable and there is no repetitive behavior based on personal profile information.

[0690] Meanwhile, the processor (175) can determine the risk level of vehicle driving to be a second level if habitual behavior exists based on personal profile information.

[0691] Meanwhile, the processor (175) can determine the risk level of vehicle driving to be a third level if there is repeated risk behavior based on personal profile information and there is a history of past personal medical history or accidents.

[0692] Meanwhile, the processor (175) can determine the risk level of vehicle driving to be a fourth level if it is confident of the risk based on the behavioral results and medical history based on personal profile information.

[0693] For example, the processor (175) can determine the risk level of vehicle driving as a first level when the passenger's behavior is in a stable state (e.g., wearing a seatbelt) based on passenger behavior information.

[0694] Meanwhile, the processor (175) can determine the risk level of driving the vehicle to be a second level if there is a mild voice stimulus (e.g., talking) based on passenger behavior information.

[0695] Meanwhile, the processor (175) can determine the risk level of driving the vehicle to a third level if the driver's attention is distracted due to shouting or physical interference based on passenger behavior information.

[0696] Meanwhile, the processor (175) can determine the risk level of vehicle driving to be a fourth level if there is strong interference or panic inducement or interference with the driver's hand operation based on passenger behavior information.

[0697] For example, the processor (175) can determine the risk level of vehicle driving as a first level if the driver can operate it based on driver driving ability information.

[0698] Meanwhile, the processor (175) can determine the risk level of vehicle driving to be a second level based on driver driving ability information, when the driver is able to operate the vehicle but is slightly distracted.

[0699] Meanwhile, the processor (175) can determine the risk level of vehicle driving to be a third level based on driver driving ability information, if the driver is able to operate but there is a delay in reaction or increased stress.

[0700] Meanwhile, the processor (175) can determine the risk level of vehicle driving to be the fourth level based on driver driving ability information, if the driver is unable to operate the vehicle, is unresponsive, or is in a panic state.

[0701] Meanwhile, the processor (175) can determine the risk level of the vehicle driving as the first level when driving while looking ahead, when there is no abnormality in the vital signs, or when stopping next to the U-turn line.

[0702] Meanwhile, the processor (175) can determine the risk level of driving the vehicle to be a second level if the vehicle is driven with one hand while holding a drink, the heart rate increases, or the vehicle stops after crossing the U-turn line.

[0703] Meanwhile, the processor (175) can determine the risk level of the vehicle driving to be the third level if there is a history of accidents such as sudden braking while driving with one hand holding a drink, or if there is a history of heart rate increase, irregular breathing, or past cardiac arrest, or if there is a stop beyond the U-turn line and an oncoming motorcycle.

[0704] Meanwhile, the processor (175) can determine the risk level of driving the vehicle to be the fourth level if, while startled by spilling a drink, the gaze is directed downward rather than forward and the hand is removed from the steering wheel, or if there is cardiac arrest and steering, or if there is no response or misoperation of the pedal, or if the vehicle stops beyond the U-turn line and an oncoming vehicle is coming.

[0705] Meanwhile, when the risk level of vehicle driving is the first level, the authority adjustment module (915), safety extension module (910), and sensitivity adjustment module (920) within the processor (175) can all remain in a non-intervention state.

[0706] Meanwhile, when the risk level of vehicle driving is at level 2, the authority adjustment module (915) within the processor (175) can learn and monitor operation patterns in a standby state, the safety extension module (910) can perform preparations to relax functional constraints of vehicle driving assistance control or autonomous driving in a standby state, and the sensitivity adjustment module (920) can perform simple visual or auditory warnings by enhancing sensitivity.

[0707] Meanwhile, when the risk level of vehicle driving is at the third level, the authority adjustment module (915) within the processor (175) may ignore or begin to correct some errors such as blinking mistakes, the safety extension module (910) may partially relax the functional restrictions of vehicle driving assistance control or autonomous driving, or gradually increase the threshold, and the sensitivity adjustment module (920) may further strengthen the sensitivity, perform multiple warnings, perform adaptive warnings based on personalization, actively perform warnings in case of behavioral risk, or perform warning mitigation in case of health risk.

[0708] Meanwhile, when the risk level of vehicle driving is at level 4, the authority adjustment module (915) within the processor (175) can immediately intervene to perform steering control or acceleration control, and the safety extension module (910) can completely release functional restrictions on vehicle driving assistance control or autonomous driving, or set the threshold value to the maximum, and the sensitivity adjustment module (920) can maintain heightened sensitivity or suppress warnings to prevent confusion. Accordingly, the situation inside and outside the vehicle can be quickly and accurately determined to perform adaptive vehicle control.

[0709] Meanwhile, the processor (175) can increase the intervention level when the driver has a history of heart-related or other diseases from the database when the Adaptive Cruise Control (ACC) or Lane Centering System (LCS) is executed. Accordingly, the vehicle control can be performed adaptively based on the driver's history of diseases.

[0710] Meanwhile, the processor (175) can increase the intervention level when there is a diagnostic history related to the driver's vision from the database when the Adaptive Cruise Control (ACC) or Lane Centering System (LCS) is executed. Accordingly, it is possible to adaptively control the vehicle based on the driver's diagnostic history.

[0711] FIG. 16 is an example of a flowchart illustrating a method of operation of a signal processing device according to an embodiment of the present disclosure.

[0712] Referring to the drawings, the processor (175) in the signal processing device (170) according to an embodiment of the present disclosure can obtain context data of the driver inside the vehicle based on the vehicle's internal sensor device or internal camera (195i), etc. (S1402).

[0713] Meanwhile, the processor (175) can obtain driver profile information or driver characteristic data based on the database (762) (S1403).

[0714] Meanwhile, the processor (175) can obtain context data of the occupant based on the vehicle's internal sensor device or internal camera (195i), etc. (S1404).

[0715] Meanwhile, the processor (175) can obtain vehicle external context data based on vehicle external sensor devices, etc. (S1405).

[0716] Meanwhile, the processor (175) can compute driving ability evaluation data based on the driver's context data, driver profile information or driver's characteristic data, passenger's context data, and vehicle external context data (S1410).

[0717] Next, the processor (175) can calculate emergency level evaluation data based on driving ability evaluation data (S1420).

[0718] Alternatively, unlike the drawing, the processor (175) can calculate emergency level evaluation data related to the safety level of vehicle driving based on the driver's context data, driver profile information or driver characteristic data, passenger's context data, and vehicle external context data.

[0719] Meanwhile, the processor (175) determines whether the emergency level evaluation data related to the safety level of vehicle driving is at the first level (S1422), and if so, can control it to remain as is without separate interference (S1420).

[0720] If not applicable in step 1422 (S1422), the processor (175) determines whether the emergency level evaluation data related to the safety level of vehicle driving is at the second level (S1424), and if applicable, performs a safety extension corresponding to the second level (S1431), performs an authority adjustment corresponding to the second level (S1432), and performs a sensitivity adjustment corresponding to the second level (S1433).

[0721] If not applicable in step 1442 (S1424), the processor (175) determines whether the emergency level evaluation data related to the safety level of vehicle driving is at the third level (S1426), and if applicable, performs a safety extension corresponding to the third level (S1434), performs an authority adjustment corresponding to the third level (S1435), and performs a sensitivity adjustment corresponding to the third level (S1436).

[0722] If not applicable in step 1426 (S1426), the processor (175) determines whether the emergency level evaluation data related to the safety level of vehicle driving is at the fourth level (S1428), and if applicable, performs a safety extension corresponding to the fourth level (S1437), performs an authority adjustment corresponding to the fourth level (S1438), and performs a sensitivity adjustment corresponding to the fourth level (S1439).

[0723] Accordingly, it becomes possible to perform adaptive vehicle control by rapidly and accurately determining the situation inside and outside the vehicle. In particular, it enables adaptive vehicle control by rapidly and accurately determining the situation inside and outside the vehicle based on multimodal context data.

[0724] FIG. 17 is another example of a flowchart illustrating a method of operation of a signal processing device according to an embodiment of the present disclosure.

[0725] Referring to the drawings, the processor (175) in the signal processing device (170) according to an embodiment of the present disclosure can obtain context data of the driver inside the vehicle based on the vehicle interior sensor device or interior camera (195i), etc. (S1402).

[0726] Meanwhile, the processor (175) can obtain driver profile information or driver characteristic data based on the database (762) (S1403).

[0727] Meanwhile, the processor (175) can obtain context data of the occupant based on the vehicle's internal sensor device or internal camera (195i), etc. (S1404).

[0728] Meanwhile, the processor (175) can obtain vehicle external context data based on vehicle external sensor devices, etc. (S1405).

[0729] Meanwhile, the processor (175) can compute driving ability evaluation data based on the driver's context data, driver profile information or driver's characteristic data, passenger's context data, and vehicle external context data (S1410).

[0730] Next, the processor (175) can calculate emergency level evaluation data based on driving ability evaluation data (S1420).

[0731] Alternatively, unlike the drawing, the processor (175) can calculate emergency level evaluation data related to the safety level of vehicle driving based on the driver's context data, driver profile information or driver characteristic data, passenger's context data, and vehicle external context data.

[0732] Meanwhile, the processor (175) determines whether the emergency level evaluation data related to the safety level of vehicle driving is at the first level (S1422), and if so, can perform safety extension, authority adjustment, and sensitivity adjustment corresponding to the first level (S1510).

[0733] If it does not apply in step 1422 (S1422), the processor (175) determines whether the emergency level evaluation data related to the safety level of vehicle driving is at the second level (S1424), and if it is, can perform safety extension, authority adjustment, and sensitivity adjustment corresponding to the second level (S1520).

[0734] If it does not apply in step 1442 (S1424), the processor (175) determines whether the emergency level evaluation data related to the safety level of vehicle driving is at the third level (S1426), and if it is, can perform safety extension, authority adjustment, and sensitivity adjustment corresponding to the third level (S1530).

[0735] If it does not apply in step 1426 (S1426), the processor (175) determines whether the emergency level evaluation data related to the safety level of vehicle driving is at the fourth level (S1428), and if it is, can perform safety extension, authority adjustment, and sensitivity adjustment corresponding to the fourth level (S1540).

[0736] Accordingly, it becomes possible to perform adaptive vehicle control by rapidly and accurately determining the situation inside and outside the vehicle. In particular, it enables adaptive vehicle control by rapidly and accurately determining the situation inside and outside the vehicle based on multimodal context data.

[0737] FIG. 18 is another example of a block diagram of a signal processing device according to an embodiment of the present disclosure.

[0738] Referring to the drawings, a signal processing device (170mm) according to an embodiment of the present disclosure includes a processor (175) that receives camera data outside the vehicle, camera data inside the vehicle, and sensor data inside the vehicle.

[0739] Meanwhile, the processor (175) according to an embodiment of the present disclosure includes a recognizer (720), an insightor (750), and an illustrator (780).

[0740] The internal recognizer (725) within the recognizer (720) can recognize gaze, position, gesture, vital signs, facial expression, facial identity, or voice.

[0741] The external recognizer (730) within the recognizer (720) can recognize road participants, traffic signs, roads, or scenes.

[0742] Meanwhile, the recognizer (720) can recognize vehicle interior state information such as distraction, impaired cognitive function, drowsiness, behavior, or health based on data from the internal recognizer (725).

[0743] Meanwhile, the multimodal context engine (755) within the insightor (750) can generate real-time context data (905) based on vehicle internal state information, external recognition data by the external recognizer (730), and vehicle and external data (718).

[0744] Meanwhile, the multimodal context engine (755) within the insightor (750) can generate multimodal context data based on real-time context data (905) and feature data (907).

[0745] Meanwhile, the AI ​​orchestrator (760) within the insightor (750) can generate a prompt for vehicle driving control based on multimodal context data from the multimodal context engine (755), profile information from the knowledge device (762), etc.

[0746] In particular, the prompt manager (763) within the AI ​​orchestrator (760) can generate a prompt for vehicle driving control based on multimodal context data from the multimodal context engine (755), profile information from the knowledge device (762), etc.

[0747] And, the generated prompt can be transmitted to an AI model set (765) or a safety assistance engine (757).

[0748] Unlike Fig. 7, the insightor (750) of Fig. 18 may not include an AI model set (765) and a safety assistance engine (757).

[0749] That is, according to FIG. 18, an AI model set (765) and a safety assistance engine (757) may be provided separately from the insightor (750) in the processor (175).

[0750] The AI ​​model set (765) may include an interface (766) for data exchange between an on-device AI model (767) and an AI model (769) within the server (400).

[0751] The AI ​​model set (765) can perform inference or response based on a prompt received from the insightor (750) and output an inference result or a response result.

[0752] Meanwhile, the safety assist engine (757) can determine whether the current situation is a safe situation or a dangerous situation based on the prompt received from the insightor (750).

[0753] For example, the safety assist engine (757) can calculate the risk level of vehicle driving based on a prompt received from the insightor (750).

[0754] As another example, the safety assist engine (757) can calculate the risk level of vehicle driving based on multimodal context data, profile information, etc. received from the insightor (750).

[0755] Meanwhile, the illustrator (780) may include a multimodal output encoder (781), a visual interface (782) including AR (783) and MR (784), and an audio / display / haptic interface (786).

[0756] Meanwhile, unlike Fig. 7, the safety application (790) within the illustrator (780) is not included and can be provided or executed separately within the processor (175).

[0757] Meanwhile, the illustrator (780) outputs a vehicle control signal based on the calculated risk level. Accordingly, the situation inside and outside the vehicle can be determined quickly and accurately, and adaptive vehicle control can be performed.

[0758] Meanwhile, the illustrator (780) can output a vehicle control signal, a warning message, or a guide message based on the calculated risk level. Accordingly, it is possible to quickly and accurately determine the situation inside and outside the vehicle and adaptively perform vehicle control.

[0759] Although preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above. Various modifications are possible by those skilled in the art without departing from the essence of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical spirit or perspective of the present disclosure.

Claims

1. A processor that receives camera data from outside the vehicle, camera data from inside the vehicle, and sensor data inside the vehicle; and The above processor is, Based on the camera data outside the vehicle mentioned above, object detection is performed, and Based on camera data inside the vehicle, the condition of passengers including the driver is monitored, and Based on the monitored occupant status information, the detected object information, the sensor data, and the driver's driving pattern information stored in the database, multimodal context data corresponding to a driving context related to vehicle driving is extracted, and A signal processing device for setting a mode for vehicle driving assistance control or autonomous driving based on the above-described multimodal context data.

2. In Paragraph 1, The above processor is, A signal processing device that controls the operation of the vehicle driving assistance control or autonomous driving in an off mode, on mode, or safety mode based on the extracted multimodal context data.

3. In Paragraph 1, The above processor is, A signal processing device that calculates a risk level of the vehicle driving based on the extracted multimodal context data, and sets the vehicle driving assistance control or autonomous driving to one of an off mode, an on mode, or a safety mode based on the calculated risk level.

4. In Paragraph 1, The above processor is, A signal processing device that sets a mode for vehicle driving assistance control or autonomous driving based on the extracted multimodal context data, and controls authority adjustment, safety extension, or sensitivity adjustment for vehicle control based on the set mode.

5. In Paragraph 1, The above processor is, Based on the above-mentioned extracted multimodal context data, determine whether it is a risk precursor stage or a risk situation stage, and A signal processing device that controls the operation mode of the vehicle driving assistance control or the autonomous driving to be variable according to the above danger precursor stage or the above danger situation stage.

6. In Paragraph 1, The above processor is, A signal processing device that calculates a risk level of the vehicle driving based on the extracted multimodal context data and outputs a vehicle control signal based on the calculated risk level.

7. In Paragraph 6, The above processor is, A signal processing device that classifies the risk level of the above-mentioned vehicle driving into levels 1 through 4.

8. In Paragraph 1, The above processor is, A signal processing device that calculates the risk level of the vehicle driving based on the extracted multimodal context data, and controls the authority adjustment, safety extension, or sensitivity adjustment for vehicle control based on the calculated risk level.

9. In Paragraph 1, The above processor is, Based on the extracted multimodal context data, the risk level of the vehicle driving is calculated, and based on the calculated risk level, authority adjustment data for vehicle control, safety extension data for safety extension, and sensitivity adjustment data for sensitivity adjustment are calculated. A signal processing device that generates the vehicle control signal based on the above authority adjustment data, safety extension data, and sensitivity adjustment data.

10. In Paragraph 1, The above processor is, A signal processing device that calculates the risk level of the vehicle driving based on the extracted multimodal context data, performs learning based on the calculated risk level and occupant profile information, and outputs the vehicle control signal, outputs a warning message, or outputs a guide message based on the result data of the learning.

11. In Paragraph 1, The above processor is, Based on the context data of the above driver and the characteristic data of the above driver, driving ability evaluation data and emergency level evaluation data are calculated, and A signal processing device that calculates the risk level of vehicle driving based on the above driving ability evaluation data and emergency level evaluation data.

12. In Paragraph 1, The above processor is, Based on the driver's context data, the driver's characteristic data, the passenger's context data, and the passenger's characteristic data, driving ability evaluation data and emergency level evaluation data are calculated, and A signal processing device that calculates the risk level of vehicle driving based on the above driving ability evaluation data and emergency level evaluation data.

13. In Paragraph 1, The above processor is, A signal processing device that classifies the risk level of the vehicle's driving into a plurality of levels based on vehicle internal context data, vehicle external context data, occupant characteristic information, occupant behavior information, and the driver's driving ability information.

14. In Paragraph 1, The above processor is, Based on the extracted multimodal context data, the risk level of the vehicle driving is calculated, and the risk level of the vehicle driving is classified into multiple levels. When the risk level of the above vehicle driving is Level 1, control is maintained so that the authority adjustment, safety extension, and sensitivity adjustment for vehicle control remain as they are, and A signal processing device that, when the risk level of the vehicle driving is a second level higher than the first level, switches the authority adjustment and safety extension for vehicle control to a standby state and increases the sensitivity of the sensitivity adjustment above the first level.

15. In Paragraph 14, The above processor is, A signal processing device that, when the risk level of the vehicle driving is a third level higher than the second level, ignores or supplements the operation input within the authority adjustment for vehicle control, increases the threshold value within the safety extension, or raises the sensitivity of the sensitivity adjustment above the second level.

16. In Paragraph 14, The above processor is, A signal processing device that, when the risk level of the vehicle driving is at its highest level, controls steering and acceleration for the authority adjustment for vehicle control, sets the threshold value within the safety extension to the maximum value, and sets the sensitivity of the sensitivity adjustment to the maximum value.

17. In Paragraph 1, A neural processor that executes inference or a response based on the above prompt; further comprising The above processor is, Obtaining an inference result or response result executed in the neural processor based on the above prompt, and A signal processing device that calculates the risk level of the vehicle driving based on the above inference result or response result, and outputs a vehicle control signal based on the calculated risk level.

18. In Paragraph 1, The above processor is, Run a hypervisor, and on the hypervisor, run a driving control virtualization machine, and The above driving control virtualization machine is, Through shared memory within the hypervisor, camera data from outside the vehicle, camera data from inside the vehicle, and sensor data from inside the vehicle are received, and A signal processing device that executes a driving control service or a driving control application based on camera data outside the vehicle, camera data inside the vehicle, sensor data, and driving pattern information of the driver stored in the database.

19. A vehicle control device comprising a signal processing device according to any one of paragraphs 1 through 18.

Images