Signal processing device and vehicle control device including same

Abstract

A signal processing device and a vehicle control device including same, according to an embodiment of the present disclosure, comprise a processor configured to receive extra-vehicle sensing data from a vehicle exterior sensor and intra-vehicle sensing data from a vehicle interior sensor, wherein the processor: collects extra-vehicle situation information, intra-vehicle situation information, and occupant situation information on the basis of the extra-vehicle sensing data and the intra-vehicle sensing data; generates a prompt for calculating a risk level of vehicle driving on the basis of each piece of the situation information; classifies the risk level of vehicle driving into a plurality of levels on the basis of an inference result or a response result obtained on the basis of the generated prompt; and controls a vehicle on the basis of the classified risk level. Accordingly, it is possible to rapidly and accurately determine situations inside and outside the vehicle and adaptively control the vehicle.

Description

Signal processing device, and vehicle control device having the same

[0001] The present disclosure relates to a signal processing device and a vehicle control device equipped with the same, and more specifically, to a signal processing device capable of rapidly and accurately determining the situation inside and outside the vehicle and adaptively performing vehicle control, and a vehicle control device equipped with the same.

[0002] A vehicle is a device that moves the user in the desired direction. A typical example is an automobile.

[0003] Meanwhile, for the convenience of users, a vehicle control device is installed inside the vehicle.

[0004] The vehicle control unit includes a signal processing unit and can perform signal processing based on sensor data from various internal sensor devices.

[0005] The prior art U.S. Patent No. 7719431 discloses a system for detecting and responding to drowsy driving, which determines the drowsy driving state of a vehicle based on a determination of whether there is a lane violation, and outputs a driver alarm signal, etc. based thereon.

[0006] However, according to prior art, there is a problem in that real-time monitoring of the vehicle interior cannot be performed and the situation inside the vehicle cannot be accurately recognized. Consequently, a problem arises where driver alarm signals unrelated to the situation inside the vehicle are output.

[0007] The problem that the present disclosure aims to solve is to provide a signal processing device capable of rapidly and accurately determining the situation inside and outside the vehicle and adaptively performing vehicle control, and a vehicle control device equipped with the same.

[0008] Another problem that the present disclosure aims to solve is to provide a vehicle control device capable of adaptively performing vehicle control based on the driver's skill level.

[0009] To solve the above technical problem, a signal processing device according to one embodiment of the present disclosure and a vehicle control device equipped therewith include a processor that receives vehicle external sensing data from a vehicle external sensor and vehicle internal sensing data from a vehicle internal sensor, and the processor collects vehicle external situation information, vehicle internal situation information, and occupant situation information based on the vehicle external sensing data and vehicle internal sensing data, generates a prompt for calculating a risk level of vehicle driving based on each situation information, classifies the risk level of vehicle driving into a plurality of levels based on an inference result or response result obtained based on the generated prompt, and controls the vehicle based on the classified risk level.

[0010] Meanwhile, the processor can classify the risk level of vehicle driving into a first level corresponding to a normal situation, a second level corresponding to a caution situation, a third level corresponding to a danger precursor situation, and a fourth level corresponding to a danger situation.

[0011] Meanwhile, the processor can generate a prompt for calculating the risk level of vehicle driving based on each situation information and previously stored driving pattern information, classify the risk level of vehicle driving into multiple levels based on the inference result or response result obtained based on the generated prompt, and control the vehicle based on the classified risk level.

[0012] Meanwhile, the processor can control authority adjustment, safety extension, or sensitivity adjustment for vehicle control based on distinguished risk levels.

[0013] Meanwhile, the processor can relax the allowable operating range during safety expansion, perform authority adjustment between the operator and the system during authority adjustment, or adjust sensitivity based on the risk level during sensitivity adjustment.

[0014] Meanwhile, the processor can increase the warning sensitivity of Autonomous Emergency Braking (AEB) or Lane Keeping Assist (LKA) when the risk level of vehicle driving is Level 2.

[0015] Meanwhile, when the risk level of vehicle driving is at level 3, the processor can control the vehicle to gradually increase the allowable lateral acceleration, gradually increase the target distance for object tracking, gradually increase the allowable steering torque value, gradually increase the accelerator pedal override threshold, gradually increase the steering override torque, gradually increase the sensitivity to the course of a front collision, gradually decrease the target speed according to road curvature, or track the center of the lane more precisely.

[0016] Meanwhile, when the risk level of vehicle driving is the fourth level, the processor can increase the allowable lateral acceleration, increase the target distance for object following, decrease the lane width limit, increase the allowable steering torque value, limit acceleration based on accelerator pedal override, ignore the condition for deactivating adaptive cruise control by the brake pedal, ignore the overtaking function by the turn signal, increase the steering override torque, ignore the condition for deactivating the lane centering system (LCS) by the turn signal, increase the sensitivity of the forward collision course, decrease the target speed based on road curvature, or control to follow the center of the lane more precisely.

[0017] Meanwhile, when the risk level of vehicle driving is at level 4, the processor compares a first allowable range of acceleration / deceleration during Adaptive Cruise Control (ACC) with a longitudinal input value based on an input signal, and if the input value is included within the first allowable range, it can lower the intervention level, and if the input value is outside the first allowable range, it can raise the intervention level.

[0018] Meanwhile, the processor can compare a second allowable range of acceleration / deceleration during adaptive cruise control (ACC) with a lateral input value based on an input signal when the risk level of vehicle driving is at the fourth level, and if the input value is included within the second allowable range, it can lower the intervention level, and if the input value is outside the second allowable range, it can raise the intervention level.

[0019] Meanwhile, the processor may lower the intervention level when the longitudinal input or lateral input is within the allowable range and there is no event in the external vehicle situation information, maintain the intervention level when the longitudinal input or lateral input is within the allowable range and there is an event requiring attention, and raise the intervention level when the longitudinal input or lateral input is outside the allowable range and there is an event requiring attention.

[0020] Meanwhile, the processor can determine the driver's proficiency by comparing the input value and the allowable range during the operation of Adaptive Cruise Control (ACC) or Lane Centering System (LCS).

[0021] Meanwhile, the processor compares the input value with the allowable range during the execution of Adaptive Cruise Control (ACC) or Lane Centering System (LCS), and if the input value deviates from the allowable range, it can raise the risk level of vehicle driving.

[0022] Meanwhile, the processor may lower the risk level of vehicle driving when the input value during the execution of Adaptive Cruise Control (ACC) or Lane Centering System (LCS) is within the allowable range and there are no events in the external vehicle situation information; maintain the risk level of vehicle driving when the input value during the execution of Adaptive Cruise Control (ACC) or Lane Centering System (LCS) is within the allowable range and there are events requiring attention in the external vehicle situation information; and raise the risk level of vehicle driving when the input value during the execution of Adaptive Cruise Control (ACC) or Lane Centering System (LCS) is outside the allowable range and there are events requiring attention.

[0023] Meanwhile, the processor calculates a risk level of vehicle driving based on an acquired inference result or response result, calculates authority adjustment data for vehicle control, safety extension data for safety extension, and sensitivity adjustment data for sensitivity adjustment based on the calculated risk level, and can generate a vehicle control signal based on the authority adjustment data, safety extension data, and sensitivity adjustment data.

[0024] Meanwhile, the processor may calculate the risk level of vehicle driving based on the acquired inference result or response result, perform learning based on the calculated risk level and occupant profile information, and output a vehicle control signal, a warning message, or a guide message based on the result data of the learning.

[0025] Meanwhile, the processor can compute driving ability evaluation data and emergency level evaluation data based on the driver's context data and driver's characteristic data, and compute the risk level of vehicle driving based on the driving ability evaluation data and emergency level evaluation data.

[0026] Meanwhile, the processor can calculate driving ability evaluation data and emergency level evaluation data based on the driver's context data, driver's characteristic data, passenger's context data, and passenger's characteristic data, and calculate the risk level of vehicle driving based on the driving ability evaluation data and emergency level evaluation data.

[0027] Meanwhile, the processor can classify the risk level of vehicle driving into multiple levels based on vehicle internal context data, vehicle external context data, occupant characteristic information, occupant behavior information, and driver's driving ability information.

[0028] Meanwhile, the processor calculates the risk level of vehicle driving based on the acquired inference result or response result, and based on the calculated risk level, if the risk level of vehicle driving is at the first level, controls the authority adjustment, safety extension, and sensitivity adjustment for vehicle control to remain as they are, and if the risk level of vehicle driving is at the second level which is higher than the first level, switches the authority adjustment and safety extension for vehicle control to a standby state and can increase the sensitivity of the sensitivity adjustment above the first level.

[0029] Meanwhile, if the risk level of vehicle driving is at a third level higher than the second level, the processor may ignore or supplement the operation input within the authority adjustment for vehicle control, increase the threshold value within the safety extension, or raise the sensitivity of the sensitivity adjustment above the second level.

[0030] Meanwhile, when the risk level of vehicle driving is at its highest level, the processor can control steering and acceleration to adjust authority for vehicle control, set the threshold within the safety extension to the maximum value, and set the sensitivity of the sensitivity adjustment to the maximum value.

[0031] Meanwhile, the processor calculates the risk level of vehicle driving based on the acquired inference result or response result, and calculates the risk level of vehicle driving as a first level when the driver's driving ability information among the monitored passenger status information indicates that the driver can operate the vehicle, calculates the risk level of vehicle driving as a second level higher than the first level when the driver's driving ability information indicates that the driver can operate the vehicle but is in a distracted state, calculates the risk level of vehicle driving as a third level higher than the second level when the driver's driving ability information indicates that the driver cannot operate the vehicle, and calculates the risk level of vehicle driving as a fourth level higher than the third level when the driver's driving ability information indicates that the driver cannot operate the vehicle.

[0032] Meanwhile, the processor may calculate a risk level of vehicle driving based on an acquired inference result or response result, and calculate a risk level of vehicle driving as a first level if the occupant characteristic information does not include habit behavior information or is not personalized habit behavior information, calculate a risk level of vehicle driving as a second level higher than the first level if the occupant characteristic information includes habit behavior information that interferes with driving, calculate a risk level of vehicle driving as a third level higher than the second level if the occupant characteristic information includes repetitive risk behavior and medical history or accident history, and calculate a risk level of vehicle driving as a fourth level higher than the third level if the occupant characteristic information includes risk information based on medical history.

[0033] Meanwhile, the processor may calculate a risk level of vehicle driving based on the acquired inference result or response result, and if the passenger's behavior information includes stable state information, calculate the risk level of vehicle driving as a first level; if the passenger's behavior information includes voice information at a level below a threshold, calculate the risk level of vehicle driving as a second level higher than the first level; if the passenger's behavior information includes repeated voice information at a level above a threshold, calculate the risk level of vehicle driving as a third level higher than the second level; and if the passenger's behavior information includes actions that interfere with the driver's driving operation, calculate the risk level of vehicle driving as a fourth level higher than the third level.

[0034] A signal processing device according to one embodiment of the present disclosure and a vehicle control device equipped therewith further include a neural processor that executes inference or response based on a prompt, and the processor generates a prompt for calculating a risk level of vehicle driving based on multimodal context data, obtains an inference result or response result executed by the neural processor based on the prompt, calculates a risk level of vehicle driving based on the obtained inference result or response result, and outputs a vehicle control signal based on the calculated risk level.

[0035] Meanwhile, the processor executes a hypervisor and executes a driving control virtualization machine on the hypervisor, and the driving control virtualization machine receives camera data outside the vehicle, camera data inside the vehicle, and sensor data inside the vehicle through shared memory within the hypervisor, and can execute a driving control service or a driving control application based on the camera data outside the vehicle, camera data inside the vehicle, sensor data, and driver's driving pattern information stored in a database.

[0036] A signal processing device according to one embodiment of the present disclosure and a vehicle control device equipped therewith comprises a processor that receives vehicle external sensing data from a vehicle external sensor and vehicle internal sensing data from a vehicle internal sensor. The processor collects vehicle external situation information, vehicle internal situation information, and occupant situation information based on the vehicle external sensing data and vehicle internal sensing data, generates a prompt for calculating a risk level of vehicle driving based on each situation information, classifies the risk level of vehicle driving into a plurality of levels based on an inference result or response result obtained based on the generated prompt, and controls the vehicle based on the classified risk level. Accordingly, the situation inside and outside the vehicle can be determined quickly and accurately, and adaptive vehicle control can be performed.

[0037] Meanwhile, the processor can classify the risk levels of vehicle driving into a first level corresponding to normal situations, a second level corresponding to caution situations, a third level corresponding to danger precursor situations, and a fourth level corresponding to danger situations. Accordingly, it becomes possible to quickly and accurately determine the situation inside and outside the vehicle and perform adaptive vehicle control.

[0038] Meanwhile, the processor generates a prompt for calculating the risk level of vehicle driving based on each situation information and previously stored driving pattern information, classifies the risk level of vehicle driving into multiple levels based on the inference result or response result obtained based on the generated prompt, and controls the vehicle based on the classified risk level. Accordingly, it becomes possible to rapidly and accurately determine the situation inside and outside the vehicle and perform adaptive vehicle control.

[0039] Meanwhile, the processor can control authority adjustment, safety extension, or sensitivity adjustment for vehicle control based on distinguished risk levels. Accordingly, it becomes possible to adaptively perform vehicle control by rapidly and accurately determining the situation inside and outside the vehicle.

[0040] Meanwhile, the processor can relax the allowable operating range during safety extension, perform authority adjustments between the driver and the system during authority adjustment, or adjust sensitivity based on risk levels during sensitivity adjustment. Accordingly, it becomes possible to rapidly and accurately determine the situation inside and outside the vehicle and perform adaptive vehicle control.

[0041] Meanwhile, the processor can increase the warning sensitivity of Autonomous Emergency Braking (AEB) or Lane Keeping Assist (LKA) when the risk level of vehicle driving is Level 2. Accordingly, it becomes possible to quickly and accurately determine the situation inside and outside the vehicle and perform adaptive vehicle control.

[0042] Meanwhile, when the risk level of vehicle driving is at level 3, the processor can control the vehicle by gradually increasing the allowable lateral acceleration, gradually increasing the target distance for object tracking, increasing the allowable steering torque value, gradually increasing the accelerator pedal override threshold, gradually increasing the steering override torque, gradually increasing the sensitivity to the course of a frontal collision, gradually decreasing the target speed according to road curvature, or tracking the center of the lane more precisely. Accordingly, it becomes possible to perform adaptive vehicle control by quickly and accurately determining the situation inside and outside the vehicle.

[0043] Meanwhile, when the risk level of vehicle driving is Level 4, the processor can control the vehicle by increasing the allowable lateral acceleration, increasing the target distance for object tracking, decreasing the lane width limit, increasing the allowable steering torque value, limiting acceleration based on accelerator pedal override, ignoring the condition for deactivating adaptive cruise control via the brake pedal, ignoring the overtaking function via the turn signal, increasing the steering override torque, ignoring the condition for deactivating the lane centering system (LCS) via the turn signal, increasing the sensitivity to forward collision, decreasing the target speed based on road curvature, or tracking the center of the lane more precisely. Accordingly, it becomes possible to quickly and accurately determine the situation inside and outside the vehicle and perform adaptive vehicle control.

[0044] Meanwhile, when the risk level of vehicle driving is at the fourth level, the processor compares a first allowable range of acceleration / deceleration during Adaptive Cruise Control (ACC) with a longitudinal input value based on an input signal, and if the input value falls within the first allowable range, it can lower the intervention level, and if the input value falls outside the first allowable range, it can raise the intervention level. Accordingly, adaptive vehicle control can be performed based on the driver's proficiency.

[0045] Meanwhile, when the risk level of vehicle driving is at the fourth level, the processor compares the second allowable range of acceleration / deceleration during Adaptive Cruise Control (ACC) with a lateral input value based on an input signal, and if the input value is included within the second allowable range, it can lower the intervention level, and if the input value is outside the second allowable range, it can raise the intervention level. Accordingly, it is possible to adaptively control the vehicle based on the driver's proficiency.

[0046] Meanwhile, the processor may lower the intervention level when the longitudinal or lateral input is within the acceptable range and there are no events in the external vehicle situation information, maintain the intervention level when the longitudinal or lateral input is within the acceptable range and there are events requiring attention, and raise the intervention level when the longitudinal or lateral input is outside the acceptable range and there are events requiring attention. Accordingly, adaptive vehicle control can be performed based on the driver's proficiency.

[0047] Meanwhile, the processor can determine the driver's proficiency by comparing the input value with the allowable range during the execution of Adaptive Cruise Control (ACC) or the Lane Centering System (LCS). Accordingly, it becomes possible to perform adaptive vehicle control based on the driver's proficiency.

[0048] Meanwhile, the processor compares the input value with the allowable range during the execution of Adaptive Cruise Control (ACC) or the Lane Centering System (LCS), and can raise the risk level of vehicle driving if the input value deviates from the allowable range. Accordingly, adaptive vehicle control can be performed based on the driver's proficiency.

[0049] Meanwhile, the processor may lower the risk level of vehicle driving when the input value during the execution of Adaptive Cruise Control (ACC) or Lane Centering System (LCS) is within an acceptable range and there are no events in the external vehicle situation information; maintain the risk level of vehicle driving when the input value during the execution of Adaptive Cruise Control (ACC) or Lane Centering System (LCS) is within an acceptable range and there are events requiring attention in the external vehicle situation information; and raise the risk level of vehicle driving when the input value during the execution of Adaptive Cruise Control (ACC) or Lane Centering System (LCS) exceeds an acceptable range and there are events requiring attention. Accordingly, adaptive vehicle control can be performed based on the driver's proficiency.

[0050] Meanwhile, the processor calculates a risk level of vehicle driving based on acquired inference results or response results, calculates authority adjustment data for vehicle control, safety extension data for safety extension, and sensitivity adjustment data for sensitivity adjustment based on the calculated risk level, and generates a vehicle control signal based on the authority adjustment data, safety extension data, and sensitivity adjustment data. Accordingly, it becomes possible to rapidly and accurately determine the situation inside and outside the vehicle and adaptively perform vehicle control.

[0051] Meanwhile, the processor can calculate the risk level of vehicle driving based on the acquired inference results or response results, perform learning based on the calculated risk level and occupant profile information, and output a vehicle control signal, a warning message, or a guide message based on the learning result data. Accordingly, it becomes possible to quickly and accurately determine the situation inside and outside the vehicle and adaptively perform vehicle control.

[0052] Meanwhile, the processor can compute driving capability evaluation data and emergency level evaluation data based on the driver's context data and driver characteristic data, and compute the risk level of vehicle driving based on the driving capability evaluation data and emergency level evaluation data. Accordingly, it becomes possible to rapidly and accurately determine the situation inside and outside the vehicle and perform adaptive vehicle control.

[0053] Meanwhile, the processor can compute driving capability evaluation data and emergency level evaluation data based on the driver's context data, driver's characteristic data, passenger's context data, and passenger's characteristic data, and compute the risk level of vehicle driving based on the driving capability evaluation data and emergency level evaluation data. Accordingly, it becomes possible to rapidly and accurately determine the situation inside and outside the vehicle and perform adaptive vehicle control.

[0054] Meanwhile, the processor can classify the risk level of vehicle driving into multiple levels based on vehicle internal context data, vehicle external context data, occupant characteristic information, occupant behavior information, and driver driving ability information. Accordingly, it becomes possible to rapidly and accurately determine the situation inside and outside the vehicle and perform adaptive vehicle control.

[0055] Meanwhile, the processor calculates the risk level of vehicle driving based on the acquired inference result or response result, and based on the calculated risk level, if the risk level of vehicle driving is at the first level, controls the authority adjustment, safety extension, and sensitivity adjustment for vehicle control to remain unchanged; and if the risk level of vehicle driving is at the second level, which is higher than the first level, switches the authority adjustment and safety extension for vehicle control to a standby state and can increase the sensitivity of the sensitivity adjustment above the first level. Accordingly, it becomes possible to quickly and accurately determine the situation inside and outside the vehicle and perform adaptive vehicle control.

[0056] Meanwhile, if the risk level of vehicle driving is at a third level higher than the second level, the processor may ignore or supplement the operation input within the authority adjustment for vehicle control, increase the threshold value within the safety extension, or raise the sensitivity of the sensitivity adjustment above the second level. Accordingly, it becomes possible to quickly and accurately determine the situation inside and outside the vehicle and perform adaptive vehicle control.

[0057] Meanwhile, when the risk level of vehicle driving is at its highest level, the processor can control steering and acceleration to adjust authority for vehicle control, set the threshold within the safety extension to the maximum value, and set the sensitivity of the sensitivity adjustment to the maximum value. Accordingly, it becomes possible to quickly and accurately determine the situation inside and outside the vehicle and perform adaptive vehicle control.

[0058] Meanwhile, the processor calculates a risk level of vehicle driving based on the acquired inference result or response result. If the driver's driving ability information among the monitored occupant status information indicates that the driver can operate the vehicle, the risk level of vehicle driving is calculated as a first level; if the driver's driving ability information indicates that the driver can operate the vehicle but is in a distracted state, the risk level of vehicle driving is calculated as a second level higher than the first level; if the driver's driving ability information indicates that the driver can operate the vehicle but is in a delayed reaction state, the risk level of vehicle driving is calculated as a third level higher than the second level; and if the driver's driving ability information indicates that the driver cannot operate the vehicle, the risk level of vehicle driving is calculated as a fourth level higher than the third level. Accordingly, it becomes possible to quickly and accurately determine the situation inside and outside the vehicle and perform adaptive vehicle control.

[0059] Meanwhile, the processor calculates the risk level of vehicle driving based on the acquired inference result or response result. If the occupant characteristic information does not include habitual behavior information or is not personalized habitual behavior information, the risk level of vehicle driving is calculated as a first level. If the occupant characteristic information includes habitual behavior information that interferes with driving, the risk level of vehicle driving is calculated as a second level higher than the first level. If the occupant characteristic information includes repetitive risky behaviors and medical history or accident history, the risk level of vehicle driving is calculated as a third level higher than the second level. If the occupant characteristic information includes risk information based on medical history, the risk level of vehicle driving is calculated as a fourth level higher than the third level. Accordingly, it becomes possible to quickly and accurately determine the situation inside and outside the vehicle and perform adaptive vehicle control.

[0060] Meanwhile, the processor calculates a risk level of vehicle driving based on the acquired inference result or response result, and calculates the risk level of vehicle driving as a first level if the passenger's behavioral information includes stable state information; calculates the risk level of vehicle driving as a second level higher than the first level if the passenger's behavioral information includes voice information at a level below a threshold; calculates the risk level of vehicle driving as a third level higher than the second level if the passenger's behavioral information includes repeated voice information at a level above a threshold; and calculates the risk level of vehicle driving as a fourth level higher than the third level if the passenger's behavioral information includes actions that interfere with the driver's driving operation. Accordingly, it becomes possible to quickly and accurately determine the situation inside and outside the vehicle and perform adaptive vehicle control.

[0061] A signal processing device according to one embodiment of the present disclosure and a vehicle control device equipped therewith further include a neural processor that executes inference or response based on a prompt. The processor generates a prompt for calculating a risk level of vehicle driving based on multimodal context data, obtains an inference result or response result executed by the neural processor based on the prompt, calculates a risk level of vehicle driving based on the obtained inference result or response result, and outputs a vehicle control signal based on the calculated risk level. Accordingly, it is possible to rapidly and accurately determine the situation inside and outside the vehicle and adaptively perform vehicle control.

[0062] Meanwhile, the processor executes a hypervisor and executes a driving control virtualization machine on the hypervisor. The driving control virtualization machine receives camera data from outside the vehicle, camera data from inside the vehicle, and sensor data from inside the vehicle through shared memory within the hypervisor. Based on the camera data from outside the vehicle, camera data from inside the vehicle, sensor data, and driver's driving pattern information stored in a database, the driving control service or driving control application can be executed. Accordingly, it becomes possible to quickly and accurately determine the situation inside and outside the vehicle and adaptively perform vehicle control.

[0063] Figure 1 is a drawing illustrating an example of the exterior and interior of a vehicle.

[0064] Figure 2 is a diagram illustrating an example of the architecture of a vehicle control device.

[0065] FIG. 3a is a drawing illustrating an example of the arrangement of displays inside a vehicle.

[0066] Figure 3b is a drawing illustrating another example of the arrangement of displays inside a vehicle.

[0067] FIG. 4 is an example of an internal block diagram of a vehicle control device according to an embodiment of the present disclosure.

[0068] FIG. 5 is an example of a configuration diagram of a signal processing device according to an embodiment of the present disclosure.

[0069] FIG. 6 is an example of a block diagram of a vehicle control device according to an embodiment of the present disclosure.

[0070] FIG. 7 is an example of a block diagram of a signal processing device according to an embodiment of the present disclosure.

[0071] FIGS. 8a to 8c are drawings referenced in the description of FIG. 7.

[0072] FIG. 9 is another example of a block diagram of a signal processing device according to an embodiment of the present disclosure.

[0073] FIGS. 10 to 13d are drawings referenced in the description of FIG. 9.

[0074] FIG. 14a is an example of a signal processing system according to an embodiment of the present disclosure.

[0075] FIG. 14b is another example of a signal processing system according to an embodiment of the present disclosure.

[0076] FIGS. 14c to 15 are drawings referenced in the description of FIG. 14a or FIG. 14b.

[0077] FIG. 16 is an example of a flowchart illustrating a method of operation of a signal processing device according to an embodiment of the present disclosure.

[0078] The present disclosure will be described in more detail below with reference to the drawings.

[0079] The suffixes "module" and "part" for components used in the following description are assigned solely for the ease of drafting this specification and do not inherently confer any particularly significant meaning or role. Accordingly, the terms "module" and "part" may be used interchangeably.

[0080] Figure 1 is a drawing illustrating an example of the exterior and interior of a vehicle.

[0081] Referring to the drawing, the vehicle (200) is operated by a plurality of wheels (103FR, 103FL, 103RL,...) that rotate by a power source, and a steering wheel (150) for controlling the direction of travel of the vehicle (200).

[0082] Meanwhile, the vehicle (200) may further be equipped with a camera (195), etc., for acquiring an image of the front of the vehicle.

[0083] Meanwhile, the vehicle (200) may be equipped with a plurality of displays (180a, 180b) for displaying images, information, etc. inside.

[0084] In FIG. 1, a cluster display (180a) and an IVI (In-Vehicle Infotainment) display (180b) are exemplified as multiple displays (180a, 180b). In addition, a HUD (Head Up Display) and the like are also possible.

[0085] Meanwhile, the IVI (In-Vehicle Infotainment) display (180b) may also be named a Center Information Display or an AVN (Audio Video Navigation) display.

[0086] Meanwhile, the vehicle (200) described in this specification may be a concept that includes all of the following: a vehicle equipped with an engine as a power source, a hybrid vehicle equipped with an engine and an electric motor as a power source, an electric vehicle equipped with an electric motor as a power source, etc.

[0087] Figure 2 is a diagram illustrating an example of the architecture of a vehicle control device.

[0088] Referring to the drawing, the architecture (300a) of the vehicle control device can correspond to a zone-based architecture.

[0089] Accordingly, sensor devices and processors inside the vehicle may be placed in each of the multiple zones (Z1 to Z4), and a signal processing device (170a) including a gateway (GWDa) may be placed in the central area of ​​the multiple zones (Z1 to Z4).

[0090] Meanwhile, the signal processing device (170a) may additionally include an autonomous driving control module (ACC), a cockpit control module (CPG), etc., in addition to the gateway (GWDa).

[0091] The gateway (GWDa) within the signal processing device (170a) may be a High Performance Computing (HPC) gateway.

[0092] That is, the signal processing device (170a) of FIG. 2 is an integrated HPC and can exchange data with an external communication module (not shown) or a processor (not shown) in a plurality of zones (Z1 to Z4).

[0093] FIG. 3a is a drawing illustrating an example of the arrangement of displays inside a vehicle.

[0094] Referring to the drawing, the vehicle interior may be equipped with a cluster display (180a), an IVI (In-Vehicle Infotainment) display (180b), a rear seat entertainment display (180c, 180d), a rearview mirror display (not shown), etc.

[0095] Meanwhile, in addition to the display, an interior camera (195i) may be installed inside the vehicle.

[0096] Figure 3b is a drawing illustrating another example of the arrangement of displays inside a vehicle.

[0097] A vehicle control device (100) according to an embodiment of the present disclosure may include a plurality of displays (180a to 180b) and a signal processing device (170) that performs signal processing for displaying images, information, etc. on the plurality of displays (180a to 180b) and outputs an image signal to at least one display (180a to 180b).

[0098] Among the plurality of displays (180a to 180b), the first display (180a) is a cluster display (180a) for displaying driving status, operation information, etc., and the second display (180b) may be an IVI (In-Vehicle Infotainment) display (180b) for displaying vehicle operation information, navigation map, various entertainment information or video.

[0099] The signal processing device (170) has a processor (175) inside and can execute a first virtualization machine to a third virtualization machine (not shown) on a hypervisor (not shown) within the processor (175).

[0100] A second virtualization machine (not shown) operates for the first display (180a), and a third virtualization machine (not shown) can operate for the second display (180b).

[0101] Meanwhile, the first virtualization machine (not shown) within the processor (175) can be controlled to set up a shared memory (508) based on a hypervisor (505) for the same data transmission to the second virtualization machine (not shown) and the third virtualization machine (not shown). Accordingly, the same information or the same image can be synchronized and displayed on the first display (180a) and the second display (180b) within the vehicle.

[0102] Meanwhile, the first virtualization machine (not shown) within the processor (175) shares at least a portion of the data with the second virtualization machine (not shown) and the third virtualization machine (not shown) for data sharing processing. Accordingly, data can be shared and processed by multiple virtualization machines for multiple displays within the vehicle.

[0103] Meanwhile, the first virtualization machine (not shown) within the processor (175) can receive and process wheel speed sensor data of the vehicle and transmit the processed wheel speed sensor data to at least one of the second virtualization machine (not shown) or the third virtualization machine (not shown). Accordingly, the wheel speed sensor data of the vehicle can be shared with at least one virtualization machine, etc.

[0104] Meanwhile, the vehicle control device (100) according to the embodiment of the present disclosure may further include a rear seat entertainment display (180c) for displaying driving status information, simple navigation information, various entertainment information or images.

[0105] The signal processing device (170) can control the RSE display (180c) by running a fourth virtualization machine (not shown) in addition to the first to third virtualization machines (not shown) on a hypervisor (not shown) within the processor (175).

[0106] Accordingly, various displays (180a to 180c) can be controlled using a single signal processing device (170).

[0107] Meanwhile, some of the multiple displays (180a to 180c) operate under a Linux OS, and others can operate under a Web OS.

[0108] A signal processing device (170) according to an embodiment of the present disclosure can control displays (180a to 180c) operating under various operating systems (OS) to synchronize and display the same information or the same image.

[0109] Meanwhile, FIG. 3b illustrates that a vehicle speed indicator (212a) and a vehicle interior temperature indicator (213a) are displayed on a first display (180a), a home screen (222) including a plurality of applications, a vehicle speed indicator (212b), and a vehicle interior temperature indicator (213b) is displayed on a second display (180b), and a second home screen (222b) including a plurality of applications and a vehicle interior temperature indicator (213c) is displayed on a third display (180c).

[0110] FIG. 4 is an example of an internal block diagram of a vehicle control device according to an embodiment of the present disclosure.

[0111] Referring to the drawings, a vehicle control device (100) according to an embodiment of the present disclosure may include an input device (110), a communication unit (120) for communication with an external device, a plurality of communication modules (EMa~EMd) for internal communication, a memory (140), a signal processing device (170), a plurality of displays (180a~180c), an audio output unit (185), and a power supply unit (190).

[0112] Multiple communication modules (EMa~EMd) can be placed in each of the multiple zones (Z1~Z4) of FIG. 2, for example.

[0113] Meanwhile, the signal processing device (170) may have a communication switch (736b) inside for data communication with each communication module (EM1~EM4).

[0114] Each communication module (EM1~EM4) can perform data communication with a plurality of sensor devices (SN), ECU (770), or area signal processing device (170Z).

[0115] Meanwhile, a plurality of sensor devices (SN) may include a camera (195), lidar (196), radar (197), or position sensor (198).

[0116] The input device (110) may be equipped with physical buttons, pads, etc. for button input, touch input, etc.

[0117] Meanwhile, the input device (110) may be equipped with a microphone (112) for user voice input.

[0118] The communication unit (120) can exchange data wirelessly with a mobile terminal (600) or a server (400).

[0119] In particular, the communication unit (120) can wirelessly exchange data with the vehicle driver's mobile terminal. Various data communication methods are possible as wireless data communication methods, such as Bluetooth, WiFi, WiFi Direct, and APiX.

[0120] The communication unit (120) can receive weather information, road traffic condition information, for example, TPEG (Transport Protocol Expert Group) information from a mobile terminal (600) or a server (400). To this end, the communication unit (120) may be equipped with a mobile communication module (not shown).

[0121] A plurality of communication modules (EM1~EM4) can receive sensor data, etc. from an ECU (770), a sensor device (SN), or a region signal processing device (170Z), and transmit the received sensor data to the signal processing device (170).

[0122] Here, the sensor data may include at least one of vehicle direction data, vehicle location data (GPS data), vehicle angle data, vehicle speed data, vehicle acceleration data, vehicle tilt data, vehicle forward / reverse data, battery data, fuel data, tire data, vehicle lamp data, vehicle interior temperature data, and vehicle interior humidity data.

[0123] Such sensor data can be obtained from a heading sensor, a yaw sensor, a gyro sensor, a position module, a vehicle forward / reverse sensor, a wheel sensor, a vehicle speed sensor, a vehicle body inclination sensor, a battery sensor, a fuel sensor, a tire sensor, a steering sensor based on steering wheel rotation, a vehicle interior temperature sensor, a vehicle interior humidity sensor, etc.

[0124] Meanwhile, the position module may include a GPS module or a position sensor (198) for receiving GPS information.

[0125] Meanwhile, at least one of the multiple communication modules (EM1 to EM4) can transmit location information data sensed from a GPS module or a location sensor (198) to a signal processing device (170).

[0126] Meanwhile, at least one of the plurality of communication modules (EM1 to EM4) can receive vehicle front image data, vehicle side image data, vehicle rear image data, and obstacle distance information around the vehicle from a camera (195), lidar (196), radar (197), etc., and transmit the received information to a signal processing device (170).

[0127] The memory (140) can store various data for the overall operation of the vehicle control device (100), such as a program for processing or controlling the signal processing device (170).

[0128] For example, memory (140) can store data regarding a hypervisor, a first virtualization machine to a third virtualization machine, for execution within a processor (175).

[0129] The audio output unit (185) converts an electrical signal from the signal processing device (170) into an audio signal and outputs it. To do this, a speaker or the like may be provided.

[0130] The power supply unit (190) can supply power necessary for the operation of each component under the control of the signal processing unit (170). In particular, the power supply unit (190) can receive power from a battery inside the vehicle, etc.

[0131] The signal processing unit (170) controls the overall operation of each unit within the vehicle control unit (100).

[0132] For example, the signal processing device (170) may include a processor (175) that performs signal processing for a vehicle display (180a, 180b).

[0133] The processor (175) can run a first virtualization machine to a third virtualization machine (not shown) on a hypervisor (505 in FIG. 5) within the processor (175).

[0134] Among the first to third virtual machines (not shown), the first virtual machine (not shown) may be named a Server Virtual Machine, and the second to third virtual machines (not shown) may be named a Guest Virtual Machine.

[0135] For example, a first virtualization machine (not shown) within a processor (175) can receive sensor data from a plurality of sensor devices, such as vehicle sensor data, location information data, camera image data, audio data, or touch input data, and process or modify it to output it.

[0136] In this way, by performing most of the data processing in the first virtualization machine (not shown), 1:N data sharing becomes possible.

[0137] As another example, the first virtualization machine (not shown) can directly receive and process CAN data, Ethernet data, audio data, radio data, USB data, and wireless communication data for the second virtualization machine to the third virtualization machine (not shown).

[0138] And, the first virtualization machine (not shown) can transmit the processed data to the second virtualization machine to the third virtualization machine (not shown).

[0139] Accordingly, among the first to third virtualization machines (not shown), only the first virtualization machine (not shown) receives sensor data, communication data, or external input data from a plurality of sensor devices and performs signal processing, thereby reducing the signal processing burden on other virtualization machines and enabling 1:N data communication, which enables synchronization when sharing data.

[0140] Meanwhile, the first virtualization machine (not shown) can control the sharing of the same data with the second virtualization machine (not shown) and the third virtualization machine (not shown) by writing data to the shared memory (508 in FIG. 5).

[0141] For example, the first virtualization machine (not shown) can record vehicle sensor data, the location information data, the camera image data, or the touch input data in a shared memory (508) and control the sharing of the same data with the second virtualization machine (not shown) and the third virtualization machine (not shown). Accordingly, data sharing in a 1:N manner becomes possible.

[0142] Ultimately, by performing most of the data processing on the first virtualization machine (not shown), 1:N data sharing becomes possible.

[0143] Meanwhile, the first virtualization machine (not shown) within the processor (175) can control the second virtualization machine (not shown) and the third virtualization machine (not shown) to set up a shared memory (508) based on the hypervisor (505) for the same data transmission.

[0144] Meanwhile, the signal processing device (170) can process various signals such as audio signals, video signals, and data signals. To this end, the signal processing device (170) can be implemented in the form of a System On Chip (SOC).

[0145] Meanwhile, the signal processing device (170) in the display device (100) of FIG. 4 may be the same as the signal processing device (170) of the vehicle control device of FIG. 5 and below.

[0146] FIG. 5 is an example of a configuration diagram of a signal processing device according to an embodiment of the present disclosure.

[0147] Referring to the drawings, a signal processing device (170) according to an embodiment of the present disclosure includes a processor (175).

[0148] Meanwhile, the signal processing device (170) may be named as an HPC (High Performance Computing) signal processing device.

[0149] Meanwhile, the signal processing device (170) according to an embodiment of the present disclosure may further include a neural processor (179).

[0150] Meanwhile, the neural processor (179) may also be referred to as an on-device-based learning processor.

[0151] Meanwhile, the processor (175) in the signal processing device (170) can execute the hypervisor (505) and execute the first to third virtualization machines (510 to 530) on the hypervisor (505).

[0152] The first virtualization machine (510) may be a gateway virtualization machine corresponding to the gateway (GWDa) of FIG. 2.

[0153] The second virtualization machine (520) may be a driving control virtualization machine corresponding to the autonomous driving control module (ACC) of FIG. 2.

[0154] The driving control virtualization machine (520) at this time can control the vehicle driving assistance (ADAS) or the autonomous driving (AD).

[0155] The third virtualization machine (530) may be a display virtualization machine corresponding to the cockpit control module (CPG) of FIG. 2 or the display (180a, 180b, 180c) of FIG. 3.

[0156] For example, the third virtualization machine (530) may include a cluster virtualization machine (530b) for a cluster display (180a) and an IVI virtualization machine (530b) for an IVI display (180b).

[0157] Meanwhile, the third virtualization machine (530) may further include a HUD virtualization machine (not shown) for a HUD display (180c).

[0158] Meanwhile, the processor (175) can share data between each virtualization machine (510~530) through shared memory (505) within the hypervisor (505).

[0159] Meanwhile, the processor (175) can share data with a plurality of sensor devices (SN) or ECUs (770) or area signal processing devices (170Z) through shared memory (505) within the hypervisor (505).

[0160] Meanwhile, the processor (175) can exchange data with an external server (400) or mobile terminal (600) through the communication device (120) of FIG. 4.

[0161] Meanwhile, the data received by the processor (175) within the signal processing device (170) may include camera data or sensor data.

[0162] For example, sensor data within the vehicle may include at least one of vehicle wheel speed data, vehicle direction data, vehicle location data (GPS data), vehicle angle data, vehicle speed data, vehicle acceleration data, vehicle tilt data, vehicle forward / reverse data, battery data, fuel data, tire data, vehicle lamp data, vehicle interior temperature data, vehicle interior humidity data, vehicle external radar data, and vehicle external lidar data.

[0163] Meanwhile, camera data may include external vehicle camera data and internal vehicle camera data.

[0164] Meanwhile, the processor (175) within the signal processing device (170) can execute multiple virtualization machines (510 to 530) based on safety standards.

[0165] Meanwhile, the processor (175) in the signal processing unit (170a) can execute the hypervisor (505) and, on the hypervisor (505), execute the first to third virtualization machines (510 to 530) according to the automotive safety integrity level (Automotive SIL and ASIL).

[0166] For example, the first virtualization machine (510) may be a virtualization machine corresponding to ASIL C or ASIL D, in which the sum of Severity, Exposure, and Controllability in the Automotive Safety Integrity Level (ASIL) is 9 or 10.

[0167] Meanwhile, ASIL D can correspond to the grade requiring the highest safety level.

[0168] The first virtualization machine (510) can run a safety operating system (not shown) and an application (not shown) on the safety operating system.

[0169] Meanwhile, the first virtualization machine (510) may run a container runtime (not shown) and a container runtime (not shown) on a safety operating system.

[0170] Meanwhile, unlike the drawing, the first virtualization machine (510) may also be executed through a separate processor core instead of the processor (175).

[0171] Meanwhile, the second virtualization machine (520) may be a virtualization machine corresponding to ASIL A or ASIL B, in which the sum of Severity, Exposure, and Controllability in the Automotive Safety Integrity Level (ASIL) is 7 or 8.

[0172] Meanwhile, the second virtualization machine (520) can run an operating system (not shown), a container runtime (not shown) on the operating system (not shown), and a container (not shown) on the container runtime.

[0173] Alternatively, the second virtualization machine (520) can run an operating system (not shown) and an application on the operating system (not shown).

[0174] Meanwhile, the third virtualization machine (530) may be a virtualization machine corresponding to Quality Management (QM), which is the lowest safety level and non-mandatory grade in the Automotive Safety Integrity Level (ASIL).

[0175] Meanwhile, the third virtualization machine (530) can run an operating system (not shown), a container runtime (not shown) on the operating system (not shown), and a container (not shown) on the container runtime.

[0176] Alternatively, the third virtualization machine (530) can run an operating system (not shown) and an application on the operating system (not shown).

[0177] FIG. 6 is an example of a block diagram of a vehicle control device according to an embodiment of the present disclosure.

[0178] Referring to the drawings, a vehicle control device (900) according to an embodiment of the present disclosure comprises a signal processing device (170).

[0179] A vehicle control device (900) according to an embodiment of the present disclosure may further include at least one display.

[0180] In the drawing, at least one display is exemplified as a cluster display (180a) and an IVI display (180b).

[0181] Meanwhile, the vehicle control device (900) may further include a plurality of area signal processing devices (170Z1 to 170Z4).

[0182] The signal processing device (170) at this time is a high-performance centralized signal processing and control device having a plurality of CPUs (175), GPUs (178), NPUs (179), etc., and can be named as a High Performance Computing (HPC) signal processing device or a central signal processing device.

[0183] Multiple area signal processing devices (170Z1~170Z4) and signal processing device (170) are connected by wired cables (CB1~CB4).

[0184] Meanwhile, multiple area signal processing devices (170Z1~170Z4) can be connected to each other by wired cables (CBa~CBd).

[0185] The wired cable (CBa~CBd) at this time may include a CAN communication cable, an Ethernet communication cable, or a PCI Express cable.

[0186] Meanwhile, the signal processing device (170) according to the embodiment of the present disclosure may have at least one processor (175, 178, 177) and a large-capacity storage device (925).

[0187] For example, a signal processing device (170) according to an embodiment of the present disclosure may include a central processor (175, 177), a graphics processor (178), and a neural processor (179).

[0188] Meanwhile, sensor data can be transmitted from at least one of the multiple area signal processing devices (170Z1 to 170Z4) to the signal processing device (170). In particular, the sensor data can be stored in a storage device (925) within the signal processing device (170).

[0189] The sensor data at this time may include at least one of camera data, lidar data, radar data, vehicle direction data, vehicle position data (GPS data), vehicle angle data, vehicle speed data, vehicle acceleration data, vehicle tilt data, vehicle forward / reverse data, battery data, fuel data, tire data, vehicle lamp data, vehicle interior temperature data, and vehicle interior humidity data.

[0190] In the drawing, camera data from a camera (195a) and lidar data from a lidar sensor (196) are input to a first area signal processing device (170Z1), and the camera data and lidar data are transmitted to a signal processing device (170) via a second area signal processing device (170Z2) and a third area signal processing device (170Z3), etc.

[0191] Meanwhile, since the data reading or writing speed to the storage device (925) is faster than the network speed when sensor data is transmitted from at least one of the multiple area signal processing devices (170Z1~170Z4) to the signal processing device (170), it is desirable to perform multipath routing so that network bottlenecks do not occur.

[0192] To this end, the signal processing device (170) according to an embodiment of the present disclosure can perform multipath routing based on a Software Defined Network (SDN). Accordingly, a stable network environment can be secured when reading or writing data of the storage device (925). Furthermore, since data can be transmitted to the storage device (925) using multiple paths, data can be transmitted by dynamically changing the network configuration.

[0193] Data communication between a plurality of area signal processing devices (170Z1~170Z4) and a signal processing device (170) within a vehicle control device (900) according to an embodiment of the present disclosure is preferably Peripheral Component Interconnect Express communication or Ethernet communication for high-bandwidth, low-latency communication.

[0194] FIG. 7 is an example of a block diagram of a signal processing device according to an embodiment of the present disclosure.

[0195] Referring to the drawings, the signal processing system (700) according to an embodiment of the present disclosure may be referred to as an IVEX (In Vehicle Experience) system.

[0196] Meanwhile, the signal processing system (700) according to an embodiment of the present disclosure may include an edge sensor group (SN), a signal processing device (170) in a vehicle, and a server (400).

[0197] Meanwhile, the signal processing device (170) according to an embodiment of the present disclosure includes a recognizer (720), an insightor (750), and an illustrator (780).

[0198] Meanwhile, the edge sensor group (SN) can correspond to a plurality of sensor devices (SN) of FIG. 4.

[0199] Meanwhile, the recognizer (720), the insightor (750), and the illustrator (780) may be included in the signal processing device (170) of the vehicle display device (100) of FIG. 4.

[0200] In particular, the recognizer (720), the insightor (750), and the illustrator (780) may be included in the processor (175) of the signal processing device (170).

[0201] Meanwhile, the edge sensor group (SN) may include vehicle interior sensors (710) and vehicle exterior sensors (715).

[0202] Meanwhile, the edge sensor group (SN) may further include a data receiving unit (718) for receiving vehicle internal data or vehicle external data. The data receiving unit (718) may correspond to the communication device (120) of FIG. 2.

[0203] Meanwhile, the vehicle interior sensors (710) are sensors placed inside the vehicle (200) and may include a front camera (195), lidar (196), radar (197), interior camera (195i), vehicle interior temperature sensor, or vehicle interior humidity sensor.

[0204] The vehicle external sensors (715) are sensors positioned on the exterior of the vehicle (200) and may include an external camera, lidar (196), radar (197), heading sensor, yaw sensor, gyro sensor, position module, vehicle forward / reverse sensor, wheel sensor, vehicle speed sensor, vehicle body inclination sensor, battery sensor, fuel sensor, tire sensor, or steering sensor based on steering wheel rotation. The position module may include a GPS module or a position sensor (198) for receiving GPS information.

[0205] Sensing data may include vehicle interior sensing data and vehicle exterior sensing data.

[0206] The vehicle interior sensing data may be data sensed by the vehicle interior sensors (710).

[0207] Vehicle interior sensing data may include at least one of vehicle interior temperature data, vehicle interior humidity data, battery data, fuel data, vehicle lamp data, tire data, or vehicle interior camera data, or audio data received through a microphone.

[0208] The vehicle external sensing data may be data sensed by the vehicle external sensors (715).

[0209] Vehicle external sensing data may include at least one of vehicle location data (GPS), vehicle direction data, vehicle angle data, vehicle speed data, vehicle acceleration data, vehicle tilt data, whether the vehicle is moving forward or backward, front camera data, or rear camera data.

[0210] Meanwhile, the recognizer (720) can recognize the vehicle situation based on sensing data received from the edge sensor group (SN). In this regard, the recognizer (720) may be referred to as a situation recognition unit.

[0211] The vehicle situation at this time may include external vehicle conditions and internal vehicle conditions.

[0212] Meanwhile, the recognizer (720) can recognize the vehicle situation based on the vehicle internal sensing data or the vehicle internal sensing data, and can generate vehicle situation information regarding the recognized vehicle situation.

[0213] For example, the vehicle situation information (740) may include at least one of the external situation information of the vehicle and the situation information of the vehicle's occupants.

[0214] As another example, the vehicle situation information (740) may include at least one of the vehicle's external situation information, the vehicle's internal situation information, and the vehicle's occupant situation information.

[0215] Meanwhile, the recognizer (720) can recognize external situation information of the vehicle based on external sensing data from the edge sensor group (SN), recognize internal situation information of the vehicle based on internal sensing data of the vehicle, or recognize situation information of the vehicle's occupant based on internal sensing data of the vehicle.

[0216] Meanwhile, the recognizer (720) can output external situation information of the vehicle based on internal vehicle sensing data or external vehicle sensing data, output internal situation information of the vehicle based on internal vehicle sensing data or external vehicle sensing data, or output situation information of the vehicle's occupant based on internal vehicle sensing data or external vehicle sensing data.

[0217] Meanwhile, the recognizer (720) can perform preprocessing and calibration of the vehicle interior sensing data or the vehicle interior sensing data, and can obtain vehicle situation information based on the preprocessed and calibrated sensing data.

[0218] For example, the internal recognition unit (725) within the recognition unit (720) can perform preprocessing and calibration (721) on the vehicle internal sensing data, and can obtain internal situation information of the vehicle based on the preprocessed and calibrated vehicle internal sensing data.

[0219] Specifically, the internal recognizer (725) can perform preprocessing and calibration (721) on the vehicle interior sensing data, perform primitive detection (722) or sensor fusion (723), and based on this, obtain information about the vehicle's interior situation.

[0220] For example, an external recognizer (730) within the recognizer (720) can perform preprocessing and calibration (721) on the external vehicle sensing data, and can obtain external situation information of the vehicle based on the preprocessed and calibrated external vehicle sensing data.

[0221] Specifically, the external recognizer (730) can perform preprocessing and calibration (731) on external vehicle sensing data, execute a perception engine (732), perform sensor fusion (733), or perform segmentation (734), and based on this, obtain external situation information of the vehicle.

[0222] Meanwhile, the passenger recognition device (730) within the recognition device (720) can perform passenger face identification (741), recognize distraction (742), recognize gaze, position, gesture (743), recognize emotion (744), recognize whether alcohol or medication is consumed (746), recognize drowsiness (747), or recognize information (748) regarding other actions based on the vehicle interior sensing data, and can obtain situational information of the vehicle's passenger based thereon.

[0223] Meanwhile, the situational information of the occupant may include at least one of the following: a face ID of the occupant inside the vehicle (200), distraction information, gaze information, position information, gesture information, emotion information, information on whether alcohol or medication has been taken, drowsiness information, or other information regarding actions.

[0224] Meanwhile, the recognizer (720) can transmit the generated vehicle situation information to the insightor (750).

[0225] Meanwhile, the insightor (750) can generate an inference result or a response result based on the situation information of the vehicle received from the recognizer (720).

[0226] Meanwhile, the insightor (750) can generate an inference result or a response result based on the vehicle situation information received from the recognizer (720), and can generate or control a service to be executed based on the inference result or the response result.

[0227] Accordingly, the insightor (750) may be named a response result generation unit or a service generation unit.

[0228] Meanwhile, the insightor (750) may include a multimodal context engine (755), a context controller (754), a safety assistance engine (757), a user characteristic engine (758), an AI orchestrator (760), and an AI model set (765).

[0229] Meanwhile, the multimodal context engine (755) can generate multimodal context data based on the vehicle situation information received from the recognizer (720).

[0230] For example, multimodal context data may include at least one of text data or image data describing a vehicle situation generated based on vehicle situation information.

[0231] Meanwhile, the context controller (754) may be a component included in the multimodal context engine (755) or provided separately from the multimodal context engine (755).

[0232] Meanwhile, the context controller (754) can execute the process of generating multimodal context data when it receives a passenger query from the AI ​​orchestrator (760).

[0233] For example, the passenger curie may be a voice recognition result corresponding to a voice command spoken by the passenger.

[0234] Meanwhile, the user characteristic engine (758) can generate user context data or passenger context data based on user information or passenger information.

[0235] Meanwhile, user information or passenger information may be referred to as user persona or passenger persona.

[0236] Meanwhile, user information or passenger information may include at least one of the user or passenger's nationality, age, gender, occupation, personality, or psychological type (Myers-Briggs Type Indicator, MBTI).

[0237] Meanwhile, the AI ​​orchestrator (760) can generate a prompt based on at least one of multimodal context data or passenger context data.

[0238] Meanwhile, the AI ​​orchestrator (760) can send the generated prompt to the AI ​​model set (765).

[0239] Meanwhile, the AI ​​model set (765) may include at least one AI model.

[0240] For example, the AI ​​model set (765) may include an on-device AI model (767).

[0241] Meanwhile, the on-device AI model (767) may include at least one AI model. For example, the on-device AI model (767) may include a Small Language Model (LLM).

[0242] Meanwhile, the AI ​​model set (765) may include an interface (766) for data exchange with the AI ​​model (769) in the server (400).

[0243] Meanwhile, the AI ​​model (769) within the server (400) may include at least one AI model. For example, the AI ​​model (769) within the server (400) may include a Large Language Model (LLM).

[0244] That is, the on-device AI model (767) may have a smaller capacity or size than the AI ​​model (769) in the server (400).

[0245] Meanwhile, the AI ​​model set (765) can output an inference result in response to a prompt received from the AI ​​orchestrator (760) and can transmit the inference result to the AI ​​orchestrator (760).

[0246] Meanwhile, the AI ​​orchestrator (760) can generate additional prompts based on the inference results and send the additional prompts to the AI ​​model set (765).

[0247] Meanwhile, the AI ​​model set (765) that receives the additional prompt can output an additional inference result in response to the additional prompt and can transmit the additional inference result to the AI ​​orchestrator (760).

[0248] The AI ​​orchestrator (760) can obtain an inference result or additional inference result received from the AI ​​model set (765) as a response result, and can output the obtained response result to the illustrator (780).

[0249] Meanwhile, the safety assist engine (757) can determine whether the current situation is a safe situation or a dangerous situation based on the vehicle situation information received from the recognizer (720).

[0250] Meanwhile, the safety assist engine (757) can transmit a driver assistance control command or a warning notification output command to the safety application (790) of the illustrator (780) if the current situation is determined to be a dangerous situation.

[0251] Meanwhile, the safety assistance engine (757) can set a mode for vehicle driving assistance control or autonomous driving based on inference results or response results received from the AI ​​orchestrator (760) or the AI ​​model set (765).

[0252] For example, the safety assistance engine (757) can distinguish the mode of vehicle driving assistance control or autonomous driving as an off mode, an on mode, or a safety mode based on the inference result or response result received from the AI ​​orchestrator (760) or the AI ​​model set (765).

[0253] Meanwhile, the safety assistance engine (757) calculates the risk level of vehicle driving based on the inference result or response result received from the AI ​​orchestrator (760) or the AI ​​model set (765), and based on the calculated risk level, can be set to one of the off mode, on mode, or safety mode of vehicle driving assistance control or autonomous driving.

[0254] Meanwhile, the safety assistance engine (757) can set a mode for vehicle driving assistance control or autonomous driving based on inference results or response results received from the AI ​​orchestrator (760) or the AI ​​model set (765), and can control authority adjustment, safety extension, or sensitivity adjustment for vehicle control based on the set mode.

[0255] Meanwhile, the safety assistance engine (757) can determine whether it is a danger precursor stage or a danger situation stage based on inference results or response results received from the AI ​​orchestrator (760) or the AI ​​model set (765), and control the operation mode of the vehicle driving assistance control or autonomous driving to vary according to the danger precursor stage or danger situation stage.

[0256] For example, the safety assistance engine (757) can control the vehicle driving assistance control or autonomous driving operation mode to remain off or on based on inference results or response results received from the AI ​​orchestrator (760) or the AI ​​model set (765), when the vehicle driving situation information corresponds to a danger warning stage.

[0257] As another example, the safety assistance engine (757) can control the vehicle driving assistance control or autonomous driving operation mode to operate in safety mode by disabling the off mode or on mode based on inference results or response results received from the AI ​​orchestrator (760) or the AI ​​model set (765), if the vehicle driving situation information corresponds to a dangerous situation stage. Accordingly, it is possible to quickly and accurately determine the situation inside and outside the vehicle and adaptively perform vehicle control.

[0258] Meanwhile, the safety assist engine (757) may be included within the safety area (756) as shown in the drawing, but is not limited thereto, and may also be provided separately within the insightor (750) outside the safety area (756).

[0259] Meanwhile, the illustrator (780) can execute a service, output service information, execute an application, or output application information based on the response result output from the insightor (750).

[0260] For example, the illustrator (780) can execute a navigation service, execute a vehicle driving assistance control service, execute an autonomous driving service, or execute a display-related service based on the response result output from the insightor (750).

[0261] As another example, the illustrator (780) can run a navigation application, run a vehicle driving assistance control application, run an autonomous driving application, or run a display-related application based on the response result output from the insightor (750).

[0262] Meanwhile, the illustrator (780) may include a multimodal output encoder (781), a visual interface (782), an audio interface (785), and a safety application (790).

[0263] Meanwhile, the multimodal output encoder (781) can encode the response result output from the insightor (750) and output the encoded response result data to the visual interface (782) or audio interface (785).

[0264] Meanwhile, the visual interface (782) or audio interface (785) may be referred to as an output interface.

[0265] Meanwhile, the visual interface (782) can output response result data output from the multimodal output encoder (781).

[0266] Accordingly, at least one of the plurality of displays (180a to 180c) of FIG. 4 can display an image based on response result data from the visual interface (782).

[0267] Meanwhile, the visual interface (782) can output augmented reality (AR) video based on response result data or mixed reality (MR) video data based on response result data.

[0268] Meanwhile, the audio interface (785) can output response result data in the form of audio.

[0269] Accordingly, the audio output unit (185) of FIG. 4 can output a sound corresponding to the response result data from the audio interface (785).

[0270] Meanwhile, the safety application (790) can perform Advanced Driver Assistance System (ADAS) control based on the driver assistance control command or the response result received from the insightor (750).

[0271] For example, the safety application (790) can output warning notification data according to a warning notification output command.

[0272] In response to this, at least one of the electronic control unit (770) of FIG. 4 or a plurality of displays (180a to 180c) can output warning notification data.

[0273] FIGS. 8a to 8c are drawings referenced in the description of FIG. 7.

[0274] First, FIG. 8a is a drawing referenced in the description of the insight of FIG. 7.

[0275] Referring to the drawing, the insightor (750) may include a multimodal context engine (755), an AI orchestrator (760), and a multimodal LLM (767).

[0276] Meanwhile, the multimodal context engine (755) may include a multimodal signal adapter (751), a multimodal indexer (812), a multimodal context buffer (752), a multimodal context retriever (814), a multimodal context descriptor (815), a multimodal event monitor (811), and a context controller (754).

[0277] Unlike Fig. 7, the multimodal context engine (755) may include a context controller (754).

[0278] Meanwhile, the multimodal signal adapter (751) can generate pre-processed multimodal data by filtering, cleaning, synchronizing, and reformulating data received from various sensors or vehicle situation information received from the recognizer (720).

[0279] Meanwhile, the multimodal signal adapter (751) can receive at least one of audio data received through a microphone, front image data captured through a front camera, ADAS information obtained from the front image data or vehicle sensor, location data, IVI (In-Vehicle Infotainment) system data, IVI display information, DMS (Driver Monitoring System) information or IMS (Interior Monitoring System) information based on image data captured through an interior camera (195i), and biometric data obtained from a biometric sensor.

[0280] Meanwhile, the multimodal indexer (812) can generate multimodal processing data by dividing the preprocessed multimodal data into chunks and can index the multimodal processing data.

[0281] Meanwhile, the multimodal indexer (812) can obtain an encoding vector or keyword representing the attribute (or meaning) of the multimodal processed data divided into chunks as an index.

[0282] Meanwhile, the multimodal context buffer (752) can store multimodal processing data and an index corresponding to the multimodal processing data.

[0283] Meanwhile, the multimodal context retriever (814) can search for multimodal processing data most related to the passenger query through an index from the multimodal context buffer (752), and can select the searched multimodal processing data as a context candidate for the creation of multimodal context data.

[0284] Meanwhile, the multimodal context descriptor (815) can reconfigure multimodal processing data selected as a context candidate into multimodal context data having a prompt form that the multimodal LLM (767) can interpret.

[0285] Meanwhile, the multimodal event monitor (811) can monitor whether an index matching the trigger condition is entered when a trigger condition for multimodal data is registered. If an index matching the trigger condition is entered, the multimodal event monitor (811) can generate a trigger event to generate a proactive service query.

[0286] Meanwhile, the context controller (754) can control the overall operation of the multimodal context engine (755).

[0287] Meanwhile, the context controller (754) can execute the process of generating multimodal context data when it receives a passenger query from the AI ​​orchestrator (760).

[0288] Meanwhile, the context controller (754) can generate a preemptive service query when it receives a trigger event from the multimodal event monitor (811).

[0289] Meanwhile, the AI ​​orchestrator (760) can generate a prompt by combining the passenger query and the multimodal context, and can send the generated prompt to the multimodal LLM (767).

[0290] Meanwhile, the passenger curry may be a query in the form of recognized text based on a voice command spoken by the passenger. The voice command may be received through a microphone, and the voice command may be converted into text through an Automatic Speech Recognition (ASR) process.

[0291] Meanwhile, passenger curry may be text converted through an Automatic Speech Recognition (ASR) process.

[0292] Meanwhile, multimodal LLM (767) may be an example of an AI model included in the set of AI models (765) of FIG. 6a.

[0293] Meanwhile, the multimodal LLM (767) can output an inference result from a prompt received from the AI ​​orchestrator (760) and can transmit the inference result to the AI ​​orchestrator (760).

[0294] Meanwhile, the inference result may include a result representing a response to the prompt.

[0295] For example, the inference result may include an API call result regarding whether an API call corresponding to a driving assistance function was successfully performed, and a feedback generation result regarding whether feedback corresponding to the API call was successfully generated.

[0296] Next, Fig. 8b is a drawing referenced in the description of the AI ​​orchestrator of Fig. 7.

[0297] Referring to the drawing, the AI ​​orchestrator (760) may include a task arbitrator (771), a prompt manager (763), a sub-agent set (761), a knowledge database (762), a workflow controller (772), and a tool set / adapter set (764).

[0298] Meanwhile, the task arbitrator (771) can determine one of the multiple sub-agents based on the voice recognition result and the multimodal context data output from the multimodal context engine (755).

[0299] Meanwhile, the prompt manager (763) can generate a prompt for the operation of the determined sub-agent. The prompt manager (763) can generate a prompt based on multimodal context data and passenger queries.

[0300] Meanwhile, the prompt manager (763) can generate a prompt based on information about the functions that the determined sub-agent can perform, multimodal context data, the results of previously performed functions, conversation history and passenger query.

[0301] Meanwhile, the sub-agent set (761) may include multiple sub-agents.

[0302] For example, the sub-agent set (761) may include a navigation agent for navigation services, a vehicle function agent for providing vehicle functions, a telephony agent for automated telephone answering services, and a Q&A agent for providing response services to queries.

[0303] Meanwhile, the sub-agent determined by the task arbitrator (771) can call the cloud AI model (769) or on-device AI model (767) assigned to it.

[0304] Meanwhile, the cloud AI model (769) or on-device AI model (767) may be a Large Language Model (LLM).

[0305] Meanwhile, a sub-agent within a sub-agent set (761) can call a cloud AI model (769) or an on-device AI model (767) assigned to it to obtain an inference result corresponding to a prompt from the model.

[0306] Meanwhile, a sub-agent in the sub-agent set (761) can call the knowledge database (762) to provide additional information based on the acquired inference result, obtain additional information from the knowledge database (762), specify the name of the function to be executed and the parameter value of the function, and determine the feedback phrase to be provided to the user.

[0307] Meanwhile, a sub-agent in the sub-agent set (761) can transmit to the workflow controller (772) a parsing result including additional information called from the knowledge database (762), the name of the function to be executed, the parameter value of the function, and a feedback phrase, which is generated by parsing the inference result received from the model.

[0308] Meanwhile, the knowledge database (762) can store additional information and information about functions. The information about functions may include the name of the function and the parameter values ​​of the function.

[0309] Meanwhile, the workflow controller (772) can generate control commands to perform the corresponding function based on the parsing results and can store the results of the conversation and function performed by the AI ​​model and sub-agent.

[0310] Meanwhile, the tool set / adapter set (764) can call an API corresponding to a control command generated by the workflow controller (772). The tool set / adapter set (764) can convert the control command into an execution command of an actual function or an API call command that the IVI system can understand and execute, and can execute the converted command.

[0311] Fig. 8c is an example of an internal block diagram of the server of Fig. 7.

[0312] Referring to the drawing, the server (400) may represent a device that trains an artificial neural network using a machine learning algorithm or uses a trained artificial neural network.

[0313] Here, the server (400) may be composed of multiple servers to perform distributed processing. Alternatively, the server (400) may be defined as a 5G network.

[0314] The server (400) may be included as part of the vehicle (200) and may perform at least some of the AI ​​processing together.

[0315] The server (400) may include a communication interface (410), memory (430), a learning processor (440), and a processor (470).

[0316] The communication interface (410) can transmit and receive data with the vehicle (200) or an external device.

[0317] The memory (430) may include a model storage unit (431). The model storage unit (431) may store a model (or artificial neural network, 531a) that is being learned or has been learned through the learning processor (440).

[0318] The learning processor (440) can train the artificial neural network (431a) using the training data.

[0319] For example, the learning processor (440) can execute a learning model. The learning model may include a cloud AI model (769) such as FIG. 7.

[0320] The learning model may be used while mounted on the server (400) of the artificial neural network, or may be used while mounted on an external device such as a vehicle (200).

[0321] The learning model may be implemented in hardware, software, or a combination of hardware and software. If part or all of the learning model is implemented in software, one or more instructions constituting the learning model may be stored in memory (430).

[0322] The processor (470) can infer a result value for new input data using a learning model and generate a response or control command based on the inferred result value.

[0323] Meanwhile, the processor (470) can control the learning processor (440) to execute the cloud AI model (769) based on requests from the signal processing device (170) in the vehicle (200), etc.

[0324] FIG. 9 is another example of a block diagram of a signal processing device according to an embodiment of the present disclosure.

[0325] Referring to the drawings, a signal processing device (170) according to an embodiment of the present disclosure includes a processor (175) that receives camera data outside the vehicle, camera data inside the vehicle, and sensor data inside the vehicle.

[0326] Meanwhile, the processor (175) according to an embodiment of the present disclosure performs object detection based on camera data outside the vehicle and monitors the state of a passenger, including a driver, based on camera data inside the vehicle.

[0327] Meanwhile, the processor (175) according to an embodiment of the present disclosure extracts multimodal context data corresponding to a driving context related to vehicle driving based on monitored passenger status information, detected object information, sensor data, and driver's driving pattern information stored in a database.

[0328] Meanwhile, the processor (175) according to an embodiment of the present disclosure generates a prompt for vehicle driving control based on extracted multimodal context data, obtains an inference result or response result based on an artificial intelligence model based on the generated prompt, and sets a mode for vehicle driving assistance control or autonomous driving based on the obtained inference result or response result.

[0329] Accordingly, it becomes possible to perform adaptive vehicle control by rapidly and accurately determining the situation inside and outside the vehicle. In particular, it enables adaptive vehicle control by rapidly and accurately determining the situation inside and outside the vehicle based on multimodal context data.

[0330] Meanwhile, the processor (175) according to the embodiment of the present disclosure can be controlled to operate in an off mode, on mode, or safety mode for vehicle driving assistance control or autonomous driving based on the acquired inference result or response result. Accordingly, it is possible to quickly and accurately determine the situation inside and outside the vehicle and adaptively perform vehicle control.

[0331] Meanwhile, the processor (175) according to the embodiment of the present disclosure calculates a risk level of vehicle driving based on the acquired inference result or response result, and can set the vehicle driving assistance control or autonomous driving to an off mode, on mode, or safety mode based on the calculated risk level. Accordingly, it is possible to quickly and accurately determine the situation inside and outside the vehicle and adaptively perform vehicle control.

[0332] Meanwhile, the processor (175) according to an embodiment of the present disclosure can set a mode for vehicle driving assistance control or autonomous driving based on the acquired inference result or response result, and control authority adjustment, safety extension, or sensitivity adjustment for vehicle control based on the set mode. Accordingly, it is possible to quickly and accurately determine the situation inside and outside the vehicle and adaptively perform vehicle control.

[0333] Meanwhile, the processor (175) according to the embodiment of the present disclosure can determine whether it is a danger precursor stage or a danger situation stage based on the acquired inference result or response result, and can control the operation mode of the vehicle driving assistance control or autonomous driving to be variable according to the danger precursor stage or danger situation stage. Accordingly, it is possible to quickly and accurately determine the situation inside and outside the vehicle and adaptively perform vehicle control.

[0334] Meanwhile, the processor (175) according to an embodiment of the present disclosure can obtain an inference result or a response result based on extracted multimodal context data, calculate a risk level of vehicle driving based on the obtained inference result or response result, and output a vehicle control signal based on the calculated risk level. Accordingly, it is possible to quickly and accurately determine the situation inside and outside the vehicle and adaptively perform vehicle control.

[0335] Meanwhile, the processor (175) in the signal processing device (170) according to an embodiment of the present disclosure may include a recognizer (720), an insightor (750), and an illustrator (780).

[0336] Meanwhile, the recognizer (720) performs object detection based on camera data outside the vehicle and monitors the status of passengers, including the driver, based on camera data inside the vehicle.

[0337] Meanwhile, the recognizer (720) can recognize the vehicle situation based on sensing data received from the edge sensor group (SN). In this regard, the recognizer (720) may be referred to as a situation recognition unit. The vehicle situation at this time may include the vehicle external situation and the vehicle internal situation.

[0338] Meanwhile, the edge sensor group (SN) can correspond to a plurality of sensor devices (SN) of FIG. 4.

[0339] Meanwhile, the edge sensor group (SN) may include a microphone (112) which is an example of an input device (110), a vehicle interior camera (195i), a vehicle front camera (195), and a position sensor (198) that receives GPS data, etc.

[0340] Among these, the microphone (112) and the vehicle interior camera (195i) may be included in the vehicle interior sensors (710) of FIG. 7.

[0341] Meanwhile, the vehicle front camera (195) or position sensor (198) may be included in the vehicle external sensors (715).

[0342] Meanwhile, the recognizer (720) can recognize the situation inside the vehicle based on the vehicle interior sensing data (725) or the vehicle interior situation information (740a), and can generate the vehicle's situation information based on the recognized vehicle interior situation information.

[0343] The vehicle interior sensing data (725) at this time may be body sensing data of an occupant, including a driver or a passenger. For example, the body sensing data may include heart rate sensing data, respiration sensing data, etc.

[0344] Meanwhile, the vehicle interior situation information (740a) may include motion information and voice information of a passenger, including a driver or co-passenger, from the vehicle interior camera (195i) and microphone (112).

[0345] Meanwhile, the vehicle situation information (740 in FIG. 7) may include external vehicle situation information in addition to internal vehicle situation information.

[0346] Meanwhile, the vehicle's internal situation information may include occupant situation information. For example, the vehicle's internal situation information may include driver situation information and passenger situation information.

[0347] Meanwhile, the recognizer (720) can recognize external situation information of the vehicle based on external sensing data from the edge sensor group (SN), recognize internal situation information of the vehicle based on internal sensing data of the vehicle, or recognize situation information of the vehicle's occupant based on internal sensing data of the vehicle.

[0348] Meanwhile, the recognizer (720) can output external situation information of the vehicle based on internal vehicle sensing data or external vehicle sensing data, output internal situation information of the vehicle based on internal vehicle sensing data or external vehicle sensing data, or output situation information of the vehicle's occupant based on internal vehicle sensing data or external vehicle sensing data.

[0349] Meanwhile, the recognizer (720) can perform vehicle sensing data or preprocessing and calibration of vehicle sensing data, and can obtain vehicle situation information based on the preprocessed and calibrated sensing data.

[0350] Meanwhile, vehicle sensing data may include vehicle interior sensing data and vehicle exterior sensing data.

[0351] Meanwhile, vehicle situation information may include internal vehicle situation information and external vehicle situation information.

[0352] For example, the recognizer (720) can perform preprocessing and calibration of the vehicle interior sensing data, and can obtain information on the vehicle's interior situation based on the preprocessed and calibrated vehicle interior sensing data.

[0353] For example, the recognizer (720) can perform preprocessing and calibration on the vehicle external sensing data, and can obtain external situation information of the vehicle based on the preprocessed and calibrated vehicle external sensing data.

[0354] Meanwhile, the recognition device (720) can perform facial identification of the occupant based on the vehicle interior sensing data, recognize distraction, recognize gaze, position, gesture, recognize emotion, recognize whether alcohol or medication has been taken, recognize drowsiness, or recognize information (748) about other actions, and obtain situational information of the vehicle occupant based thereon.

[0355] Meanwhile, the situational information of the occupant may include at least one of the following: a face ID of the occupant inside the vehicle (200), distraction information, gaze information, position information, gesture information, emotion information, information on whether alcohol or medication has been taken, drowsiness information, or other information regarding actions.

[0356] Meanwhile, the recognizer (720) can transmit the generated vehicle situation information to the insightor (750).

[0357] Meanwhile, the recognizer (720) receives camera data from outside the vehicle from the vehicle's external camera (195) and performs object detection based on the camera data from outside the vehicle.

[0358] For example, the recognizer (720) can receive vehicle front image data from the vehicle's external camera (195), perform segmentation on the vehicle front image data, and detect multiple objects (735).

[0359] Meanwhile, the recognizer (720) can receive location information from the location sensor (198 in FIG. 4) and transmit the location information to the insightor (750).

[0360] Meanwhile, the insightor (750) extracts multimodal context data corresponding to a driving context related to vehicle driving based on monitored passenger status information, detected object information, sensor data, and driver's driving pattern information stored in a database.

[0361] Meanwhile, the insightor (750) generates a prompt for vehicle driving control based on the extracted multimodal context data.

[0362] Meanwhile, the insightor (750) obtains an inference result or a response result based on the extracted multimodal context data, and sets a mode for vehicle driving assistance control or autonomous driving based on the obtained inference result or response result.

[0363] Meanwhile, the insightor (750) can be controlled to operate in an off mode, on mode, or safety mode of vehicle driving assistance control or autonomous driving based on the acquired inference result or response result.

[0364] For example, the insightor (750) can calculate a risk level of vehicle driving based on an inference result or response result received from an AI orchestrator (760 in FIG. 7) or an AI model set (765 in FIG. 7), and based on the calculated risk level, set the vehicle driving assistance control or autonomous driving to an off mode, on mode, or safety mode.

[0365] Meanwhile, the insightor (750) can calculate the risk level of vehicle driving based on the extracted multimodal context data.

[0366] Meanwhile, the insightor (750) can obtain an inference result or a response result based on the extracted multimodal context data, and calculate the risk level of vehicle driving based on the obtained inference result or response result.

[0367] Meanwhile, the insightor (750) can generate an inference result or a response result based on the situation information of the vehicle received from the recognizer (720).

[0368] Meanwhile, the insightor (750) can generate an inference result or a response result based on the vehicle situation information received from the recognizer (720), and can generate or control a service to be executed based on the inference result or the response result.

[0369] Accordingly, the insightor (750) may be named a response result generation unit or a service generation unit.

[0370] Meanwhile, the insightor (750) may include a multimodal context engine (755) and a safety assistance engine (757).

[0371] Meanwhile, the multimodal context engine (755) can generate multimodal context data based on the vehicle situation information received from the recognizer (720).

[0372] Specifically, the multimodal context engine (755) can generate multimodal context data corresponding to a driving context related to vehicle driving based on monitored passenger status information, detected object information, sensor data, and driver's driving pattern information stored in a database.

[0373] For example, multimodal context data may include at least one of text data, image data, or audio data describing a vehicle situation generated based on vehicle situation information.

[0374] Meanwhile, the multimodal context engine (755) can generate multimodal context data based on real-time context data (905) and feature data (907).

[0375] Meanwhile, real-time context data (905) may include monitored passenger status information, detected object information, and in-vehicle sensor data.

[0376] For example, real-time context data (905) may include monitored driver status information, detected object information, and in-vehicle sensor data.

[0377] As another example, real-time context data (905) may include monitored driver status information, monitored passenger status information, detected object information, and in-vehicle sensor data.

[0378] Meanwhile, the characteristic data (907) may include the characteristic data of the occupant.

[0379] For example, characteristic data (907) may include characteristic data of the driver.

[0380] As another example, characteristic data (907) may include characteristic data of the driver and characteristic data of the passenger.

[0381] Meanwhile, the safety assistance engine (757) may include an on-device AI model (767), a safety extension module (910), an authority adjustment module (915), and a sensitivity adjustment module (920).

[0382] Meanwhile, the safety assistance engine (757) can set a mode for vehicle driving assistance control or autonomous driving based on the extracted multimodal context data.

[0383] Meanwhile, the on-device AI model (767) can output an inference result or response result corresponding to the prompt.

[0384] Meanwhile, the safety assistance engine (757) can set a mode for vehicle driving assistance control or autonomous driving based on the inference result or response result received from the on-device AI model (767).

[0385] For example, the safety assistance engine (757) can distinguish the mode of vehicle driving assistance control or autonomous driving as an off mode, an on mode, or a safety mode based on the inference result or response result received from the on-device AI model (767).

[0386] Meanwhile, the safety assistance engine (757) calculates the risk level of vehicle driving based on the inference result or response result received from the on-device AI model (767), and based on the calculated risk level, can set the vehicle driving assistance control or autonomous driving to an off mode, on mode, or safety mode.

[0387] Meanwhile, the safety assistance engine (757) can set a mode for vehicle driving assistance control or autonomous driving based on the inference result or response result received from the on-device AI model (767), and control authority adjustment, safety extension, or sensitivity adjustment for vehicle control based on the set mode.

[0388] Meanwhile, the safety assistance engine (757) can determine whether it is a danger warning stage or a danger situation stage based on the inference result or response result received from the on-device AI model (767), and control the operation mode of the vehicle driving assistance control or autonomous driving to be variable according to the danger warning stage or danger situation stage.

[0389] For example, the safety assistance engine (757) can control the vehicle driving assistance control or autonomous driving operation mode to remain off or on based on the inference result or response result received from the on-device AI model (767), if the vehicle driving situation information corresponds to a danger warning stage.

[0390] As another example, the safety assistance engine (757) can control the vehicle driving assistance control or autonomous driving operation mode to operate in safety mode by disabling the off mode or on mode based on inference results or response results received from the on-device AI model (767), if the vehicle driving situation information corresponds to a dangerous situation stage. Accordingly, it is possible to quickly and accurately determine the situation inside and outside the vehicle and adaptively perform vehicle control.

[0391] Meanwhile, the safety assist engine (757) can calculate the risk level of vehicle driving based on the extracted multimodal context data.

[0392] Meanwhile, the safety assist engine (757) can obtain an inference result or a response result based on the extracted multimodal context data, and calculate the risk level of vehicle driving based on the obtained inference result or response result.

[0393] Specifically, the on-device AI model (767) can perform inference or response based on the extracted multimodal context data and output the inference result or response result.

[0394] And, the safety assistance engine (757) can calculate the risk level of vehicle driving based on the inference result or response result obtained from the on-device AI model (767).

[0395] Meanwhile, the safety assist engine (757) can classify the risk level of vehicle driving into multiple levels.

[0396] For example, the safety assist engine (757) can classify the risk level of vehicle driving into levels 1 through 4.

[0397] Meanwhile, the safety assist engine (757) can calculate the risk level of vehicle driving as the first level when the driver's driving ability information among the monitored passenger status information indicates that the driver can operate the vehicle.

[0398] Meanwhile, the safety assist engine (757) can calculate the risk level of vehicle driving to a second level higher than the first level when the driver's driving ability information indicates that the driver is capable of operating the vehicle or is distracted.

[0399] Meanwhile, the safety assist engine (757) can calculate the risk level of vehicle driving to a third level higher than the second level when the driver's driving ability information indicates that the driver can operate the vehicle or that the reaction is delayed.

[0400] Meanwhile, the safety assist engine (757) can calculate the risk level of vehicle driving to a fourth level higher than the third level when the driver's driving ability information indicates that the driver cannot operate the vehicle. Accordingly, it is possible to quickly and accurately determine the situation inside and outside the vehicle and adaptively perform vehicle control.

[0401] Meanwhile, the safety assist engine (757) can calculate the risk level of vehicle driving as the first level if the occupant characteristic information does not include habit behavior information or is not personalized habit behavior information.

[0402] Meanwhile, the safety assist engine (757) can calculate the risk level of vehicle driving as a second level higher than the first level when the occupant characteristic information includes habitual behavior information that interferes with driving.

[0403] Meanwhile, the safety assist engine (757) can calculate the risk level of vehicle driving as a third level higher than the second level if the occupant characteristic information includes repeated dangerous behaviors and medical history or accident history.

[0404] Meanwhile, the safety assist engine (757) can calculate the risk level of vehicle driving as a fourth level higher than the third level when the occupant characteristic information includes risk information based on medical history. Accordingly, it is possible to quickly and accurately determine the situation inside and outside the vehicle and adaptively perform vehicle control.

[0405] Meanwhile, the safety assist engine (757) can calculate the risk level of vehicle driving as a first level when the passenger's behavior information includes stable state information.

[0406] Meanwhile, the safety assist engine (757) can calculate the risk level of vehicle driving as a second level higher than the first level when the passenger's behavior information includes voice information at a level below the threshold.

[0407] Meanwhile, the safety assist engine (757) can calculate the risk level of vehicle driving as a third level higher than the second level when the passenger's behavior information includes voice information at a level higher than the repeated threshold.

[0408] Meanwhile, the safety assist engine (757) can calculate the risk level of vehicle driving as a fourth level higher than the third level if the passenger's behavior information includes actions that interfere with the driver's driving operation. Accordingly, it is possible to quickly and accurately determine the situation inside and outside the vehicle and adaptively perform vehicle control.

[0409] Meanwhile, the safety assist engine (757) can control the authority adjustment, safety extension, or sensitivity adjustment for vehicle control based on the risk level of vehicle driving.

[0410] Meanwhile, the authority adjustment module (915) can calculate authority adjustment data for vehicle control based on the risk level of vehicle driving.

[0411] Meanwhile, the authority adjustment module (915) can control the authority adjustment for vehicle control to remain unchanged when the risk level of vehicle driving is at the first level.

[0412] Meanwhile, the authority adjustment module (915) can switch the authority adjustment for vehicle control to a standby state when the risk level of vehicle driving is a second level higher than the first level.

[0413] Meanwhile, the authority adjustment module (915) can ignore or supplement the operation input within the authority adjustment for vehicle control when the risk level of vehicle driving is a third level higher than the second level.

[0414] Meanwhile, the authority adjustment module (915) can control steering and acceleration for authority adjustment for vehicle control when the risk level of vehicle driving is at the highest level, level 4.

[0415] Meanwhile, the safety extension module (910) can calculate safety extension data for safety extension based on the risk level of vehicle driving.

[0416] Meanwhile, the safety extension module (910) can control the safety extension to remain as is when the risk level of vehicle driving is at the first level.

[0417] Meanwhile, the safety extension module (910) can switch the safety extension for vehicle control to a standby state when the risk level of vehicle driving is at a second level higher than the first level.

[0418] Meanwhile, the safety extension module (910) can increase the threshold value within the safety extension for vehicle control when the risk level of vehicle driving is a third level higher than the second level.

[0419] Meanwhile, the safety extension module (910) can set the threshold value within the safety extension to the maximum value when the risk level of vehicle driving is the highest level, which is the fourth level.

[0420] Meanwhile, the sensitivity adjustment module (920) can calculate sensitivity adjustment data for sensitivity adjustment based on the risk level of vehicle driving.

[0421] Meanwhile, the sensitivity adjustment module (920) can control the sensitivity adjustment for vehicle control to remain unchanged when the risk level of vehicle driving is at the first level.

[0422] Meanwhile, the sensitivity adjustment module (920) can switch the sensitivity adjustment for vehicle control to a standby state or an initial state when the risk level of vehicle driving is a second level higher than the first level.

[0423] Meanwhile, the sensitivity adjustment module (920) can increase the sensitivity of the sensitivity adjustment above the second level when the risk level of vehicle driving is a third level higher than the second level.

[0424] Meanwhile, the sensitivity adjustment module (920) can set the sensitivity of the sensitivity adjustment to the maximum value when the risk level of vehicle driving is the highest level, which is the fourth level.

[0425] Meanwhile, the illustrator (780) can execute a service, output service information, execute an application, or output application information based on the inference result or response result output from the insightor (750).

[0426] For example, the illustrator (780) can execute an autonomous driving service or a driving assistance service based on the response result output from the insightor (750).

[0427] Meanwhile, the illustrator (780) outputs a vehicle control signal based on the risk level calculated by the insightor (750). Accordingly, the situation inside and outside the vehicle can be determined quickly and accurately to perform adaptive vehicle control. In particular, the situation inside and outside the vehicle can be determined quickly and accurately based on multimodal context data to perform adaptive vehicle control.

[0428] Meanwhile, the illustrator (780) can generate a vehicle control signal based on authority adjustment data, safety extension data, and sensitivity adjustment data from the insightor (750).

[0429] For example, the illustrator (780) can generate vehicle control signals for controlling Lane Centering Systems (LCS), Adaptive Cruise Control (ACC), Lane Keeping Assist (LKA), Autonomous Emergency Braking (AEB), or Emergency Stop System (ESS), based on authority adjustment data, safety extension data, and sensitivity adjustment data from the insightor (750). Accordingly, it is possible to quickly and accurately determine the situation inside and outside the vehicle and adaptively perform vehicle control.

[0430] Meanwhile, the illustrator (780) can output a vehicle control signal, a warning message, or a guide message based on the calculated risk level and occupant profile information. Accordingly, it is possible to quickly and accurately determine the situation inside and outside the vehicle and adaptively control the vehicle.

[0431] FIGS. 10 to 13d are drawings referenced in the description of FIG. 9.

[0432] First, FIG. 10 is an example of a block diagram of a signal processing system according to an embodiment of the present disclosure.

[0433] Referring to the drawings, the signal processing system according to an embodiment of the present disclosure may be referred to as an IVEX (In Vehicle Experience) system or a vehicle control device.

[0434] Meanwhile, the signal processing system (900) according to an embodiment of the present disclosure includes an edge sensor group (SN) and a signal processing device (170) in a vehicle.

[0435] Meanwhile, the signal processing system according to the embodiment of the present disclosure may further include a server (400).

[0436] Meanwhile, the signal processing device (170) according to an embodiment of the present disclosure includes a recognizer (720), an insightor (750), and an illustrator (780).

[0437] Meanwhile, the edge sensor group (SN) can correspond to a plurality of sensor devices (SN) of FIG. 4.

[0438] Meanwhile, the edge sensor group (SN) may include an internal vehicle sensor (710) and an external vehicle sensor (715).

[0439] The vehicle interior sensor (710) may include a microphone (112 in FIG. 4), a vehicle interior camera (195i), etc.

[0440] The vehicle external sensor (715) may include a vehicle front camera (195), a position sensor (198), etc.

[0441] Meanwhile, the recognizer (720) can recognize the vehicle situation based on sensing data received from the edge sensor group (SN). In this regard, the recognizer (720) may be referred to as a situation recognition unit.

[0442] The vehicle situation at this time may include external vehicle conditions and internal vehicle conditions.

[0443] Meanwhile, the vehicle interior recognition unit (725) within the recognition unit (720) can recognize the vehicle interior situation based on vehicle interior sensing data from the vehicle interior sensor (710).

[0444] For example, the vehicle interior recognition device (725) within the recognition device (720) can monitor the condition of the occupants, including the driver, based on camera data inside the vehicle.

[0445] Meanwhile, the vehicle external recognition unit (730) within the recognition unit (720) can recognize the external vehicle situation based on the external vehicle sensing data from the external vehicle sensor (715).

[0446] For example, the vehicle external recognizer (730) within the recognizer (720) can perform object detection based on camera data from outside the vehicle.

[0447] Meanwhile, the integrated scene understandr (935) within the recognizer (720) can recognize vehicle situation information including an integrated scene inside and outside the vehicle based on vehicle inside situation information from the vehicle inside recognizer (725) and vehicle outside situation information from the vehicle outside recognizer (730).

[0448] Meanwhile, vehicle situation information may include vehicle interior situation information, vehicle exterior situation information, and vehicle occupant situation information.

[0449] Meanwhile, the integrated scene comprehension unit (935) within the recognition unit (720) can recognize vehicle situation information that further includes warning signs of the vehicle occupant.

[0450] Meanwhile, the insightor (750) can generate an inference result or a response result based on the situation information of the vehicle received from the recognizer (720).

[0451] Meanwhile, the insightor (750) can generate an inference result or a response result based on the vehicle situation information received from the recognizer (720), and can generate or control a service to be executed based on the inference result or the response result.

[0452] Accordingly, the insightor (750) may be named a response result generation unit or a service generation unit.

[0453] Meanwhile, the insightor (750) may include a multimodal context engine (755), a risk level determination unit (940), and a personalization module (945).

[0454] Meanwhile, the multimodal context engine (755) can generate multimodal context data based on the vehicle situation information received from the recognizer (720).

[0455] For example, multimodal context data may include at least one of text data, image data, or audio data describing a vehicle situation generated based on vehicle situation information.

[0456] Meanwhile, the multimodal context engine (755) may include a database (752) that stores driver's driving pattern information.

[0457] At this time, the driver's driving pattern information may include risk response information or risk response pattern information.

[0458] Meanwhile, the multimodal context engine (755) may include a database (752) that stores driver's driving pattern information and passenger's pattern information.

[0459] Meanwhile, the multimodal context engine (755) can extract multimodal context data corresponding to a driving context related to vehicle driving based on monitored passenger status information, detected object information, sensor data, and driver's driving pattern information stored in a database.

[0460] Meanwhile, the multimodal context engine (755) can transmit individual driving pattern information collected to the server (400).

[0461] Meanwhile, the risk level determination unit (940) can calculate the risk level of vehicle driving based on the extracted multimodal context data.

[0462] Meanwhile, the risk level determination unit (940) can obtain an inference result or a response result based on the extracted multimodal context data, and calculate the risk level of vehicle driving based on the obtained inference result or response result.

[0463] For example, the risk level determination unit (940) can classify the risk level of vehicle driving into levels 1 through 4.

[0464] Meanwhile, the risk level determination unit (940) can calculate driving ability evaluation data and emergency level evaluation data based on the driver's context data and driver's characteristic data.

[0465] Meanwhile, the risk level determination unit (940) can calculate driving ability evaluation data and emergency level evaluation data based on the driver's context data, the driver's characteristic data, the passenger's context data, and the passenger's characteristic data.

[0466] Meanwhile, the risk level determination unit (940) can calculate driving ability evaluation data and emergency level evaluation data based on the driver's context data and driver's characteristic data, and calculate the risk level of vehicle driving based on the driving ability evaluation data and emergency level evaluation data.

[0467] Meanwhile, the risk level determination unit (940) can calculate driving ability evaluation data and emergency level evaluation data based on the driver's context data, driver's characteristic data, passenger's context data, and passenger's characteristic data, and calculate the risk level of vehicle driving based on the driving ability evaluation data and emergency level evaluation data.

[0468] Meanwhile, the risk level determination unit (940) can classify the risk level of vehicle driving into multiple levels based on vehicle internal context data, vehicle external context data, and occupant profile information.

[0469] Meanwhile, the risk level determination unit (940) can classify the risk level of vehicle driving into multiple levels based on vehicle internal context data, vehicle external context data, occupant characteristic information, occupant behavior information, and driver's driving ability information.

[0470] Meanwhile, the risk level determination unit (940) can transmit risk level information of vehicle driving to the personalization module (945).

[0471] Meanwhile, the personalization module (945) can receive individual profile information from the server (400).

[0472] Meanwhile, the learning model (767) within the personalization module (945) can perform learning based on the calculated risk level information and occupant profile information, and output the result data of the learning.

[0473] Meanwhile, the personalization module (945) can compute authority adjustment data for vehicle control, safety extension data for safety extension, and sensitivity adjustment data for sensitivity adjustment based on the result data of the learning.

[0474] Meanwhile, the insightor (750) can set a mode for vehicle driving assistance control or autonomous driving based on the risk level information of vehicle driving determined by the risk level determination unit (940).

[0475] For example, the insightor (750) can classify the vehicle driving assistance control or autonomous driving mode into one of the off mode, on mode, or safety mode based on the risk level of vehicle driving from the risk level determination unit (940).

[0476] Specifically, the insightor (750) can set the vehicle driving assistance control or autonomous driving mode to off mode when the risk level of vehicle driving from the risk level determination unit (940) is the first level.

[0477] Meanwhile, the insightor (750) can set the vehicle driving assistance control or autonomous driving mode to ON mode when the risk level of the vehicle driving from the risk level determination unit (940) is the second level or the third level.

[0478] Meanwhile, the insightor (750) can set the vehicle driving assistance control or autonomous driving mode to safety mode when the risk level of the vehicle driving from the risk level determination unit (940) is the fourth level.

[0479] Unlike the above, the insightor (750) can control the vehicle driving assistance control or autonomous driving mode to remain in off mode or on mode when the risk level of vehicle driving from the risk level determination unit (940) is at a first to third level.

[0480] Meanwhile, the insightor (750) can control the vehicle driving assistance control or autonomous driving mode to end the on or off mode when the risk level of the vehicle driving from the risk level determination unit (940) is the fourth level, and to enter the safety mode.

[0481] Meanwhile, the insightor (750) can control the authority adjustment, safety extension, or sensitivity adjustment for vehicle control based on the set mode.

[0482] Meanwhile, the insightor (750) determines whether it is a danger warning stage or a danger situation stage based on the danger level of the vehicle driving from the danger level judgment unit (940), and can control the operation mode of the vehicle driving assistance control or autonomous driving to be variable according to the danger warning stage or danger situation stage.

[0483] For example, the insightor (750) can determine a danger warning stage when the risk level of vehicle driving is at a second or third level, and can control the on mode of vehicle driving assistance control or autonomous driving mode in response to the danger warning stage.

[0484] Specifically, the insightor (750) can control the safety extension module (910), the authority adjustment module (915), and the sensitivity adjustment module (920) in response to the danger precursor stage.

[0485] As another example, the insightor (750) can determine the dangerous situation stage when the risk level of vehicle driving is at level 4, and can control the vehicle driving assistance control or autonomous driving mode to operate in safety mode in response to the dangerous situation stage.

[0486] Specifically, the insightor (750) can control the safety expansion module (910), the authority adjustment module (915), and the sensitivity adjustment module (920) in response to the risk situation stage.

[0487] Meanwhile, the illustrator (780) can execute a service, output service information, execute an application, or output application information based on the inference result or response result output from the insightor (750).

[0488] Meanwhile, the illustrator (780) can output a vehicle control signal based on the risk level calculated by the insightor (750).

[0489] Meanwhile, the illustrator (780) can output a vehicle control signal based on the authority adjustment data for vehicle control, the safety extension data for safety extension, and the sensitivity adjustment data for sensitivity adjustment calculated in the insightor (750).

[0490] Meanwhile, the illustrator (780) may include a multimodal output encoder (781) including a modality manager (795), a line suggestion application (931), a warning application (932), a system control (930), etc.

[0491] Meanwhile, the multimodal output encoder (781) can encode the inference result or response result output from the insightor (750).

[0492] Meanwhile, the first proposal application (931), the warning application (932), and the system control (930) can each be executed based on encoded inference result data or response result data from the multimodal output encoder (781).

[0493] Meanwhile, the multimodal output encoder (781) can encode data corresponding to the calculated risk level.

[0494] Meanwhile, the first proposal application (931), the warning application (932), and the system control (930) can each be executed based on data corresponding to the calculated risk level from the multimodal output encoder (781).

[0495] For example, the system control (930) can output a vehicle control signal based on data corresponding to the calculated risk level.

[0496] Meanwhile, the pre-proposal application (931) can generate and output a guide message related to a pre-proposal for vehicle control.

[0497] For example, the pre-proposal application (931) can output a guide message for vehicle speed reduction, a guide message for lane change, etc.

[0498] Meanwhile, the pre-proposal application (931) can output a guide message related to a pre-proposal for vehicle control as Augmented Reality (AR) or video data (783) or Mixed Reality (MR) video data (784).

[0499] Meanwhile, the pre-proposal application (931) can output a guide message related to a pre-proposal for vehicle control as audio, such as voice.

[0500] Meanwhile, the warning application (932) can output a warning message audio, visual, or physically.

[0501] Meanwhile, the multimodal output encoder (781) can encode the result data of the learning from the personalization module (945).

[0502] Meanwhile, the first proposal application (931), the warning application (932), and the system control (930) can each be executed based on the result data of learning from the multimodal output encoder (781).

[0503] For example, the pre-proposal application (931) can output a guide message related to a pre-proposal for vehicle control based on the result data of the learning.

[0504] Meanwhile, the warning application (932) can output a warning message based on the result data of the learning.

[0505] Meanwhile, the system control (930) can output a vehicle control signal based on the result data of the learning.

[0506] Meanwhile, the multimodal output encoder (781) can encode authority adjustment data for vehicle control, safety extension data for safety extension, and sensitivity adjustment data for sensitivity adjustment.

[0507] Meanwhile, the first proposal application (931), the warning application (932), and the system control (930) can be executed based on the authority adjustment data for vehicle control, the safety extension data for safety extension, and the sensitivity adjustment data for sensitivity adjustment from the multimodal output encoder (781), respectively.

[0508] Meanwhile, the pre-proposal application (931) can output a guide message based on authority adjustment data for vehicle control, safety extension data for safety extension, and sensitivity adjustment data for sensitivity adjustment.

[0509] Meanwhile, the warning application (932) can output a warning message based on authority adjustment data for vehicle control, safety extension data for safety extension, and sensitivity adjustment data for sensitivity adjustment.

[0510] Meanwhile, the system control (930) can output a vehicle control signal based on authority adjustment data for vehicle control, safety extension data for safety extension, and sensitivity adjustment data for sensitivity adjustment.

[0511] FIG. 11a is a drawing referenced in the description of the operation of a processor according to one embodiment of the present disclosure.

[0512] Referring to the drawings, a processor (175) in a signal processing device (170) according to one embodiment of the present disclosure may include a risk level determination unit (940) and a level corresponding control module (950).

[0513] The processor (175) receives vehicle internal data from vehicle internal sensors (710), receives vehicle external data from vehicle external sensors (715), and can receive data around the vehicle through the data receiving unit (718).

[0514] Meanwhile, the processor (175) performs object detection based on camera data outside the vehicle and monitors the status of passengers, including the driver, based on camera data inside the vehicle.

[0515] Meanwhile, the processor (175) extracts multimodal context data corresponding to a driving context related to vehicle driving based on monitored passenger status information, detected object information, sensor data, and driver's driving pattern information stored in a database.

[0516] Meanwhile, the risk level determination unit (940) within the processor (175) calculates the risk level of vehicle driving based on the extracted multimodal context data.

[0517] Meanwhile, the level-corresponding control module (950) within the processor (175) outputs a vehicle control signal based on the calculated risk level. Accordingly, the situation inside and outside the vehicle can be determined quickly and accurately, and adaptive vehicle control can be performed.

[0518] Meanwhile, the risk level determination unit (940) within the processor (175) can obtain an inference result or a response result based on the extracted multimodal context data, and calculate the risk level of vehicle driving based on the obtained inference result or response result.

[0519] That is, the processor (175) can generate a prompt for calculating the risk level of vehicle driving based on multimodal context data, calculate the risk level of vehicle driving based on the inference result or response result obtained based on the prompt, and output a vehicle control signal based on the calculated risk level.

[0520] Meanwhile, a signal processing device (170) according to one embodiment of the present disclosure may further include a neural processor (179) that executes inference or a response based on a prompt.

[0521] Meanwhile, the processor (175) can generate a prompt for calculating the risk level of vehicle driving based on multimodal context data, calculate the risk level of vehicle driving based on the inference result or response result executed by the neural processor (179) based on the prompt, and output a vehicle control signal based on the calculated risk level. Accordingly, the situation inside and outside the vehicle can be determined quickly and accurately to perform adaptive vehicle control.

[0522] Meanwhile, the risk level determination unit (940) within the signal processing unit (170) is included in the processor (175), as in FIG. 10 or FIG. 11a, and can classify the risk level of vehicle driving into a plurality of levels based on vehicle internal context data, vehicle external context data, and occupant profile information.

[0523] Meanwhile, the risk level determination unit (940) within the signal processing device (170) may be a concept including a processor (175) and a neural processor (179), unlike FIG. 10 or FIG. 11a.

[0524] Meanwhile, the risk level determination unit (940) within the signal processing unit (170) can operate as a hybrid type in the processor (175) and the neural processor (179), respectively.

[0525] For example, the processor (175) can calculate the risk level of vehicle driving for the event based on vehicle internal context data, vehicle external context data, and occupant profile information when an event occurs that requires urgent and immediate judgment, i.e., when the judgment completion period must be within the first set period.

[0526] In particular, the processor (175) can execute rule-based logic when urgent and immediate judgment is required to calculate the risk level of vehicle driving for the event.

[0527] Meanwhile, the processor (175) can control the neural processor (179) to calculate the risk level of vehicle driving when urgent and immediate judgment is not required.

[0528] For example, the processor (175) can control the neural processor (179) to calculate the risk level of vehicle driving when the situation is complex and various contexts must be considered together, that is, when an event occurs where the judgment completion period may exceed the first set period.

[0529] Specifically, the processor (175) can transmit vehicle internal context data, vehicle external context data, and passenger profile information to the neural processor (179) when an event occurs in which the situation is complex and various contexts must be considered together.

[0530] Meanwhile, the neural processor (179) can calculate the risk level of vehicle driving for the event based on the received vehicle internal context data, vehicle external context data, and occupant profile information.

[0531] For example, the neural processor (179) can calculate the risk level of vehicle driving for the event using a safety large-scale language model (LLM), etc.

[0532] Based on this hybrid method of risk level calculation, immediate or accurate risk level calculation becomes possible.

[0533] Meanwhile, the processor (175) can run a hypervisor (505 in FIG. 5) and run a driving control virtualization machine (520 in FIG. 5) on the hypervisor (505).

[0534] Meanwhile, the driving control virtualization machine (520) receives camera data outside the vehicle, camera data inside the vehicle, and sensor data inside the vehicle through the shared memory (508) within the hypervisor (505), and can execute a driving control service or a driving control application based on the camera data outside the vehicle, camera data inside the vehicle, and sensor data inside the vehicle. Accordingly, it is possible to quickly and accurately determine the situation inside and outside the vehicle and adaptively perform vehicle control.

[0535] Meanwhile, the risk level determination unit (940) within the processor (175) can classify the risk level of vehicle driving into multiple levels based on vehicle internal context data, vehicle external context data, and occupant profile information.

[0536] Meanwhile, the processor (175) can classify the risk level of vehicle driving into multiple levels based on vehicle internal context data, vehicle external context data, occupant characteristic information, occupant behavior information, and driver's driving ability information.

[0537] Meanwhile, the risk level determination unit (940) within the processor (175) can classify the risk level of vehicle driving into multiple levels.

[0538] Specifically, the risk level determination unit (940) within the processor (175) can classify the risk level of vehicle driving into levels 1 through 4.

[0539] For example, the risk level determination unit (940) within the processor (175) can calculate the risk level of vehicle driving as a first level if the driver's driving ability information among the monitored passenger status information indicates that the driver can operate the vehicle.

[0540] As another example, the risk level determination unit (940) within the processor (175) can calculate the risk level of vehicle driving as a second level higher than the first level when the driver's driving ability information indicates that the driver is capable of operating the vehicle or is in a distracted state.

[0541] As another example, the risk level determination unit (940) within the processor (175) can calculate the risk level of vehicle driving as a third level higher than the second level when the driver's driving ability information indicates that the driver is capable of driving or is in a delayed response state.

[0542] As another example, the risk level determination unit (940) within the processor (175) can calculate the risk level of vehicle driving as a fourth level higher than the third level when the driver's driving ability information indicates that the driver cannot operate the vehicle. Accordingly, the situation inside and outside the vehicle can be determined quickly and accurately to enable adaptive vehicle control.

[0543] Meanwhile, the risk level determination unit (940) within the processor (175) can calculate the risk level of vehicle driving as a first level if the occupant characteristic information does not include habit behavior information or is not personalized habit behavior information.

[0544] Meanwhile, the risk level determination unit (940) within the processor (175) can calculate the risk level of vehicle driving as a second level higher than the first level when the occupant characteristic information includes habitual behavior information that interferes with driving.

[0545] For example, the risk level determination unit (940) within the processor (175) can calculate the risk level of vehicle driving as a second level higher than the first level if the occupant characteristic information includes habitual behavior information that interferes with driving.

[0546] Meanwhile, the risk level determination unit (940) within the processor (175) can calculate the risk level of vehicle driving as a third level higher than the second level if the occupant characteristic information includes repeated dangerous behaviors and medical history or accident history.

[0547] For example, the risk level determination unit (940) within the processor (175) can calculate the risk level of vehicle driving as a third level higher than the second level if the occupant characteristic information includes repeated dangerous behaviors and medical history or accident history that interfere with driving.

[0548] Meanwhile, the risk level determination unit (940) within the processor (175) can calculate the risk level of vehicle driving as a fourth level higher than the third level when the occupant characteristic information includes risk information based on medical history. Accordingly, it is possible to quickly and accurately determine the situation inside and outside the vehicle and adaptively perform vehicle control.

[0549] Meanwhile, the risk level determination unit (940) within the processor (175) can calculate the risk level of vehicle driving as a first level when the passenger's behavior information includes stable state information.

[0550] Meanwhile, the risk level determination unit (940) within the processor (175) can calculate the risk level of vehicle driving as a second level higher than the first level when the passenger's behavior information includes voice information at a level below the threshold.

[0551] For example, the risk level determination unit (940) within the processor (175) can calculate the risk level of vehicle driving as a second level higher than the first level when the passenger's behavior information includes voice stimuli that distract concentration on driving, such as voice information at a level below a reference value.

[0552] Meanwhile, the risk level determination unit (940) within the processor (175) can calculate the risk level of vehicle driving as a third level higher than the second level when the passenger's behavior information includes voice information at a level higher than a repeated threshold.

[0553] For example, the risk level determination unit (940) within the processor (175) can calculate the risk level of vehicle driving as a third level higher than the second level if the passenger's behavior information includes voice stimuli that interfere with driving, such as voice information at a level higher than a threshold.

[0554] Meanwhile, the risk level determination unit (940) within the processor (175) can calculate the risk level of vehicle driving as a fourth level higher than the third level if the passenger's behavior information includes actions that interfere with the driver's driving operation. Accordingly, the situation inside and outside the vehicle can be determined quickly and accurately to enable adaptive vehicle control.

[0555] Meanwhile, the risk level determination unit (940) within the processor (175) can calculate driving ability evaluation data and emergency level evaluation data based on the driver's context data and driver's characteristic data, and calculate the risk level of vehicle driving based on the driving ability evaluation data and emergency level evaluation data.

[0556] Meanwhile, the risk level determination unit (940) within the processor (175) can calculate driving ability evaluation data and emergency level evaluation data based on the driver's context data, the driver's characteristic data, the passenger's context data, and the passenger's characteristic data.

[0557] Meanwhile, the level-corresponding control module (950) within the processor (175) can control authority adjustment, safety extension, or sensitivity adjustment for vehicle control based on the risk level of vehicle driving.

[0558] Meanwhile, the level-corresponding control module (950) within the processor (175) can calculate authority adjustment data for vehicle control, safety extension data for safety extension, and sensitivity adjustment data for sensitivity adjustment based on the risk level of vehicle driving.

[0559] Meanwhile, the level-corresponding control module (950) within the processor (175) can generate a vehicle control signal based on authority adjustment data, safety extension data, and sensitivity adjustment data. Accordingly, it is possible to quickly and accurately determine the situation inside and outside the vehicle and adaptively perform vehicle control.

[0560] Meanwhile, the level-corresponding control module (950) within the processor (175) can control the authority adjustment, safety extension, and sensitivity adjustment for vehicle control to remain unchanged when the risk level of vehicle driving is at the first level.

[0561] Meanwhile, the level-corresponding control module (950) within the processor (175) can switch the authority adjustment and safety expansion for vehicle control to a standby state and increase the sensitivity of the sensitivity adjustment above the first level when the risk level of vehicle driving is at a second level higher than the first level. Accordingly, it is possible to quickly and accurately determine the situation inside and outside the vehicle and adaptively perform vehicle control.

[0562] Meanwhile, the level-corresponding control module (950) within the processor (175) can ignore or supplement the operation input within the authority adjustment for vehicle control, increase the threshold value within the safety extension, or raise the sensitivity of the sensitivity adjustment above the second level when the risk level of vehicle driving is at a third level higher than the second level. Accordingly, it is possible to quickly and accurately determine the situation inside and outside the vehicle and adaptively perform vehicle control.

[0563] Meanwhile, the level-corresponding control module (950) within the processor (175) can control steering and acceleration for authority adjustment for vehicle control when the risk level of vehicle driving is at the highest level (Level 4), set the threshold value within the safety extension to the maximum value, and set the sensitivity of the sensitivity adjustment to the maximum value. Accordingly, it is possible to quickly and accurately determine the situation inside and outside the vehicle and adaptively perform vehicle control.

[0564] Meanwhile, the processor (175) can perform learning based on the calculated risk level and occupant profile information, and output a vehicle control signal, a warning message, or a guide message based on the result data of the learning. Accordingly, it is possible to quickly and accurately determine the situation inside and outside the vehicle and adaptively perform vehicle control.

[0565] Figure 11b is a drawing referenced in the description of the integrated scene analyzer of Figure 10.

[0566] Referring to the drawing, the integrated scene understandr (935) may include a vehicle interior context understandr (960) and a vehicle exterior context understandr (970).

[0567] The vehicle internal context understandr (960) may include a driver state information extractor (962) and a driver behavior information extractor (964).

[0568] For example, the driver state information extractor (962) can extract driver state information such as body-skeleton information, hands-on information, distraction information, behavior information, and health information through camera data from an internal camera (195i) and sensing data from a bio-sensor device.

[0569] Meanwhile, the driver behavior information extractor (964) can extract driver behavior information such as spilling, child crying, driver disturbance, and heart attack based on various information or data from the driver state information extractor (962).

[0570] The vehicle external context understandr (970) may include a setting external information extractor (972) and a real-time external information extractor (974).

[0571] Meanwhile, the external information extractor (972) may be named the Operational Design Domain (ODD).

[0572] The external information extractor (972) can extract surrounding environment information including a vehicle or a vulnerable road user (VRU), extract road context data, or extract highway context data based on communication data from the data receiver (718) and external vehicle sensing data from the external vehicle sensors (715).

[0573] The real-time external information extractor (974) can extract real-time external information such as traffic jam information, crowded vulnerable road user (Crowded VRU) information, and uncongested highway information based on information or data from the set external information extractor (972).

[0574] Meanwhile, the risk level determination unit (940) may include a vehicle interior risk determination unit (952) and a driving risk determination unit (954).

[0575] The vehicle interior risk determination unit (952) can calculate the risk level inside the vehicle based on vehicle interior context data from the vehicle interior context understandr (960). Meanwhile, the risk level inside the vehicle can be transmitted to the safety assist engine (757).

[0576] Meanwhile, the driving risk determination unit (954) can calculate the risk level outside the vehicle based on the vehicle external context data from the vehicle external context understandr (970). Meanwhile, the risk level outside the vehicle can be transmitted to the safety assist engine (757).

[0577] Meanwhile, the safety assist engine (757) may include the level corresponding control module (950) of FIG. 11a.

[0578] Meanwhile, the authority adjustment module (915) within the safety assistance engine (757) can adjust driving authority based on the risk level of driving the vehicle, including the risk level inside the vehicle and the risk level of driving the vehicle.

[0579] For example, the authority adjustment module (915) within the safety assist engine (757) can partially restrict or supplement the vehicle operation authority when the driver's ability to operate in a dangerous situation is below a standard level.

[0580] As another example, the authority adjustment module (915) within the safety assist engine (757) can control the driver's ability to operate in a dangerous situation to return to its original state by reducing the intervention of the driving authority by the system when the driver's ability to operate returns to a level exceeding the threshold. Accordingly, the situation inside and outside the vehicle can be determined quickly and accurately to perform adaptive vehicle control.

[0581] Meanwhile, the safety extension module (910) within the safety assistance engine (757) can perform safety extension control based on the risk level of vehicle driving, including the risk level inside the vehicle and the risk level of vehicle driving.

[0582] For example, the safety extension module (910) within the safety assist engine (757) can activate the system or adjust functional limitations regarding the allowable operating range in response to the risk level. Accordingly, it is possible to quickly and accurately determine the situation inside and outside the vehicle and perform adaptive vehicle control.

[0583] Meanwhile, the sensitivity adjustment module (920) within the safety assist engine (757) can perform sensitivity adjustment based on the risk level of the vehicle, including the risk level inside the vehicle and the risk level of the vehicle driving.

[0584] For example, the sensitivity adjustment module (920) within the safety assist engine (757) can control the sensitivity to be increased when the risk level exceeds a set value, and to be returned to its original state when the risk level returns to below the set value. Accordingly, the situation inside and outside the vehicle can be determined quickly and accurately to perform adaptive vehicle control.

[0585] FIG. 11c is a drawing referenced in the description of the operation of the personalization module of FIG. 10.

[0586] Referring to the drawing, the personalization module (945) can receive database data for initial profile generation from the user characteristic engine (758).

[0587] Meanwhile, the personalization module (945) can generate initial profile information based on the received database data.

[0588] At this time, the database data may include at least one of the user or passenger's nationality, age, gender, occupation, personality, or psychological type (Myers-Briggs Type Indicator, MBTI).

[0589] For example, the personalization module (945) can classify personal tendencies by category based on at least one of the nationality, age, gender, occupation, personality, or psychological type (Myers-Briggs Type Indicator, MBTI) of the user or passenger, and generate initial profile information based on the classified personal tendencies information.

[0590] Meanwhile, the personalization module (945) can perform mapping suitable for the vehicle domain based on initial profile information.

[0591] Meanwhile, the user characteristic engine (758) may be provided within the insightor (750) as in FIG. 7. Alternatively, the user characteristic engine (758) may be provided within the server (400) as in FIG. 7.

[0592] FIG. 11d is a diagram referenced in the description of the operation of a mode for vehicle driving assistance control or autonomous driving.

[0593] Referring to the drawings, a signal processing system (1600) according to an embodiment of the present disclosure includes an edge sensor group (SNa), a database (1610), and a signal processing device (170), as an IVEX (In Vehicle Experience) system or vehicle control device.

[0594] The edge sensor group (SNa) may be equipped with a plurality of sensors for vehicle driving assistance control or autonomous driving.

[0595] Meanwhile, the edge sensor group (SNa) may include an internal vehicle sensor (710) and an external vehicle sensor (715).

[0596] For example, the vehicle interior sensor (710) may include a microphone (112 in FIG. 4), a vehicle interior camera (195i), a biosensor (not shown) attached to the steering wheel (150), etc.

[0597] The vehicle external sensor (715) may include a vehicle front camera (195), lidar (196), radar (197), or position sensor (198), etc.

[0598] Meanwhile, the database (1610) can provide vehicle driving record data, driver medical history record data, disease symptom data, driver accident history data, driving habit data, or driving environment data stored internally in the signal processing device (170).

[0599] Meanwhile, the signal processing device (170) may include a processor (175).

[0600] The recognizer (725) in the processor (175) receives vehicle interior sensing data from the vehicle interior sensor (710) and can detect the occupant's face identification (Face ID, 741), gaze, head position, body skeleton, emotion, voice, heart beat, or object.

[0601] The recognizer (725) in the processor (175) can receive external vehicle sensing data from the external vehicle sensor (715) and detect a lane, road border, vehicle, traffic sign, or traffic light.

[0602] Meanwhile, the recognizer (725) in the processor (175) can calculate or extract driver behavior information, emergency situation information, or risk level of vehicle driving based on vehicle internal sensing data from the vehicle internal sensor (710), vehicle external sensing data from the vehicle external sensor (715), or vehicle internal driving record data, driver medical history record data, disease symptom data, driver accident history data, driving habit data, or driving environment data from the database (1610).

[0603] Meanwhile, the driving system control unit (1620) within the processor (175) can be controlled to operate in an off mode, on mode, or safety mode for vehicle driving assistance control or autonomous driving.

[0604] For example, the driving system control unit (1620) within the processor (175) can be controlled to operate in any one of the off mode, on mode, or safety mode of vehicle driving assistance control or autonomous driving based on an input signal.

[0605] Meanwhile, the action recognition unit (1730) within the processor (175) can perform driver action recognition based on an accelerator pedal input signal, a brake operation input signal, a turn signal input signal, or a steering wheel input signal.

[0606] Meanwhile, the processor (175) can generate a normal driving state as context data for each individual based on in-vehicle driving record data, driving habit data, or driving environment data, and detect signs of abnormal behavior based on this.

[0607] For example, the behavior recognition unit (1730) within the processor (175) can detect signs of abnormal behavior of the driver based on driver medical history data, disease symptom data, or driver accident history data, and accelerator pedal input signal, brake operation input signal, turn signal input signal, or steering wheel input signal, etc.

[0608] Meanwhile, the processor (175) can vary the maximum speed, maximum lateral acceleration, limit distance, minimum lane width, etc. when detecting signs of abnormal behavior.

[0609] For example, the system range setting unit (1635) within the processor (175) can vary the maximum speed, maximum lateral acceleration, limit distance, minimum lane width, etc. when abnormal behavior signs are detected.

[0610] Meanwhile, the driving operation unit (1640) within the processor (175) can set the mode for vehicle driving assistance control or autonomous driving to safety mode when abnormal behavior signs are detected, and output a vehicle control signal according to the safety mode.

[0611] For example, the driving operation unit (1640) within the processor (175) may, when operating in safety mode, limit override, temporarily relax the allowable range of system operation during vehicle driving assistance control or autonomous driving, or strengthen the system control sensitivity during vehicle driving assistance control or autonomous driving.

[0612] Meanwhile, the driving operation unit (1640) within the processor (175), when operating in safety mode, controls the lane centering system (LCS) and adaptive cruise control (ACC) to increase the maximum allowable lateral acceleration or increase the operating speed limit, and controls the emergency braking system (ESS) to operate immediately. Accordingly, it is possible to quickly and accurately determine the situation inside and outside the vehicle and adaptively control the vehicle.

[0613] Meanwhile, the driving operation unit (1640) within the processor (175), when operating in safety mode, controls the lane centering system (LCS) and lane keeping assist (LKA) to increase the steering override torque or to maintain the lane even in the event of a turn signal malfunction, controls the adaptive cruise control (ACC) to increase the accelerator pedal override torque or to respond to a dangerous object ahead even in the event of a brake pedal malfunction or to prevent an OTF (Overtaking Function) even in the event of a turn signal malfunction, and controls the emergency braking system (ESS) so that the emergency braking system operation is performed even if the brake pedal, accelerator pedal, or steering override torque increases. Accordingly, it is possible to quickly and accurately determine the situation inside and outside the vehicle and adaptively perform vehicle control.

[0614] Meanwhile, the driving operation unit (1640) within the processor (175), when operating in safety mode, can control the Lane Centering System (LCS) to increase the target distance level for object tracking or decrease the lane width limit, control the Lane Keeping Assist (LKA) to increase the distance between the vehicle and the lane, control the Adaptive Cruise Control (ACC) to change the Time to Collision (TTC) or change the ACC Early Warning, or decrease the target speed according to curves or road types, and control the Automatic Emergency Braking (AEB) to change the Time to Collision (TTC) or change the ACC Early Warning. Accordingly, it is possible to quickly and accurately determine the situation inside and outside the vehicle and adaptively perform vehicle control.

[0615] Meanwhile, the driving operation unit (1640) within the processor (175), when operating in safety mode, can perform steering control or acceleration control, completely remove functional restrictions on vehicle driving assistance control or autonomous driving, set a threshold value to the maximum, maintain heightened sensitivity, or suppress warnings to prevent confusion. Accordingly, it is possible to quickly and accurately determine the situation inside and outside the vehicle and adaptively perform vehicle control.

[0616] Meanwhile, the driving operation unit (1640) within the processor (175) can control the lane centering system (LCS) and the adaptive cruise control (ACC) when operating in the vehicle driving assistance control or autonomous driving on mode, thereby gradually increasing the allowable maximum lateral acceleration or gradually increasing the operating limit speed, and can keep the emergency braking system (ESS) in a ready state.

[0617] Meanwhile, the driving operation unit (1640) within the processor (175) can control the lane centering system (LCS) and lane keeping assist (LKA) to gradually increase the steering torque when operating in the vehicle driving assistance control or autonomous driving on mode, or control the adaptive cruise control (ACC) to gradually increase the steering torque, and the emergency braking system (ESS) can be kept in a ready state.

[0618] Meanwhile, the driving operation unit (1640) within the processor (175), when operating in the vehicle driving assistance control or autonomous driving on mode, can control the lane centering system (LCS) and lane keeping assistance (LKA) to gradually lower the lane width limit, or control the adaptive cruise control (ACC) to gradually increase the distance to the vehicle ahead, and can keep the emergency braking system (ESS) in a ready state.

[0619] Meanwhile, the driving operation unit (1640) within the processor (175) can, when operating in the ON mode of vehicle driving assistance control or autonomous driving, ignore or start correcting some errors such as blinking mistakes, partially relieve functional limitations of vehicle driving assistance control or autonomous driving, gradually increase the threshold, increase sensitivity further, perform multiple warnings, perform adaptive warnings based on personalization, actively perform warnings in case of behavioral risk, or relieve warnings in case of health risk.

[0620] FIG. 12 is a diagram illustrating a vehicle driving assistance function corresponding to a driving state.

[0621] Referring to the drawing, the operating state can be classified into normal mode, first emergency mode, second emergency mode, and emergency release mode.

[0622] Meanwhile, Normal Mode or Emergency Release Mode can correspond to Level 1 of the risk levels of vehicle driving.

[0623] Meanwhile, the first emergency mode can respond to the second or third level of the risk level of vehicle driving.

[0624] Meanwhile, the second emergency mode can respond to Level 4, which is the highest level among the risk levels of vehicle driving.

[0625] Meanwhile, in the case of the first emergency mode, the safety extension module (910) within the processor (175) controls the lane centering system (LCS) and the adaptive cruise control (ACC) to gradually increase the maximum allowable lateral acceleration or gradually increase the operating limit speed, and the emergency braking system (ESS) can be kept in a ready state.

[0626] Meanwhile, in the case of the first emergency mode, the authority adjustment module (915) within the processor (175) can control the lane centering system (LCS) and lane keeping assist (LKA) to gradually increase the steering torque, or control the adaptive cruise control (ACC) to gradually increase the steering torque, and keep the emergency braking system (ESS) in a ready state.

[0627] Meanwhile, in the case of the first emergency mode, the sensitivity adjustment module (920) within the processor (175) can control the lane centering system (LCS) and lane keeping assist (LKA) to gradually lower the lane width limit, or control the adaptive cruise control (ACC) to gradually increase the distance to the vehicle ahead, and the emergency braking system (ESS) can be kept in a ready state.

[0628] Meanwhile, in the case of the second emergency mode, the safety extension module (910) within the processor (175) can control the lane centering system (LCS) and the adaptive cruise control (ACC) to increase the maximum allowable lateral acceleration or increase the operating speed limit, and can control the emergency braking system (ESS) to operate immediately. Accordingly, it is possible to quickly and accurately determine the situation inside and outside the vehicle and adaptively control the vehicle.

[0629] Meanwhile, in the case of the second emergency mode, the authority adjustment module (915) within the processor (175) controls the Lane Centering System (LCS) and Lane Keeping Assist (LKA) to increase the steering override torque or to maintain the lane even in the event of a turn signal malfunction, controls the Adaptive Cruise Control (ACC) to increase the accelerator pedal override torque or to respond to a hazard ahead even in the event of a brake pedal malfunction or to prevent an Overtaking Function (OTF) even in the event of a turn signal malfunction, and controls the Emergency Braking System (ESS) so that the Emergency Braking System operation is performed even if the brake pedal, accelerator pedal, or steering override torque increases. Accordingly, the vehicle control can be performed adaptively by quickly and accurately determining the situation inside and outside the vehicle.

[0630] Meanwhile, in the case of the second emergency mode, the sensitivity adjustment module (920) within the processor (175) can control the Lane Centering System (LCS) to increase the target distance level for object tracking or to decrease the lane width limit, control the Lane Keeping Assist (LKA) to increase the distance between the vehicle and the lane, control the Adaptive Cruise Control (ACC) to change the Time to Collision (TTC) or change the ACC Early Warning, or to decrease the target speed depending on the curve or road type, and control the Automatic Emergency Braking (AEB) to change the Time to Collision (TTC) or change the ACC Early Warning. Accordingly, it is possible to quickly and accurately determine the situation inside and outside the vehicle and adaptively control the vehicle.

[0631] Meanwhile, in the case of emergency release mode, the safety extension module (910) within the processor (175) controls the lane centering system (LCS) and adaptive cruise control (ACC) to gradually restore or reduce the maximum allowable lateral acceleration or gradually restore or reduce the operating limit speed, and the emergency braking system (ESS) can be immediately released.

[0632] Meanwhile, in the case of emergency release mode, the authority adjustment module (915) within the processor (175) can control the lane centering system (LCS), lane keeping assist (LKA), and adaptive cruise control (ACC) to gradually restore or reduce the brake pedal, accelerator pedal, or steering override torque, or restore the turn signal dependency function, and the emergency braking system (ESS) can be immediately released.

[0633] Meanwhile, in the case of emergency release mode, the sensitivity adjustment module (920) within the processor (175) can control the lane centering system (LCS) and lane keeping assist (LKA) to gradually restore or reduce the target distance, lane width limit, parameters, etc., and can control the adaptive cruise control (ACC) to gradually restore or reduce the Time to Collision (TTC).

[0634] FIG. 13a is a drawing referenced in the description of the vehicle's operation in normal mode of FIG. 12.

[0635] Referring to the drawing, the vehicle (200) can operate in normal mode at points Pma, Pmb, and Pmc.

[0636] Meanwhile, the processor (175) can disable the lane centering system (LCS) and adaptive cruise control (ACC) in normal mode.

[0637] For example, in normal mode, when the lane centering system (LCS) is deactivated, the direction of travel of the vehicle (200) is changed based on the driver's steering wheel operation input.

[0638] As another example, in normal mode, when the lane centering system (LCS) is deactivated, the turn signal is turned on or off based on the driver's turn signal operation input.

[0639] In the drawing, based on the driver's steering wheel operation input in normal mode, the vehicle (200) is shown passing through the center line of the lane between the lanes, such as at point Pmb.

[0640] Meanwhile, in Normal mode, if the driver's steering input corresponds to a leftward movement input, the vehicle proceeds to the left of the lane rather than the center line, as at the Pmc point.

[0641] Meanwhile, in Normal mode, if the driver's steering input corresponds to a centering input, the vehicle will continue to deviate to the left of the lane rather than the center line, as seen at the Pmd point.

[0642] FIG. 13b is a drawing referenced in the description of the operation of the vehicle in the second emergency mode of FIG. 12.

[0643] Referring to the drawing, the vehicle (200) can operate in normal mode at point Pna, and then operate in a second emergency mode at points Pnb, Pnc, and Pnd.

[0644] For example, the processor (175) can disable the lane centering system (LCS) and adaptive cruise control (ACC) in normal mode.

[0645] Meanwhile, after the Pna point, if the risk level of vehicle driving changes, for example, if an emergency situation occurs, the processor (175) can switch to a second emergency mode.

[0646] Accordingly, the processor (175) can activate the lane centering system (LCS) and adaptive cruise control (ACC) in the second emergency mode.

[0647] At this time, the processor (175) can control the lane centering system (LCS) and adaptive cruise control (ACC) to remain active and not deactivated despite the driver's steering wheel operation input or the driver's turn signal operation input.

[0648] For example, the processor (175) can ignore the input of the accelerator pedal based on the lane centering system (LCS) and adaptive cruise control (ACC) when there is an unintended vehicle operation input at the Pnc point in the second emergency mode, for example, when there is an input of the accelerator pedal.

[0649] Accordingly, as at point Pnd, the vehicle (200) can maintain its lane so that it passes the center line, while maintaining the distance from the vehicle ahead.

[0650] As another example, the processor (175) can ignore the driver's steering wheel operation input in the second emergency mode, based on the lane centering system (LCS) and adaptive cruise control (ACC). Accordingly, the vehicle (200) can maintain its lane so as to pass the center line.

[0651] As another example, the processor (175) can ignore the driver's turn signal operation input in the second emergency mode based on the lane centering system (LCS) and adaptive cruise control (ACC). Accordingly, the vehicle (200) can maintain the lane so as to pass the center line.

[0652] FIG. 13c is a drawing illustrating an example of various behaviors of a passenger inside a vehicle.

[0653] Referring to the drawing, the processor (175) can monitor various behaviors of the occupants in the vehicle and calculate the risk level of the vehicle's driving based on the monitored behavior information of the occupants.

[0654] Meanwhile, the processor (175) monitors various behaviors of the occupants in the vehicle and, based on the monitored behavior information of the occupants, can classify them into use cases and non-use cases.

[0655] Meanwhile, the processor (175) may not output a vehicle control signal if the behavior information of the occupant in the vehicle corresponds to a non-use case.

[0656] Meanwhile, the processor (175) can disable the Lane Centering System (LCS), Lane Keeping Assist (LKA), Adaptive Cruise Control (ACC), or Emergency Braking System (ESS) when the behavior information of the occupant in the vehicle corresponds to a non-use case.

[0657] Meanwhile, the processor (175) can activate at least one of the lane centering system (LCS), lane keeping assist (LKA), adaptive cruise control (ACC), or emergency braking system (ESS) when the behavior information of the occupant in the vehicle corresponds to a use case.

[0658] Meanwhile, the processor (175) can monitor various behaviors of the occupants in the vehicle and, based on the monitored behavior information of the occupants, classify the risk level of the vehicle driving into levels 1 through 4.

[0659] At this time, the first level can correspond to a non-use case, and the second to fourth levels can correspond to a use case.

[0660] That is, the processor (175) monitors various behaviors of the occupants in the vehicle and, based on the monitored behavior information of the occupants, can distinguish between non-use cases and use cases.

[0661] Meanwhile, the processor (175) can monitor various behaviors of the occupants in the vehicle and, based on the monitored behavior information of the occupants, control the execution of any one of a normal mode, a first emergency mode, a second emergency mode, or an emergency release mode.

[0662] At this time, the normal mode and the emergency release mode can correspond to non-use cases, and the first emergency mode and the second emergency mode can correspond to use cases.

[0663] For example, the processor (175) can be classified as a non-use case in the case of smoking, talking, laughing, singing, navigation, object catching, calling, and phone usage.

[0664] Meanwhile, the processor (175) can classify the case of drinking a warm beverage during drinking as a non-use case.

[0665] Meanwhile, the processor (175) can classify the case as a use case when visual distraction occurs due to liquid spread or movement while drinking.

[0666] Meanwhile, the processor (175) can classify the case of eating warm food as a non-use case.

[0667] Meanwhile, the processor (175) can classify the case as a use case when visual distraction occurs due to food dispersion or movement during eating.

[0668] Meanwhile, the processor (175) can classify the case where eye scratching occurs as a use case.

[0669] Meanwhile, the processor (175) can classify cases where sneezing occurs as non-use cases.

[0670] Meanwhile, the processor (175) can classify the case of visual obstruction caused by sneezing (Sneezing while not looking forward) as a use case.

[0671] Meanwhile, the processor (175) can classify cases where yawning occurs as non-use cases.

[0672] Meanwhile, the processor (175) can classify the case of yawning while closing eyes as a use case.

[0673] FIG. 13d is a drawing illustrating different examples of various behaviors of occupants inside a vehicle.

[0674] Referring to the drawing, the processor (175) can be classified into non-use cases such as object searching, grooming, object dropping, and object placing.

[0675] Meanwhile, the processor (175) can classify cases of emergency and driver disturbance as use cases.

[0676] Meanwhile, in the case of the use case of FIG. 13c to FIG. 13d, the processor (175) can activate at least one of the lane centering system (LCS), lane keeping assist (LKA), adaptive cruise control (ACC), or emergency braking system (ESS).

[0677] Meanwhile, in the case of the non-use cases of FIGS. 13c to 13d, the processor (175) may not activate the lane centering system (LCS), lane keeping assist (LKA), adaptive cruise control (ACC), or emergency braking system (ESS), or may not operate them even if the conditions are met.

[0678] FIG. 14a is an example of a signal processing system according to an embodiment of the present disclosure.

[0679] Referring to the drawings, the signal processing system (1400a) according to an embodiment of the present disclosure may be referred to as an IVEX (In Vehicle Experience) system.

[0680] Meanwhile, a signal processing system (1400a) according to an embodiment of the present disclosure includes an edge sensor group (SN) and a signal processing device (170) in a vehicle.

[0681] Meanwhile, the signal processing device (170) according to the embodiment of the present disclosure includes a processor (175) that receives vehicle external sensing data from a vehicle external sensor (715) and vehicle internal sensing data from a vehicle internal sensor (710).

[0682] Meanwhile, the processor (175) collects external vehicle situation information, internal vehicle situation information, and passenger situation information based on external vehicle sensing data and internal vehicle sensing data, generates a prompt for calculating the risk level of vehicle driving based on each situation information, classifies the risk level of vehicle driving into multiple levels based on the inference result or response result obtained based on the generated prompt, and controls the vehicle based on the classified risk level. Accordingly, it is possible to quickly and accurately determine the situation inside and outside the vehicle and adaptively control the vehicle.

[0683] Meanwhile, the processor (175) according to an embodiment of the present disclosure includes a recognizer (720), an insightor (750), and an illustrator (780).

[0684] Meanwhile, the edge sensor group (SN) includes an internal vehicle sensor (710), an external vehicle sensor (715), and a database (1610) that stores driver's driving pattern information.

[0685] Meanwhile, the edge sensor group (SN) may further include vehicle driving sensors (718), such as an acceleration sensor, a brake sensor, a torque sensor, and a steering sensor.

[0686] Meanwhile, the vehicle interior sensor (710) is a sensor placed inside the vehicle (200) and may include a front camera (195), lidar (196), radar (197), interior camera (195i), microphone (112), biosensor, vehicle interior temperature sensor, or vehicle interior humidity sensor.

[0687] The vehicle external sensor (715) is a sensor positioned on the exterior of the vehicle (200) and may include an external camera, lidar (196), radar (197), heading sensor, yaw sensor, gyro sensor, position module, vehicle forward / reverse sensor, wheel sensor, vehicle speed sensor, vehicle body inclination sensor, battery sensor, fuel sensor, tire sensor, or steering sensor based on steering wheel rotation. The position module may include a GPS module or a position sensor (198) for receiving GPS information.

[0688] The vehicle interior sensing data may be data sensed by the vehicle interior sensor (710).

[0689] Vehicle interior sensing data may include at least one of vehicle interior temperature data, vehicle interior humidity data, battery data, fuel data, vehicle lamp data, tire data, or vehicle interior camera data, or audio data received through a microphone.

[0690] The vehicle external sensing data may be data sensed by the vehicle external sensor (715).

[0691] Vehicle external sensing data may include at least one of vehicle location data (GPS), vehicle direction data, vehicle angle data, vehicle speed data, vehicle acceleration data, vehicle tilt data, whether the vehicle is moving forward or backward, front camera data, or rear camera data.

[0692] The vehicle driving sensing data may be data sensed by the vehicle driving sensor (718).

[0693] Meanwhile, the recognizer (720) in the processor (175) can receive vehicle external sensing data from the vehicle external sensor (715), vehicle internal sensing data from the vehicle internal sensor (710), vehicle driving sensing data from the vehicle driving sensor (718), and driver driving pattern information from the database (1610).

[0694] The recognizer (720) can collect external vehicle situation information, internal vehicle situation information, and passenger situation information based on external vehicle sensing data from the external vehicle sensor (715), internal vehicle sensing data from the internal vehicle sensor (710), driving vehicle sensing data from the driving vehicle sensor (718), and driver driving pattern information from the database (1610).

[0695] Meanwhile, occupant situation information may include driver situation information, passenger situation information, etc.

[0696] For example, the passenger status recognizer (1413) within the recognizer (720) can perform passenger face identification, recognize distraction, recognize gaze, position, gesture, recognize action, recognize emotion, recognize health status, and obtain situational information of the passenger in the vehicle based on the above, based on the vehicle interior sensing data.

[0697] For example, the driving environment recognizer (1417) within the recognizer (720) can obtain driving situation information such as objects around the vehicle, lanes in front of the vehicle, traffic signs, traffic lights, road types, tunnels, traffic conditions, and weather.

[0698] That is, the recognizer (720) collects external vehicle situation information, internal vehicle situation information, and passenger situation information.

[0699] Meanwhile, the recognition device (720) can transmit the collected vehicle external situation information, vehicle internal situation information, and passenger situation information to the insight device (750).

[0700] Meanwhile, the insightor (750) within the processor (175) generates a prompt for calculating the risk level of vehicle driving based on each situation information, and classifies the risk level of vehicle driving into multiple levels based on the inference result or response result obtained based on the generated prompt. Accordingly, the situation inside and outside the vehicle can be determined quickly and accurately to perform adaptive vehicle control.

[0701] Meanwhile, the insightor (750) within the processor (175) can classify the risk level of vehicle driving into a first level corresponding to a normal situation, a second level corresponding to a caution situation, a third level corresponding to a danger warning situation, and a fourth level corresponding to a danger situation. Accordingly, it is possible to quickly and accurately determine the situation inside and outside the vehicle and adaptively perform vehicle control.

[0702] Meanwhile, the vehicle interior situation recognition device (960) within the insight device (750) can recognize vehicle interior situation information based on passenger situation information from the passenger status recognition device (1413), etc.

[0703] For example, the vehicle interior situation recognition device (960) within the insightor (750) can classify the risk level of the occupant, particularly the driver, based on occupant situation information, etc., into a first level corresponding to a general situation, a second level corresponding to a caution situation, a third level corresponding to a danger warning situation, and a fourth level corresponding to a danger situation.

[0704] Meanwhile, the vehicle external situation recognition unit (970) within the insightor (750) can calculate the risk level of the driving environment based on driving situation information from the driving environment recognition unit (1417).

[0705] Meanwhile, the driver feedback judgment unit (989) within the insightor (750) can receive driver risk level information from the vehicle interior situation recognizer (960) and receive risk level information regarding the driving environment from the vehicle exterior situation recognizer (970).

[0706] And, the driver feedback judgment unit (989) can calculate a plurality of levels of risk level for vehicle driving based on the driver's risk level information from the vehicle interior situation recognizer (960) and the risk level information regarding the driving environment from the vehicle exterior situation recognizer (970).

[0707] For example, the driver feedback judgment unit (989) can classify the risk level of vehicle driving into a first level corresponding to a normal situation, a second level corresponding to a caution situation, a third level corresponding to a danger warning situation, and a fourth level corresponding to a danger situation.

[0708] Meanwhile, the driver feedback judgment unit (989) receives a vehicle control signal from the illustrator (780) or an input signal from the driver's operation, and can determine the driver's ability to respond to an emergency situation based on the received vehicle control signal or input signal.

[0709] For example, the driver feedback judgment unit (989) can determine that the driver's ability to handle emergency situations is above a threshold value when the input value of the received input signal is within an allowable range, and can maintain or lower the intervention level during autonomous driving or vehicle driving assistance control.

[0710] As another example, the driver feedback judgment unit (989) can determine that the driver's ability to handle emergency situations is below a threshold value when the input value of the received input signal is outside the allowable range, and can increase the intervention level during autonomous driving or vehicle driving assistance control.

[0711] Meanwhile, the driver feedback judgment unit (989) can transmit information about the risk level of vehicle driving to the safety intervention judgment unit (757).

[0712] Meanwhile, the safety intervention judgment unit (757) can determine safety intervention during autonomous driving or vehicle driving assistance control based on the risk level of vehicle driving.

[0713] For example, the safety intervention judgment unit (757) can control the authority adjustment, safety extension, or sensitivity adjustment for vehicle control based on the distinguished risk level. Accordingly, it is possible to quickly and accurately determine the situation inside and outside the vehicle and adaptively perform vehicle control.

[0714] As another example, the safety intervention judgment unit (757) can relax the allowable operating range during safety expansion, perform authority adjustment between the driver and the system during authority adjustment, or adjust sensitivity based on the risk level during sensitivity adjustment. Accordingly, it is possible to quickly and accurately determine the situation inside and outside the vehicle and adaptively perform vehicle control.

[0715] To this end, the safety intervention judgment unit (757) may include a safety extension module (910), an authority adjustment module (915), and a sensitivity adjustment module (920).

[0716] For example, the sensitivity adjustment module (920) can increase the warning sensitivity of Autonomous Emergency Braking (AEB) or Lane Keeping Assist (LKA) when the risk level of vehicle driving is at the second level. Accordingly, it is possible to quickly and accurately determine the situation inside and outside the vehicle and adaptively perform vehicle control.

[0717] Meanwhile, the illustrator (780) can execute a service, output service information, execute an application, or output application information based on the response result output from the insightor (750).

[0718] Meanwhile, the illustrator (780) can generate vehicle control signals for controlling Lane Centering Systems (LCS), Adaptive Cruise Control (ACC), Lane Keeping Assist (LKA), Autonomous Emergency Braking (AEB), or Emergency Stop System (ESS) based on authority adjustment data, safety extension data, and sensitivity adjustment data from the insightor (750). Accordingly, it is possible to quickly and accurately determine the situation inside and outside the vehicle and adaptively perform vehicle control.

[0719] Meanwhile, the illustrator (780) can transmit an input signal by driver operation to the driver feedback judgment unit (989) when performing Lane Centering Systems (LCS), Adaptive Cruise Control (ACC), Lane Keeping Assist (LKA), Autonomous Emergency Braking (AEB), or Emergency Stop System (ESS).

[0720] FIG. 14b is another example of a signal processing system according to an embodiment of the present disclosure.

[0721] Referring to the drawings, the signal processing system (1400b) according to an embodiment of the present disclosure is similar to the signal processing system (1400a) of FIG. 14a, but differs in some operation of the processor (175) within the signal processing device (170). The following description focuses on the differences.

[0722] Meanwhile, the signal processing system (1400b) according to an embodiment of the present disclosure includes an edge sensor group (SN) and a signal processing device (170) in a vehicle.

[0723] Meanwhile, the edge sensor group (SN) includes vehicle driving sensors (718), such as an internal vehicle sensor (710), an external vehicle sensor (715), an acceleration sensor, a brake sensor, a torque sensor, and a steering sensor.

[0724] The processor (175) in the signal processing unit (170) includes a recognizer (720), an insightor (750), and an illustrator (780).

[0725] The recognizer (720) may include a risk situation determination unit (1412) that performs a risk situation determination based on vehicle external sensing data from a vehicle external sensor (715), vehicle internal sensing data from a vehicle internal sensor (710), vehicle driving sensing data from a vehicle driving sensor (718), and driver driving pattern information from a database (1610 in FIG. 14a), and a driving environment determination unit (1414) that performs a driving environment determination.

[0726] Meanwhile, the risk situation judgment unit (1412) generates a prompt for calculating the risk level of vehicle driving based on each situation information, and classifies the risk level of vehicle driving into multiple levels based on the inference result or response result obtained based on the generated prompt. Accordingly, the situation inside and outside the vehicle can be quickly and accurately determined to perform adaptive vehicle control.

[0727] Meanwhile, the danger situation judgment unit (1412) can classify the driver's danger level into a first level corresponding to a general situation, a second level corresponding to a caution situation, a third level corresponding to a danger warning situation, and a fourth level corresponding to a danger situation. Accordingly, the situation inside and outside the vehicle can be judged quickly and accurately to perform adaptive vehicle control.

[0728] Meanwhile, the danger situation judgment unit (1412) can classify the danger level of vehicle driving into a first level corresponding to a general situation, a second level corresponding to a caution situation, a third level corresponding to a danger warning situation, and a fourth level corresponding to a danger situation. Accordingly, the situation inside and outside the vehicle can be judged quickly and accurately to enable adaptive vehicle control.

[0729] Meanwhile, the driving environment judgment unit (1414) can determine whether there is road curvature, a tunnel ahead, or traffic congestion.

[0730] Meanwhile, the recognition unit (720) can transmit risk level information from the risk situation judgment unit (1412) and driving environment information from the driving environment judgment unit (1414) to the insightor (750), respectively.

[0731] Meanwhile, the insightor (750), unlike FIG. 14a, may be equipped with a safety intervention judgment unit (757) and a driver feedback judgment unit (989).

[0732] The safety intervention judgment unit (757) can control the adjustment of authority, safety extension, or sensitivity for vehicle control based on the risk level information from the risk situation judgment unit (1412) and the driving environment information from the driving environment judgment unit (1414). Accordingly, it is possible to quickly and accurately determine the situation inside and outside the vehicle and adaptively perform vehicle control.

[0733] Meanwhile, the safety intervention judgment unit (757) may include a safety expansion module (910), an authority adjustment module (915), and a sensitivity adjustment module (920).

[0734] The driver feedback judgment unit (989) can receive risk level information from the risk situation judgment unit (1412), a vehicle control signal from the illustrator (780), or an input signal from the driver's operation.

[0735] Meanwhile, the driver feedback judgment unit (989) can determine the driver's ability to respond to an emergency situation based on the risk level information and the received vehicle control signal or input signal.

[0736] Meanwhile, the driver feedback judgment unit (989) can transmit information on the driver's ability to handle emergency situations to the safety intervention judgment unit (757).

[0737] Meanwhile, the safety intervention judgment unit (757) can adjust the intervention level or the risk level of vehicle driving based on the information on the former's ability to handle emergency situations. Accordingly, vehicle control can be performed adaptively based on the driver's proficiency.

[0738] Meanwhile, the illustrator (780) can generate vehicle control signals for controlling Lane Centering Systems (LCS), Adaptive Cruise Control (ACC), Lane Keeping Assist (LKA), Autonomous Emergency Braking (AEB), or Emergency Stop System (ESS) based on authority adjustment data, safety extension data, and sensitivity adjustment data from the insightor (750). Accordingly, it is possible to quickly and accurately determine the situation inside and outside the vehicle and adaptively perform vehicle control.

[0739] FIGS. 14c to 15 are drawings referenced in the description of FIG. 14a or FIG. 14b.

[0740] FIG. 14c is a drawing referenced in the description of the operation of the safety intervention judgment unit of FIG. 14a or FIG. 14b.

[0741] Referring to the drawing, the safety extension module (910) within the safety intervention judgment unit (757) can adjust the allowable operating range.

[0742] For example, the safety extension module (910) can gradually increase the allowable lateral acceleration when the risk level of vehicle driving is at the third level during the operation of adaptive cruise control (ACC).

[0743] Meanwhile, the safety extension module (910) can gradually increase the allowable lateral acceleration or gradually increase the target distance for object tracking when the risk level of vehicle driving is at the third level during the execution of the lane centering system (LCS).

[0744] Meanwhile, the authority adjustment module (915) within the safety intervention judgment unit (757) can restrict or supplement some or all of the vehicle operation authority based on driving situation information or driver situation information.

[0745] For example, the authority adjustment module (915) can gradually increase the accelerator pedal override threshold when the risk level of vehicle driving is at the third level during the execution of adaptive cruise control (ACC).

[0746] Meanwhile, the authority adjustment module (915) can gradually increase the steering override torque when the risk level of vehicle driving is at the third level during the execution of the lane centering system (LCS).

[0747] Meanwhile, the sensitivity adjustment module (920) within the safety intervention judgment unit (757) can adjust the sensitivity.

[0748] For example, the sensitivity adjustment module (920) can gradually increase the forward collision trajectory sensitivity, gradually decrease the target speed according to the road curvature, or increase the distance level when the risk level of the vehicle driving is at the third level during the operation of Adaptive Cruise Control (ACC).

[0749] Meanwhile, the sensitivity adjustment module (920) can control the lane centering system (LCS) to follow the lane center more precisely than normal operation when the risk level of vehicle driving is at the third level during operation.

[0750] Meanwhile, the sensitivity adjustment module (920) can increase the warning sensitivity of the Autonomous Emergency Braking (AEB) or Lane Keeping Assist (LKA) when the risk level of vehicle driving is at the second level during the execution of Autonomous Emergency Braking (AEB) or Lane Keeping Assist (LKA).

[0751] For example, the safety extension module (910) can increase the allowable lateral acceleration when the risk level of vehicle driving is at the fourth level during the operation of adaptive cruise control (ACC).

[0752] Meanwhile, the safety extension module (910) can increase the allowable lateral acceleration, increase the target distance for object tracking, decrease the lane width limit, or increase the allowable steering torque value when the risk level of vehicle driving is at the fourth level during the operation of the lane centering system (LCS).

[0753] For example, the authority adjustment module (915) can limit acceleration based on accelerator pedal override, ignore the condition for deactivating adaptive cruise control by the brake pedal, or ignore the overtaking function by the turn signal when the risk level of vehicle driving is at the fourth level during the execution of adaptive cruise control (ACC).

[0754] Meanwhile, the authority adjustment module (915) can increase the steering override torque or ignore the deactivation condition of the lane centering system (LCS) by the turn signal when the risk level of vehicle driving is at the fourth level during the execution of the lane centering system (LCS).

[0755] For example, the sensitivity adjustment module (920) can increase the forward collision risk sensitivity, decrease the target speed according to road curvature, or increase the distance level when the risk level of vehicle driving is at the fourth level during the operation of Adaptive Cruise Control (ACC).

[0756] Meanwhile, the sensitivity adjustment module (920) can control the lane centering system (LCS) to follow the lane center more precisely when the risk level of the vehicle driving is at the fourth level during the operation of the LCS.

[0757] Meanwhile, the safety expansion module (910), the authority adjustment module (915), and the sensitivity adjustment module (920) can gradually relax or remove the restriction conditions when the dangerous situation is released.

[0758] For example, the safety extension module (910), the authority adjustment module (915), and the sensitivity adjustment module (920) can lower the risk level of vehicle driving from any of the second to fourth levels to the first level when a dangerous situation is released, and gradually relax or release the restriction conditions. Accordingly, it becomes possible to quickly and accurately determine the situation inside and outside the vehicle and perform adaptive vehicle control.

[0759] Fig. 14d is a drawing referenced in the explanation of driver feedback.

[0760] Referring to the drawing, the processor (175) can feed back an input signal from the driver's operation to prevent driver surprise or secondary accidents caused by excessive ADAS safety intervention.

[0761] Meanwhile, the processor (175) compares the acceleration / deceleration data during adaptive cruise control (ACC) with the longitudinal input value based on the driver's operation-based input signal.

[0762] For example, the processor (175) can lower the intervention level to prevent excessive intervention during autonomous driving or vehicle driving assistance control when longitudinal control is performed well in an emergency situation.

[0763] As another example, the processor (175) can increase the intervention level to prevent excessive intervention during autonomous driving or vehicle driving assistance control when longitudinal control is not performed well in an emergency situation.

[0764] Meanwhile, the processor (175) compares the acceleration / deceleration data during adaptive cruise control (ACC) with the lateral input value based on the driver's operation-based input signal.

[0765] For example, the processor (175) can lower the intervention level to prevent excessive intervention during autonomous driving or vehicle driving assistance control when lateral control is performed well in an emergency situation.

[0766] As another example, the processor (175) can increase the intervention level to prevent excessive intervention during autonomous driving or vehicle driving assistance control when lateral control is not performed well in an emergency situation.

[0767] FIG. 14e is a diagram illustrating intervention levels based on various driver feedback evaluations.

[0768] Referring to the drawing, the processor (175) can raise, maintain, or lower the intervention level based on an input signal or input value resulting from the driver's operation and an allowable range corresponding to each operation.

[0769] Meanwhile, the processor (175) can raise, maintain, or lower the intervention level based on an input signal or input value resulting from the driver's operation, an allowable range corresponding to each operation, and an event that occurs.

[0770] The event at this time may include an event corresponding to an external vehicle situation, an event corresponding to an internal vehicle situation, or an event corresponding to a vehicle driving sensor.

[0771] For example, the processor (175) can determine that the driver's control ability is above a reference value (e.g., good) when the longitudinal input value based on the driver's operation is within the first allowable range.

[0772] As another example, the processor (175) can determine that the driver's control ability is below a reference value (e.g., poor) when the longitudinal input value based on the driver's operation falls outside the first allowable range.

[0773] Meanwhile, the processor (175) can determine that the driver's control ability is above a standard value (e.g., good) when the lateral input value based on the driver's operation is within the second allowable range.

[0774] Meanwhile, the processor (175) can determine that the driver's control ability is below a reference value (e.g., poor) if the lateral input value based on the driver's operation falls outside the second allowable range.

[0775] Meanwhile, the processor (175) can evaluate the risk of the driving environment based on artificial intelligence.

[0776] For example, the processor (175) can evaluate the risk of the driving environment based on a driver personalization database, road environment or weather or driving route or construction section or tunnel, etc.

[0777] Meanwhile, the processor (175) can evaluate the risk level of the driving environment corresponding to the driver's operation input.

[0778] For example, the processor (175) can evaluate the longitudinal driving risk based on a driver personalization database, road environment or weather or driving route or construction section or tunnel, etc., in response to a longitudinal input value based on the driver's operation.

[0779] As another example, the processor (175) can evaluate the lateral driving risk based on a driver personalization database, road environment or weather or driving route or construction section or tunnel, etc., in response to a lateral input value based on the driver's operation.

[0780] Meanwhile, the processor (175) can distinguish the level of longitudinal driving risk evaluated or calculated based on the first reference level.

[0781] For example, the processor (175) can classify the level of longitudinal driving risk evaluated or calculated as a risk level (e.g., "high" level) if the level is greater than or equal to a first reference level.

[0782] As another example, the processor (175) may classify the level of longitudinal driving risk evaluated or calculated as a non-risk level (e.g., "low" level) if it is less than a first reference level.

[0783] Meanwhile, the processor (175) can classify the level of lateral driving risk evaluated or calculated as a risk level (e.g., "high" level) if it is higher than the second reference level.

[0784] As another example, the processor (175) may classify the level of lateral driving risk evaluated or calculated as a non-risk level (e.g., "low" level) if it is less than a second reference level.

[0785] Meanwhile, the processor (175) can maintain the intervention level for longitudinal control as is when the longitudinal input value based on the driver's operation is within the first allowable range and the level of the longitudinal driving risk is above the first reference level.

[0786] Meanwhile, the processor (175) can maintain the intervention level for lateral control as is when the lateral input value based on the driver's operation is within the second allowable range and the level of lateral driving risk is above the second reference level.

[0787] Meanwhile, the processor (175) can mitigate or reduce the intervention level for lateral control when the lateral input value based on the driver's operation is within the second allowable range and the level of lateral driving risk is below the second reference level.

[0788] Meanwhile, the processor (175) can mitigate or reduce the intervention level for longitudinal control when the longitudinal input value based on the driver's operation is within the first allowable range and the level of longitudinal driving risk is below the first reference level.

[0789] Meanwhile, the processor (175) can strengthen or increase the intervention level for lateral control when the lateral input value based on the driver's operation exceeds the second allowable range and the level of lateral driving risk is higher than the second reference level.

[0790] Meanwhile, the processor (175) can maintain the intervention level for lateral control as is when the lateral input value based on the driver's operation is outside the second allowable range and the level of lateral driving risk is below the second reference level.

[0791] Meanwhile, the processor (175) can strengthen or increase the intervention level for longitudinal control when the longitudinal input value based on the driver's operation exceeds the first allowable range and the level of longitudinal driving risk is greater than or equal to the first reference level.

[0792] Meanwhile, the processor (175) can maintain the intervention level for longitudinal control as is when the longitudinal input value based on the driver's operation is outside the first allowable range and the level of the longitudinal driving risk is below the first reference level.

[0793] Meanwhile, the processor (175) can maintain the intervention level for longitudinal control when the longitudinal input value based on the driver's operation is within the first allowable range and there is an event where a curved road appears ahead while driving straight. Accordingly, vehicle control can be performed adaptively based on the driver's proficiency.

[0794] Meanwhile, the processor (175) can increase the intervention level for lateral control when there is an event where the lateral input value based on the driver's operation exceeds the second allowable range and a curved road appears ahead while driving straight. Accordingly, vehicle control can be performed adaptively based on the driver's proficiency.

[0795] Meanwhile, the processor (175) can reduce or lower the intervention level for longitudinal control when the longitudinal input value based on the driver's operation is within the first allowable range, the vehicle is driving straight, and there is an event with road conditions ahead similar to the current situation. Accordingly, vehicle control can be performed adaptively based on the driver's proficiency.

[0796] Meanwhile, the processor (175) can maintain an intervention level for lateral control when the lateral input based on the driver's operation exceeds the second allowable range, and when the vehicle is driving straight and there is an event in the road conditions ahead that is similar to the current situation. Accordingly, the vehicle can be adaptedly controlled based on the driver's proficiency.

[0797] Meanwhile, the processor (175) can increase or raise the intervention level for longitudinal control when the longitudinal input value based on the driver's operation is within the first allowable range, the vehicle is driving straight, and there is an event corresponding to traffic congestion or a tunnel ahead. Accordingly, vehicle control can be performed adaptively based on the driver's proficiency.

[0798] Meanwhile, the processor (175) can increase the intervention level for lateral control when the lateral input value based on the driver's operation exceeds the second allowable range and there is an event corresponding to traffic congestion or a tunnel ahead while driving straight. Accordingly, adaptive vehicle control can be performed based on the driver's proficiency.

[0799] Meanwhile, the processor (175) can reduce or lower the intervention level for longitudinal control when the longitudinal input value based on the driver's operation is within the first allowable range and there is no special event in front of the vehicle. Accordingly, vehicle control can be performed adaptively based on the driver's proficiency.

[0800] Meanwhile, the processor (175) can reduce or lower the intervention level for lateral control when the lateral input value based on the driver's operation is within the second allowable range and there is no special event in front of the vehicle. Accordingly, vehicle control can be performed adaptively based on the driver's proficiency.

[0801] Meanwhile, the processor (175) can maintain the intervention level for longitudinal control as is when the longitudinal input value based on the driver's operation is within the first allowable range and there is an event requiring the driver's attention. Accordingly, vehicle control can be performed adaptively based on the driver's proficiency.

[0802] Meanwhile, the processor (175) can increase or raise the intervention level for lateral control when the lateral input value based on the driver's operation is within the second allowable range and there is an event requiring the driver's attention. Accordingly, vehicle control can be performed adaptively based on the driver's proficiency.

[0803] Meanwhile, the processor (175) can increase or raise the intervention level for longitudinal control when the longitudinal input value based on the driver's operation exceeds the first allowable range and there is an event requiring the driver's attention. Accordingly, vehicle control can be performed adaptively based on the driver's proficiency.

[0804] Meanwhile, the processor (175) can increase or raise the intervention level for lateral control when the lateral input value based on the driver's operation exceeds the second allowable range and there is an event requiring the driver's attention. Accordingly, vehicle control can be performed adaptively based on the driver's proficiency.

[0805] FIG. 14f is a diagram referenced in the description of the operation of the safety extension module, the authority adjustment module, and the sensitivity adjustment module.

[0806] Referring to the drawing, the safety extension module (910) within the processor (175) can gradually increase the maximum allowable lateral acceleration when performing the lane centering system (LCS) when the risk level of vehicle driving is a third level corresponding to a danger warning situation.

[0807] Meanwhile, the safety extension module (910) within the processor (175) can gradually increase the maximum allowable lateral acceleration when performing adaptive cruise control (ACC) when the risk level of vehicle driving is at the third level.

[0808] Meanwhile, the safety extension module (910) within the processor (175) can further lower or reduce the allowable speed required when entering a ramp during the operation of adaptive cruise control (ACC) when the risk level of vehicle driving is at the third level.

[0809] Meanwhile, the safety extension module (910) within the processor (175) can increase or raise the maximum allowable lateral acceleration when performing the lane centering system (LCS) when the risk level of vehicle driving is the fourth level corresponding to the dangerous situation.

[0810] Meanwhile, the safety extension module (910) within the processor (175) can increase or raise the maximum allowable lateral acceleration when performing adaptive cruise control (ACC) when the risk level of vehicle driving is the fourth level.

[0811] Meanwhile, the safety extension module (910) within the processor (175) can gradually lower or reduce the maximum allowable lateral acceleration when performing the lane centering system (LCS) when the risk level of vehicle driving is at a first level corresponding to normal conditions.

[0812] Meanwhile, the safety extension module (910) within the processor (175) can gradually lower or reduce the maximum allowable lateral acceleration when performing adaptive cruise control (ACC) when the risk level of vehicle driving is at the first level.

[0813] Meanwhile, the authority adjustment module (915) within the processor (175) can gradually increase the steering override torque when performing the lane centering system (LCS) when the risk level of vehicle driving is a third level corresponding to a danger warning situation.

[0814] Meanwhile, the authority adjustment module (915) within the processor (175) can gradually increase the accelerator pedal override threshold when performing adaptive cruise control (ACC) when the risk level of vehicle driving is at the third level.

[0815] Meanwhile, the authority adjustment module (915) within the processor (175) can increase or raise the steering override torque when performing the lane centering system (LCS) when the risk level of vehicle driving is the fourth level corresponding to the dangerous situation.

[0816] Meanwhile, the authority adjustment module (915) within the processor (175) can ignore the deactivation condition of the lane centering system (LCS) by the turn signal when the risk level of the vehicle driving is the fourth level corresponding to the dangerous situation.

[0817] Meanwhile, the authority adjustment module (915) within the processor (175) can limit acceleration based on accelerator pedal override when performing adaptive cruise control (ACC) when the risk level of vehicle driving is the fourth level.

[0818] Meanwhile, the authority adjustment module (915) within the processor (175) can ignore the condition for deactivating the adaptive cruise control by the brake pedal or ignore the overtaking function by the turn signal when the risk level of the vehicle driving is the fourth level during the operation of the adaptive cruise control (ACC).

[0819] Meanwhile, the authority adjustment module (915) within the processor (175) can gradually lower or reduce the steering override torque when the risk level of vehicle driving is at a first level corresponding to a normal situation.

[0820] Meanwhile, the authority adjustment module (915) within the processor (175) can restore or gradually restore each function during the performance of Adaptive Cruise Control (ACC) when the risk level of vehicle driving is at the first level.

[0821] Meanwhile, the sensitivity adjustment module (920) within the processor (175) can gradually increase the target distance for object tracking when performing the lane centering system (LCS) when the risk level of vehicle driving is at a third level corresponding to a danger warning situation.

[0822] Meanwhile, the sensitivity adjustment module (920) within the processor (175) can gradually increase the collision warning sensitivity with a forward object during the operation of adaptive cruise control (ACC) when the risk level of vehicle driving is at the third level.

[0823] Meanwhile, the sensitivity adjustment module (920) within the processor (175) can gradually lower or reduce the target speed according to the road curvature during the operation of Adaptive Cruise Control (ACC) when the risk level of vehicle driving is at the third level.

[0824] Meanwhile, the sensitivity adjustment module (920) within the processor (175) can gradually lower or reduce the brake operation time during the performance of automatic emergency braking (AEB) when the risk level of vehicle driving is at the third level.

[0825] Meanwhile, the sensitivity adjustment module (920) within the processor (175) can increase or raise the target distance for object tracking when performing the lane centering system (LCS) when the risk level of vehicle driving is the fourth level corresponding to the dangerous situation.

[0826] Meanwhile, the sensitivity adjustment module (920) within the processor (175) can lower or reduce the lane width limit when the risk level of vehicle driving is at the fourth level during the operation of the lane centering system (LCS).

[0827] Meanwhile, the sensitivity adjustment module (920) within the processor (175) can increase or raise the collision warning sensitivity with a forward object during the operation of adaptive cruise control (ACC) when the risk level of vehicle driving is the fourth level.

[0828] Meanwhile, the sensitivity adjustment module (920) within the processor (175) can lower or reduce the target speed according to the road curvature during the operation of Adaptive Cruise Control (ACC) when the risk level of vehicle driving is the fourth level.

[0829] Meanwhile, the sensitivity adjustment module (920) within the processor (175) can restore or gradually restore each function of the lane centering system (LCS) when the risk level of vehicle driving is at a first level corresponding to normal conditions.

[0830] Meanwhile, the sensitivity adjustment module (920) within the processor (175) can restore or gradually restore each function of the adaptive cruise control (ACC) when the risk level of vehicle driving is at the first level.

[0831] Fig. 14g is a diagram referenced in the explanation of driver feedback judgment.

[0832] Referring to the drawing, the processor (175) can perform control to prevent driver surprise or secondary accidents caused by excessive safety intervention during autonomous driving or vehicle driving assistance control.

[0833] For example, the processor (175) can determine the driver's proficiency by comparing the input value and the allowable range during the operation of adaptive cruise control (ACC).

[0834] Meanwhile, the processor (175) can lower the intervention level to prevent excessive intervention when the driver performs vehicle longitudinal control or vehicle lateral control above a reference value in emergency situations, etc. Accordingly, vehicle control can be performed adaptively based on the driver's proficiency.

[0835] Meanwhile, the processor (175) can increase the intervention level to facilitate smooth intervention when, in emergency situations, the driver performs vehicle longitudinal control or vehicle lateral control below a reference value. Accordingly, vehicle control can be performed adaptively based on the driver's proficiency.

[0836] As another example, the processor (175) can determine the driver's proficiency by comparing the reference steering torque during the operation of the lane centering system (LCS) with the input value from the driver's operation.

[0837] Meanwhile, the processor (175) can lower the intervention level to prevent excessive intervention when, in emergency situations, the steering torque produced by the driver's operation is within an allowable range based on the reference steering torque. Accordingly, vehicle control can be performed adaptively based on the driver's proficiency.

[0838] Meanwhile, the processor (175) can increase the intervention level to facilitate smooth intervention when, in emergency situations or the like, the steering torque caused by the driver's operation exceeds the allowable range based on the reference steering torque. Accordingly, adaptive vehicle control can be performed based on the driver's proficiency.

[0839] Meanwhile, the processor (175) compares a first allowable range of acceleration / deceleration during adaptive cruise control (ACC) with a longitudinal input value based on an input signal when the risk level of vehicle driving is at the fourth level, and if the input value is included within the first allowable range, the intervention level can be lowered.

[0840] Meanwhile, the processor (175) can increase the intervention level when the input value during adaptive cruise control (ACC) exceeds the first allowable range while the risk level of vehicle driving is at the fourth level. Accordingly, vehicle control can be performed adaptively based on the driver's proficiency.

[0841] Meanwhile, the processor (175) compares the second allowable range of acceleration / deceleration during adaptive cruise control (ACC) with the lateral input value based on the input signal when the risk level of the vehicle driving is at the fourth level, and if the input value is included within the second allowable range, the intervention level can be lowered.

[0842] Meanwhile, the processor (175) can increase the intervention level when the input value during adaptive cruise control (ACC) exceeds the second allowable range while the risk level of the vehicle driving is at the fourth level. Accordingly, adaptive vehicle control can be performed based on the driver's proficiency.

[0843] Meanwhile, the processor (175) can lower the intervention level if the longitudinal input value or the lateral input value is within the allowable range and there is no event in the external vehicle situation information.

[0844] Meanwhile, the processor (175) can maintain an intervention level when the longitudinal input value or the transverse input value is within the allowable range and there is an event requiring attention.

[0845] Meanwhile, the processor (175) can increase the intervention level when there is an event requiring attention where the longitudinal input or lateral input is outside the allowable range. Accordingly, vehicle control can be performed adaptively based on the driver's proficiency.

[0846] Meanwhile, the processor (175) can compare the input value with the allowable range during the execution of the Adaptive Cruise Control (ACC) or Lane Centering System (LCS), and if the input value deviates from the allowable range, it can increase the risk level of the vehicle's driving. Accordingly, it is possible to adaptively control the vehicle based on the driver's proficiency.

[0847] Meanwhile, the processor (175) can lower the risk level of vehicle driving when the input value during the execution of the adaptive cruise control (ACC) or lane centering system (LCS) is within the allowable range and there are no events in the external situation information of the vehicle.

[0848] Meanwhile, the processor (175) can maintain the risk level of vehicle driving when the input value during the execution of the Adaptive Cruise Control (ACC) or Lane Centering System (LCS) is within the allowable range and there is an event requiring attention within the external situation information of the vehicle.

[0849] Meanwhile, the processor (175) can increase the risk level of vehicle driving when the input value during the execution of Adaptive Cruise Control (ACC) or Lane Centering System (LCS) is outside the allowable range and there is an event requiring attention. Accordingly, vehicle control can be performed adaptively based on the driver's proficiency.

[0850] Meanwhile, the processor (175) can increase the intervention level when the driver has a history of heart-related or other diseases from the database when the Adaptive Cruise Control (ACC) or Lane Centering System (LCS) is executed. Accordingly, the vehicle control can be performed adaptively based on the driver's history of diseases.

[0851] Meanwhile, the processor (175) can increase the intervention level when there is a diagnostic history related to the driver's vision from the database when the Adaptive Cruise Control (ACC) or Lane Centering System (LCS) is executed. Accordingly, the vehicle control can be performed adaptively based on the driver's diagnostic history.

[0852] FIG. 14h is a drawing referenced in the description of multiple driving safety levels.

[0853] Referring to the drawing, the risk level determination unit (940) within the processor (175) can calculate the risk level of vehicle driving based on vehicle internal context data (941), vehicle external context data (942), and personal profile information (945).

[0854] Meanwhile, the risk level determination unit (940) within the processor (175) can classify the risk level of vehicle driving into a first level to a fourth level (951, 952, 953, 954).

[0855] Meanwhile, the level corresponding control module (950) can control the performance of corresponding operations based on the first to fourth levels (951, 952, 953, 954).

[0856] Meanwhile, the level corresponding control module (950) can control that, after performing an operation corresponding to the first level (951), if the risk level increases, an operation corresponding to the second level (952) is performed, and after performing an operation corresponding to the second level (952), if the risk level increases, an operation corresponding to the third level (953) is performed, and after performing an operation corresponding to the third level (953), if the risk level increases, an operation corresponding to the fourth level (954) is performed.

[0857] Meanwhile, the level-corresponding control module (950) can control the execution of an operation corresponding to the first level (951) when the risk level decreases after the operation corresponding to the second level (952) is performed.

[0858] Meanwhile, the level-corresponding control module (950) can control the execution of an operation corresponding to the first level (951) or the second level (952) when the risk level decreases after the operation corresponding to the third level (953) is performed.

[0859] Meanwhile, the level-corresponding control module (950) can control the execution of an operation corresponding to the first level (951) when the risk level decreases after the operation corresponding to the fourth level (954) is performed.

[0860] FIG. 15 is a drawing referenced in the description of the operation of the level-corresponding control module of FIG. 14h.

[0861] Referring to the drawing, the processor (175) can classify the risk level of vehicle driving into multiple levels based on vehicle internal context data, vehicle external context data, personal profile information, passenger behavior information, and driver driving ability information.

[0862] For example, the processor (175) can classify the risk level of vehicle driving into levels 1 through 4.

[0863] At this time, Level 1 indicates a normal state, Level 2 indicates a need for caution, Level 3 indicates imminent danger, and Level 4 may indicate a stage requiring immediate intervention.

[0864] Meanwhile, Levels 2 and 3 can correspond to the prodromal symptom stage.

[0865] For example, the processor (175) can determine the risk level of vehicle driving to be the first level when the vehicle internal behavior is normal based on the vehicle internal context data and the vehicle driving environment is stable based on the vehicle external context data.

[0866] Meanwhile, the processor (175) can determine the risk level of vehicle driving to be a second level when a single sign appears based on the vehicle internal context data or the vehicle external context data.

[0867] Meanwhile, the processor (175) can determine the risk level of vehicle driving to be a third level when a complex sign appears based on vehicle internal context data or vehicle external context data.

[0868] Meanwhile, the processor (175) can determine the risk level of vehicle driving to be a fourth level when a dangerous situation occurs due to vehicle internal context data or vehicle external context data.

[0869] For example, the processor (175) can determine the risk level of vehicle driving as the first level if there is no repetitive behavior and the habit is stable based on personal profile information.

[0870] Meanwhile, the processor (175) can determine the risk level of vehicle driving to be a second level if habitual behavior exists based on personal profile information.

[0871] Meanwhile, the processor (175) can determine the risk level of vehicle driving to be a third level if there is repeated risk behavior based on personal profile information and there is a history of past personal medical history or accidents.

[0872] Meanwhile, the processor (175) can determine the risk level of vehicle driving to be a fourth level if it is confident of the risk based on the behavioral results and medical history based on personal profile information.

[0873] For example, the processor (175) can determine the risk level of vehicle driving as a first level when the passenger's behavior is in a stable state (e.g., wearing a seatbelt) based on passenger behavior information.

[0874] Meanwhile, the processor (175) can determine the risk level of driving the vehicle to be a second level if there is a mild voice stimulus (e.g., talking) based on passenger behavior information.

[0875] Meanwhile, the processor (175) can determine the risk level of driving the vehicle to a third level if the driver's attention is distracted due to shouting or physical interference based on passenger behavior information.

[0876] Meanwhile, the processor (175) can determine the risk level of vehicle driving to be a fourth level if there is strong interference or panic inducement or interference with the driver's hand operation based on passenger behavior information.

[0877] For example, the processor (175) can determine the risk level of vehicle driving to be a first level if the driver is capable of operating the vehicle based on driver driving ability information.

[0878] Meanwhile, the processor (175) can determine the risk level of vehicle driving to be a second level based on driver driving ability information, when the driver is able to operate the vehicle but is slightly distracted.

[0879] Meanwhile, the processor (175) can determine the risk level of vehicle driving to be a third level based on driver driving ability information, if the driver is able to operate but there is a delay in reaction or increased stress.

[0880] Meanwhile, the processor (175) can determine the risk level of vehicle driving to be the fourth level if the driver is unable to operate the vehicle, is unresponsive, or is in a panic state based on driver driving ability information.

[0881] Meanwhile, the processor (175) can determine the risk level of the vehicle driving as the first level when driving while looking ahead, when there is no abnormality in the vital signs, or when stopping next to the U-turn line.

[0882] Meanwhile, the processor (175) can determine the risk level of driving the vehicle to be a second level if the vehicle is driven with one hand while holding a drink, the heart rate increases, or the vehicle stops after crossing the U-turn line.

[0883] Meanwhile, the processor (175) can determine the risk level of the vehicle driving to be the third level if there is a history of accidents such as sudden braking while driving with one hand holding a drink, or if there is a history of heart rate increase, irregular breathing, or past cardiac arrest, or if there is a stop beyond the U-turn line and an oncoming motorcycle.

[0884] Meanwhile, the processor (175) can determine the risk level of driving the vehicle to be the fourth level if, while startled by spilling a drink, the gaze is directed downward rather than forward and the hand is removed from the steering wheel, or if there is cardiac arrest and steering, or if there is no response or misoperation of the pedal, or if the vehicle stops beyond the U-turn line and an oncoming vehicle is coming.

[0885] Meanwhile, when the risk level of vehicle driving is the first level, the authority adjustment module (915), safety extension module (910), and sensitivity adjustment module (920) within the processor (175) can all remain in a non-intervention state.

[0886] Meanwhile, when the risk level of vehicle driving is at level 2, the authority adjustment module (915) within the processor (175) can learn and monitor operation patterns in a standby state, the safety extension module (910) can prepare to relax functional restrictions in a standby state, and the sensitivity adjustment module (920) can perform simple visual or auditory warnings by enhancing sensitivity.

[0887] Meanwhile, when the risk level of vehicle driving is at the third level, the authority adjustment module (915) within the processor (175) may ignore or begin to correct some misoperations, such as unintended errors in turning signals, and the safety extension module (910) may partially relax functional restrictions or gradually increase thresholds, and the sensitivity adjustment module (920) may further strengthen sensitivity, perform multiple warnings, perform adaptive warnings based on personalization, actively perform warnings in case of behavioral risk, or mitigate warnings in case of health risk.

[0888] Meanwhile, when the risk level of vehicle driving is at level 4, the authority adjustment module (915) within the processor (175) can immediately intervene to perform steering control or acceleration control, the safety extension module (910) can completely release functional restrictions or set the threshold to the maximum value, and the sensitivity adjustment module (920) can maintain heightened sensitivity or suppress warnings to prevent confusion. Accordingly, the situation inside the vehicle can be determined quickly and accurately to perform stable vehicle control.

[0889] FIG. 16 is an example of a flowchart illustrating a method of operation of a signal processing device according to an embodiment of the present disclosure.

[0890] Referring to the drawings, the processor (175) in the signal processing device (170) according to an embodiment of the present disclosure can obtain context data of the driver inside the vehicle based on the vehicle's internal sensor device or internal camera (195i), etc. (S1402).

[0891] Meanwhile, the processor (175) can obtain driver profile information or driver characteristic data based on the database (762) (S1403).

[0892] Meanwhile, the processor (175) can obtain context data of the occupant based on the vehicle's internal sensor device or internal camera (195i), etc. (S1404).

[0893] Meanwhile, the processor (175) can obtain vehicle external context data based on vehicle external sensor devices, etc. (S1405).

[0894] Meanwhile, the processor (175) can compute driving ability evaluation data based on the driver's context data, driver profile information or driver's characteristic data, passenger's context data, and vehicle external context data (S1410).

[0895] Next, the processor (175) can calculate emergency level evaluation data based on driving ability evaluation data (S1420).

[0896] Alternatively, unlike the drawing, the processor (175) can calculate emergency level evaluation data related to the safety level of vehicle driving based on the driver's context data, driver profile information or driver characteristic data, passenger's context data, and vehicle external context data.

[0897] Meanwhile, the processor (175) determines whether the emergency level evaluation data related to the safety level of vehicle driving is at the first level (S1422), and if so, can control it to remain as is without separate interference (S1420).

[0898] If not applicable in step 1422 (S1422), the processor (175) determines whether the emergency level evaluation data related to the safety level of vehicle driving is at the second level (S1424), and if applicable, performs a safety extension corresponding to the second level (S1431), performs an authority adjustment corresponding to the second level (S1432), and performs a sensitivity adjustment corresponding to the second level (S1433).

[0899] If not applicable in step 1442 (S1424), the processor (175) determines whether the emergency level evaluation data related to the safety level of vehicle driving is at the third level (S1426), and if applicable, performs a safety extension corresponding to the third level (S1434), performs an authority adjustment corresponding to the third level (S1435), and performs a sensitivity adjustment corresponding to the third level (S1436).

[0900] If not applicable in step 1426 (S1426), the processor (175) determines whether the emergency level evaluation data related to the safety level of vehicle driving is at level 4 (S1428), and if applicable, performs a safety extension corresponding to level 4 (S1437), performs an authority adjustment corresponding to level 4 (S1438), and performs a sensitivity adjustment corresponding to level 4 (S1439).

[0901] Accordingly, it becomes possible to perform adaptive vehicle control by rapidly and accurately determining the situation inside and outside the vehicle. In particular, it enables adaptive vehicle control by rapidly and accurately determining the situation inside and outside the vehicle based on multimodal context data.

[0902] Although preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above. Various modifications are possible by those skilled in the art without departing from the essence of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical spirit or perspective of the present disclosure.

Claims

1. A processor that receives external vehicle sensing data from external vehicle sensors and internal vehicle sensing data from internal vehicle sensors; and The above processor is, Based on the above vehicle external sensing data and the above vehicle internal sensing data, vehicle external situation information, vehicle internal situation information, and occupant situation information are collected, and A signal processing device that generates a prompt for calculating a risk level of vehicle driving based on each of the above situation information, classifies the risk level of vehicle driving into a plurality of levels based on an inference result or response result obtained based on the generated prompt, and controls the vehicle based on the classified risk levels.

2. In Paragraph 1, The above processor is, A signal processing device that classifies the risk level of the above vehicle driving into a first level corresponding to a normal situation, a second level corresponding to a caution situation, a third level corresponding to a danger precursor situation, and a fourth level corresponding to a danger situation.

3. In Paragraph 1, The above processor is, A signal processing device that generates a prompt for calculating the risk level of the vehicle driving based on the above-mentioned situation information and previously stored driving pattern information, classifies the risk level of the vehicle driving into a plurality of levels based on an inference result or response result obtained based on the generated prompt, and controls the vehicle based on the classified risk levels.

4. In Paragraph 1, The above processor is, A signal processing device that controls authority adjustment, safety extension, or sensitivity adjustment for vehicle control based on the above-described risk levels.

5. In Paragraph 4, The above processor is, Relaxing the allowable operating range during the above safety extension, or Performing authority adjustment between the driver and the system during the above authority adjustment, or A signal processing device that adjusts sensitivity based on the risk level during the above sensitivity adjustment.

6. In Paragraph 2, The above processor is, A signal processing device that increases the warning sensitivity of Autonomous Emergency Braking (AEB) or Lane Keeping Assist (LKA) when the risk level of the vehicle driving is the second level.

7. In Paragraph 2, The above processor is, If the risk level of the vehicle driving is the third level, gradually increase the allowable lateral acceleration, gradually increase the target distance for object tracking, or increase the allowable steering torque value, Gradually increase the accelerator pedal override threshold, or gradually increase the steering override torque, A signal processing device that controls to gradually increase the sensitivity to the course of a forward collision, gradually decrease the target speed according to road curvature, or more precisely follow the center of the lane.

8. In Paragraph 2, The above processor is, If the risk level of the vehicle driving is the fourth level, increase the allowable lateral acceleration, increase the target distance for object tracking, decrease the lane width limit, increase the allowable steering torque value, or Limiting acceleration based on accelerator pedal override, ignoring the condition for deactivating adaptive cruise control via the brake pedal, ignoring the overtaking function via the turn signal, increasing steering override torque, or ignoring the condition for deactivating the lane centering system (LCS) via the turn signal, A signal processing device that controls to increase the sensitivity of the forward collision course, decrease the target speed according to road curvature, or more precisely follow the center of the lane.

9. In Paragraph 2, The above processor is, When the risk level of the vehicle driving is at the fourth level, the first allowable range of acceleration / deceleration during Adaptive Cruise Control (ACC) and the longitudinal input value based on the input signal are compared, and If the above input value is included within the above first allowable range, the intervention level is lowered, and A signal processing device that increases the intervention level when the above input value deviates from the above first allowable range.

10. In Paragraph 2, The above processor is, When the risk level of the vehicle driving is at the fourth level, the second allowable range of acceleration / deceleration during Adaptive Cruise Control (ACC) and the lateral input value based on the input signal are compared, and If the above input value is included within the above second allowable range, the intervention level is lowered, and A signal processing device that increases the intervention level when the above input value exceeds the above second allowable range.

11. In Paragraph 1, The above processor is, If the longitudinal or lateral input is within the allowable range and there are no events in the external vehicle situation information, the intervention level is lowered, and If the above longitudinal input value or the above lateral input value is within the allowable range and there is an event requiring attention, the above intervention level is maintained, and A signal processing device that raises the intervention level when the above longitudinal input value or the above transverse input value is outside the allowable range and there is an event requiring attention.

12. In Paragraph 1, The above processor is, A signal processing device that determines the driver's proficiency by comparing the input value and the allowable range during the operation of Adaptive Cruise Control (ACC) or Lane Centering System (LCS).

13. In Paragraph 1, The above processor is, A signal processing device that compares an input value with an allowable range during the operation of Adaptive Cruise Control (ACC) or Lane Centering System (LCS), and raises the risk level of the vehicle's driving when the input value deviates from the allowable range.

14. In Paragraph 1, The above processor is, If the input value during the execution of Adaptive Cruise Control (ACC) or Lane Centering System (LCS) is within the allowable range and there are no events in the external vehicle situation information, the risk level of the vehicle driving is lowered, and When the input value during the execution of the above Adaptive Cruise Control (ACC) or Lane Centering System (LCS) is within the allowable range and there is an event requiring attention within the external vehicle situation information, the risk level of the vehicle driving is maintained, A signal processing device that raises the risk level of vehicle driving when the input value during the execution of the above Adaptive Cruise Control (ACC) or Lane Centering System (LCS) deviates from the allowable range and there is an event requiring attention.

15. In Paragraph 1, The above processor is, Based on the above-mentioned inference result or response result, the risk level of the vehicle driving is calculated, and based on the above-mentioned risk level, authority adjustment data for vehicle control, safety extension data for safety extension, and sensitivity adjustment data for sensitivity adjustment are calculated. A signal processing device that generates the vehicle control signal based on the above authority adjustment data, safety extension data, and sensitivity adjustment data.

16. In Paragraph 1, The above processor is, A signal processing device that calculates the risk level of the vehicle driving based on the inference result or response result obtained above, performs learning based on the calculated risk level and occupant profile information, and outputs the vehicle control signal, outputs a warning message, or outputs a guide message based on the result data of the learning.

17. In Paragraph 1, The above processor is, A signal processing device that classifies the risk level of the vehicle driving into a plurality of levels based on vehicle internal context data, vehicle external context data, occupant characteristic information, occupant behavior information, and the driver's driving ability information.

18. In Paragraph 1, Further comprising a neural processor that executes inference or a response based on the above prompt, The above processor is, Obtaining an inference result or response result executed in the neural processor based on the above prompt, and A signal processing device that calculates the risk level of the vehicle driving based on the above inference result or response result, and outputs a vehicle control signal based on the calculated risk level.

19. In Paragraph 1, The above processor is, Run a hypervisor, and on the hypervisor, run a driving control virtualization machine, and The above driving control virtualization machine is, Through shared memory within the hypervisor, camera data from outside the vehicle, camera data from inside the vehicle, and sensor data from inside the vehicle are received, and A signal processing device that executes a driving control service or a driving control application based on camera data outside the vehicle, camera data inside the vehicle, sensor data, and driving pattern information of the driver stored in the database.

20. A vehicle control device comprising a signal processing device according to any one of claims 1 to 19.

Images