Warning control device and warning control method

Abstract

The present invention comprises: an image acquisition unit (101) that acquires a captured image in which at least the face of the driver of a vehicle is captured; a facing-direction detection unit (102) that detects the driver's line-of-sight facing-direction or the driver's face facing-direction on the basis of the captured image; a target object detection unit (103) that detects whether there is a target object around the vehicle which could be a factor that causes the vehicle to run into an unexpected situation; and a first warning control unit (104) that, if it is determined that the target object is present and the driver is not facing in the direction of the target object, on the basis of the facing-direction detection result of the driver's line-of-sight facing-direction or the driver's face facing-direction together with the target object detection result of the target object, outputs, as first warning control information, display control information for causing a target display device to provide a display for calling attention to the target object. The target object display device is a display device (3) among a plurality of display devices (3) provided at mutually different locations in the vehicle interior, and is provided in the driver's line-of-sight facing-direction or the driver's face facing-direction detected by the facing-direction detection unit (102).

Description

Alarm control device and alarm control method

[0001] The present disclosure relates to an alarm control device and an alarm control method.

[0002] Conventionally, in a vehicle, when an object that may cause the vehicle to fall into an unexpected situation (hereinafter referred to as "target object"), such as an object that may come into contact with the vehicle, exists around the vehicle, an alarm control technique is known that issues an alarm to make the driver aware of the presence of the target object. As one of the alarm control techniques described above, for example, Patent Document 1 takes into account the case where the driver is looking at an object that may come into contact with the host vehicle but has not recognized the object, and when it is determined that the current situation is a dangerous situation with a possibility of collision with another object, a technique is disclosed for performing control to change the execution timing of support control such as the output timing of an alarm sound according to whether the driver has performed a preliminary action for avoiding a collision.

[0003] International Publication No. 2024 / 015863

[0004] In the alarm control technology as described above, when trying to make the driver recognize the presence of the target object through display, the driver cannot recognize the presence of the target object unless the driver is looking at the display device on which the display is presented, or the recognition of the presence of the target object is delayed, and as a result, there has been a problem that effective alarm control for notifying the driver of the presence of the target object may not be performed. Note that the technology disclosed in Patent Document 1 is a technology related to alarm control when the driver does not perform a preliminary action for avoiding a collision, and does not consider the problem when making the driver recognize the presence of an object that may come into contact with the host vehicle through display, and still does not solve the above-mentioned problem.

[0005] The present disclosure has been made to solve the above problems, and an object thereof is to provide an alarm control device capable of more effectively performing an alarm for notifying a driver of the presence of a target object through display than in the prior art.

[0006] The alarm control device of this disclosure includes an image acquisition unit that acquires an image capturing at least the face of the vehicle driver; a direction detection unit that detects the direction of the driver's gaze or face orientation based on the image capturing unit acquired by the image acquisition unit; an object detection unit that detects whether or not there is an object in the vicinity of the vehicle that could cause the vehicle to fall into an unforeseen situation; and a first alarm control unit that, based on the direction detection result of the direction detection unit for the driver's gaze or face orientation and the object detection result of the object detection unit for the object, determines that an object exists and the driver is not facing the object, and outputs display control information as first alarm control information to cause a target display device, which is a display device installed in the direction of the driver's gaze or face orientation detected by the direction detection unit among a plurality of display devices installed in different locations inside the vehicle, to display a message prompting attention to the object.

[0007] According to this disclosure, the alarm control device can more effectively provide a visual alarm to the driver informing them of the presence of an object than conventional systems.

[0008] This figure shows an example configuration of the alarm control device according to Embodiment 1. Figures 2A and 2B are diagrams illustrating an example of the outline of the first alarm control process by the first alarm control unit in Embodiment 1 when a target object is present but the driver is not facing the target object. This is a flowchart illustrating the operation of the alarm control device according to Embodiment 1. This is a flowchart illustrating in detail an example of the operation of the first alarm control process by the first alarm control unit of the alarm control device in step ST4 of Figure 3. This is a flowchart illustrating the operation of the alarm control device when second alarm control information is acquired from another vehicle in the alarm control device according to Embodiment 1. Figures 6A and 6B are diagrams illustrating an example of the hardware configuration of the alarm control device according to Embodiment 1.

[0009] The forms for implementing this disclosure will be described below with reference to the attached drawings.

[0010] Figure 1 shows an example of the configuration of the alarm control device 1 according to Embodiment 1. In Embodiment 1, it is assumed that the alarm control device 1 is mounted, for example, in a vehicle (not shown). The alarm control device 1 is connected to a camera 2 installed inside the vehicle and to a plurality of display devices 3 installed inside the vehicle. Note that in Figure 1, for the sake of simplicity, only one display device 3 is shown. Furthermore, the alarm control device 1 is capable of directly exchanging information with other vehicles via a communication network such as V2V (Vehicle to Vehicle).

[0011] Camera 2 is positioned inside the vehicle to capture at least the face of the vehicle's driver. Camera 2 may be shared with, for example, an imaging device in a so-called "Driver Monitoring System (DMS)" that is mounted in the vehicle to monitor the driver's condition inside the vehicle. Camera 2 is either an infrared camera or a visible light camera. In the following Embodiment 1, the vehicle's driver is also simply referred to as "driver". Camera 2 outputs the captured image of the driver to the alarm control device 1.

[0012] Multiple display devices 3 are provided in different locations within the vehicle interior. In Embodiment 1, the display device 3 includes various in-vehicle display devices capable of displaying information within the vehicle interior, such as a rearview mirror, side mirrors, meters, head-up display, LED display, liquid crystal display, and indicators. In Embodiment 1, "displaying" by the display device 3 includes, for example, displaying information such as icons on the display, or making an indicator provided on the display device 3 visually recognizable by lighting or flashing the indicator. The display device 3 performs display based on display control information output from the alarm control device 1. Details of the display control information output from the alarm control device 1 will be described later.

[0013] The alarm control device 1 according to Embodiment 1 detects the direction of the driver's gaze or face orientation based on the image captured by the camera 2. The alarm control device 1 also detects whether or not there is an object in the vicinity of the vehicle that could cause the vehicle to fall into an unforeseen situation. If the alarm control device 1 determines, for example, that an object exists and the driver is not facing the object, it outputs display control information as first alarm control information to cause one of the multiple display devices 3, which is located in the direction of the driver's gaze or face orientation, to display a message prompting the driver to pay attention to the object. In Embodiment 1, the alarm control device 1 warns the driver that an object exists by, for example, displaying a message prompting the driver to pay attention to the object on the target display device. In the following Embodiment 1, the vehicle in which the driver is the target of the alarm control device 1's warning, i.e., the vehicle in which the driver is the target of the

[0014] As shown in Figure 1, the alarm control device 1 comprises an image acquisition unit 101, a direction detection unit 102, a target object detection unit 103, a first alarm control unit 104, a behavior detection unit 105, a driving aptitude determination unit 106, a second alarm control unit 107, a vehicle control unit 108, an other vehicle alarm control information acquisition unit 109, and a cooperation control unit 110.

[0015] The image acquisition unit 101 acquires the captured image from the camera 2. The image acquisition unit 101 outputs the acquired captured image to the orientation detection unit 102.

[0016] The orientation detection unit 102 detects the direction of the driver's gaze or face orientation based on the captured image acquired by the image acquisition unit 101. In Embodiment 1, the process by which the orientation detection unit 102 detects the direction of the driver's gaze or face orientation is referred to as the orientation detection process. The orientation detection unit 102 can detect the direction of the driver's gaze or face orientation from the captured image acquired by the image acquisition unit 101 using a known method for detecting the direction of a person's gaze or face orientation captured on the image based on the image. Therefore, a detailed explanation of the method by which the orientation detection unit 102 detects the direction of the driver's gaze or face orientation based on the captured image is omitted here. For example, the orientation detection unit 102 can extract the driver's facial features (hereinafter referred to as "facial features") based on the captured image using a known image recognition technique, and then detect the direction of the driver's gaze or face orientation from the extracted facial features. Facial features include, for example, the distance between facial feature points such as the outer corner of the eye, the inner corner of the eye, and the eyelid, the height of the eyebrows, or the degree to which the eyes are open. The orientation detection unit 102 can detect the driver's gaze direction or face orientation by inputting extracted facial features into a model trained using a machine learning algorithm (hereinafter referred to as the "machine learning model"). The machine learning model is, for example, a model that takes facial features as input and outputs information indicating the gaze direction or face orientation.

[0017] In Embodiment 1, the direction of the driver's gaze or face orientation is represented, for example, by an angle with respect to a reference direction. In Embodiment 1, the reference direction is, for example, a straight line in real space passing through the center of the driver's head and a point directly in front of the center of the head. The point that will be the center of the head is predetermined. For example, it is predetermined that the center of the head will be the center of the eyes. Since the installation position and field of view of the camera 2 are known in advance, the orientation detection unit 102 can calculate the position of the driver's head center by extracting the driver's head center from the captured image. The position of the driver's head center is a point in real space and is represented, for example, by coordinate values ​​that can be mapped on a map. The direction of the driver's gaze or face orientation is represented, for example, by horizontal and vertical angles with respect to the above-mentioned reference direction. In detail, the direction of the driver's gaze or face is defined as the reference point (0 degrees) when the driver is facing directly in front of the vehicle's direction of travel. The left-right direction of the driver's gaze or face is represented by an angle that increases as the driver turns their gaze or face to the right relative to the vehicle's direction of travel from the forward-facing position, and decreases as the driver turns their gaze or face to the left relative to the vehicle's direction of travel from the forward-facing position. The up-down direction of the driver's gaze or face is represented by an angle that increases as the driver turns their gaze or face upward relative to the vehicle's direction of travel from the forward-facing position, and decreases as the driver turns their gaze or face downward relative to the vehicle's direction of travel from the forward-facing position. In Embodiment 1, "front" is not strictly limited to being directly in front, but includes approximately the front.

[0018] Furthermore, when the orientation detection unit 102 detects the direction of the driver's gaze or face orientation, it calculates the position that the driver is presumed to be looking at (hereinafter referred to as the "estimated viewing position") based on the detected direction of the driver's gaze or face orientation. For example, the estimated viewing position is expressed as the angle and distance from the position of the vehicle, in other words, in which direction and at what distance from the position of the vehicle. The estimated viewing position is a point in real space. In Embodiment 1, the point that will be considered the position of the vehicle is predetermined. For example, the position of the vehicle is a single point on the front of the vehicle. The installation position of the camera 2, the field of view of the camera 2, the direction of the driver's gaze or face orientation, and the positional relationship between the driver's head position, which serves as the reference for the direction of the driver's gaze or face orientation, and the installation position of the camera 2 are known. Also, the positional relationship between the position of the vehicle and the installation position of the camera 2 is known in advance. The orientation detection unit 102 can calculate the driver's estimated viewing position from the direction of the driver's gaze or face orientation detected based on the captured image. For example, the orientation detection unit 102 estimates the driver's viewing direction as the intersection of a virtual straight line indicating the viewing direction of the right eye and a virtual straight line indicating the viewing direction of the left eye, and calculates the coordinates of the estimated viewing position. The orientation detection unit 102 outputs information (hereinafter referred to as "orientation information") that associates the detected direction of the driver's gaze or face orientation with the calculated estimated viewing position of the driver to the first alarm control unit 104. The orientation detection unit 102 also stores the orientation information in chronological order in a storage unit (not shown). The storage unit (not shown) may be provided in the alarm control device 1, or it may be provided outside the alarm control device 1 in a location accessible to the alarm control device 1.

[0019] The object detection unit 103 detects whether or not there is an object in the vicinity of the vehicle that could cause the vehicle to fall into an unforeseen situation. In Embodiment 1, the process by which the object detection unit 103 detects whether or not an object exists is called the object detection process. In Embodiment 1, an object is an object that the driver needs to pay attention to in accordance with the driving environment of the vehicle, and which, if present, requires the driver to take some kind of action such as confirmation or avoidance. Objects include, for example, other vehicles, pedestrians, bicycles, motorcycles, animals, road obstacles such as fallen objects or construction equipment, utility poles, guardrails, or median strips.

[0020] For example, the target object detection unit 103 acquires surrounding situation information (hereinafter referred to as "surrounding situation information") from a surrounding situation collection device (not shown) that collects information about the surrounding conditions of the vehicle, and based on the acquired surrounding situation information, detects whether or not a target object exists around the vehicle according to pre-set target object detection conditions. The surrounding situation collection device includes, for example, an external camera (not shown), an external radar (not shown), an external ultrasonic sensor (not shown), a map database (not shown), or another vehicle (not shown). The external camera is an imaging device that images the surroundings of the vehicle. The target object detection unit 103 acquires, for example, an image captured by the external camera of the surroundings of the vehicle (hereinafter referred to as "external image") as surrounding situation information. The external radar is, for example, a millimeter-wave radar that detects objects present around the vehicle. The target object detection unit 103 acquires, for example, information about the object detected by the external radar as surrounding situation information. The ultrasonic sensor, for example, is installed on the side of the vehicle and measures the distance to other vehicles traveling in the adjacent lane. The object detection unit 103 acquires information indicating the distance to other vehicles traveling in the adjacent lane, measured by the ultrasonic sensor, as surrounding situation information. The object detection unit 103 also acquires information about objects present around other vehicles (hereinafter referred to as "other vehicle surrounding information") from other vehicles, for example, through vehicle-to-vehicle communication, as vehicle surrounding situation information. The object detection unit 103 also acquires map information from a map database, for example, as vehicle surrounding situation information.

[0021] The target object detection unit 103 calculates the position, speed, or direction of travel of objects, including other vehicles, present around the vehicle based on surrounding situation information acquired from the surrounding situation collection device. For example, the target object detection unit 103 can constantly update and calculate the distance to other vehicles ahead, the position of other vehicles in adjacent lanes, the speed of other vehicles approaching from behind, etc., based on the surrounding situation information. Then, after calculating the position, speed, or direction of travel of objects present around the vehicle, the target object detection unit 103 detects whether or not a target object exists around the vehicle according to the target object detection conditions. The target object detection conditions include, for example, the following conditions (1) or (2). <Condition (1)> If there is another vehicle ahead whose distance from the vehicle is below a preset threshold (hereinafter referred to as the "first threshold"), that other vehicle is considered a target object. <Condition (2)> If there is another vehicle in the adjacent lane whose distance from the vehicle is below a predetermined threshold (hereinafter referred to as the "second threshold"), that other vehicle shall be considered the target object.

[0022] The target object detection conditions are set as appropriate by the developers. For example, the developers set the target object detection conditions considering situations in which the presence of an object is presumed to indicate that the vehicle may be in danger of an unforeseen incident. When the target object detection unit 103 detects the presence of a target object, it is presumed that the vehicle may be in danger of an unforeseen incident. Note that the target object detection conditions described above are merely examples. Other conditions besides the above-mentioned <Condition (1)> and <Condition (2)> may also be set for the target object detection conditions. For example, the target object detection conditions may include conditions that take into account the speed of the vehicle and the speed of other vehicles, or conditions that take into account the type of target object (vehicle, pedestrian, obstacle, etc.). The target object detection unit 103 can obtain information on the speed of the vehicle from, for example, a vehicle speed sensor, and information on the speed of other vehicles from other vehicles via vehicle-to-vehicle communication. Furthermore, the target object detection unit 103 can determine the type of target object using, for example, known image recognition technology based on surrounding situation information.

[0023] The target object detection unit 103 outputs information indicating the result of detecting a target object (hereinafter referred to as "target object detection result") to the first alarm control unit 104. The target object detection result includes, for example, information indicating whether or not a target object has been detected, and, if a target object has been detected, information indicating the relative position or relative speed of the target object with respect to the vehicle. The target object detection result may further include, for example, information such as the type of target object (vehicle, pedestrian, obstacle, etc.), the size of the target object, or the direction of movement of the target object.

[0024] The first alarm control unit 104 performs a first alarm control process to output first alarm control information to a display device 3 (hereinafter referred to as "target display device") among a plurality of display devices 3 installed in different locations inside the vehicle, which is installed in the direction of the driver's gaze or face as detected by the direction detection unit 102, to display a message prompting attention to the target object. This information is based on the direction detection result of the direction detection unit 102, which detects the direction of the driver's gaze or face. In Embodiment 1, the first alarm control information is information for warning that a target object is present.

[0025] The first alarm control process by the first alarm control unit 104 will now be explained in detail. First, the first alarm control unit 104 determines whether or not the target object has been detected as present by the target object detection unit 103. Based on the target object detection result, the first alarm control unit 104 can determine whether or not the target object has been detected as present.

[0026] If the system detects the presence of a target object, the first alarm control unit 104 determines whether the driver is facing the target object. In Embodiment 1, "the driver is facing the target object" means that the driver is directing their gaze toward the target object. The first alarm control unit 104 can determine whether the driver is facing the target object based on whether the direction of the driver's gaze or face is directed toward the target object. For example, the first alarm control unit 104 determines that the driver is facing the target object if the target object is located within a range (hereinafter referred to as the "visibility determination range") that extends horizontally and vertically from the estimated viewing position by a predetermined distance relative to the direction of travel of the vehicle, centered on the estimated viewing position. On the other hand, if the first alarm control unit 104 determines that the driver is not facing the target object if the target object is not located within the visibility determination range. The first alarm control unit 104 can set the visibility determination range based on the direction detection result. Furthermore, the first alarm control unit 104 can determine the position of the target object based on the target object detection result, and can determine whether or not the target object is within the range for visual detection.

[0027] <When a target object exists and the driver is not facing the target object> If the first alarm control unit 104 determines that a target object exists and that the driver is not facing the target object, it outputs display control information as first alarm control information to cause the target display device to display a message prompting the driver to pay attention to the target object. For example, the driver's gaze direction or face direction may be associated with each display device 3 in advance by the developer or the like, and conditions (hereinafter referred to as "target display device determination conditions") may be set that define which display device 3 will be used as the target display device depending on the range within which the driver's gaze direction or face direction is. The first alarm control unit 104 simply needs to determine the target display device by comparing the driver's gaze direction or face direction with the target display device determination conditions. The first alarm control unit 104 then outputs the above display control information to the target display device. The display control information includes, for example, information for displaying an icon to draw attention to the target object, or information for making an indicator on the target display device visually recognizable by lighting or flashing the indicator. For example, it is predetermined which display device 3 is the target display device, in what form the display to draw attention to the target object will be displayed, or in which area of ​​the target display device the display to draw attention to the target object will be displayed. Furthermore, for example, the first alarm control unit 104 may change the display form of the display to draw attention to the target object depending on the type of target object or the position of the target object as seen from the vehicle.

[0028] <When a target object exists and the driver is facing the target object> On the other hand, if the first alarm control unit 104 determines that a target object exists and that the driver is facing the target object, it outputs display control information (hereinafter referred to as "suppressed display control information") as first alarm control information to cause one of the multiple display devices 3 to display a display prompting the driver to pay attention to the target object in a form that is more suppressed than the display based on the above-mentioned display control information, that is, in a form that is more suppressed than the display that is displayed when it is determined that the driver is not facing the target object, prompting the driver to pay attention to the target object. In Embodiment 1, a form that is more suppressed than the display based on the display control information includes, for example, reducing the size of the icon to be displayed, lowering the intensity of the light to be lit or flashed, or changing the color to a less conspicuous color. Also, for example, a form that is more suppressed than the display based on the display control information may be, for example, if the display of the icon based on the display control information is already being displayed, then not displaying it. Furthermore, since the direction of the driver's gaze or face relative to the target object differs depending on whether the driver is facing the target object or not, the target display device 3 used when it is determined that the driver is facing the target object is a different display device 3 than the target display device used when it is determined that the driver is not facing the target object.

[0029] Here, the first alarm control unit 104 outputs display control information or suppression display control information as first alarm control information for causing the target display device to display something. However, the output method of information prompting the driver to pay attention to the target object is not limited to display. For example, in addition to the display control information or suppression display control information, the first alarm control unit 104 may output audio output control information to an audio output device (not shown) as first alarm control information for causing the driver to output an audio prompting them to pay attention to the target object. Alternatively, it may output vibration control information to a vibration generator (not shown) as first alarm control information for generating vibrations to prompt the driver to pay attention to the target object. The first alarm control unit 104 only needs to output at least display control information or suppression display control information as first alarm control information.

[0030] <When the target object does not exist> If it is determined that the presence of the target object has not been detected, the first alarm control unit 104 does not need to output first alarm control information prompting the driver to pay attention to the target object.

[0031] Here, an example of the outline of the first alarm control process by the first alarm control unit 104 will be explained using drawings. Here, an example of the outline of the first alarm control process by the first alarm control unit 104 will be explained when a target object is present, but the driver is not facing the target object. Figures 2A and 2B are diagrams illustrating an example of the outline of the first alarm control process by the first alarm control unit 104 in Embodiment 1 when a target object is present, but the driver is not facing the target object. In Figures 2A and 2B, the driver is indicated by "D". In Figures 2A and 2B, the vehicle is assumed to be a right-hand drive vehicle as an example.

[0032] Figure 2A shows, as an example, a situation where, despite the presence of another vehicle (indicated as "Ta" in Figure 2A) ahead (see 201a in Figure 2A), the driver is checking a motorcycle behind them in the rearview mirror (see 202a in Figure 2A) and is not looking ahead. In this case, the first warning control unit 104 outputs display control information as first warning control information to the rearview mirror, which is an object display device located in the direction of the driver's gaze or face, to display a message prompting the driver to pay attention to the other vehicle Ta ahead. As a result, the rearview mirror displays an icon (indicated as "Aa" in Figure 2A) prompting the driver to pay attention to the object ahead, in this case the other vehicle Ta ahead (see 203a in Figure 2A).

[0033] Figure 2B shows, as an example, a situation where, despite the presence of another vehicle (indicated as "Tb" in Figure 2B) to the left rear (see 201b in Figure 2B), the driver is checking the right rear using the right side mirror (202b in Figure 2B) and not looking to the left rear. In Figure 2B, "G" indicates an area visible to the driver, and "N" indicates an area not visible to the driver. In this case, the first alarm control unit 104 outputs display control information as first alarm control information to the right side mirror, which is an object display device located in the direction of the driver's gaze or face, to display a warning about the other vehicle Tb to the left rear. As a result, an icon (indicated as "Ab" in Figure 2B) prompting the driver to pay attention to the object to the left rear, in this case the other vehicle Tb to the left rear, is displayed on the right side mirror (see 203b in Figure 2B).

[0034] For example, in the example in Figure 2B, if the driver can see the target object, which is another vehicle, in the left side mirror, it is determined that the driver is facing the direction of the target object, and suppression display control information is output to the left side mirror, which is the target display device. In Embodiment 1, "the driver is facing the direction of the target object" includes directing one's gaze towards the target object via the display device 3. That is, in Embodiment 1, "the driver is facing the direction of the object" includes directing one's gaze towards the object via the display device 3. For example, in the example in Figure 2A, it can be said that the driver is facing the direction of the motorcycle traveling behind them via the rearview mirror. For example, conditions (hereinafter referred to as "visibility display device determination conditions") that define which range the target object is within and which direction the driver's gaze or face is directed towards in order for it to be determined that the driver is facing the direction of the target object are pre-set by the developers, etc. The first alarm control unit 104 may, based on the direction detection result of the direction detection unit 102 for the direction of the driver's gaze or face, and the target object detection result of the target object detection unit 103 for the target object, determine whether the driver is looking in the direction of the target object or not, according to the conditions for determining the visibility display device, if the direction of the driver's gaze or face is not directly facing the direction of the target object, that is, if the target object is not within the visibility determination range.

[0035] Let us return to the explanation of the example configuration of the alarm control device 1 using Figure 1. When the first alarm control unit 104 outputs the first alarm control information, it outputs information to the action detection unit 105 indicating that the first alarm control information has been output (hereinafter referred to as "first alarm output information"). The first alarm output information includes, for example, information indicating the content of the first alarm control information (for example, whether it is display control information or suppression display control information, or what kind of icon etc. was displayed), information that can identify the output destination of the first alarm control information, i.e., the target display device (for example, an ID that has been assigned to each display device 3 in advance), the orientation detection result output from the orientation detection unit 102, and the target object detection result output from the target object detection unit 103.

[0036] The behavior detection unit 105 detects whether the driver has performed an action presumed to be an action to confirm the target object or an action to avoid the target object, after the first alarm control unit 104 has caused a display for warning purposes based on the display control information or suppression display control information as the first alarm control information.In Embodiment 1, the process by which the behavior detection unit 105 detects whether the driver has performed an action presumed to be an action to confirm the target object or an action to avoid the target object is called the behavior detection process.For example, the behavior detection unit 105 detects whether the driver has performed an action presumed to be an action to confirm the target object or an action to avoid the target object within a predetermined period after the first alarm control unit 104 has output the first alarm control information.For example, the behavior detection unit 105 may use the time at which it obtained the first alarm output information from the first alarm control unit 104 as the time at which the first alarm control unit 104 output the first alarm control information.

[0037] In Embodiment 1, the action of confirming the target object is assumed to be, for example, the direction in which the target object is located, or the movement of the gaze to a side mirror or rearview mirror from which the target object can be seen. For example, the action detection unit 105 can detect whether or not the driver has performed the action of confirming the target object based on the orientation information for the above-mentioned preset period, which is stored in a storage unit (not shown) by the orientation detection unit 102. Alternatively, for example, the action detection unit 105 can determine which side mirror or rearview mirror can be seen from which the target object can be seen based on the orientation information, the target object detection result, and the above-mentioned conditions for determining the visibility display device. Also, in Embodiment 1, the action of avoiding the target object is assumed to be, for example, steering wheel operation, braking operation, accelerator operation, shift position operation, hazard light operation, or turn signal operation for lane change. For example, the action detection unit 105 can acquire information indicating steering wheel operation, brake operation, shift position switching, and hazard lamp or turn signal illumination status from a steering wheel angle sensor (not shown), a brake sensor (not shown), an accelerator sensor (not shown), a shift position sensor (not shown), and a sensor (not shown) that detects the illumination status of hazard lamps or turn signals, and then detect whether the driver took evasive action towards the target object based on the sensor information.

[0038] The action detection unit 105 outputs information (hereinafter referred to as "action detection result") to the second alarm control unit 107 indicating whether or not the driver performed an action that is presumed to be an action to confirm the target object or an action to avoid the target object. The action detection result includes, for example, information indicating whether or not the driver performed an action that is presumed to be an action to confirm or avoid the object, information indicating the content of the action that the driver performed if the action was presumed to be an action to confirm or avoid the object, and first alarm output information output from the first alarm control unit 104.

[0039] The driving aptitude determination unit 106 determines the driver's driving aptitude based on the driver's state. In Embodiment 1, the process by which the driving aptitude determination unit 106 determines the driver's driving aptitude is called the driving aptitude determination process. In Embodiment 1, the driving aptitude determined by the driving aptitude determination unit 106 refers not to the driver's inherent driving ability, but to the degree to which the driver is in a state suitable for driving at the time of determination. In Embodiment 1, the driver's state that forms the basis for determining the driver's driving aptitude includes, for example, the degree of drowsiness, the degree of distraction, or age. For example, the driving aptitude determination unit 106 can determine the driver's state based on the captured image of the driver taken by the camera 2 using a known image recognition process or a machine learning model (hereinafter referred to as the "state determination model"). The state determination model is, for example, a model that takes an captured image as input and outputs information indicating the state of the person captured in the captured image. The driver aptitude determination unit 106 can then determine the driver's driving aptitude using either pre-set driving aptitude determination conditions or a machine learning model (hereinafter referred to as the "driving aptitude determination model"). The driving aptitude determination conditions are defined by the developers, etc., specifying what level of driving aptitude is expected under different driver conditions. The driving aptitude determination model, for example, takes information indicating the driver's condition as input and outputs information indicating the driver's driving aptitude. Driving aptitude may be classified into multiple levels, for example, or expressed as "high" or "low." The driver aptitude determination unit 106 outputs the determined information regarding the driver's driving aptitude (hereinafter referred to as "driving aptitude information") to the second alarm control unit 107 and the vehicle control unit 108. The driving aptitude information includes information indicating the driver's driving aptitude. The driving aptitude information may further include information indicating the driver's condition.

[0040] The second alarm control unit 107 performs a second alarm control process, outputting information to the target display device to prompt a warning based on display control information or suppression display control information as first alarm control information (hereinafter referred to as "level change instruction information"), or information to prompt a warning in a way different from the warning based on display control information or suppression display control information as first alarm control information (hereinafter referred to as "alternative alarm control information"), based on the detection result by the action detection unit 105 of whether or not the driver has performed an action that is presumed to be an action that confirms the target object or an action that is presumed to be an action that avoids the target object.

[0041] <When neither confirmation action nor avoidance action is performed> For example, if the action detection unit 105 detects that the driver has not performed either an action presumed to be a confirmation action or an avoidance action, the second alarm control unit 107 outputs level change instruction information to raise the level of the display that prompts a warning based on the display control information or suppression display control information as first alarm control information for the target display device, or the above-mentioned separate alarm control information, as second alarm control information.

[0042] In Embodiment 1, raising the level of the warning display means, for example, changing the currently used warning display—in this case, the display of an icon or the like on the target display device based on the display control information or suppression display control information as first alarm control information by the first alarm control unit 104—to a more attention-grabbing form. Specifically, raising the level of the above display means, for example, changing the color of the display currently displayed based on the display control information or suppression display control information to a more noticeable color, increasing the intensity of the light that illuminates or flashes the indicator, shortening the flashing interval of the indicator, or increasing the size of the display. The second alarm control unit 107 outputs level change instruction information as second alarm control information to the target display device, for example, to change the color of the display to a more noticeable color, to increase the intensity of the light that illuminates or flashes the indicator, to shorten the flashing interval of the indicator, or to increase the size of the display. Furthermore, if the first alarm control unit 104 is outputting first alarm control information that prompts the driver to pay attention to the target object in a form other than display, the second alarm control unit 107 may increase the level of the warning output in accordance with that output form. For example, if the first alarm control unit 104 is outputting audio output control information as first alarm control information for outputting a warning sound, the second alarm control unit 107 may output level change instruction information to increase the volume as second alarm control information, or if the first alarm control unit 104 is outputting vibration control information as first alarm control information for generating a warning vibration, the second alarm control unit 107 may output level change instruction information to increase the vibration as second alarm control information.

[0043] Furthermore, in Embodiment 1, providing a warning in a method other than the warning based on display control information or suppression display control information means, for example, outputting information other than display control information (for example, audio output control information that outputs sound, or vibration control information that generates vibration) to prompt the driver to pay attention to the target object, in addition to the current warning display, or outputting information to other vehicles to prompt them to pay attention to the vehicle itself. For example, the second alarm control unit 107 outputs audio output control information or vibration control information that prompts attention to the target object, which is separate alarm control information, as second alarm control information to an audio output device or vibration generator (not shown). Alternatively, for example, the second alarm control unit 107 may output information that prompts other vehicles to pay attention to the vehicle itself, which is separate alarm control information, as second alarm control information via vehicle-to-vehicle communication. Note that the separate alarm control information includes, for example, the target object detection result output from the target object detection unit 103. The second alarm control unit 107 can, for example, obtain the target object detection result from the first alarm control unit 104 via the action detection unit 105.

[0044] <When a confirmation or avoidance action is taken> On the other hand, if the second alarm control unit 107 detects, for example, that the driver has taken an action that is presumed to be a confirmation or avoidance action, it may output level change instruction information as second alarm control information to lower the level of the warning display based on the display control information or suppression display control information as first alarm control information for the target display device. In Embodiment 1, lowering the level of the warning display means, for example, changing the currently displayed warning display, in this case the display of an icon or the like on the target display device based on the display control information or suppression display control information as first alarm control information by the first alarm control unit 104, to a less conspicuous form, or deleting the display. For example, suppose the first alarm control unit 104 determines that the driver is not looking in the direction of the target object, and as a result of outputting the first alarm control information as display control information to the target display device, the driver turns their gaze in the direction of the target object. The action detection unit 105 then detects that the driver has performed an action that is presumed to be a confirmation action, based on the direction of the driver's gaze or the direction of their face. In this case, the second alarm control unit 107 outputs level change instruction information to delete the display of icons, etc., that were displayed on the target display device, more specifically, the display device 3 that has displayed a message prompting the driver to pay attention to the target object based on the first alarm control information, and which is located in a direction where the target object is not present at the time of the display. If, for example, the first alarm control unit 104 has output the first alarm control information prompting the driver to pay attention to the target object in a form other than a display, the second alarm control unit 107 should lower the level of the warning output in accordance with that output form. For example, if the first alarm control unit 104 outputs audio output control information as first alarm control information for outputting a warning sound, the second alarm control unit 107 may output level change instruction information to lower the volume as second alarm control information. Alternatively, if the first alarm control unit 104 outputs vibration control information as first alarm control information for generating a warning vibration, the second alarm control unit 107 may output level change instruction information to weaken the vibration as second alarm control information.

[0045] <Second Alarm Control Processing Considering Driving Aptitude> For example, the second alarm control unit 107 may output the level change instruction information or the separate alarm control information as the second alarm control information as described above, based on the driver's driving aptitude determined by the driving aptitude determination unit 106. For example, driving aptitude alarm conditions are defined in advance by a developer or the like, which define, for a given level of the driver's driving aptitude, how to change the level of alerting based on the display control information or the suppression display control information as the first alarm control information for the target display device (e.g., whether to raise or lower the alert level), or whether to prompt other vehicles to alert to the host vehicle. The second alarm control unit 107 may output the level change instruction information or the separate alarm control information as the second alarm control information based on the driving aptitude information output from the driving aptitude determination unit 106 and the driving aptitude alarm conditions.

[0046] When the driving aptitude determination unit 106 determines that the driver's driving aptitude is low, the vehicle control unit 108 outputs vehicle control information for causing the host vehicle to perform vehicle control that prevents the occurrence of an unexpected situation. In the first embodiment, the process of outputting the vehicle control information by the vehicle control unit 108 is referred to as vehicle control processing. Note that, for example, when the driver's driving aptitude is classified into a plurality of levels, the condition for determining that the driving aptitude is low at a certain level of the driver's driving aptitude is determined in advance. More specifically, the vehicle control information includes, for example, information for causing a driving control device (not shown) that performs driving assistance such as automated driving to apply automatic brakes, information for performing steering operation support that assists the driver in avoidance steering, information for performing follow-up traveling while maintaining an appropriate inter-vehicle distance with a preceding vehicle, or information for issuing an alarm when the host vehicle is about to depart from the lane and performing assistance to return the vehicle into the lane as necessary.

[0047] The other-vehicle alarm control information acquisition unit 109 acquires the second alarm control information output from another vehicle via inter-vehicle communication. The other-vehicle alarm control information acquisition unit 109 outputs the acquired second alarm control information to the cooperative control unit 110.

[0048] When the other-vehicle alarm control information acquisition unit 109 acquires the second alarm control information from another vehicle, the cooperative control unit 110 outputs cooperative control information that is information for avoiding a target object. In the first embodiment, the process of outputting the cooperative control information by the cooperative control unit 110 is referred to as a cooperative control process. The cooperative control information is, for example, information for causing a driver to perform an avoidance action on a target object, or information for causing the host vehicle to perform vehicle control for avoiding the target object. For example, when the cooperative control information is information for causing the driver to perform an avoidance action on the target object, the cooperative control unit 110 outputs the cooperative control information to the display device 3, and causes the display device 3 to display an icon or the like that prompts the driver to pay attention to the target object. At this time, for example, the cooperative control unit 110 may identify a target display device provided in the direction of the driver's line of sight or the direction of the driver's face from orientation information stored in a storage unit (not shown), and output the cooperative control information to the identified target display device. For example, when the cooperative control information is information for causing the host vehicle to perform vehicle control for avoiding the target object, the cooperative control unit 110 outputs, to a driving control device (not shown), information for applying an automatic brake, information for performing steering operation support that assists the driver's avoidance steering, information for performing follow-up traveling while maintaining an appropriate inter-vehicle distance with a preceding vehicle, or information for issuing a warning when the host vehicle is about to depart from the lane and performing assistance to return the host vehicle into the lane as necessary.

[0049] To give a specific example, suppose the other vehicle warning control information acquisition unit 109 acquires second warning control information as separate warning control information from another vehicle traveling in front of its own vehicle (hereinafter referred to as "the vehicle in front"), indicating that a target object exists in front of the vehicle in front. In this case, the cooperation control unit 110 outputs cooperation control information to avoid the target object present in front of the vehicle in front. For example, the cooperation control unit 110 outputs cooperation control information to display an icon or the like on the target display device to prompt the driver to pay attention to the target object present in front of the vehicle in front. Alternatively, for example, the cooperation control unit 110 outputs cooperation control information to a driving control device (not shown) to cause it to perform vehicle control to avoid the target object present in front of the vehicle in front.

[0050] Furthermore, the cooperation control unit 110 may, for example, output the cooperation control information and also output the second warning control information output from another vehicle to another vehicle. To explain using the specific example described above, for example, the cooperation control unit 110 may output the second warning control information output from the vehicle in front to another vehicle traveling behind its own vehicle (hereinafter referred to as the "rear vehicle") via vehicle-to-vehicle communication.

[0051] The operation of the alarm control device 1 according to Embodiment 1 will now be described. Figure 3 is a flowchart illustrating the operation of the alarm control device 1 according to Embodiment 1. Figure 3 is a flowchart illustrating the operation of the alarm control device 1 when it detects a target object. For example, when the engine of the vehicle is turned on, the alarm control device 1 starts the operation shown in the flowchart of Figure 3, and repeats the operation shown in the flowchart of Figure 3 until the engine of the vehicle is turned off.

[0052] The image acquisition unit 101 acquires an image captured from the camera 2 (step ST1). The image acquisition unit 101 outputs the acquired image captured to the orientation detection unit 102.

[0053] The orientation detection unit 102 performs orientation detection processing to detect the direction of the driver's gaze or face orientation based on the captured image acquired by the image acquisition unit 101 in step ST1 (step ST2). The orientation detection unit 102 outputs the orientation information to the first alarm control unit 104. The orientation detection unit 102 also stores the orientation information in chronological order in a storage unit (not shown).

[0054] The object detection unit 103 performs object detection processing to detect whether or not there is an object in the vicinity of the vehicle that could cause the vehicle to fall into an unforeseen situation (step ST3). The object detection unit 103 outputs the object detection result to the first alarm control unit 104.

[0055] The first alarm control unit 104 performs a first alarm control process (step ST4) in which it outputs first alarm control information to cause one of the multiple display devices 3 located in different places inside the vehicle to display a message prompting the driver to pay attention to the target object, based on the direction detection result of the direction detection unit 102 in step ST2 and the target object detection result of the target object detection unit 103 in step ST3.

[0056] In step ST4, the action detection unit 105 performs an action detection process to detect whether the driver performed an action that is presumed to be an action to confirm the target object or an action to avoid the target object, after the first alarm control unit 104 has caused a display for warning based on the display control information as the first alarm control information (step ST5). The action detection unit 105 outputs the action detection result to the second alarm control unit 107.

[0057] The driver aptitude determination unit 106 performs a driver aptitude determination process to determine the driver's driving aptitude based on the driver's condition (step ST6). The driver aptitude determination unit 106 outputs the driver aptitude information to the second alarm control unit 107 and the vehicle control unit 108.

[0058] The second alarm control unit 107 performs a second alarm control process (step ST7) in which, based on the detection result in step ST5 by the action detection unit 105 of whether or not the driver performed an action that is presumed to be an action to confirm the target object or an action to avoid the target object, it outputs as second alarm control information level change instruction information to change the level of the display that prompts a warning based on the display control information or suppression display control information as first alarm control information for the target display device, or alternative alarm control information to prompt a warning in a way different from the warning based on the display control information or suppression display control information as first alarm control information. Alternatively, in the second alarm control process, for example, the second alarm control unit 107 may output as second alarm control information level change instruction information or alternative alarm control information based on the driver's driving aptitude determined by the driving aptitude determination unit 106 in step ST6.

[0059] If the vehicle control unit 108 determines in step ST6 that the driver's driving ability is low, it performs vehicle control processing (step ST8) to output vehicle control information to cause the vehicle to perform vehicle control to prevent unforeseen incidents.

[0060] Figure 4 is a flowchart illustrating in detail an example of the operation of the first alarm control process by the first alarm control unit 104 of the alarm control device 1 in step ST4 of Figure 3.

[0061] The first alarm control unit 104 first determines whether or not the target object has been detected by the target object detection unit 103 (step ST11). If it is determined that the target object does not exist (if the result of step ST11 is "NO"), the alarm control device 1 skips the processing in steps ST12 to ST14 and then terminates the operation as shown in the flowchart of Figure 4.

[0062] If it is determined that the target object is present (if the answer is "YES" in step ST11), the first alarm control unit 104 determines whether the driver is facing the direction of the target object (step ST12).

[0063] If the first alarm control unit 104 determines that the driver is not facing the target object (in the case of "NO" in step ST12), it outputs display control information as first alarm control information to cause the target display device to display a message prompting attention to the target object (step ST14).

[0064] On the other hand, if the first alarm control unit 104 determines that the driver is facing the target object (in the case of "YES" in step ST12), it outputs suppressed display control information as first alarm control information to cause one of the multiple display devices 3 to display a warning about the target object in a form that is more suppressed than the display based on the above display control information, that is, in a form that is more suppressed than the display that is displayed when it is determined that the driver is not facing the target object to warn the driver about the target object (step ST13).

[0065] In step ST14, the first alarm control unit 104 may output audio output control information or vibration control information in addition to display control information. Also, in step ST13, the first alarm control unit 104 may output suppression audio output control information or suppression vibration control information in addition to suppression display control information.

[0066] Furthermore, while the flowchart in Figure 3 assumes that steps ST1 to ST2 and step ST3 are performed in parallel, this is merely an example. For example, step ST3 may be performed after steps ST1 to ST2, or vice versa. It is sufficient that steps ST1 to ST3 are completed before step ST4 is performed. Also, while the flowchart in Figure 3 assumes that step ST6 is performed after step ST5, this is merely an example. For example, the order of steps ST5 and ST6 may be reversed, or steps ST5 and ST6 may be performed in parallel. Furthermore, while the flowchart in Figure 3 assumes that step ST8 is performed after step ST7, this is merely an example. For example, the order of steps ST7 and ST8 may be reversed, or steps ST7 and ST8 may be performed in parallel.

[0067] Figure 5 is a flowchart illustrating the operation of the alarm control device 1 when it acquires second alarm control information from another vehicle in the alarm control device 1 according to Embodiment 1. For example, the operation of the alarm control device 1 as described using the flowchart in Figure 5 is performed in parallel with the operation of the alarm control device 1 as described using the flowchart in Figure 3.

[0068] The other vehicle warning control information acquisition unit 109 waits until the other vehicle outputs the second warning control information (if "NO" is selected in step ST101), and when the other vehicle outputs the second warning control information (if "YES" is selected in step ST101), it acquires the second warning control information output from the other vehicle via vehicle-to-vehicle communication. The other vehicle warning control information acquisition unit 109 outputs the acquired second warning control information to the cooperation control unit 110.

[0069] If the other vehicle warning control information acquisition unit 109 acquires second warning control information from another vehicle in step ST101, the cooperation control unit 110 performs cooperation control processing to output cooperation control information, which is information for avoiding the target object (step ST102).

[0070] In the above embodiment 1, the alarm control device 1 is configured to include an action detection unit 105, a driver aptitude determination unit 106, a second alarm control unit 107, a vehicle control unit 108, an other vehicle alarm control information acquisition unit 109, and a linkage control unit 110, but this is merely one example. It is not essential that the alarm control device 1 includes the action detection unit 105, the driver aptitude determination unit 106, the second alarm control unit 107, the vehicle control unit 108, the other vehicle alarm control information acquisition unit 109, and the linkage control unit 110, and the alarm control device 1 may be configured without the action detection unit 105, the driver aptitude determination unit 106, the second alarm control unit 107, the vehicle control unit 108, the other vehicle alarm control information acquisition unit 109, or the linkage control unit 110. If the alarm control device 1 is configured without the action detection unit 105, the processing of step ST5 can be omitted in the operation of the alarm control device 1 described using the flowchart in Figure 3. If the alarm control device 1 is configured not to include a driver aptitude determination unit 106, the process of step ST6 can be omitted in the operation of the alarm control device 1 as described using the flowchart in Figure 3. If the alarm control device 1 is configured not to include a second alarm control unit 107, the process of step ST7 can be omitted in the operation of the alarm control device 1 as described using the flowchart in Figure 3. If the alarm control device 1 is configured not to include a vehicle control unit 108, the process of step ST8 can be omitted in the operation of the alarm control device 1 as described using the flowchart in Figure 3. If the alarm control device 1 is configured not to include an other vehicle alarm control information acquisition unit 109 and a cooperation control unit 110, the operation of the alarm control device 1 as described using the flowchart in Figure 5 can be omitted. The alarm control device 1 is configured to include at least an image acquisition unit 101, a direction detection unit 102, a target object detection unit 103, and a first alarm control unit 104, and at least steps ST1 to ST4 can be performed in the operation of the alarm control device 1 as described using the flowchart in Figure 3.

[0071] Furthermore, in the above embodiment 1, the target object detection unit 103 may be configured to detect multiple target objects. When the target object detection unit 103 detects multiple target objects, the first alarm control unit 104 may output, for example, display control information or suppression display control information to prompt attention to each target object as first alarm control information. For example, in the alarm control device 1, when the first alarm control unit 104 determines that multiple target objects exist and the driver is not facing the direction of the target objects, it may output, as first alarm control information, display control information to cause the target display device to display icons or the like to prompt attention to each of the multiple target objects. In this case, for example, multiple icons or the like to prompt attention to the driver will be displayed on the target display device.

[0072] Furthermore, in the above embodiment 1, the alarm control device 1 is an in-vehicle device mounted on a vehicle, and the image acquisition unit 101, orientation detection unit 102, target object detection unit 103, first alarm control unit 104, action detection unit 105, driving aptitude determination unit 106, second alarm control unit 107, vehicle control unit 108, other vehicle alarm control information acquisition unit 109, and cooperation control unit 110 are provided on the in-vehicle device. However, it is not limited to this, and some of the image acquisition unit 101, orientation detection unit 102, target object detection unit 103, first alarm control unit 104, action detection unit 105, driving aptitude determination unit 106, second alarm control unit 107, vehicle control unit 108, other vehicle alarm control information acquisition unit 109, and cooperation control unit 110 may be provided on the in-vehicle device, while the others may be provided on a server connected to the in-vehicle device via a network. Furthermore, the image acquisition unit 101, orientation detection unit 102, target object detection unit 103, first alarm control unit 104, action detection unit 105, driving aptitude determination unit 106, second alarm control unit 107, vehicle control unit 108, other vehicle alarm control information acquisition unit 109, and cooperation control unit 110 may all be provided on a server.

[0073] Figures 6A and 6B show an example of the hardware configuration of the alarm control device 1 according to Embodiment 1. In Embodiment 1, the functions of the image acquisition unit 101, the orientation detection unit 102, the target object detection unit 103, the first alarm control unit 104, the action detection unit 105, the driving aptitude determination unit 106, the second alarm control unit 107, the vehicle control unit 108, the other vehicle alarm control information acquisition unit 109, and the cooperation control unit 110 are realized by the processing circuit 1001. That is, the alarm control device 1 includes a processing circuit 1001 for performing alarm control that notifies the driver of the presence of a target object by display. The processing circuit 1001 may be dedicated hardware as shown in Figure 6A, or it may be a processor 1004 that executes a program stored in memory 1005 as shown in Figure 6B.

[0074] If the processing circuit 1001 is dedicated hardware, it may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof.

[0075] When the processing circuit is a processor 1004, the functions of the image acquisition unit 101, the orientation detection unit 102, the target object detection unit 103, the first alarm control unit 104, the action detection unit 105, the driving aptitude determination unit 106, the second alarm control unit 107, the vehicle control unit 108, the other vehicle alarm control information acquisition unit 109, and the cooperation control unit 110 are realized by software, firmware, or a combination of software and firmware. The software or firmware is written as a program and stored in memory 1005. The processor 1004 reads and executes the program stored in memory 1005 to perform the functions of the image acquisition unit 101, the orientation detection unit 102, the target object detection unit 103, the first alarm control unit 104, the action detection unit 105, the driving aptitude determination unit 106, the second alarm control unit 107, the vehicle control unit 108, the other vehicle alarm control information acquisition unit 109, and the cooperation control unit 110. In other words, the alarm control device 1 includes a memory 1005 for storing a program that, when executed by the processor 1004, will result in the execution of steps ST1 to ST8 in Figure 3, or steps ST101 to ST102 in Figure 5. The program stored in the memory 1005 can also be said to cause the computer to execute the processing procedures or methods of the image acquisition unit 101, the orientation detection unit 102, the target object detection unit 103, the first alarm control unit 104, the action detection unit 105, the driving aptitude determination unit 106, the second alarm control unit 107, the vehicle control unit 108, the other vehicle alarm control information acquisition unit 109, and the cooperation control unit 110. Here, memory 1005 refers to non-volatile or volatile semiconductor memory such as RAM, ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), magnetic disks, flexible disks, optical disks, compact disks, minidiscs, DVDs (Digital Versatile Discs), etc.

[0076] Furthermore, the functions of the image acquisition unit 101, the orientation detection unit 102, the target object detection unit 103, the first alarm control unit 104, the action detection unit 105, the driving aptitude determination unit 106, the second alarm control unit 107, the vehicle control unit 108, the other vehicle alarm control information acquisition unit 109, and the cooperation control unit 110 may be partially implemented by dedicated hardware and partially by software or firmware. For example, the functions of the image acquisition unit 101 and the other vehicle alarm control information acquisition unit 109 can be implemented by a processing circuit 1001 as dedicated hardware, while the functions of the orientation detection unit 102, the target object detection unit 103, the first alarm control unit 104, the action detection unit 105, the driving aptitude determination unit 106, the second alarm control unit 107, the vehicle control unit 108, and the cooperation control unit 110 can be implemented by the processor 1004 reading and executing a program stored in the memory 1005. The storage unit, which is not shown, is composed of, for example, memory. Furthermore, the alarm control device 1 includes a camera 2, a display device 3, an audio output device (not shown), or a vibration generator (not shown), and an input interface device 1002 and an output interface device 1003 for wired or wireless communication.

[0077] As described above, the alarm control device 1 according to Embodiment 1 is configured to include an image acquisition unit 101 that acquires an image of at least the face of the vehicle driver, a direction detection unit 102 that detects the direction of the driver's gaze or face orientation based on the image acquired by the image acquisition unit 101, an object detection unit 103 that detects whether or not there is an object in the vicinity of the vehicle that could cause the vehicle to fall into an unforeseen situation, and a first alarm control unit 104 that, based on the direction detection result of the direction of the driver's gaze or face orientation by the direction detection unit 102 and the object detection result of the object detection unit 103, determines that an object exists and the driver is not facing the object, and outputs display control information as first alarm control information to cause a target display device, which is a display device 3 located in the direction of the driver's gaze or face orientation detected by the direction detection unit 102 among a plurality of display devices 3 located in different places inside the vehicle, to display a message prompting attention to the object. Therefore, the alarm control device 1 can more effectively provide a display-based alarm to the driver informing them of the presence of the target object than in conventional systems. As a result, the alarm control device 1 can prevent unforeseen incidents from occurring.

[0078] Furthermore, the alarm control device 1 may be configured to include an action detection unit 105 that detects whether the driver has performed an action presumed to be an action to confirm the target object or an action to avoid the target object, after the first alarm control unit 104 has caused a display for warning purposes based on the display control information as first alarm control information, and a second alarm control unit 107 that outputs level change instruction information to change the level of the display based on the display control information as first alarm control information to the target display device, or alternative alarm control information to cause warnings in a way other than the display based on the display control information, as second alarm control information, based on the detection result by the action detection unit 105 of whether the driver has performed an action presumed to be an action to confirm the target object or an action to avoid the target object.

[0079] For example, in the alarm control device 1, if the action detection unit 105 detects that the driver has not performed any actions presumed to be confirmation actions or avoidance actions, the second alarm control unit 107 may be configured to output level change instruction information, or separate alarm control information, as second alarm control information, which causes the level of display based on the display control information as first alarm control information for the target display device. This allows the alarm control device 1 to make the driver aware of the seriousness of the situation that could lead to a more unforeseen incident if the warning from the previous display was insufficient.

[0080] Furthermore, for example, in the alarm control device 1, the second alarm control unit 107 may be configured to output level change instruction information as second alarm control information, which lowers the display level based on the display control information as first alarm control information for the target display device, when the action detection unit 105 detects that the driver has performed an action that is presumed to be a confirmation action or an avoidance action. This allows the alarm control device 1 to reduce the annoyance of the driver caused by the warning, when it is presumed that the driver has become aware of the possibility of an unforeseen situation due to the warning from the aforementioned display.

[0081] Furthermore, in the alarm control device 1, if the first alarm control unit 104 determines, based on the orientation detection result and the target object detection result, that a target object exists and the driver is facing the target object, it outputs suppressed display control information as first alarm control information to the target display device among the multiple display devices 3, which displays a warning about the target object in a more suppressed form than the display based on the display control information. As a result, the alarm control device 1 can consider the possibility that the driver is aware of the target object and prompt the driver to pay attention to the target object in a display form that does not cause inconvenience to the driver.

[0082] Furthermore, in the alarm control device 1, the separate alarm control information may include information that prompts other vehicles to pay attention to the vehicle (the own vehicle), and the second alarm control unit 107 may be configured to output the separate alarm control information as the second alarm control information to other vehicles. This allows the alarm control device 1 to reduce the possibility of unforeseen incidents occurring for other vehicles in the vicinity of the own vehicle.

[0083] Furthermore, the alarm control device 1 may include a driving aptitude determination unit 106 that determines the driver's driving suitability based on the driver's state, including the degree of drowsiness, the degree of distraction, or age, and the second alarm control unit 107 may be configured to output level change instruction information or other alarm control information as second alarm control information based on the driver's driving suitability determined by the driving aptitude determination unit 106. This allows the alarm control device 1 to take the driver's driving suitability into consideration and make the driver aware of the seriousness of situations that could lead to more unforeseen circumstances.

[0084] Furthermore, the warning control device 1 may also include a vehicle control unit 108 that outputs vehicle control information to the vehicle (its own vehicle) to prevent the occurrence of unforeseen incidents if the driver aptitude determination unit 106 determines that the driver's aptitude is low. This allows the warning control device 1 to take the driver's aptitude into consideration and prevent unforeseen incidents from occurring.

[0085] Furthermore, the alarm control device 1 may be configured to include a vehicle alarm control information acquisition unit 109 that acquires second alarm control information output from other vehicles, and a cooperation control unit 110 that, when the vehicle alarm control information acquisition unit 109 acquires second alarm control information from other vehicles, outputs cooperation control information to cause the driver to take evasive action towards the target object, or to cause the vehicle (its own vehicle) to perform vehicle control to avoid the target object. This allows the alarm control device 1 to share information with other vehicles and prevent unforeseen incidents from occurring.

[0086] Furthermore, in the alarm control device 1, the coordination control unit 110 may be configured to output coordination control information and also to output second alarm control information output from other vehicles to other vehicles. This allows the alarm control device 1 to share information with other vehicles and prevent unforeseen incidents from occurring.

[0087] Furthermore, this disclosure allows for modifications of any component of the embodiment, or the omission of any component of the embodiment.

[0088] The alarm control device described herein can more effectively provide a display-based alarm to the driver informing them of the presence of an object than conventional devices.

[0089] 1. Alarm control device, 101 Image acquisition unit, 102 Orientation detection unit, 103 Target object detection unit, 104 First alarm control unit, 105 Action detection unit, 106 Driving aptitude determination unit, 107 Second alarm control unit, 108 Vehicle control unit, 109 Other vehicle alarm control information acquisition unit, 110 Coordination control unit, 2. Camera, 3. Display device, 1001 Processing circuit, 1002 Input interface device, 1003 Output interface device, 1004 Processor, 1005 Memory.

Claims

1. An alarm control device comprising: an image acquisition unit that acquires an image of at least the face of a vehicle driver; a direction detection unit that detects the direction of the driver's gaze or face orientation based on the image acquired by the image acquisition unit; an object detection unit that detects whether or not there is an object in the vicinity of the vehicle that could cause the vehicle to fall into an unforeseen situation; and a first alarm control unit that, based on the direction detection result of the direction of the driver's gaze or face orientation by the direction detection unit and the object detection result of the object detection unit, determines that an object exists and the driver is not facing the object, and outputs, as first alarm control information, display control information to cause a target display device, which is one of a plurality of display devices installed in different locations inside the vehicle, that is installed in the direction of the driver's gaze or face orientation detected by the direction detection unit, to display a warning about the object.

2. The alarm control device according to claim 1, further comprising: an action detection unit that, after the first alarm control unit causes the display for warning purposes to be made based on the display control information as first alarm control information, the driver has performed an action that is presumed to be an action to confirm the target object or an action to avoid the target object; and a second alarm control unit that, based on the detection result by the action detection unit of whether the driver has performed an action that is presumed to be an action to confirm the target object or an action to avoid the target object, outputs as second alarm control information level change instruction information to change the level of the display based on the display control information as first alarm control information to the target display device, or alternative alarm control information to cause the driver to perform the warning in a manner different from the display based on the display control information.

3. The alarm control device according to claim 2, characterized in that, when the action detection unit detects that the driver has not performed either the action presumed to be the confirmation action or the action presumed to be the avoidance action, the second alarm control unit outputs, as second alarm control information, level change instruction information for raising the level of the display based on the display control information as first alarm control information for the target display device, or the separate alarm control information.

4. The alarm control device according to claim 2, characterized in that, when the action detection unit detects that the driver has performed an action presumed to be the confirmation action or the avoidance action, the second alarm control unit outputs level change instruction information as second alarm control information for the target display device, which causes the level of the display to be lowered based on the display control information as first alarm control information.

5. The alarm control device according to any one of claims 1 to 4, characterized in that, if the first alarm control unit determines, based on the orientation detection result and the target object detection result, that the target object exists and the driver is facing the direction of the target object, it outputs suppressed display control information as first alarm control information to cause one of the plurality of display devices, the target display device, to display the warning to the target object in a form that is more suppressed than the display based on the display control information.

6. The alarm control device according to claim 2, characterized in that the action detection unit detects whether or not it is presumed that the driver has performed the confirmation action or the avoidance action, based on the direction the driver is facing or the operation of the vehicle by the driver.

7. The alarm control device according to claim 3, wherein the separate alarm control information includes information that prompts other vehicles to give the vehicle the warning, and the second alarm control unit outputs the separate alarm control information as the second alarm control information to the other vehicles.

8. The alarm control device according to claim 2, comprising a driving aptitude determination unit that determines the driver's driving aptitude based on the driver's condition, including the degree of drowsiness, the degree of distraction, or age, wherein the second alarm control unit outputs the level change instruction information or the separate alarm control information as the second alarm control information based on the driver's driving aptitude determined by the driving aptitude determination unit.

9. The alarm control device according to claim 8, further comprising a vehicle control unit that, when the driver aptitude determination unit determines that the driver's aptitude is low, outputs vehicle control information for causing the vehicle to perform vehicle control to prevent the occurrence of the unforeseen incident.

10. The alarm control device according to claim 7, comprising: an other vehicle alarm control information acquisition unit that acquires the second alarm control information output from the other vehicle; and a cooperation control unit that, when the other vehicle alarm control information acquisition unit acquires the second alarm control information from the other vehicle, outputs cooperation control information to cause the driver to perform the avoidance action toward the target object, or to cause the vehicle to perform vehicle control to avoid the target object.

11. The alarm control device according to claim 10, characterized in that the linkage control unit outputs the linkage control information and also outputs the second alarm control information output from the other vehicle to another vehicle.

12. The alarm control device according to claim 1, characterized in that the first alarm control unit outputs, in addition to the display control information, voice output control information for outputting a voice prompting the warning to the target object, or vibration control information for generating vibration, as the first alarm control information to a voice output device or vibration generating device installed in the vehicle interior.

13. An alarm control method comprising: a step of an image acquisition unit acquiring an image in which at least the face of the vehicle's driver is captured; a step of a direction detection unit detecting the direction of the driver's gaze or face orientation based on the image acquired by the image acquisition unit; a step of a target object detection unit detecting whether or not there is a target object in the vicinity of the vehicle that could cause the vehicle to fall into an unforeseen situation; and a step of a first alarm control unit determining, based on the direction detection result of the direction of the driver's gaze or face orientation by the direction detection unit and the target object detection result of the target object detection unit, that the target object exists and the driver is not facing the direction of the target object, outputting display control information as first alarm control information to cause a target display device, which is one of a plurality of display devices installed in different locations inside the vehicle, that is installed in the direction of the driver's gaze or face orientation detected by the direction detection unit, to display a warning about the target object.

Images