Driving assistance device, driving assistance method, and recording medium

Abstract

Provided are a driving assistance device, a driving assistance method, and a recording medium whereby it is possible to select a more appropriate countermeasure against a decrease in the wakefulness of a driver.　The driving assistance device comprises: an acquisition means that acquires a facial image of a driver of a vehicle; a measurement means that measures the wakefulness of the driver on the basis of at least the degree of openness of the eyes in the facial image of the driver; a situation comprehension means that comprehends the travel environment of a road on which the vehicle is traveling; and a selection means that selects a countermeasure for the driver on the basis of the travel environment and the wakefulness level of the driver.

Description

Driving support device, driving support method, and recording medium

[0001] The present invention relates to a driving support device, a driving support method, and a recording medium.

[0002] Patent Document 1 discloses an arousal level maintenance device that can suppress fluctuations in a driver's arousal level and maintain a relatively high arousal level. The arousal level maintenance device described in this document detects at least one of the driver's actual arousal level and the expected change in arousal level, and performs an operation of giving the driver the stimulation information necessary to maintain the driver's arousal level according to the difference in the driver's arousal level.

[0003] Japanese Patent Application Laid-Open No. 2019-159711

[0004] In the arousal level maintenance device of Patent Document 1, since it does not consider the distinction between day and night, road conditions, etc., there is a possibility of giving unnecessary stimulation or not being able to give appropriate stimulation when necessary.

[0005] An object of the present disclosure is to provide a driving support device, a driving support method, and a recording medium that can select more appropriate countermeasures against a decrease in a driver's arousal level.

[0006] According to a first aspect, there is provided a driving support device including an acquisition unit that acquires a face image of a driver of a vehicle, a measurement unit that measures the arousal level of the driver based at least on the degree of opening of the eyes in the face image of the driver, a situation grasping unit that grasps the driving environment of the road on which the vehicle is traveling, and a selection unit that selects a countermeasure for the driver based on the driving environment and the arousal level of the driver.

[0007] According to a second aspect, there is provided a driving support method including acquiring a face image of a driver of a vehicle, measuring the arousal level of the driver based at least on the degree of opening of the eyes in the face image of the driver, grasping the driving environment of the road on which the vehicle is traveling, and selecting a countermeasure for the driver based on the driving environment and the arousal level of the driver.

[0008] From a third perspective, a recording medium is provided that contains a program that causes a computer to execute the following: a process for acquiring a facial image of a vehicle driver; a process for measuring the driver's level of alertness based at least on the degree of eye opening in the facial image of the driver; a process for understanding the driving environment of the road on which the vehicle is traveling; and a process for selecting an action to take towards the driver based on the driving environment and the driver's level of alertness.

[0009] According to this disclosure, it will be possible to provide a driver assistance device, a driver assistance method, and a recording medium that can select a more appropriate response to a decrease in the driver's level of alertness.

[0010] This is a diagram showing one configuration of the present disclosure. This is a flowchart showing the operation of the present disclosure. This is a diagram for explaining the operation of the present disclosure. This is another diagram for explaining the operation of the present disclosure. This is a diagram showing one configuration of the present disclosure. This is a functional block diagram showing the configuration of the driver assistance server of the present disclosure. This is an example of a table for selecting a response (message) from an alertness level value. This is another example of a table for selecting a response (message) from an alertness level value. This is a sequence diagram for explaining the operation of the present disclosure. This is a diagram for explaining the operation of the present disclosure. This is another diagram for explaining the operation of the present disclosure. This is a functional block diagram showing the configuration of the driver assistance server of the present disclosure. This is another example of a table for selecting a response (message) from an alertness level value. This is a functional block diagram showing the configuration of the driver assistance server of the present disclosure. This is an example of a table for selecting a response (message) from an alertness level value. This is a functional block diagram showing the configuration of the driver assistance server of the present disclosure. This is an example of a table for selecting a response (message) from an alertness level value. This is a diagram showing the configuration of the computer constituting the driver monitoring device or driver monitoring server of the present disclosure.

[0011] First, an overview of one embodiment of this disclosure will be described with reference to the drawings. In this disclosure, the drawings are associated with one or more embodiments. The reference numerals in the drawings appended to this overview are provided for convenience as examples to aid understanding and are not intended to limit this disclosure to the illustrated embodiments. In addition, the connecting lines between blocks in the drawings and other references referred to in the following description include both bidirectional and unidirectional lines. Unidirectional arrows schematically indicate the flow of the main signal (data) and do not exclude bidirectionality. The program is executed via a computer device, which includes, for example, a processor, a storage device, an input device, a communication interface, and a display device as needed. This computer device is also configured to communicate with devices (including computers) inside or outside the device via the communication interface, whether wired or wireless. In addition, there are ports or interfaces at the input / output connection points of each block in the figures, but these are omitted from the illustration.

[0012] In one embodiment, this disclosure can be realized by a driving assistance device 10, as shown in Figure 1, comprising an acquisition means 11, a measuring means 12, a situation understanding means 14, and a selection means 15. More specifically, the acquisition means 11 acquires, for example, a facial image of the vehicle's driver from a camera C installed in the vehicle. The measuring means 12 measures the driver's level of alertness based at least on the degree of eye opening in the driver's facial image. The situation understanding means 14 grasps the driving environment of the road on which the vehicle is traveling. The selection means 15 then selects an action to take towards the driver based on the driving environment and the driver's level of alertness.

[0013] The driver assistance device 10 configured as described above operates as follows. First, the driver assistance device 10 acquires a facial image of the vehicle's driver (step S01 in Figure 2).

[0014] Next, the driver assistance device 10 measures the driver's level of alertness based on the degree of eye opening in the driver's facial image (step S02 in Figure 2).

[0015] Next, the driver assistance device 10 grasps the driving environment of the road on which the vehicle is traveling (step S03 in Figure 2). This driving environment includes the type of road, time of day, weather, and road congestion.

[0016] Next, the driver assistance device 10 selects an action to take for the driver based on the driving environment and the driver's level of alertness (step S04 in Figure 2). This action to take for the driver may include various actions to prevent a decrease in the driver's alertness, such as voice, light, and text warnings, activation of a vibrator, changes in room temperature (lowering the temperature), and ventilation. Another form of action to take for the driver may involve controlling the vehicle, such as reducing speed, activating hazard lights, changing lanes, or safely stopping the vehicle. The driver assistance device 10 or the action recipient instructed by the driver assistance device 10 will then implement the selected action.

[0017] Figure 3 is a diagram illustrating the operation of the present disclosure. In Figure 3, the vertical axis represents the level of alertness, and the horizontal axis represents the elapsed time. In the example in Figure 3, the driver's level of alertness decreases as time passes. The driver assistance device 10 changes the threshold for taking action against the driver depending on the weather conditions obtained as the driving environment. For example, if it is raining, the driver assistance device 10 uses a threshold TH2 that is higher than the threshold TH1 used in sunny weather to determine whether or not to take action against the driver. This makes it possible to take action against the driver earlier in response to a decrease in alertness in the case of rain.

[0018] Figure 4 is another diagram illustrating the operation of the present disclosure. In Figure 4, the vertical axis represents the level of alertness, and the horizontal axis represents the elapsed time. In the example in Figure 4, similar to Figure 3, the driver's level of alertness decreases as time passes. The driver assistance device 10 changes the threshold for taking action against the driver depending on the type of road obtained as the driving environment. For example, if the road being driven on is a highway, the driver assistance device 10 determines whether or not to take action against the driver using a threshold TH12 that is higher than the threshold TH11 used when the road being driven on is a general road. This makes it possible to take action against the driver earlier in response to a decrease in alertness when driving on a highway.

[0019] In the above example, the information obtained as the driving environment was described as weather and road type, but the driving environment is not limited to these. For example, if the information obtained as the driving environment is the time of day or road congestion, similarly, by changing the threshold, it will be possible to take action in more dangerous driving environments more quickly. For example, the driver assistance system 10 will use a higher threshold than the daytime threshold to determine whether or not to take action for the driver in the early morning or at night. Similarly, for example, the driver assistance system 10 will use a higher threshold than the congestion threshold to determine whether or not to take action for the driver when the road is clear.

[0020] Furthermore, although the above example described an instance where only one type of information was obtained regarding the driving environment, multiple types of information may be obtained regarding the driving environment. For example, the driver assistance device 10 determines whether or not to take action for the driver using a higher threshold than the threshold used when driving on a regular road in sunny weather, when driving on a highway in rainy weather. Alternatively, in another configuration, the driver assistance device 10 may use one threshold and adjust the level of alertness to a lower value when driving in rainy weather or on a highway by multiplying it by a predetermined coefficient or subtracting a predetermined number. This allows for earlier implementation of countermeasures against decreased alertness in more dangerous or unfavorable driving environments.

[0021] As explained above, this embodiment makes it possible to select a more appropriate response to a decrease in the driver's alertness level, in accordance with the driving environment. In the example in Figure 2, the vehicle's driving environment is assessed after the driver's alertness level is measured, but the driver's alertness level may be measured after assessing the vehicle's driving environment. Alternatively, the measurement of the driver's alertness level and the assessment of the vehicle's driving environment may be performed in parallel.

[0022] [First Embodiment] Next, a first embodiment in which the functions corresponding to the driver assistance device 10 are located on the network side (cloud side) will be described in detail with reference to the drawings. Figure 5 is a diagram showing one configuration of the present disclosure. Referring to Figure 5, a configuration is shown in which an in-vehicle terminal 200 mounted on a vehicle V and a driver assistance server 100 are connected via a network N. A camera C capable of photographing the driver of the vehicle V is connected to the in-vehicle terminal 200, and the in-vehicle terminal 200 is capable of transmitting the driver's image to the driver assistance server 100. In the following description, it will be assumed that the driver's image is accompanied by location information indicating the shooting location (vehicle location). As such location information, location information from GPS (Global Positioning System) or GNSS (Global Navigation Satellite System) mounted on the vehicle can be used.

[0023] Figure 6 is a functional block diagram showing the detailed configuration of the driver assistance server 100 of this disclosure. Referring to Figure 6, the configuration includes an acquisition unit 101, a measurement unit 102, a recording unit 103, a situation understanding unit 104, a selection unit 105, a countermeasure implementation unit 106, and a countermeasure information storage unit 107. Reference numerals 201 and 202 indicate the display device and speaker of the vehicle V.

[0024] The acquisition unit 101 acquires a facial image of the driver of the vehicle V, captured by the camera C, via the aforementioned in-vehicle terminal 200, and sends the acquired image to the measurement unit 102. The acquisition unit 101 also extracts location information indicating the shooting location from the image and sends it to the situation assessment unit 104. This acquisition unit 101 corresponds to the acquisition means 11 described above.

[0025] The measurement unit 102 measures the driver's level of alertness based on the degree of eye opening in the driver's facial image and outputs it to the recording unit 103. This level of alertness is a quantitative measure of how alert a person is, and can be expressed as a value from 0 to 100. For example, if a person's eyes tend to close and their pupils do not move much, the level of alertness will be a low value (for example, 50). Conversely, if a person's eyes are open and moving appropriately, the level of alertness will be a high value (for example, 100). Alternatively, a similar index such as eye opening rate can be calculated and used instead of the level of alertness. This measurement unit 102 corresponds to the measurement means 12 described above.

[0026] The recording unit 103 records the changes in the driver's alertness level measured by the measurement unit 102. Such changes in alertness level may be used as data for telematics purposes. The recording unit 103 may discard old alertness level information after a certain period of time. In other words, the recording unit 103 only needs to retain enough changes in alertness level to be necessary for the selection unit 105 to make a decision.

[0027] The situation assessment unit 104 understands the driving environment, which is the environment in which the vehicle is located, based on the location information received from the acquisition unit 101. One method for understanding the driving environment is to query an external server. For example, if the situation assessment unit 104 wants to know the weather at the location where vehicle V is driving, it accesses a server that collects weather information from various locations and obtains the weather at the location where vehicle V is driving. Similarly, if the situation assessment unit 104 wants to know the road type and congestion level at the location where vehicle V is driving, it accesses a server that collects road conditions and obtains the road type and congestion level at the location where vehicle V is driving.

[0028] Another method the situation assessment unit 104 uses to assess the driving environment is to determine the driving environment based on images acquired by the acquisition unit 101 or external images received separately from the vehicle V. For example, if the image sent from the vehicle V is dark despite it being daytime, it can be inferred that the weather is bad. Also, if a highway road sign is visible in the image, it can be determined that the vehicle V is traveling on a highway. If many vehicles are visible in the image, it can be determined that the vehicle V is caught in a traffic jam. Of course, both this image-based method and the method of querying the aforementioned external server may be used in combination. This situation assessment unit 104 corresponds to the situation assessment means 14 described above.

[0029] The selection unit 105 selects an action to take regarding the driver based on the driving environment grasped by the situation assessment unit 104 and the changes in the driver's level of alertness stored in the recording unit 103.

[0030] The response information storage unit 107 stores the content of the response to be taken for the driver based on the driving environment and changes in the driver's level of alertness. In this embodiment, it is assumed that the response information storage unit 107 holds a response selection table T1 for driving on general roads and a response selection table T2 for driving on expressways.

[0031] Figure 7 shows Table T1, which is used to select a response (message) based on the difference between the current alertness level and the measurement taken n seconds ago, applicable when driving on a public road. For example, if the alertness level is 90 or higher, and the difference from the measurement taken n seconds ago is -3 or higher, the selection unit 105 selects "You are in good spirits" as the response (message). On the other hand, even if the alertness level is 90 or higher, if the difference from the measurement taken n seconds ago is less than -3, that is, if the alertness level is rapidly decreasing, the selection unit 105 selects "Your alertness level is decreasing" as the response (message). In this way, Table T1 defines the response (message) to be selected according to the difference between the current alertness level and the measurement taken n seconds ago. By using such a table, the selection unit 105 changes the judgment criteria based on the trend of increase or decrease in the alertness level and selects a response depending on whether the alertness level is higher than a predetermined threshold or below the threshold.

[0032] Figure 8 shows Table T2, which is used when driving on a highway to select a response (message) based on the difference between the current alertness level and the measurement taken n seconds ago. The difference from Table T1 when driving on a regular road is the alertness level value in the area indicated by the dotted line. For example, in Table T1 when driving on a regular road, if the alertness level is 70 or higher and the difference from the measurement taken n seconds ago is 0 or greater, the message "Your alertness level is low" is selected. On the other hand, in Table T2, if the alertness level is 75 or higher and the difference from the measurement taken n seconds ago is 0 or greater, the message "Your alertness level is low" will be selected. Similarly, in Table T1 when driving on a regular road, if the alertness level is 70 or higher and the difference from the measurement taken n seconds ago is less than 0, the message "Your alertness level is decreasing further" is selected. On the other hand, in Table T2, if the alertness level is 75 or higher and the difference from the measurement taken n seconds ago is less than 0, the message "Your alertness level is decreasing further" will be selected. In other words, even with the same alertness level, the response selected will differ when driving on a regular road and when driving on a highway. Furthermore, in a state of reduced alertness, Table T1 for general road driving stipulates "Dangerous" when the alertness level is below 60, but Table T2 requires selecting "Dangerous" when the alertness level is below 70.

[0033] The response execution unit 106 transmits the response (message) selected by the selection unit 105 to the in-vehicle terminal 200 and displays it on the vehicle's display device 201. The response execution unit 106 also transmits the response (message) selected by the selection unit 105 to the in-vehicle terminal 200 and outputs it as audio from the speaker 202.

[0034] Next, the operation of this embodiment will be described in detail with reference to the drawings. Figure 9 is a sequence diagram illustrating the operation of the present disclosure. Referring to Figure 9, first, the in-vehicle terminal 200 transmits an image with location information to the driver assistance server 100 (step S001).

[0035] Next, the driver support server 100 measures the driver's level of alertness based on the degree to which the driver's eyes are open in the image (step S002). Next, the recording unit 103 records the change in the driver's level of alertness measured by the measurement unit 102 (step S003). The change in level of alertness is recorded, for example, as shown in the graphs in Figures 10 and 11.

[0036] Next, the driver support server 100 grasps the vehicle's driving environment (step S004). Based on this vehicle's driving environment, the driver support server 100 selects a table to refer to and selects the appropriate action (message) to be taken based on the change in the driver's alertness level (step S005).

[0037] Next, the driver assistance server 100 sends a message to the in-vehicle terminal 200 indicating the action to be taken (step S006). The in-vehicle terminal 200 displays the message on the vehicle V's display device 201 according to the action to be taken (message) received from the driver assistance server 100 (step S007). In addition, the in-vehicle terminal 200 outputs the message as audio from the vehicle V's speaker 202 along with the display.

[0038] Figures 10 and 11 show a state where the alertness level, which was 81% or higher, rapidly decreased from 80% to 69%. Figure 10 shows the change in the message displayed according to table T1 in Figure 7, assuming that the vehicle V is driving on a public road. For example, when the alertness level changes from 81% to 80%, it corresponds to an alertness level of 80 or higher and a difference >= -1 in Figure 7, and the message "Normal state" is selected. Subsequently, when the alertness level changes from 80% to 75%, it corresponds to an alertness level of 70 or higher and a difference < 0 in Figure 7, and the message "Awareness level is decreasing further" is selected. Subsequently, when the alertness level changes from 75% to 74%, it corresponds to an alertness level of 70 or higher and a difference < 0 in Figure 7, and the message "Awareness level is decreasing further" is selected. Subsequently, if the level of alertness changes from 74% to 69%, it corresponds to the condition in Figure 7 where the level of alertness is 60 or higher, and the message "You are getting sleepy" is selected.

[0039] Figure 11 shows the changes in the messages displayed according to Table T2 in Figure 8, assuming that the vehicle V is driving on a highway. Since the area other than the dotted line in Figure 8 is the same as Table T1, the messages selected during the process of the alertness level changing to 81%, 80%, and 75% are the same as in Figure 10. Subsequently, when the alertness level changes from 75% to 74%, it corresponds to an alertness level of 70 or higher in Figure 8, and the message "You are getting sleepy" is selected. Subsequently, when the alertness level changes from 74% to 69%, it corresponds to an alertness level of less than 70 in Figure 7, and the message "It's dangerous!!" is selected.

[0040] As described above, according to this embodiment, even with the same change in alertness, if vehicle V is in a highly dangerous situation, a message with content that is more effective in counteracting the decrease in alertness can be displayed. Specifically, when the road on which vehicle V is traveling is a highway, the driver assistance server 100 selects a response that is more effective in counteracting the decrease in alertness than when vehicle V is traveling on a general road.

[0041] In the examples shown in Figures 7 and 8 above, the table was switched based on the type of road the vehicle V is traveling on as the driving environment. However, the table may be switched based on other driving environment factors. Using a similar principle, tables corresponding to the time of day, weather, road congestion, etc., can be created and the driving support server 100 can be made to select from them. For example, by creating a table corresponding to the weather, when vehicle V is traveling in the rain, the server can select an action that is more effective in reducing the level of alertness than when vehicle V is traveling in sunny weather.

[0042] [Second Embodiment] Next, a second embodiment will be described in which the interval for measuring the level of alertness in the driver support server 100a is changed according to the measured level of alertness. Figure 12 is a functional block diagram showing another configuration of the driver support server 100a of this disclosure. The difference from the driver support server 100 of the first embodiment shown in Figure 6 is the contents held in the response information recording unit 107a and the operation of the selection unit 105a based thereon. The other configurations are almost the same as in the first embodiment, so the differences in operation will be explained below.

[0043] Figure 13 shows two tables held in the response information recording unit 107a. The difference from table T1 used during general road driving in the first embodiment is that there are two tables: table T11 which is applied when the alertness level is 90 or higher, and table T12 which is applied when the alertness level is less than 90. Furthermore, the selection condition for messages in table T11 is the difference with the measurement value 20 seconds ago, and the selection condition for messages in table T12 is the difference with the measurement value 10 seconds ago, with the measurement interval (judgment interval) differing depending on the level of alertness.

[0044] In the second embodiment, the selection unit 105a makes a determination with a measurement interval of 20 seconds while the alertness level is 90 or higher, and selects a message. When the alertness level falls below 90, the selection unit 105a shortens the measurement interval to 10 seconds and selects a message. Similarly, the table for highway driving may also be divided into two or more tables.

[0045] By doing so, it is possible to extend the measurement interval while the arousal level is above a predetermined threshold value, and reduce the load on the driving support server 100a. In the present embodiment, it is preferably applied to a case where one driving support server 100a monitors a plurality of vehicles V. In the example of FIG. 13 described above, an example in which the table to be referred to is divided according to whether the arousal level is 90 or more has been described. However, the threshold value for dividing the table is not limited to 90, and an appropriate value can be set. In the above example, an example in which the measurement intervals (determination intervals) of the differences from the previous measurement values are set to 20 seconds and 10 seconds has been described. However, the measurement interval (determination interval) of the difference from the previous measurement value is not limited to this.

[0046] [Third Embodiment] Next, a third embodiment in which, in addition to the arousal level, a countermeasure to be applied is selected in consideration of the driver's physical condition and the previous day's sleep time will be described. FIG. 14 is a functional block diagram showing another configuration of the driving support server 100b of the present disclosure. The first difference from the driving support server 100 of the first embodiment shown in FIG. 6 is that a second acquisition unit 108 is added to the driving support server 100b. The second difference from the first embodiment is that a table for selecting a countermeasure in consideration of the driver's physical condition and the previous day's sleep time is set in the countermeasure information recording unit 107b. The third difference from the first embodiment is that the selection unit 105b selects a countermeasure in consideration of the driver's physical condition and the previous day's sleep time. Since other configurations are substantially the same as those of the first embodiment, the differences in their operations will be mainly described below.

[0047] The second acquisition unit 108 uses the sensor S provided in the vehicle V to acquire the driver's vital information and sends this as physical condition information to the selection unit 105b. When the second acquisition unit 108 acquires the previous day's sleep time of the driver, for example, the sleep time held in the driver's smartwatch or smartphone (health management application) may be acquired.

[0048] FIG. 15 shows an example of a table T2a applied in a situation where the driver's physical condition held in the countermeasure information recording unit 107b is poor. The difference from the table T1 during general road driving in the first embodiment is the threshold of arousal level indicated by the dotted line, which is generally raised. For example, in the table T1 during general road driving in the first embodiment, as a condition for displaying "You are in good condition.", the arousal level is 90 or more and the difference from the previous measurement value >= -3 was set. However, in the example of FIG. 15, the arousal level is 95 or more and the difference from the previous measurement value >= -3. Similarly, in the table T1 during general road driving in the first embodiment, as a condition for displaying "Danger!!", the arousal level less than 60 was set. However, in the example of FIG. 15, the arousal level is less than 70.

[0049] Generally, when the driver's physical condition is poor or the sleep time the previous day is short, the decrease in arousal level acts in a direction to increase the probability of encountering an accident or the like. By adding a table when the physical condition is poor or the sleep time is short as shown in FIG. 15 and allowing the selection unit 105b to refer to the table, it is possible to make a difference in the selected countermeasures between the situation where the driver's physical condition is poor and when it is not.

[0050] [Fourth Embodiment] Next, a fourth embodiment in which other operations than the output of a message are performed as the applied countermeasure will be described. FIG. 16 is a functional block diagram showing another configuration of the driving support server 100c of the present disclosure. The difference from the driving support server 100 in the first embodiment shown in FIG. 6 is the content held in the countermeasure information recording unit 107c and that the countermeasure implementation unit 106c can transmit an instruction to the vehicle's ECU (Electronic Control Unit). Since other configurations are substantially the same as those in the first embodiment, the difference in its operation will be mainly described below.

[0051] The response information recording unit 107c contains not only the content of messages to be displayed under certain conditions, but also the content of actions to be taken under certain conditions. Figure 17 shows an example of a driver table T1a held in the response information recording unit 107c. In the example in Figure 17, it is specified that an alarm sound and a vibrator should be activated when the alertness level falls between 60 and 70. In this case, the response implementation unit 106c instructs the ECU 203 to output an alarm sound and activate the vibrator 203 when the relevant conditions are met. Similarly, in the example in Figure 17, it is specified that an emergency stop should be performed when the alertness level falls below 60. In this case, the response implementation unit 106c instructs the ECU 203 to stop the vehicle when the relevant conditions are met.

[0052] As described above, in this embodiment, when the driver's level of alertness is low, it is possible to directly interact with the driver or control the vehicle to bring it to a safe stop.

[0053] While the embodiments of this disclosure have been described above, this disclosure is not limited to the embodiments described above, and further modifications, substitutions, and adjustments can be made without departing from the basic technical concept of this disclosure. For example, the network configurations, element configurations, and data representations shown in the drawings are examples to aid in understanding this disclosure and are not limited to the configurations shown in these drawings.

[0054] For example, in the first to fourth embodiments described above, the functions of the driver assistance device were described as being located on the driver assistance servers 100 to 100c. However, the driver assistance device may also be located on the edge side, which is the in-vehicle terminal side of the vehicle. In this case, the load on the in-vehicle terminal may increase, but compared to the first to fourth embodiments, there is the advantage that the driver assistance device can be configured on the vehicle alone.

[0055] (Regarding Hardware Configuration) In each embodiment of this disclosure, each component of each device represents a functional unit block. Some or all of each component of each device is realized by any combination of an information processing device 900 and a program, for example, as shown in Figure 18. Figure 18 is a block diagram showing an example of the hardware configuration of the information processing device 900 that realizes each component of each device. The information processing device 900 includes, as an example, the following configuration: ・CPU (Central Processing Unit) 901 ・ROM (Read Only Memory) 902 ・RAM (Random Access Memory) 903 ・Program 904 loaded into RAM 903 ・Storage device 905 that stores the program 904 ・Drive device 907 that reads and writes to the recording medium 906 ・Communication interface 908 that connects to a communication network 909 ・Input / output interface 910 that performs data input and output ・Bus 911 that connects each component

[0056] Each component of each device in each embodiment is realized by the CPU 901 acquiring and executing a program 904 that realizes these functions. That is, the CPU 901 in Figure 18 executes the alertness level measurement program and the treatment selection program, and performs the update process of each calculation parameter held in the RAM 903, storage device 905, etc. The program 904 that realizes the functions of each component of each device is, for example, stored in advance in the storage device 905 or ROM 902, and read by the CPU 901 as needed. The program 904 may be supplied to the CPU 901 via the communication network 909, or it may be stored in advance in the recording medium 906, and the drive device 907 may read the program and supply it to the CPU 901.

[0057] There are various variations in how each device is implemented. For example, each device may be implemented by any combination of a separate information processing device 900 and a program for each component. Alternatively, multiple components of each device may be implemented by any combination of a single information processing device 900 and a program. In other words, each part (processing means, function) of the above-described driving support device can be implemented by a computer program that causes a processor mounted on the device to execute the above-described processes using its hardware.

[0058] Furthermore, some or all of the components of each device are realized by other general-purpose or dedicated circuits, processors, etc., or combinations thereof. These may be made up of a single chip or multiple chips connected via a bus.

[0059] Some or all of the components of each device may be realized by a combination of the circuits and programs described above.

[0060] When some or all of the components of each device are implemented by multiple information processing devices or circuits, these multiple information processing devices or circuits may be centrally located or distributed. For example, the information processing devices or circuits may be implemented in a form in which each is connected via a communication network, such as a client-and-server system or a cloud computing system.

[0061] The embodiments described above are preferred embodiments of this disclosure and do not limit the scope of this disclosure to these embodiments alone. That is, a person skilled in the art can modify or substitute the embodiments described above to construct various modified forms without departing from the gist of this disclosure.

[0062] Some or all of the above embodiments may also be described as follows, but are not limited to these.

[0063] [Note 1] A driving assistance device comprising: an acquisition means for acquiring a facial image of the vehicle's driver; a measurement means for measuring the driver's level of alertness based at least on the degree of eye opening in the driver's facial image; a situation understanding means for understanding the driving environment of the road on which the vehicle is traveling; and a selection means for selecting an action for the driver based on the driving environment and the driver's level of alertness. [Note 2] The above-described driving assistance device may be configured to use one or more of the following as the driving environment: road type, time of day, weather, and road congestion. [Note 3] The selection means of the above-described driving assistance device may be configured to select an action earlier in response to a decrease in alertness when the driving environment is poor compared to when the driving environment is good. [Note 4] The selection means of the above-described driving assistance device may be configured to select an action that is more effective in responding to a decrease in alertness compared to other actions when the road on which the vehicle is traveling is an expressway compared to when the road on which the vehicle is traveling is an ordinary road. [Note 5] The selection means of the above-described driving support device may be configured to select an action that is more effective in reducing the level of alertness when the vehicle is driving in the rain than when the vehicle is driving in sunny conditions, compared to other actions. [Note 6] The above-described driving support device may further include a recording means for recording changes in the driver's level of alertness, and the selection means may be configured to select an action for the driver based on the driving environment and changes in the driver's level of alertness. [Note 7] The selection means of the above-described driving support device may be configured to change the judgment criteria based on the trend of increase or decrease in the level of alertness when the value of the level of alertness is higher than a predetermined threshold and when the value of the level of alertness is below the threshold. [Note 8] The above-described driving support device may be configured to widen the time interval for measuring the driver's level of alertness when the driver's level of alertness is above a predetermined value.[Note 9] The above-described driving assistance device further comprises a second acquisition means for acquiring the driver's physical condition information or sleep time information, and the selection means can be configured to select an action for the driver based on the driver's physical condition information or sleep time information in addition to the level of alertness. [Note 10] A driving assistance method comprising: acquiring a facial image of the driver of a vehicle; measuring the driver's level of alertness based at least on the degree of eye opening in the facial image of the driver; grasping the driving environment of the road on which the vehicle is traveling; and selecting an action for the driver based on the driving environment and the driver's level of alertness. [Note 11] A recording medium that stores a program causing a computer to execute: a process for acquiring a facial image of the driver of a vehicle; a process for measuring the driver's level of alertness based at least on the degree of eye opening in the facial image of the driver; a process for grasping the driving environment of the road on which the vehicle is traveling; and a process for selecting an action for the driver based on the driving environment and the driver's level of alertness. The forms described in each of the above notes can be combined with each other after making the necessary modifications. For example, a configuration that combines the contents described in Appendix 2 and the contents described in Appendix 3 is also included within the scope of disclosure of this specification. In this case, the driver assistance system uses one or more of the following as the driving environment: road type, time of day, weather, and road congestion. If the road being driven on is a highway, the system will select the action that is most effective in reducing the level of alertness compared to other actions. The forms described in Appendix 9 to Appendix 10 can be expanded into the forms described in Appendix 2 to Appendix 8, similar to Appendix 1.

[0064] Furthermore, each disclosure in the above-mentioned patent documents is incorporated into this document by reference and may be used as the basis or part of this disclosure as necessary. Within the framework of this disclosure (including the claims), further modifications and adjustments to the embodiments or examples are possible based on their fundamental technical concept. Also, within the framework of this disclosure, various combinations or selections (including partial deletions) of various disclosure elements (including each element of each claim, each element of each embodiment or example, each element of each drawing, etc.) are possible. In other words, this disclosure naturally includes the entire disclosure, including the claims, and various modifications and alterations that a person skilled in the art could make in accordance with the technical concept. In particular, with respect to the numerical ranges described in this document, any numerical value or sub-range included within that range should be interpreted as being specifically described, even if not otherwise stated. Furthermore, each disclosure item of the above-mentioned cited documents may, as necessary, be used in part or in whole as part of this disclosure, in accordance with the spirit of this disclosure, and this is also considered to be included in the disclosure items of this application.

[0065] 10 Driving support device 11 Acquisition means 12 Measurement means 14 Situation understanding means 15 Selection means 100, 100a to 100c Driving support server 101 Acquisition unit 102 Measurement unit 103 Recording unit 104 Situation understanding unit 105, 105a, 105b Selection unit 106, 106c Action execution unit 107, 107a, 107c Action information storage unit 200 In-vehicle terminal 201 Display device 202 Speaker 203 ECU 900 Information processing device 901 CPU (Central Processing Unit) 902 ROM (Read Only Memory) 903 RAM (Random Access Memory) 904 Program 905 Storage device 906 Recording medium 907 Drive device 908 Communication interface 909 Communication network 910 Input / output interface 911 Bus C Camera N Network T1-T2a Table V Vehicle

Claims

1. A driving assistance device comprising: an acquisition means for acquiring a facial image of a vehicle driver; a measurement means for measuring the driver's level of alertness based at least on the degree of eye opening in the facial image of the driver; a situation understanding means for understanding the driving environment of the road on which the vehicle is traveling; and a selection means for selecting an action to take towards the driver based on the driving environment and the driver's level of alertness.

2. The driving support device according to claim 1, wherein the driving environment is one or more of the following: type of road, time of day, weather, and traffic congestion on the road.

3. The driving support device according to claim 1 or 2, wherein the selection means selects the countermeasure earlier in response to a decrease in alertness when the driving environment is poor compared to when the driving environment is good.

4. The driver assistance device according to any one of claims 1 to 3, wherein the selection means selects a measure that is more effective in reducing the level of alertness compared to other measures when the road on which the vehicle is traveling is an expressway, compared to when the road on which the vehicle is traveling is a general road.

5. The driving assistance device according to any one of claims 1 to 4, wherein the selection means selects a measure that is more effective in reducing the level of alertness when the vehicle is driving in the rain than when the vehicle is driving in the clear sky, compared to other measures.

6. The driving support device according to any one of claims 1 to 5, further comprising recording means for recording changes in the driver's level of alertness, wherein the selection means selects an action to take towards the driver based on the driving environment and the changes in the driver's level of alertness.

7. The driving support device according to any one of claims 1 to 6, wherein the selection means changes the determination criteria based on the trend of increase or decrease of the alertness level depending on whether the alertness level is higher than a predetermined threshold or less than or equal to the threshold.

8. A driving support device according to any one of claims 1 to 7, wherein if the driver's level of alertness is above a predetermined value, the time interval for measuring the driver's level of alertness is extended.

9. A driving support device according to any one of claims 1 to 8, further comprising a second acquisition means for acquiring the driver's physical condition information or sleep time information, wherein the selection means selects an action to take for the driver based on the driver's physical condition information or sleep time information in addition to the level of alertness.

10. A driving assistance method comprising: acquiring a facial image of the vehicle driver; measuring the driver's level of alertness based at least on the degree of eye opening in the facial image; understanding the driving environment of the road on which the vehicle is traveling; and selecting an action for the driver based on the driving environment and the driver's level of alertness.

11. A recording medium that records a program causing a computer to execute the following: a process for acquiring a facial image of a vehicle driver; a process for measuring the driver's level of alertness based at least on the degree of eye opening in the facial image of the driver; a process for understanding the driving environment of the road on which the vehicle is traveling; and a process for selecting an action to take towards the driver based on the driving environment and the driver's level of alertness.

Images