Awakening support system, awakening support device, and awakening support program

Abstract

An awakening support system (1) (2) includes: an estimating unit (41) that estimates a drowsiness level of a driver (D) on the basis of biometric information of the driver (D); a response request information output unit (42) that outputs, to the driver (D), response request information for requesting a response from the driver (D) if it is determined that the drowsiness level is higher than a reference value; a determining unit (44) that determines whether the driver (D) is in an awake state or a sleeping state on the basis of response information from the driver (D) with respect to the response request information; and a support demand signal output unit (45) that outputs a support demand signal to a preset external terminal (30) located outside a driver's cab (A) of a mobility vehicle (M) if it is determined that the driver (D) is in a sleeping state.

Description

Awakening support system, awakening support device, and awakening support program

[0001] The present disclosure relates to an awakening support system, an awakening support device, and an awakening support program.

[0002] In mobility such as passenger cars, buses, trucks, railway vehicles, and aircraft, countermeasures against driver drowsiness are important for safe driving. Patent Document 1 discloses a management device that determines the state of a vehicle driver when managing the operation of a vehicle.

[0003] This management device acquires a face image of the driver, analyzes the face image, and detects the eye opening degree of the driver. Based on the eye opening degree, the management device estimates the physical and mental state (drowsiness level) of the driver that affects the driving operation. The management device extracts capture information of an image in which an event that is a major factor in estimating the physical and mental state exists, and transmits the extracted capture information to the center. The operation manager at the center makes a comprehensive judgment about the state of the driver considering the capture information, and uses it for driver education and the like. In addition, it is also disclosed that the management device uses biometric information in addition to the face image of the driver.

[0004] Japanese Patent No. 4985428

[0005] In recent years, countermeasures against driver microsleep have become important. Microsleep is sleep for a very short period of a few seconds. During microsleep, the driver may keep their eyes open, and it may not be possible to judge microsleep from the driver's face image.

[0006] In Patent Document 1, the physical and mental state (drowsiness level) of the driver is estimated based on the sensing information of the driver, and capture information such as the face image of the driver is transmitted to the center. However, there are individual differences in the relationship between the physical and mental state of the driver and the sensing information of the driver. Therefore, if the physical and mental state of the driver is judged uniformly, there is a possibility that unnecessary information may be transmitted to the center even though the driver is in an awake state. Also, there is a possibility that necessary information may not be transmitted to the center even though the driver is in a sleeping state.

[0007] Furthermore, the technology disclosed in Patent Document 1 only transmits driver information to a center, and cannot resolve issues in real time, even if the driver is experiencing drowsiness.

[0008] Therefore, it is desirable to resolve drowsiness, including microsleep, in drivers in real time. Furthermore, when notifying external devices in real time, it is necessary to notify the external devices at an appropriate time according to the individual driver's situation. In other words, it is desirable that external devices not be notified when the driver is awake, but only when the driver is asleep, in real time.

[0009] This disclosure is made in light of the above background and aims to provide an awakening support system, an awakening support device, and an awakening support program that can notify an external terminal at an appropriate time according to the driver's condition when the driver experiences drowsiness, including microsleep.

[0010] A first aspect of this disclosure is an awakening support system comprising: a biometric information estimation device for estimating the biometric information of a mobility vehicle driver; and an awakening support device for assisting the driver's awakening based on the biometric information, wherein the awakening support device comprises: an estimation unit for estimating the driver's level of drowsiness based on the biometric information; a response request information output unit for outputting response request information to the driver to request a response from the driver when it is determined that the level of drowsiness is higher than a standard value; an acquisition unit for acquiring the driver's response information to the response request information; a determination unit for determining whether the driver is awake or asleep based on the response information; and a support request signal output unit that does not output a support request signal to a pre-configured external terminal located outside the driver's cab of the mobility vehicle when it is determined that the driver is awake, and outputs the support request signal to the external terminal when it is determined that the driver is asleep.

[0011] A second aspect of the present disclosure is an awakening support device for assisting the driver of a mobility vehicle to wake up based on the driver's biometric information, comprising: an estimation unit that estimates the driver's level of drowsiness based on the biometric information; a response request information output unit that outputs response request information to the driver to request a response from the driver when it is determined that the level of drowsiness is higher than a standard value; an acquisition unit that acquires the driver's response information to the response request information; a determination unit that determines whether the driver is awake or asleep based on the response information; and a support request signal output unit that does not output a support request signal to an external terminal located outside the driver's cab of the mobility vehicle and pre-configured when it is determined that the driver is awake, and outputs the support request signal to the external terminal when it is determined that the driver is asleep.

[0012] A third aspect of this disclosure is an awakening support program that supports the awakening of a driver of a mobility vehicle based on the driver's biometric information, the program comprising: estimating the driver's level of drowsiness based on the biometric information; outputting response request information to the driver to request a response from the driver if it is determined that the level of drowsiness is higher than a standard value; acquiring the driver's response information to the response request information; determining whether the driver is awake or asleep based on the response information; and not outputting a support request signal to a pre-configured external terminal located outside the driver's cab of the mobility vehicle if it is determined that the driver is awake, and outputting the support request signal to the external terminal if it is determined that the driver is asleep, the program causes a computer to perform these steps.

[0013] According to the first embodiment, the estimation unit estimates the driver's level of drowsiness based on the driver's biometric information. Therefore, it is possible to estimate that microsleep may be occurring, which may not be detectable from the driver's facial image alone.

[0014] Furthermore, if the driver's level of drowsiness exceeds a certain threshold, the response request information output unit outputs response request information to the driver. The determination unit then determines, based on the driver's response information, whether the driver is awake or asleep. If the system determines that the driver is asleep based on the driver's response information, the support request signal output unit outputs a support request signal to a pre-configured external terminal. Therefore, when a support request signal is output to an external terminal, the person operating the external terminal can take action to wake the driver. In this way, the system can wake the driver in real time if they are asleep, including during microsleep.

[0015] Furthermore, if the driver's level of drowsiness exceeds a certain threshold, instead of immediately outputting to an external terminal, the wakefulness support device can first check the driver's response to the response request information. If it is then determined that the driver is asleep based on the response information, the support request signal output unit outputs a support request signal to the external terminal. On the other hand, if it is determined that the driver is awake based on the response information, the support request signal output unit does not output a support request signal to the external terminal.

[0016] In this way, after confirming the driver's response information, an assistance request signal is output to the external terminal. Therefore, if the driver is awake, no assistance request signal is output to the external terminal, thus preventing the output of unnecessary assistance request signals. Also, if the driver is asleep, an assistance request signal is output to the external terminal. Therefore, the person operating the external terminal can take appropriate action to wake the driver in question.

[0017] Furthermore, the second and third embodiments can achieve the same effects as the first embodiment.

[0018] Based on the above, it is possible to provide an awakening support system, an awakening support device, and an awakening support program that can notify an external terminal at an appropriate time according to the driver's condition when drowsiness, including microsleep, occurs in the driver.

[0019] Figure 1 shows the wakefulness support system in Embodiment 1. Figure 2 shows the change over time of the heart rate interval for estimating the degree of drowsiness. Figure 3 shows the hardware of the wakefulness support device that constitutes the wakefulness support system. Figure 4 shows the functions of the wakefulness support device. Figure 5 shows the wakefulness support system in Embodiment 6.

[0020] (Embodiment 1) 1. Basic Awakening Support System 1 The Awakening Support System 1 will be described with reference to Figure 1. The Awakening Support System 1 assists the driver D of Mobility M in real time to wake up when the driver D is experiencing drowsiness. Mobility M includes automobiles, motorbikes, railway vehicles, aircraft, ships, etc., and is a means of transporting people and goods. Automobiles include passenger cars, buses, trucks, special vehicles, motorcycles, etc. Driver D is a person who drives these mobility M. Aircraft include airplanes and helicopters (including multicopters).

[0021] Mobility M is configured to operate through the driving of driver D. In other words, one Mobility M has a driver's cab A inside the Mobility M, and is configured for driver D to operate from driver D in driver's cab A. The other Mobility M has a driver's cab A located outside the Mobility M, and is configured for driver D to operate remotely from driver D in driver's cab A.

[0022] The arousal support system 1 is configured to assist driver D in becoming awake when driver D is asleep. Sleep states include microsleep, which is a momentary sleep state, and sleep states that last longer than microsleep. In addition to assisting driver D in becoming awake, the arousal support system 1 may also assist driver D in becoming awake in real time when driver D is not asleep but is experiencing drowsiness or in other situations requiring assistance.

[0023] When the driver is asleep or in need of assistance, it is effective for another person to converse with driver D in order to bring driver D to an awakened state. Therefore, one method of awakening support by the awakening support system 1 is to enable driver D to converse with another person.

[0024] 2. Details of the Awakening Support System 1 In the Awakening Support System 1, mobility M includes the driver's cab A. As shown in Figure 1, the Awakening Support System 1 includes a biological information estimation device 10 and an awakening support device 20. The Awakening Support System 1 may further include an external terminal 30. However, the external terminal 30 may be configured separately from the Awakening Support System 1.

[0025] The biometric information estimation device 10 is configured to estimate the biometric information of the driver D of the mobility vehicle M. The biometric information estimation device 10 includes a sensor 11 and a biometric information estimation unit 12.

[0026] Sensor 11 detects information for estimating the driver D's biometric information. Sensor 11 is, for example, a wearable device attached to driver D. Wearable devices include watch-type devices worn on driver D's arm, glasses-type devices worn on the face, devices that hook onto the ears, devices that hook onto the neck, and devices that are attached to the upper or lower body.

[0027] Biological information includes heart rate or respiration. Here, heart rate essentially includes pulse. Sensor 11 detects information resulting from the heartbeat or pulse as information for estimating driver D's heart rate. Sensor 11 may be configured, for example, to detect pressure received from driver D's skin, and may also detect pressure fluctuations resulting from driver D's pulse. Sensor 11 uses known detection methods, such as capacitance or piezoelectric elements.

[0028] The biological information estimation unit 12 estimates biological information based on the information detected by the sensor 11. Here, the biological information estimation unit 12 can estimate biological information using software such as a program, or hardware such as an electronic circuit. In the former case, the biological information estimation unit 12 is composed of a computer including a processor and a storage device. The biological information estimation unit 12 may also include memory, input / output devices, communication interfaces, etc.

[0029] The biometric information estimation unit 12 acquires information detected by the sensor 11. Based on the acquired information, the biometric information estimation unit 12 estimates the driver D's biometric information, such as heart rate or respiratory rate. The biometric information estimation unit 12 estimates the biometric information based on the acquired information, for example, by having the processor execute a program.

[0030] The information detected by sensor 11 includes noise unrelated to biological information. Therefore, the biological information estimation unit 12 estimates biological information by removing the noise from the information detected by sensor 11. The noise may include the driver's body movements, vibrations of mobility M, etc.

[0031] The awakening support device 20 is configured to support the awakening of driver D based on biometric information. The awakening support device 20 may be, for example, a mobile terminal of driver D. A mobile terminal may include, for example, a smartphone, tablet, or wearable device. The awakening support device 20 may be a device mounted in the driver's cab A of mobility M. This device may include, for example, a navigation system, a drive recorder, a digital tachometer, or a steering wheel.

[0032] The awakening support device 20 acquires biometric information estimated by the biometric information estimation unit 12. Based on the biometric information, the awakening support device 20 estimates the driver D's level of drowsiness. The awakening support device 20 then assists the driver D in becoming awake according to the level of drowsiness. Conversation with another person is an effective method of awakening support. Therefore, if it is determined that the driver D needs awakening support, the awakening support device 20 outputs a support request signal to the external terminal 30 to enable the driver D to communicate with the operator of the external terminal 30 or the like.

[0033] The external terminal 30 is located outside the driver's cab A of mobility M and is pre-configured to be linked to the alertness support device 20. The external terminal 30 may, for example, be the terminal of the driver of another mobility vehicle. In this case, the alertness support device 20 assists driver D in communicating with the other driver. It is preferable that the other driver is someone with whom driver D has a good relationship.

[0034] Furthermore, the external terminal 30 may be a terminal of the organization to which driver D belongs, a terminal of a driver support organization, etc. The organization to which driver D belongs includes, for example, the management department of the company or organization to which driver D belongs, driver D's family, relatives, and other driver-related parties. The driver support organization includes, for example, a call center that provides driver support services, an external server equipped with artificial intelligence (voice recognition, language generation, and output functions), etc. In other words, the awakening support device 20 assists driver D in making calls to a person operating a terminal in the management department of the company or organization to which he belongs, a person at a call center, an external server equipped with artificial intelligence (voice recognition, language generation, and output functions), etc.

[0035] The external terminal 30 can be configured for each awakening support device 20. Furthermore, the awakening support device 20 can configure multiple external terminals 30 with a priority order. In this case, if it is not possible to communicate with the person operating the higher-priority external terminal 30, it is possible to communicate with the person operating the lower-priority external terminal 30.

[0036] 3. The degree of drowsiness will be explained with reference to Figure 2. As described above, the wakefulness support device 20 estimates the degree of drowsiness of driver D based on biological information such as heart rate and respiration.

[0037] The degree of drowsiness is estimated, for example, based on heart rate. In particular, the degree of drowsiness is estimated based on the time interval between heartbeats (called the heartbeat interval). The degree of drowsiness is estimated based on the change in the heartbeat interval over time. For example, the degree of drowsiness is calculated based on the number of times a given change in the heartbeat interval occurs over a given period of time. A larger heartbeat interval means that driver D's heart is beating slowly. A smaller heartbeat interval means that driver D's heart is beating quickly.

[0038] The waveform shown in Figure 2 represents a predetermined change in the heart rate interval over time and is a criterion for estimating the degree of drowsiness of driver D. The waveform shown in section T is an example of a criterion for estimating the degree of drowsiness of driver D. Criteria for determining the degree of drowsiness are disclosed, for example, in Japanese Patent Publication No. 2018-192128.

[0039] The waveform shown in section T includes the waveforms of sections T1, T2, and T3. The first section T1 is the section in which the heart rate interval changes to a value smaller than or equal to a first predetermined value. When driver D feels drowsy, driver D tries to resist the drowsiness. For example, driver D tries to resist closing their eyelids by tensing their eyelids. In addition to tensing their eyelids, driver D may tense other parts of their body to resist drowsiness. As a result, driver D's heart rate changes to increase. When the heart rate increases, the heartbeat becomes faster and the heart rate interval decreases. The first section T1 corresponds to this state in which the heart rate interval decreases.

[0040] The second interval T2 occurs after the first interval T1 and is a period in which the heart rate interval increases by more than a second predetermined value. In the first interval T1, drowsiness is suppressed, but then drowsiness returns, causing the heart rate interval to lengthen. The second interval T2 corresponds to this state of increasing heart rate interval. For example, the heart rate interval at the end of the second interval T2 is greater than the heart rate interval at the beginning of the first interval T1.

[0041] The third section, T3, occurs after the second section, T2, and is a section in which the heart rate interval decreases and then increases again. Driver D resists drowsiness again, and when drowsiness returns, the heart rate interval decreases and then increases again. The third section, T3, corresponds to this fluctuation in heart rate interval.

[0042] Thus, the waveform corresponding to interval T shows that the heartbeat interval decreases by a first predetermined value or more, then increases by a second predetermined value or more, and then fluctuates for at least one period. Here, the numerical values ​​of the heartbeat interval used to determine each of intervals T1 to T3 can be set as appropriate. In other words, if a slight change in the heartbeat interval occurs due to noise, it can be prevented from being determined as being in each of intervals T1 to T3.

[0043] Then, the drowsiness level can be the number of detections of the waveform corresponding to the section T at a predetermined measurement time. For example, when the waveform corresponding to the section T occurs n times during several tens of seconds as the predetermined measurement time, the drowsiness level is n. Also, the drowsiness level can be a value having a positive correlation with the number of detections of the waveform corresponding to the section T at a predetermined measurement time. Further, the drowsiness level can be a calculated value obtained from the waveform corresponding to the section T at a predetermined measurement time. The calculated value is, for example, an integrated value of the waveform. Also in this case, the more the waveform corresponding to the section T occurs at the predetermined measurement time, the larger the calculated value as the drowsiness level becomes. That is, the calculated value is a value having a positive correlation with the number of detections of the waveform corresponding to the section T. In the above example, the drowsiness level as the calculated value is a value having a positive correlation with n. Thus, the drowsiness level can be the number of detections of the waveform corresponding to the section T at a predetermined measurement time, or a value having a positive correlation with the number of detections of the waveform.

[0044] Incidentally, the drowsiness level can be estimated using the heartbeat intervals in the first section T1, the second section T2, and the third section T3. In addition to this, the drowsiness level may be estimated using the heartbeat intervals in the first section T1 and the second section T2. Also, the drowsiness level can be estimated using the heartbeat intervals including a fourth section (not shown) in which a waveform similar to the third section T3 is further added in addition to the sections T1 to T3. In this case, the waveform corresponding to the section T will fluctuate in a wavy manner for a plurality of cycles after the heartbeat interval decreases by a first predetermined value or more and then increases by a second predetermined value or more.

[0045] 4. Hardware of the arousal support device 20 The hardware of the arousal support device 20 will be described with reference to FIG. 3. The arousal support device 20 is configured by a computer. For example, the arousal support device 20 includes a processor 21 and a storage device 22. The arousal support device 20 includes a memory 23, an input / output device 24, a communication interface 25, and the like.

[0046] The memory device 22 stores an awakening support program P to be executed by a computer. The awakening support program P realizes the functions of the awakening support device 20 when executed by the processor 21 of the computer.

[0047] The input device in the input / output device 24 is, for example, at least one of a microphone for inputting sound, a touch panel for inputting a touch operation, a button for inputting a pushing or pulling operation, and a lever. The output device in the input / output device 24 is, for example, at least one of a speaker for outputting sound or voice, a display device, and a device for generating an operation such as vibration.

[0048] 5. Functional Configuration of Awakening Support Device 20 The functional configuration of the awakening support device 20 will be described with reference to FIG. 4. The awakening support device 20 is configured to function by the processor 21 executing the awakening support program P.

[0049] FIG. 3 shows the functional configuration of the awakening support device 20. Also, the parentheses of each element in FIG. 3 indicate each step to be executed by the awakening support program. By executing the target step by the awakening support program, the target function is realized.

[0050] The awakening support device 20 includes an estimation unit 41, a response request information output unit 42, an acquisition unit 43, a determination unit 44, a support request signal output unit 45, an input information acquisition unit 46, a driving situation acquisition unit 47, and a reference value adjustment unit 48 when the processor 21 executes the awakening support program P. The awakening support device 20 includes a reference value storage unit 51 as one of the memory devices 22.

[0051] The estimation unit 41 estimates the drowsiness level of the driver D based on the biological information estimated by the biological information estimation unit 12. The estimation of the drowsiness level is as described with reference to FIG. 2. For example, the drowsiness level can be the number of detections of the waveform corresponding to the section T shown in FIG. 2 within a predetermined measurement time. Also, the drowsiness level can be a calculated value obtained from the waveform corresponding to the section T shown in FIG. 2 within a predetermined measurement time. In the latter case, the drowsiness level is a value having a positive correlation with the number of detections of the waveform corresponding to the section T shown in FIG. 2 within a predetermined measurement time.

[0052] The response request information output unit 42 compares the drowsiness level with a reference value. The reference value is stored in the reference value storage unit 51. The reference value may be set, for example, to a value corresponding to the second or later detection count of the waveform corresponding to section T shown in Figure 2 during a predetermined measurement time. If the response request information output unit 42 determines that the drowsiness level is higher than the reference value, it outputs response request information to driver D to request a response from driver D. On the other hand, if the response request information output unit 42 determines that the drowsiness level is below the reference value, it does not output response request information.

[0053] The response request information output unit 42 outputs at least one of voice, sound, screen display, and vibration as response request information to the driver D. The response request information output unit 42 automatically outputs at least one of the voice, sound, screen display, and vibration that has been set in advance. In other words, the response request information output unit 42 outputs response request information without requiring human operation. The response request information may consist of one of voice, sound, screen display, and vibration, or a combination of these.

[0054] The response request information output unit 42 controls the speaker so that it outputs voice or sound if the response request information is voice or sound. The response request information output unit 42 controls the display device so that the target screen is displayed on the display device if the response request information is a screen display. The response request information output unit 42 controls the device so that it vibrates if the response request information is vibration.

[0055] The response request information is preferably information that driver D will respond to if driver D is awake. In other words, the response request information is information for determining whether driver D is awake or not. If driver D is not awake, this includes a sleep state. Furthermore, if driver D is not awake, this may also include a state where the driver is not sleepy but is experiencing drowsiness. Here, an intermediate state between awake and sleep, where the driver is not sleepy but is experiencing drowsiness, is defined as a state requiring assistance. However, the response request information may also be information for determining whether driver D is awake, sleepy, or in a state requiring assistance.

[0056] Examples of response requests being in the form of voice include: "You seem drowsy. Are you alright? Please respond," "Are you alright? If you are not drowsy, please say 'I'm alright'," and "We have determined that you are drowsy. Are you feeling drowsy?" In these examples, the response request is information that requests driver D to respond by voice.

[0057] The response request information may be information that requests driver D to respond with an action. For example, the response request information may be voice messages such as, "Are you okay? If you are not drowsy, please press the button," "Are you okay? If you are not drowsy, please turn the lever," or "Are you okay? If you are not drowsy, please turn your head."

[0058] The response request information can request an audio response indicating that driver D does not need wakefulness assistance, such as "I'm fine" or "No assistance needed," if driver D is not drowsy. The response request information can also request an action response indicating that wakefulness assistance is not needed, such as pressing a button, operating a lever, or turning one's head, if driver D is not drowsy.

[0059] If driver D is asleep or in need of assistance, driver D will either not respond at all or will respond in a way that differs from that of a awake state.

[0060] When requesting a response to determine whether the person is asleep or in need of assistance, the response request information can be as follows: For example, the response request information could be: "Are you okay? If you are not sleepy, please say 'I'm okay.' If you are sleepy, please say 'I'm sleepy.'"

[0061] In this case, if driver D is asleep, driver D is unlikely to respond at all. On the other hand, if driver D requires assistance, driver D will make voice responses such as "I'm sleepy," "I'm not okay," "I need assistance," or "Please."

[0062] The response request information may be a predetermined sound that indicates drowsiness detection. In this case, driver D needs to be aware that the predetermined sound indicates drowsiness detection. For example, the action of stopping the sound can be an action that indicates that driver D is alert.

[0063] The response request information may be a predetermined vibration that indicates drowsiness detection. In this case, driver D needs to be aware that the predetermined vibration indicates drowsiness detection. For example, the action of stopping the vibration can be an action that indicates that driver D is awake. The device that generates the vibration should be a part that is in contact with driver D. For example, a wearable device, seat, or steering wheel should vibrate. Alternatively, the device that generates the vibration may be a portable terminal carried by driver D.

[0064] The response request information may be displayed on the display device as predetermined content indicating drowsiness detection. For example, the display device may display content requesting a voice response, such as, "You seem drowsy. Are you okay? Please respond," or "Are you okay? If you are not drowsy, please say 'I'm okay'." Alternatively, the display device may display content requesting an action response, such as, "Are you okay? If you are not drowsy, please press the button."

[0065] By using voice, sound, and vibration as response request information, it becomes possible to encourage driver D to become alert if he is in need of assistance. For example, if driver D is drowsy, the occurrence of voice, sound, and vibration may cause driver D to become alert.

[0066] The acquisition unit 43 acquires driver D's response information to the response request information. The acquisition unit 43 acquires at least one of the following as response information: the voice spoken by driver D, the operation performed by driver D, and the body movements of driver D. As described above, the response request information is information requesting a response from driver D. However, the type of response from driver D can be set as appropriate. Therefore, the acquisition unit 43 acquires the response information corresponding to the response request information. By setting the system in advance to recognize driver D's voice, it is also possible to distinguish between driver D's voice and the voices of others. In addition, the acquisition unit 43 may be configured to acquire the voice following a predetermined command when that command is spoken as a trigger.

[0067] The determination unit 44 determines, based on the response information, whether driver D is awake or asleep. The determination unit 44 may also determine whether driver D is awake or in any other state. In other words, the determination unit 44 may determine whether driver D is awake, asleep, or in need of assistance.

[0068] Furthermore, the determination unit 44 may determine, based on the response information, whether the driver is awake, asleep, or in need of assistance. In this case, the determination unit 44 distinguishes between awake, asleep, and in need of assistance.

[0069] In this scenario, if the response request information is "You seem sleepy. Are you okay? Please respond," and driver D is awake, driver D may give various responses such as "I'm fine," "No problem," or "I'm fine." Also, if driver D is in need of assistance, driver D may give various responses such as "I'm sleepy" or "I might be at my limit."

[0070] Therefore, the determination unit 44 may analyze the response information by applying natural language processing. By analyzing the response information, the determination unit 44 can determine whether driver D is awake, asleep, or in need of assistance. The determination unit 44 can make the determination using an artificial intelligence model. In this case, the determination unit 44 may update the artificial intelligence model by learning driver D's response information. As a result, it becomes possible to make a determination that is appropriate for driver D.

[0071] If the response request information is information requesting an action response, the determination unit 44 can make a determination based on the action response. In this case, the determination unit 44 may also consider the voice response in addition to the action response. Similar to the voice response, the determination unit 44 can update its artificial intelligence model by learning the response information of driver D in the case of the action response.

[0072] The support request signal output unit 45 performs processing according to the determination result of the determination unit 44. The support request signal output unit 45 does not output a support request signal to the external terminal 30 if it is determined that the driver D is awake. On the other hand, the support request signal output unit 45 outputs a support request signal to the external terminal 30 if it is determined that the driver D is asleep. The support request signal output unit 45 may also output a support request signal to the external terminal 30 if it is determined that the driver D is in need of assistance.

[0073] The support request signal is, for example, a call signal. In this case, the support request signal output unit 45 outputs a call signal as the support request signal, with the external terminal 30 as the destination for the call. This allows the operator of the external terminal 30 to understand that driver D is asleep or in need of assistance. Furthermore, since the external terminal 30 and the awakening support device 20 are in a call state, the operator of the external terminal 30 can speak to driver D and guide driver D into an awakened state.

[0074] The support request signal output unit 45 may emit a machine voice from its speaker to the driver D until a call is initiated between the awakening support device 20 and the external terminal 30. For example, the support request signal output unit 45 may be configured to converse with the driver D based on the driver D's response information using an artificial intelligence model. This allows the driver D to be guided into an awakened state during the time it takes to connect with the operator of the external terminal 30.

[0075] The input information acquisition unit 46 acquires information for adjusting the reference value, specifically input information from driver D. The input information includes driver D's physical condition, driving route, continuous driving time from the present, and continuous driving time up to the present. The input information may also include information that directly changes the reference value.

[0076] The driving status acquisition unit 47 acquires information detected by sensors installed on the mobility M, which is used to adjust reference values. This information includes brake operation, accelerator operation, steering operation, distance to the mobility vehicle in front, distance to the driving lane, etc. If driver D is drowsy, at least one of the following is likely to occur: for example, sudden deceleration, sudden acceleration, sudden steering, changes in the distance to the mobility vehicle in front, or changes in the distance to the lane. In other words, the driving status acquisition unit 47 detects actions caused by driver D's drowsiness.

[0077] The reference value adjustment unit 48 adjusts the reference value stored in the reference value storage unit 51 based on the determination result of the determination unit 44. Here, the relationship between changes in biological information over time and the state of wakefulness, sleep, and need for assistance differs from person to person. Therefore, it is desirable for the response request information output unit 42 to determine whether or not to output response request information using a reference value that is appropriate for the target driver D.

[0078] For example, the reference value adjustment unit 48 adjusts the reference value to increase when the determination unit 44 determines that driver D is awake. The reference value adjustment unit 48 also adjusts the reference value to decrease when the determination unit 44 determines that driver D is asleep. The reference value adjustment unit 48 does not change the reference value when the determination unit 44 determines that driver D is in need of assistance.

[0079] By raising the threshold value when driver D is awake, response request information will not be output when driver D is awake in the future. Conversely, by lowering the threshold value when driver D is asleep, response request information will be output before driver D falls asleep in the future.

[0080] However, the reference value adjustment unit 48 may adjust to raise the reference value when it is determined that driver D is awake, and not change the reference value in other states. Alternatively, the reference value adjustment unit 48 may adjust to lower the reference value when it is determined that driver D is asleep, and not change the reference value in other states.

[0081] The reference value adjustment unit 48 may adjust the reference value based on the input information acquired by the input information acquisition unit 46. For example, if information indicating that driver D is in poor health is input, the reference value adjustment unit 48 will adjust the reference value to lower it.

[0082] The reference value adjustment unit 48 may adjust the reference value based on the information acquired by the driving condition acquisition unit 47. For example, if sudden deceleration or sudden acceleration occurs more frequently than the reference value, the reference value adjustment unit 48 will adjust the reference value to be lowered.

[0083] 6. According to the Awakening Support System 1, the estimation unit 41 estimates the driver D's level of drowsiness based on the driver D's biological information. Therefore, it is possible to estimate that microsleep may be occurring, which may not be detectable from the driver D's facial image.

[0084] According to the Awakening Support System 1, if driver D's level of drowsiness is higher than a standard value, the response request information output unit 42 outputs response request information to driver D. Then, the determination unit 44 determines whether driver D is awake or asleep based on driver D's response information. If it is determined that driver D is asleep based on driver D's response information, the support request signal output unit 45 outputs a support request signal to a pre-configured external terminal 30. Therefore, when a support request signal is output to the external terminal 30, the person operating the external terminal 30 can take action to wake driver D. In this way, driver D can be woken in real time when he is asleep, including in a microsleep state.

[0085] Furthermore, if driver D's level of drowsiness is higher than the standard value, instead of immediately outputting to the external terminal 30, the wakefulness support device 20 can first check driver D's response information to the response request information. Then, if it is determined that driver D is in a sleep state based on the response information, the support request signal output unit 45 outputs a support request signal to the external terminal 30. On the other hand, if it is determined that driver D is awake based on the response information, the support request signal output unit 45 does not output a support request signal to the external terminal 30.

[0086] In this way, after confirming the driver D's response information, an assistance request signal is output to the external terminal 30. Therefore, if driver D is awake, no assistance request signal is output to the external terminal 30, thus preventing the output of unnecessary assistance request signals. Also, if driver D is asleep, an assistance request signal is output to the external terminal 30. Therefore, the person operating the external terminal 30 can take appropriate action to wake up the driver D.

[0087] Therefore, if driver D experiences drowsiness, including microsleep, the external terminal 30 can be notified at an appropriate time according to driver D's condition.

[0088] Furthermore, the determination unit 44 determines, based on the response information, whether driver D is awake, asleep, or in need of assistance. The assistance request signal output unit 45 outputs an assistance request signal to the external terminal 30 if it is determined that driver D is in need of assistance. If driver D is neither awake nor asleep, there is a possibility that they are experiencing drowsiness. If driver D is in such a state requiring assistance, an assistance request signal is output. Therefore, the person operating the external terminal 30 can take action to wake up driver D if they are experiencing drowsiness.

[0089] Furthermore, the response request information output unit 42 outputs at least one of the following to the driver D as response request information: voice, sound, screen display, and vibration. This allows the driver D to respond to the response request information.

[0090] Furthermore, the acquisition unit 43 acquires at least one of the following as response information: the voice spoken by driver D, the operation performed by driver D, and the body movements of driver D. Based on this, the determination unit 44 can determine whether driver D is awake or not, etc., based on the response information of driver D.

[0091] Furthermore, the reference value adjustment unit 48 adjusts the reference value to increase when the determination unit 44 determines that driver D is awake. This prevents response request information from being output when driver D is awake in the future. Also, the reference value adjustment unit 48 adjusts the reference value to decrease when the determination unit 44 determines that driver D is asleep. This allows response request information to be output before driver D falls asleep in the future.

[0092] Furthermore, the reference value adjustment unit 48 does not change the reference value if the determination unit 44 determines that driver D is in need of assistance. In this case, the reference value is appropriate and therefore does not need to be adjusted. In this way, the reference value can be adjusted to suit driver D, and response request information can be appropriately output to driver D.

[0093] Furthermore, the input information acquisition unit 46 acquires information such as driver D's physical condition, driving route, continuous driving time from the present, continuous driving time to the present, or information to change the reference value as input information from driver D. The reference value adjustment unit 48 adjusts the reference value based on the input information. This makes it possible to output appropriate response request information to driver D based on the information that driver D himself is aware of.

[0094] Furthermore, the external terminal 30 is the terminal of another driver, the terminal of the organization to which driver D belongs, or the terminal of a driver support organization. The support request signal output unit 45 outputs a call signal as a support request signal, designating the external terminal 30 as the destination for the call. As a result, the call is initiated without driver D having to perform any operation. Therefore, the person operating the external terminal 30 or an external server equipped with artificial intelligence (speech recognition, language generation, and output functions) can begin a call with driver D without driver D having to perform any operation.

[0095] Furthermore, the support request signal output unit 45 is configured to use an artificial intelligence model to converse with the driver D based on the driver D's response information from the time it outputs the support request signal until a call with the external terminal 30 is initiated. This makes it possible to guide the driver D into an alert state before a call is initiated with the person operating the external terminal 30.

[0096] Furthermore, the awakening support device 20 is either a portable terminal of driver D or a device mounted in the driver's cab A of mobility M. These devices ensure that the above can be reliably achieved.

[0097] Furthermore, the biometric information estimation device 10 includes a wearable device attached to the driver D as a sensor for detecting information to estimate biometric information. This allows for the accurate estimation of the driver D's biometric information.

[0098] Furthermore, by using heart rate as the biometric information, the degree of driver D's drowsiness can be appropriately estimated.

[0099] Furthermore, the degree of drowsiness is calculated based on the number of times a predetermined change in the heart rate interval occurs within a predetermined time period. This allows for an accurate estimation of driver D's drowsiness level.

[0100] (Modified form) In Embodiment 1, the awakening support device 20 includes a reference value adjustment unit 48, but it does not have to include the reference value adjustment unit 48. In this case, the awakening support device 20 does not have to include an input information acquisition unit 46 and an operating status acquisition unit 47.

[0101] The reference value adjustment unit 48 may adjust the reference value based solely on the determination result of the determination unit 44. In this case, the awakening support device 20 does not need to include the input information acquisition unit 46 and the operating status acquisition unit 47.

[0102] Furthermore, the reference value adjustment unit 48 may adjust the reference value based only on the input information acquired by the input information acquisition unit 46. In this case, the awakening support device 20 does not need to include the operating status acquisition unit 47.

[0103] Furthermore, the reference value adjustment unit 48 may adjust the reference value based solely on the information acquired by the operating status acquisition unit 47. In this case, the awakening support device 20 does not need to include the input information acquisition unit 46.

[0104] Furthermore, although the determination unit 44 is configured to determine whether the person is awake, asleep, or in need of assistance, it may be configured to determine only the awake and asleep states, and not to determine whether the person is in need of assistance.

[0105] (Embodiment 2) In Embodiment 2, the support request signal output unit 45 of the awakening support device 20 differs from that in Embodiment 1. The support request signal output unit 45 is a signal to request the external terminal 30 to initiate a call from the external terminal 30 to the driver D's communication terminal.

[0106] In this case, the awakening support device 20 may constitute the driver D's communication terminal. Alternatively, the driver D's communication terminal may be a separate terminal from the awakening support device 20.

[0107] (Embodiment 3) In Embodiment 3, the support request signal output unit 45 of the awakening support device 20 differs from that in Embodiment 1. When the support request signal output unit 45 determines that the driver D is in a sleep state, it outputs a first support request signal to the external terminal 30 as a support request signal. When the support request signal output unit 45 determines that the driver D is in need of support, it outputs a second support request signal to the external terminal 30 as a support request signal, which is different from the first support request signal.

[0108] The first and second support request signals can, for example, be signals that include urgency information. The first support request signal is a signal with a higher degree of urgency than the second support request signal. Alternatively, the first and second support request signals can be, for example, call signals originating from different external terminals 30. For example, the first support request signal may be a call signal originating from the management company or call center because of its relatively higher degree of urgency, while the second support request signal may be a call signal originating from another driver because of its relatively lower degree of urgency.

[0109] (Embodiment 4) In Embodiment 4, the sensor 11 of the awakening support system 1 differs from that in Embodiment 1. The sensor 11 is a device attached to the seat in which the driver D sits. The sensor 11 may be placed on the seat surface or positioned inside the seat surface. The sensor 11 detects the pulsation of blood vessels and cardiac pulsation in the buttocks or thighs of the driver D seated in the seat as biological information.

[0110] Furthermore, the sensor 11 may be placed not only on the seat surface but also on the backrest. In this case, it may be placed on the surface of the backrest of the seat, or it may be placed inside the backrest of the seat.

[0111] (Embodiment 5) In Embodiment 5, the sensor 11 of the awakening support system 1 differs from that in Embodiment 1. The sensor 11 is positioned on the grip portion of the steering wheel of the mobility M. In this case, the sensor 11 detects the pulsation of the blood vessels in the driver D's hand as biological information.

[0112] (Embodiment 6) The awakening support system 2 in Embodiment 6 will be described with reference to Figure 5. The awakening support system 2 includes a biological information estimation device 50. The biological information estimation device 50 includes a sensor 11 and a biological information estimation unit 52.

[0113] Sensor 11 is the same as sensor 11 in Embodiment 1. The biometric information estimation unit 52 has the same functions as biometric information estimation unit 12 in Embodiment 1. However, the biometric information estimation unit 52 is not located in the driver's cab A of Mobility M, but is located outside the driver's cab A. The biometric information estimation unit 52 is located, for example, in the cloud and is configured to communicate with sensor 11. The biometric information estimation unit 52 is also configured to communicate with the awakening support device 20.

[0114] By placing the biometric information estimation unit 52 in the cloud, the biometric information estimation unit 52 can, for example, perform complex and high-performance computational processing.

[0115] (Other) Sensor 11 may also be a camera, optical sensor, radar, etc. The camera, for example, captures an image of driver D. In this case, the biometric information estimation unit 12 can estimate driver D's biometric information based on the image captured by the camera. The optical sensor or radar, for example, detects the movement of driver D's skin. In this case, the biometric information estimation unit 12 can estimate driver D's biometric information based on the detected skin movement.

[0116] In the above embodiments, the awakening support systems 1 and 2 were described. The awakening support system 1 can be implemented as a whole, or only the awakening support device 20 can be implemented. It can also be considered as an awakening support program for realizing the awakening support device 20. It can also be considered as an awakening support method for realizing the awakening support device 20.

Claims

1. Awakening support system (1, 2) comprising: a biometric information estimation device (10, 50) for estimating the biometric information of a driver (D) of a mobility (M); and an awakening support device (20) for supporting the driver's awakening based on the biometric information, wherein the awakening support device comprises: an estimation unit (41) for estimating the driver's level of drowsiness based on the biometric information; a response request information output unit (42) for outputting response request information to the driver to request a response from the driver when it is determined that the level of drowsiness is higher than a standard value; an acquisition unit (43) for acquiring the driver's response information to the response request information; a determination unit (44) for determining whether the driver is awake or asleep based on the response information; and a support request signal output unit (45) for not outputting a support request signal to a pre-configured external terminal (30) located outside the driver's cab (A) of the mobility when it is determined that the driver is awake, and for outputting the support request signal to the external terminal when it is determined that the driver is asleep.

2. The wakefulness support system according to claim 1, wherein the determination unit determines, based on the response information, whether the driver is in the awake state, the sleep state, or the state requiring assistance, and the assistance request signal output unit outputs the assistance request signal to the external terminal when it is determined that the driver is in the state requiring assistance.

3. The wakefulness support system according to claim 2, wherein the support request signal output unit outputs a first support request signal to the external terminal as the support request signal when it is determined that the driver is in a sleep state, and outputs a second support request signal, different from the first support request signal, to the external terminal as the support request signal when it is determined that the driver is in a state requiring support.

4. The wakefulness support system according to claim 1, wherein the response request information output unit outputs at least one of voice, sound, screen display, and vibration to the driver as the response request information.

5. The awakening support system according to claim 1, wherein the acquisition unit acquires at least one of the following as response information: a voice spoken by the driver, an operation performed by the driver, and the driver's body movements.

6. The wakefulness support system according to claim 1, further comprising a reference value adjustment unit (48) that raises the reference value when the determination unit determines that the driver is in the wakeful state, or lowers the reference value when the determination unit determines that the driver is in the sleep state.

7. The wakefulness support system according to claim 2, further comprising a reference value adjustment unit (48), wherein the reference value adjustment unit raises the reference value when the determination unit determines that the driver is in the awake state, lowers the reference value when the determination unit determines that the driver is in the sleep state, and does not change the reference value when the determination unit determines that the driver is in the state requiring assistance.

8. The awakening support system according to claim 1, further comprising: an input information acquisition unit (46) that acquires, as input information from the driver, the driver's physical condition information, driving route, continuous driving time from the present, continuous driving time to the present, or information to change the reference value; and a reference value adjustment unit (48) that adjusts the reference value based on the input information.

9. The awakening support system according to claim 1, wherein the external terminal is the terminal of another driver, the terminal of the organization to which the driver belongs, or the terminal of a driver support organization, and the support request signal output unit outputs a call signal as the support request signal, with the external terminal as the destination of the call.

10. The awakening support system according to claim 1, wherein the external terminal is the terminal of another driver, the terminal of the organization to which the driver belongs, or the terminal of a driver support organization, and the support request signal is a signal for requesting the external terminal to initiate a call from the external terminal to the driver's communication terminal.

11. The awakening support system according to claim 9 or 10, wherein the support request signal output unit is configured to use an artificial intelligence model to converse with the driver based on the driver's response information from the time the support request signal is output until a call with the external terminal is initiated.

12. The awakening support system according to claim 1, wherein the awakening support device is the driver's portable terminal.

13. The awakening support system according to claim 1, wherein the awakening support device is a device mounted in the driver's cab of the mobility vehicle.

14. The awakening support system according to claim 1, wherein the biological information estimation device includes a wearable device attached to the driver as a sensor for detecting information for estimating the biological information.

15. The awakening support system according to claim 1, wherein the biological information estimation device includes a device attached to the seat in which the driver sits, as a sensor for detecting information for estimating the biological information.

16. The wakefulness support system according to claim 1, wherein the biological information is a heart rate.

17. The wakefulness support system according to claim 15, wherein the degree of drowsiness is the number of times a predetermined change in the interval between heartbeats occurs over a predetermined period of time, or is calculated based on the number of such occurrences.

18. The wakefulness support system according to claim 17, wherein the predetermined temporal change in the heartbeat interval is a waveform in which the heartbeat interval decreases by a first predetermined value or more, then increases by a second predetermined value or more, and then fluctuates in waves over multiple periods.

19. Awakening support device (20) for assisting the wakefulness of a driver (D) of a mobility (M) based on the driver's biological information, comprising: an estimation unit (41) for estimating the driver's level of drowsiness based on the biological information; a response request information output unit (42) for outputting response request information to the driver to request a response from the driver when it is determined that the level of drowsiness is higher than a standard value; an acquisition unit (43) for acquiring the driver's response information to the response request information; a determination unit (44) for determining whether the driver is awake or asleep based on the response information; and a support request signal output unit (45) for not outputting a support request signal to a pre-configured external terminal located outside the driver's cab of the mobility when it is determined that the driver is awake, and for outputting the support request signal to the external terminal when it is determined that the driver is asleep.

20. Awakening support program (P) for assisting the wakefulness of a driver (D) of a mobility (M) based on the driver's biometric information, the program comprising: a step (41) of estimating the driver's level of drowsiness based on the biometric information; a step (42) of outputting response request information to the driver to request a response from the driver if it is determined that the level of drowsiness is higher than a standard value; a step (43) of acquiring the driver's response information to the response request information; a step (44) of determining whether the driver is awake or asleep based on the response information; and a step (45) of not outputting a support request signal to a pre-configured external terminal located outside the driver's cab of the mobility when it is determined that the driver is awake, and outputting the support request signal to the external terminal when it is determined that the driver is asleep.

Images