Wakefulness estimation device and wakefulness estimation method

Abstract

The present invention comprises: a face image acquisition unit (102) that acquires a face image of a subject; an upper eyelid opening degree calculation unit (103) that calculates an opening degree of an upper eyelid of the subject on the basis of the face image of the subject; an upper-lower eyelid distance parameter calculation unit (104) that, when the opening degree of the upper eyelid of the subject is smaller than a first opening degree threshold value, calculates a parameter relating to the distance between the upper and lower eyelids of the subject on the basis of the face image of the subject; an eye closure transition parameter calculation unit (105) that calculates, when the opening degree of the upper eyelid of the subject is smaller than the first opening degree threshold value, a parameter relating to eye closure transition from a state in which the opening degree of the upper eyelid of the subject is greater than a second opening degree threshold value to a state in which the opening degree of the upper eyelid of the subject is smaller than the first opening degree threshold value; a downward vision determination unit (106) that determines whether or not the subject is in a downward vision state on the basis of the parameter relating to the distance between the upper and lower eyelids of the subject and the parameter relating to the eye closure transition of the subject; and a wakefulness estimation unit (107) that estimates wakefulness of the subject on the basis of the opening degree of the upper eyelid of the subject and the result of determination by the downward vision determination unit (106).

Description

Awareness level estimation device and awareness level estimation method

[0001] The present disclosure relates to an awareness level estimation device and an awareness level estimation method.

[0002] Conventionally, there has been known an awareness level estimation device that estimates the awareness level of a driver, that is, the presence or absence of drowsiness, based on the eye opening degree obtained from an image of the driver's face captured by an in-vehicle camera (see, for example, Patent Document 1).

[0003] Here, the driver may not only look ahead outside the vehicle but also direct their line of sight downward to look at the meters inside the vehicle or the screen of a navigation device. Hereinafter, directing the line of sight downward is referred to as "downward viewing". The eye opening degree during this downward viewing becomes a value close to the eye opening degree when the awareness level decreases. Therefore, in the awareness level estimation device, it is an issue to prevent the device from erroneously determining that the awareness level has decreased when the driver is performing downward viewing.

[0004] Therefore, for example, in the conventional awareness level estimation device disclosed in Patent Document 1, the awareness level is estimated from the eye opening degree of the driver, and it is determined whether or not the driver is in a downward viewing state. More specifically, in the conventional awareness level estimation device, the eye opening degree of the driver is calculated based on the distance between the highest point of the upper eyelid contour line and the lowest point of the lower eyelid contour line, and it is determined whether or not the driver is in a downward viewing state based on the shapes of the upper eyelid contour line and the lower eyelid contour line. Then, in the conventional awareness level estimation device, when it is determined that the driver is in a downward viewing state, the awareness level of the driver estimated from the eye opening degree is corrected in the direction of increasing the awareness level to prevent the above-mentioned erroneous determination.

[0005] Japanese Patent Application Laid-Open No. 2008-171065

[0006] However, when the driver is half asleep and enduring drowsiness, the distance between the driver's upper and lower eyelids can take a large value. In such a case, in the conventional awareness level estimation device, a value similar to that during downward viewing may be obtained, and there is a possibility of misdetecting it as downward viewing. Note that such an event is not limited to the case where the subject for estimating the awareness level is a driver, but can also occur with other subjects.

[0007] This disclosure was made to solve the above-mentioned problems and aims to provide an alertness estimation device that can estimate the alertness level of a subject with higher accuracy than conventional devices.

[0008] The alertness estimation device relating to this disclosure includes: a face image acquisition unit that acquires a face image of a subject; an upper eyelid opening calculation unit that calculates the degree of upper eyelid opening of the subject based on the face image of the subject acquired by the face image acquisition unit; an upper and lower eyelid distance parameter calculation unit that calculates a parameter relating to the distance between the subject's upper and lower eyelids based on the face image of the subject acquired by the face image acquisition unit when the degree of upper eyelid opening of the subject calculated by the upper eyelid opening calculation unit is less than a first opening threshold; and a second unit that calculates a parameter relating to the distance between the subject's upper and lower eyelids when the degree of upper eyelid opening of the subject calculated by the upper eyelid opening calculation unit is less than a first opening threshold. The system is characterized by comprising: a closed-eye transition parameter calculation unit that calculates parameters relating to the transition of closed eyes from a state greater than an opening threshold to a state less than the first opening threshold; a downward gaze determination unit that determines whether the subject is in a downward gaze state based on parameters relating to the distance between the subject's upper and lower eyelids calculated by an upper and lower eyelid distance parameter calculation unit and parameters relating to the subject's closed-eye transition calculated by a closed-eye transition parameter calculation unit; and an alertness estimation unit that estimates the subject's alertness level based on the opening of the subject's upper eyelids calculated by an upper eyelid opening calculation unit and the determination result by the downward gaze determination unit.

[0009] According to this disclosure, the above configuration makes it possible to estimate the level of alertness of the subject with higher accuracy than in the past.

[0010] This is a block diagram showing an example configuration of the alertness estimation device according to Embodiment 1. Figures 2A and 2B are diagrams showing an example of how to calculate percentile values ​​by the upper and lower eyelid distance parameter calculation unit in Embodiment 1, where Figure 2A is a diagram for explaining percentile values ​​and Figure 2B is a diagram for explaining linear interpolation. This is a flowchart showing an example of operation of the alertness estimation device according to Embodiment 1. This diagram is for explaining the difference in eye-closing transition time between when the occupant is drowsy and when the occupant looks downwards. This is a block diagram showing an example configuration of the alertness estimation device according to Embodiment 2. This is a flowchart showing an example of operation of the alertness estimation device according to Embodiment 2. Figures 7A and 7B are block diagrams showing hardware configuration examples of the alertness estimation devices according to Embodiments 1 and 2.

[0011] The embodiments will be described in detail below with reference to the drawings. Embodiment 1. Figure 1 is a block diagram showing an example configuration of the arousal level estimation device 1 according to Embodiment 1. The arousal level estimation device 1 is a device for estimating the arousal level of a subject. The subject is the person whose arousal level is estimated by the arousal level estimation device 1. Furthermore, the subject is not limited to one person, but may be multiple people.

[0012] The following explanation will use the example of the alertness level estimation device 1 estimating the alertness level of the vehicle's occupants. In this case, the subjects are the vehicle's occupants. Examples of vehicle occupants whose alertness level the alertness level estimation device 1 will estimate include the driver, but any occupant of the vehicle is acceptable, and it may include occupants other than the driver. Furthermore, the vehicle occupants whose alertness level the alertness level estimation device 1 will estimate are not limited to one person, but may include multiple people. For example, the subjects could be the driver and the passenger in the front seat, the driver and the passenger in the back seat, or all occupants of the vehicle.

[0013] Furthermore, the alertness estimation device 1 may be installed in the vehicle, for example, or it may be built on a portable device that can be brought into the vehicle, such as a mobile phone, smartphone, or portable navigation device. Also, part or all of the alertness estimation device 1 may be built on a server that can communicate with the vehicle.

[0014] As shown in Figure 1, for example, this alertness estimation device 1 includes an imaging unit 101, a face image acquisition unit 102, an upper eyelid opening degree calculation unit 103, an upper and lower eyelid distance parameter calculation unit 104, an eye-closing transition parameter calculation unit 105, a downward gaze determination unit 106, and an alertness estimation unit 107.

[0015] The imaging unit 101 obtains images by imaging the interior of the vehicle. This imaging unit 101 is positioned to capture images of the faces of occupants whose alertness level the alertness level estimation device 1 is trying to estimate. This imaging unit 101 takes images continuously or at predetermined intervals.

[0016] Although Figure 1 shows the imaging unit 101 being located inside the alertness estimation device 1, the imaging unit 101 may be located outside the alertness estimation device 1. For example, if the alertness estimation device 1 is built on a server capable of communicating with the vehicle, the imaging unit 101 will be located outside the alertness estimation device 1.

[0017] The face image acquisition unit 102 acquires the face images of the crew members. If there are multiple crew members, the face image acquisition unit 102 acquires a face image for each crew member. The face image acquisition unit 102 acquires the face images of the crew members continuously or at predetermined intervals.

[0018] For example, the face image acquisition unit 102 may acquire a face image of a crew member by using conventionally known face recognition technology to extract the region in which the crew member's face is captured from the captured image obtained by the imaging unit 101. Alternatively, the process of extracting the region in which the crew member's face is captured from the captured image using the above-mentioned face recognition technology may be performed by the imaging unit 101 instead of the face image acquisition unit 102. In this case, the face image acquisition unit 102 only needs to acquire the face image of the crew member by acquiring the region in which the crew member's face is captured extracted by the imaging unit 101.

[0019] The upper eyelid opening degree calculation unit 103 calculates the degree of upper eyelid opening of a crew member based on the crew member's facial image acquired by the facial image acquisition unit 102. If there are multiple crew members, the upper eyelid opening degree calculation unit 103 calculates the degree of upper eyelid opening for each crew member. The upper eyelid opening degree calculation unit 103 calculates the degree of upper eyelid opening of crew members continuously or at predetermined intervals.

[0020] For example, first, the upper eyelid opening degree calculation unit 103 detects the position of the inner corner of the eye, the position of the outer corner of the eye, and the position of the vertex of the upper eyelid from the face image of the occupant acquired by the face image acquisition unit 102. The vertex of the upper eyelid is the point on the upper eyelid contour line that is furthest from the line connecting the inner corner and outer corner of the eye on the vertical line. Next, the upper eyelid opening degree calculation unit 103 calculates a first distance, which is the distance between the line connecting the inner corner and outer corner of the eye and the position of the vertex of the upper eyelid. Next, the upper eyelid opening degree calculation unit 103 calculates the opening degree of the upper eyelid based on this first distance. In this case, for example, first, the upper eyelid opening degree calculation unit 103 calculates the flattening ratio by dividing the first distance by the distance between the inner corner and outer corner of the eye. Furthermore, the upper eyelid opening degree calculation unit 103 may use the value obtained by dividing the flattening ratio by a reference value as the upper eyelid opening degree.

[0021] Furthermore, the upper eyelid opening calculation unit 103 may use, for example, a predetermined fixed value as a reference value for the eye's flatness. Also, if the alertness estimation device 1 can identify individual occupants through facial recognition using facial images, the upper eyelid opening calculation unit 103 may use, for example, different values ​​for each occupant as a reference value for the eye's flatness. By using different reference values ​​for each occupant, it is possible to suppress the influence of individual differences in eye size on the calculation result of the upper eyelid opening.

[0022] The upper and lower eyelid distance parameter calculation unit 104 calculates a parameter related to the distance between the upper and lower eyelids of a occupant based on the occupant's face image acquired by the face image acquisition unit 102, when the occupant's upper eyelid opening degree calculated by the upper eyelid opening degree calculation unit 103 is smaller than a first opening degree threshold. If there are multiple occupants, the upper and lower eyelid distance parameter calculation unit 104 calculates the parameter related to the distance between the upper and lower eyelids of each occupant.

[0023] Furthermore, the interval in which the opening of the crew member's upper eyelid, calculated by the upper eyelid opening calculation unit 103, is smaller than the first opening threshold is also called the closed-eye interval.

[0024] For example, the upper and lower eyelid distance parameter calculation unit 104 may calculate the percentile value of the distance between the upper and lower eyelids of a crew member as a parameter relating to the distance between the crew member's upper and lower eyelids. In this case, for example, first, the upper and lower eyelid distance parameter calculation unit 104 calculates the distance between the upper and lower eyelids of each crew member based on multiple crew member face images taken in the closed-eyes section of the face images of the crew member acquired by the face image acquisition unit 102. At this time, for example, the upper and lower eyelid distance parameter calculation unit 104 calculates the distance between the upper and lower eyelids as the distance between the vertex of the upper eyelid and the intersection of the vertical line of the vertex of the upper eyelid and the contour line of the lower eyelid. Then, the upper and lower eyelid distance parameter calculation unit 104 calculates the percentile value based on the multiple distances between the upper and lower eyelids in the closed-eyes section. Note that the percentile value can be any percentile value; for example, the upper and lower eyelid distance parameter calculation unit 104 may calculate the 50th percentile value, i.e., the median.

[0025] Here, the method for calculating percentile values ​​by the upper and lower eyelid distance parameter calculation unit 104 will be explained with reference to Figure 2. For example, suppose there is a data set {a0, a1, ..., an} consisting of n+1 data points, and each data point is arranged in ascending order of value. In this case, the percentile value (ai) corresponding to each data point is expressed by the following equation (1): ai = (100 / n) × i (1)

[0026] Figure 2 shows an example where n = 9 (10 data points). To find the 100th percentile from the above data, we extract the data point in Figure 2 that has a percentile value of 100. In the example in Figure 2, the data point with a percentile value of 100 is a9, so the 100th percentile is a9 = 12. To find the 20th percentile from the above data, there is no data point with a percentile value of 20 in the data point in Figure 2. Therefore, we perform linear interpolation using the values ​​before and after 20 (11.1 and 22.2 in the example in Figure 2) to find the 20th percentile. Linear interpolation is expressed by the following equation (2). In equation (2), y represents the percentile value we want to find, y0 represents the percentile value immediately preceding y in the percentile values ​​of the data set, y1 represents the percentile value immediately following y in the percentile values ​​of the data set, x0 represents the data value corresponding to y0, and x1 represents the data value corresponding to y1. In the example in Figure 2, substituting y=20, y0=11.1, y1=22.2, x0=1, and x1=3 into equation (2) gives x=2.6036... x={(y-y0)(x0-x1) / (y0-y1)}+x0 (2)

[0027] Alternatively, for example, the upper and lower eyelid distance parameter calculation unit 104 may calculate the average value of the distance between the upper and lower eyelids of the occupants as a parameter related to the distance between the occupants' upper and lower eyelids. In this case, for example, first, the upper and lower eyelid distance parameter calculation unit 104 calculates the distance between the upper and lower eyelids of each occupant based on multiple occupant face images acquired by the face image acquisition unit 102 during the closed-eyes period. Then, the upper and lower eyelid distance parameter calculation unit 104 calculates the average value based on the multiple upper and lower eyelid distances during the closed-eyes period.

[0028] The eye-closing transition parameter calculation unit 105 calculates parameters related to the eye-closing transition of a occupant when the upper eyelid opening degree of the occupant calculated by the upper eyelid opening degree calculation unit 103 is smaller than a first opening degree threshold. The eye-closing transition of an occupant refers to the change from a state in which the opening degree of the occupant's upper eyelid is greater than a second opening degree threshold to a state in which it is smaller than the first opening degree threshold. The second opening degree threshold is a threshold value greater than the first opening degree threshold. In other words, the eye-closing transition of an occupant refers to the change from an open eye state to a closed eye state. If there are multiple occupants, the eye-closing transition parameter calculation unit 105 calculates parameters related to the eye-closing transition for each occupant.

[0029] For example, the eye-closed transition parameter calculation unit 105 may calculate the time required to transition from the open-eye state to the closed-eye state as a parameter related to the eye-closed transition. The time required to transition from the open-eye state to the closed-eye state is also called the eye-closed transition time.

[0030] For example, first, the eye-closing transition parameter calculation unit 105 calculates the degree of upper eyelid opening from multiple facial images of the occupant in any interval before closing their eyes. If there is another eye-closing interval within that interval, it calculates the degree of upper eyelid opening from multiple facial images in the interval after that eye-closing interval. Then, the eye-closing transition parameter calculation unit 105 normalizes the degree of upper eyelid opening based on the above-mentioned degrees of upper eyelid opening from multiple upper eyelid openings in any interval before the occupant closes their eyes. Through this normalization, the frame in which the eyes were widest open becomes 1, and the frame in which the eyes were closed becomes 0. Then, the eye-closing transition parameter calculation unit 105 defines the time required for the normalized degree of upper eyelid opening to decrease from an arbitrary value to an arbitrary value as the eye-closing transition time.

[0031] Furthermore, for example, the closed-eye transition parameter calculation unit 105 may calculate the speed at which the eyes transition from the open-eye state to the closed-eye state as a parameter related to the closed-eye transition. That is, since the closed-eye transition time can also be determined by detecting the speed, the closed-eye transition parameter calculation unit 105 may calculate this speed. The speed calculated by the closed-eye transition parameter calculation unit 105 may be a representative value such as the maximum or minimum value of the speed at which the eyes transition from the open-eye state to the closed-eye state, or the average value of the speed at which the eyes transition from the open-eye state to the closed-eye state.

[0032] The downward gaze detection unit determines whether a occupant is in a downward gaze state based on the parameter related to the distance between the occupant's upper and lower eyelids calculated by the upper and lower eyelid distance parameter calculation unit 104, and the parameter related to the eye-closing transition calculated by the eye-closing transition parameter calculation unit 105. If there are multiple occupants, the downward gaze detection unit determines whether each occupant is in a downward gaze state.

[0033] Here, for example, if the parameter related to the eye-closing transition calculated by the eye-closing transition parameter calculation unit 105 is the eye-closing transition time, the downward gaze determination unit 106 determines that the occupant is in a downward gaze state if the parameter related to the distance between the occupant's upper and lower eyelids calculated by the upper and lower eyelid distance parameter calculation unit 104 is greater than the distance threshold, and the eye-closing transition time calculated by the eye-closing transition parameter calculation unit 105 is shorter than the time threshold.

[0034] The determination by the downward gaze determination unit 106 is performed when the opening of the occupant's upper eyelids, calculated by the upper eyelid opening degree calculation unit 103, is smaller than the first opening degree threshold, that is, when the calculation process is performed by the upper and lower eyelid distance parameter calculation unit 104 and the eye-closing transition parameter calculation unit 105.

[0035] Furthermore, the downward gaze determination unit 106 may use, for example, a predetermined fixed value as the distance threshold for the distance between the upper and lower eyelids. Also, if the alertness estimation device 1 can identify individual occupants by facial recognition using facial images, the downward gaze determination unit 106 may use, for example, different values ​​for each occupant as the distance threshold for the distance between the upper and lower eyelids. By using different distance thresholds for each occupant, it is possible to suppress the influence of individual differences in eye size when looking downwards on the determination result for the distance between the upper and lower eyelids.

[0036] The alertness estimation unit 107 estimates the alertness level of the crew member based on the degree of upper eyelid opening calculated by the upper eyelid opening calculation unit 103 and the determination result by the downward gaze determination unit 106. If there are multiple crew members, the alertness estimation unit 107 estimates the alertness level for each crew member.

[0037] In this case, the alertness estimation unit 107 estimates that the crew member's alertness level is reduced if the opening of the crew member's upper eyelid is smaller than the first opening threshold and the crew member is not looking downwards. On the other hand, in all other cases, the alertness estimation unit 107 estimates that the crew member's alertness level is not reduced.

[0038] Next, an example of the operation of the alertness level estimation device 1 according to Embodiment 1 shown in Figure 1 will be explained with reference to Figure 3.

[0039] In an example of the operation of the alertness estimation device 1 according to Embodiment 1 shown in Figure 1, for example, as shown in Figure 3, first, the imaging unit 101 obtains an image by imaging the inside of the vehicle (step ST101).

[0040] Next, the facial image acquisition unit 102 acquires facial images of the crew members (step ST102).

[0041] Next, the upper eyelid opening degree calculation unit 103 calculates the degree of upper eyelid opening of the occupant based on the occupant's face image acquired by the face image acquisition unit 102 (step ST103).

[0042] Next, the alertness estimation device 1 determines whether the opening of the occupant's upper eyelids, calculated by the upper eyelid opening calculation unit 103, is smaller than the first opening threshold (step ST104).

[0043] In this step ST104, when the arousal level estimation device 1 determines that the opening degree of the upper eyelid of the occupant calculated by the upper eyelid opening degree calculation unit 103 is not smaller than the first opening degree threshold value, that is, when it is determined that the opening degree of the upper eyelid of the occupant calculated by the upper eyelid opening degree calculation unit 103 is equal to or greater than the first opening degree threshold value, the sequence proceeds to step ST108.

[0044] On the other hand, in step ST104, when the arousal level estimation device 1 determines that the opening degree of the upper eyelid of the occupant calculated by the upper eyelid opening degree calculation unit 103 is smaller than the first opening degree threshold value, the upper and lower eyelid distance parameter calculation unit 104 calculates a parameter related to the distance between the upper and lower eyelids of the occupant based on the face image of the occupant acquired by the face image acquisition unit 102 (step ST105).

[0045] Further, the closed-eye transition parameter calculation unit 105 calculates a parameter related to the closed-eye transition until the opening degree of the upper eyelid of the subject changes from a state greater than the second opening degree threshold value to a state smaller than the first opening degree threshold value (step ST106).

[0046] Next, the downward gaze detection unit determines whether or not the occupant is in a downward gaze state based on the parameter related to the distance between the upper and lower eyelids of the occupant calculated by the upper and lower eyelid distance parameter calculation unit 104 and the parameter related to the closed-eye transition calculated by the closed-eye transition parameter calculation unit 105 (step ST107).

[0047] Next, the arousal level estimation unit 107 estimates the arousal level of the occupant based on the opening degree of the upper eyelid of the occupant calculated by the upper eyelid opening degree calculation unit 103 and the determination result by the downward gaze determination unit 106 (step ST108).

[0048] Here, FIG. 4 is a diagram for explaining the difference in the eye-closure transition time when the occupant is sleepy and when the occupant looks downward. As shown in FIG. 4, the eye-closure transition time is the time from when the opening degree of the upper eyelid of the occupant falls below the second opening degree threshold value until it falls below the first opening degree threshold value. And when the occupant is fighting sleep with half-closed eyes, for example, as indicated by reference numeral 41 in FIG. 4, from the open-eye state, after the drowsy state continues, it becomes the closed-eye state and then the open-eye state again. Therefore, when the occupant is sleepy like this, it is assumed that the time from the open-eye state to the closed-eye state becomes longer. On the other hand, when the occupant looks downward, for example, as indicated by reference numeral 42 in FIG. 4, from the open-eye state, it immediately becomes the downward-looking state and then the open-eye state again. Therefore, when the occupant looks downward like this, it is assumed that the time from the open-eye state to the closed-eye state becomes shorter.

[0049] Therefore, in the arousal level estimation device 1 according to Embodiment 1, a parameter related to the transition from the open-eye state to the closed-eye state of the occupant is calculated, and based on this parameter, it is determined whether the occupant is in the downward-looking state. Thereby, in the arousal level estimation device 1 according to Embodiment 1, it is possible to suppress misdetecting the half-closed eye state due to sleepiness as the downward-looking state compared to Embodiment 1.

[0050] In the above description, the case where the arousal level estimation device 1 estimates the arousal level of the vehicle occupant has been described as an example. However, it is not limited to this, and the arousal level estimation device 1 may be configured as a device that estimates the arousal level of a subject other than the vehicle occupant. For example, the arousal level estimation device 1 may be provided in a learning facility and configured as a device that estimates the arousal level of a person learning at the learning facility.

[0051] As described above, according to this embodiment 1, the arousal level estimation device 1 includes a face image acquisition unit 102 that acquires a face image of a subject, an upper eyelid opening degree calculation unit 103 that calculates the degree of upper eyelid opening of the subject based on the face image of the subject acquired by the face image acquisition unit 102, an upper and lower eyelid distance parameter calculation unit 104 that calculates a parameter relating to the distance between the subject's upper and lower eyelids based on the face image of the subject acquired by the face image acquisition unit 102 when the degree of upper eyelid opening of the subject calculated by the upper eyelid opening degree calculation unit 103 is smaller than a first opening threshold, and when the degree of upper eyelid opening of the subject calculated by the upper eyelid opening degree calculation unit 103 is smaller than a first opening threshold, the upper eyelid opening of the subject is smaller than the first opening The device includes: a closed-eye transition parameter calculation unit 105 that calculates parameters related to the eye-closing transition from a state greater than a second eye-opening threshold greater than the threshold to a state less than the first eye-opening threshold; a downward gaze determination unit 106 that determines whether the subject is in a downward gaze state based on parameters related to the distance between the subject's upper and lower eyelids calculated by the upper and lower eyelid distance parameter calculation unit 104 and parameters related to the subject's eye-closing transition calculated by the closed-eye transition parameter calculation unit 105; and an alertness estimation unit 107 that estimates the subject's alertness level based on the upper eyelid opening degree of the subject calculated by the upper eyelid opening degree calculation unit 103 and the determination result by the downward gaze determination unit 106. As a result, the alertness estimation device 1 according to Embodiment 1 can estimate the subject's alertness level with higher accuracy than conventional devices.

[0052] Furthermore, according to this embodiment 1, the eye-closing transition parameter calculation unit 105 calculates the time required for the subject's upper eyelid opening to transition from a state greater than a second opening threshold to a state less than a first opening threshold, as a parameter related to the eye-closing transition. Also, according to this embodiment 1, the downward gaze determination unit 106 determines that the subject is in a downward gaze state if the parameter related to the distance between the subject's upper and lower eyelids, calculated by the upper and lower eyelid distance parameter calculation unit 104, is greater than a distance threshold, and the time calculated by the eye-closing transition parameter calculation unit 105 is shorter than a time threshold. As a result, the alertness estimation device 1 according to embodiment 1 can estimate the subject's alertness with higher accuracy than conventional devices.

[0053] Furthermore, according to this embodiment 1, the eye-closing transition parameter calculation unit 105 calculates the speed at which the subject's upper eyelid opening transitions from a state greater than a second opening threshold to a state less than a first opening threshold, as a parameter related to the eye-closing transition. As a result, the alertness estimation device 1 according to embodiment 1 can estimate the subject's alertness with higher accuracy than conventional devices.

[0054] Furthermore, according to this embodiment 1, the alertness estimation method includes the steps of: the face image acquisition unit 102 acquires a face image of the subject; the upper eyelid opening degree calculation unit 103 calculates the degree of opening of the subject's upper eyelids based on the face image of the subject acquired by the face image acquisition unit 102; the upper and lower eyelid distance parameter calculation unit 104 calculates a parameter relating to the distance between the subject's upper and lower eyelids based on the face image of the subject acquired by the face image acquisition unit 102 when the degree of opening of the subject's upper eyelids calculated by the upper eyelid opening degree calculation unit 103 is smaller than a first opening threshold; and the eye-closing transition parameter calculation unit 105 calculates a parameter relating to the distance between the subject's upper and lower eyelids based on the face image of the subject acquired by the face image acquisition unit 102 when the degree of opening of the subject's upper eyelids calculated by the upper eyelid opening degree calculation unit 103 is smaller than a first opening threshold. The method includes the steps of: calculating parameters related to the eye-closing transition from a state in which the opening of the upper eyelid is greater than a second opening threshold which is greater than a first opening threshold, to a state in which it is less than the first opening threshold; the downward gaze determination unit 106 determining whether the subject is in a downward gaze state based on the parameters related to the distance between the subject's upper and lower eyelids calculated by the upper and lower eyelid distance parameter calculation unit 104 and the parameters related to the eye-closing transition of the subject calculated by the eye-closing transition parameter calculation unit 105; and the alertness estimation unit 107 estimating the alertness of the subject based on the opening of the subject's upper eyelid calculated by the upper eyelid opening calculation unit 103 and the determination result by the downward gaze determination unit 106.

[0055] Embodiment 2. Figure 5 shows an example of the configuration of the alertness level estimation device 1 according to Embodiment 2. In the alertness level estimation device 1 according to Embodiment 2 shown in Figure 5, a face orientation detection unit 108 and a downward gaze necessity determination unit 109 are added to the alertness level estimation device 1 according to Embodiment 1 shown in Figure 1. Other configuration examples of the alertness level estimation device 1 according to Embodiment 2 shown in Figure 5 are the same as the configuration example of the alertness level estimation device 1 according to Embodiment 1 shown in Figure 1, and only the different parts will be described using the same reference numerals.

[0056] The face orientation detection unit 108 detects the orientation of the subject's face. If there are multiple occupants, the face orientation detection unit 108 detects the orientation of each occupant's face. For example, the face orientation detection unit 108 detects the orientation of a occupant's face based on the occupant's face image acquired by the face image acquisition unit 102.

[0057] The downward gaze determination unit 109 determines whether or not to perform a downward gaze determination in the downward gaze determination unit 106 based on the orientation of the subject's face detected by the face orientation detection unit 108. If there are multiple occupants, the downward gaze determination unit 109 determines whether or not to perform a downward gaze determination for each occupant. For example, if the face of the subject detected by the face orientation detection unit 108 is tilted to the side from the front by an angle threshold or more, the downward gaze determination unit 109 determines that a downward gaze determination in the downward gaze determination unit 106 is unnecessary.

[0058] For example, the downward gaze requirement determination unit 109 may set an angle threshold such that the occupant's eyes cannot be detected from the occupant's face image acquired by the face image acquisition unit 102. Alternatively, for example, the downward gaze requirement determination unit 109 may set an angle threshold such that the occupant's eyes can be detected from the occupant's face image acquired by the face image acquisition unit 102, but the distance between the occupant's upper and lower eyelids calculated by the upper and lower eyelid distance parameter calculation unit 104 is less than or equal to the effective distance. In other words, if the face is tilted to the side from the front, it is expected that the detected distance between the upper and lower eyelids will deviate from the actual distance between the upper and lower eyelids. Therefore, face orientations in which the distance between the upper and lower eyelids deviates significantly from the detected value may be excluded from the determination target.

[0059] In Embodiment 2, the downward gaze determination unit 106 does not perform a downward gaze determination if the downward gaze determination unit 109 determines that a downward gaze determination is unnecessary. Similarly, in Embodiment 2, the upper and lower eyelid distance parameter calculation unit 104 and the eye-closing transition parameter calculation unit 105 do not need to perform calculation processing if the downward gaze determination unit 109 determines that a downward gaze determination is unnecessary.

[0060] Next, an example of the operation of the alertness level estimation device 1 according to Embodiment 2 shown in Figure 5 will be explained with reference to Figure 6.

[0061] In the example of operation of the alertness estimation device 1 according to Embodiment 2 shown in Figure 5, for example, as shown in Figure 6, first, the imaging unit 101 obtains an image by imaging the inside of the vehicle (step ST201).

[0062] Next, the facial image acquisition unit 102 acquires facial images of the crew members (step ST202).

[0063] Next, the upper eyelid opening degree calculation unit 103 calculates the degree of upper eyelid opening of the occupant based on the occupant's face image acquired by the face image acquisition unit 102 (step ST203).

[0064] Next, the alertness estimation device 1 determines whether the opening of the occupant's upper eyelids, calculated by the upper eyelid opening calculation unit 103, is smaller than the first opening threshold (step ST204).

[0065] In step ST204, if the alertness estimation device 1 determines that the opening of the occupant's upper eyelids calculated by the upper eyelid opening calculation unit 103 is not less than the first opening threshold, that is, if it determines that the opening of the occupant's upper eyelids calculated by the upper eyelid opening calculation unit 103 is equal to or greater than the first opening threshold, the sequence proceeds to step ST210.

[0066] On the other hand, in step ST204, if the alertness estimation device 1 determines that the upper eyelid opening degree of the occupant calculated by the upper eyelid opening degree calculation unit 103 is smaller than the first opening degree threshold, the upper and lower eyelid distance parameter calculation unit 104 calculates a parameter relating to the distance between the occupant's upper and lower eyelids based on the occupant's face image acquired by the face image acquisition unit 102 (step ST205).

[0067] Furthermore, the eye-closing transition parameter calculation unit 105 calculates parameters related to the eye-closing transition from a state where the opening of the subject's upper eyelid is greater than a second opening threshold to a state where it is less than a first opening threshold (step ST206).

[0068] Furthermore, the face orientation detection unit 108 detects the orientation of the subject's face. If there are multiple occupants, the face orientation detection unit 108 detects the orientation of each occupant's face (step ST207).

[0069] Next, the downward gaze determination unit 109 determines whether or not a downward gaze determination is necessary based on the orientation of the subject's face detected by the face orientation detection unit 108 (step ST208).

[0070] In step ST208, if the downward viewing requirement determination unit 109 determines that downward viewing is not required, that is, if it determines that downward viewing is unnecessary, the sequence proceeds to step ST210.

[0071] In step ST208, if the downward gaze determination unit 109 determines that a downward gaze determination is necessary, the downward gaze detection unit determines whether the occupant is in a downward gaze state based on the parameter relating to the distance between the occupant's upper and lower eyelids calculated by the upper and lower eyelid distance parameter calculation unit 104, and the parameter relating to the eye-closing transition calculated by the eye-closing transition parameter calculation unit 105 (step ST209).

[0072] Next, the alertness estimation unit 107 estimates the alertness of the occupant based on the degree of upper eyelid opening calculated by the upper eyelid opening calculation unit 103 and the determination result by the downward gaze determination unit 106 (step ST210). If the determination by the downward gaze determination unit 106 has not been performed, the alertness estimation unit 107 estimates the alertness of the occupant based on the degree of upper eyelid opening calculated by the upper eyelid opening calculation unit 103. For example, in this case, the alertness estimation unit 107 estimates that the occupant's alertness has decreased if it determines that the degree of upper eyelid opening is smaller than the first opening threshold. On the other hand, the alertness estimation unit 107 estimates that the occupant's alertness has not decreased in all other cases.

[0073] In the above description, the upper and lower eyelid distance parameter calculation unit 104 and the eye-closing transition parameter calculation unit 105 perform calculation processing regardless of the determination result by the downward gaze necessity determination unit 109. However, the calculation processing is not limited to this case; the upper and lower eyelid distance parameter calculation unit 104 and the eye-closing transition parameter calculation unit 105 do not need to perform calculation processing if the downward gaze necessity determination unit 109 determines that downward gaze is unnecessary.

[0074] As described above, according to this second embodiment, the device includes a face orientation detection unit 108 that detects the orientation of the subject's face, and a downward gaze determination unit 109 that determines whether or not to perform a downward gaze determination in the downward gaze determination unit 106 based on the orientation of the subject's face detected by the face orientation detection unit 108. The downward gaze determination unit 106 does not perform a downward gaze determination if the downward gaze determination unit 109 determines that it is unnecessary to perform a downward gaze determination. As a result, in the alertness estimation device 1 according to the second embodiment, downward gaze determination is unnecessary when the subject is facing sideways, etc., and the alertness level of the subject can be estimated with higher accuracy than in the first embodiment.

[0075] Finally, with reference to Figure 7, hardware configuration examples of the alertness estimation device 1 according to Embodiments 1 and 2 will be described. Below, the hardware configuration example of the alertness estimation device 1 according to Embodiment 1 will be described, but the same applies to the hardware configuration example of the alertness estimation device 1 according to Embodiment 2. The functions of the face image acquisition unit 102, the upper eyelid opening degree calculation unit 103, the upper and lower eyelid distance parameter calculation unit 104, the eye-closing transition parameter calculation unit 105, the downward gaze determination unit 106, and the alertness estimation unit 107 in the alertness estimation device 1 are realized by the processing circuit 51. The processing circuit 51 may be dedicated hardware as shown in Figure 7A, or it may be a CPU (Central Processing Unit, central processing unit, processing unit, arithmetic unit, microprocessor, microcomputer, processor, or DSP (Digital Signal Processor)) 52 that executes a program stored in the memory 53, as shown in Figure 7B.

[0076] If the processing circuit 51 is dedicated hardware, it may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination thereof. The functions of each part of the face image acquisition unit 102, the upper eyelid opening degree calculation unit 103, the upper and lower eyelid distance parameter calculation unit 104, the eye-closing transition parameter calculation unit 105, the downward gaze determination unit 106, and the arousal level estimation unit 107 may be implemented by the processing circuit 51 individually, or the functions of each part may be implemented together by the processing circuit 51.

[0077] When the processing circuit 51 is a CPU 52, the functions of the face image acquisition unit 102, the upper eyelid opening degree calculation unit 103, the upper and lower eyelid distance parameter calculation unit 104, the eye-closing transition parameter calculation unit 105, the downward gaze determination unit 106, and the alertness estimation unit 107 are realized by software, firmware, or a combination of software and firmware. The software and firmware are written as programs and stored in the memory 53. The processing circuit 51 realizes the functions of each unit by reading and executing the programs stored in the memory 53. In other words, the alertness estimation device 1 has a memory for storing programs that, when executed by the processing circuit 51, result in each of the steps shown in Figure 3 being executed, for example. These programs can also be said to cause the computer to execute the procedures and methods of the face image acquisition unit 102, the upper eyelid opening degree calculation unit 103, the upper and lower eyelid distance parameter calculation unit 104, the eye-closing transition parameter calculation unit 105, the downward gaze determination unit 106, and the alertness estimation unit 107. Here, memory 53 includes, for example, non-volatile or volatile semiconductor memory such as RAM (Random Access Memory), ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable ROM), EEPROM (Electrically EPROM), magnetic disks, flexible disks, optical disks, compact disks, minidiscs, or DVDs (Digital Versatile Discs).

[0078] Furthermore, the functions of the face image acquisition unit 102, the upper eyelid opening degree calculation unit 103, the upper and lower eyelid distance parameter calculation unit 104, the eye-closing transition parameter calculation unit 105, the downward gaze determination unit 106, and the alertness level estimation unit 107 may be partially implemented by dedicated hardware and partially by software or firmware. For example, the face image acquisition unit 102 can be implemented by a processing circuit 51 as dedicated hardware, while the functions of the upper eyelid opening degree calculation unit 103, the upper and lower eyelid distance parameter calculation unit 104, the eye-closing transition parameter calculation unit 105, the downward gaze determination unit 106, and the alertness level estimation unit 107 can be implemented by the processing circuit 51 reading and executing a program stored in memory 53.

[0079] In this way, the processing circuit 51 can realize each of the above-mentioned functions through hardware, software, firmware, or a combination thereof.

[0080] Furthermore, it is possible to freely combine the embodiments, modify any component of each embodiment, or omit any component in each embodiment.

[0081] The alertness estimation device 1 according to this disclosure can estimate the alertness level of a vehicle occupant with high accuracy even when downward gaze and blinking occur in succession, and is suitable for use in alertness estimation devices 1 and the like.

[0082] 1. Awakening level estimation device, 51. Processing circuit, 52. CPU, 53. Memory, 101. Imaging unit, 102. Face image acquisition unit, 103. Upper eyelid opening degree calculation unit, 104. Upper and lower eyelid distance parameter calculation unit, 105. Eye closing transition parameter calculation unit, 106. Downward gaze determination unit, 107. Awakening level estimation unit, 108. Face orientation detection unit, 109. Downward gaze necessity determination unit.

Claims

1. A face image acquisition unit that acquires a face image of a subject; an upper eyelid opening degree calculation unit that calculates the degree of upper eyelid opening of the subject based on the face image of the subject acquired by the face image acquisition unit; an upper and lower eyelid distance parameter calculation unit that calculates parameters relating to the distance between the upper and lower eyelids of the subject based on the face image of the subject acquired by the face image acquisition unit when the degree of upper eyelid opening of the subject calculated by the upper eyelid opening degree calculation unit is less than a first opening degree threshold; an eye-closing transition parameter calculation unit that calculates parameters relating to the eye-closing transition from a state in which the degree of upper eyelid opening of the subject is greater than a second opening degree threshold (which is greater than the first opening degree threshold) to a state in which it is less than the first opening degree threshold; An alertness estimation device comprising: a downward gaze determination unit that determines whether or not a subject is in a downward gaze state based on parameters relating to the distance between the upper and lower eyelids of the subject calculated by the upper and lower eyelid distance parameter calculation unit and parameters relating to the eye-closing transition of the subject calculated by the eye-closing transition parameter calculation unit; and an alertness estimation unit that estimates the alertness of the subject based on the degree of opening of the upper eyelid of the subject calculated by the upper eyelid opening degree calculation unit and the determination result by the downward gaze determination unit.

2. The eye-closing transition parameter calculation unit calculates the time required for the subject's upper eyelid opening to transition from a state greater than a second opening threshold to a state less than a first opening threshold, as a parameter relating to the eye-closing transition, as described in claim 1.

3. The downward gaze determination unit determines that the subject is in a downward gaze state when the parameter relating to the distance between the upper and lower eyelids of the subject, calculated by the upper and lower eyelid distance parameter calculation unit, is greater than a distance threshold, and the time calculated by the eye-closing transition parameter calculation unit is shorter than a time threshold, as described in claim 2.

4. The eye-closing transition parameter calculation unit calculates the speed at which the subject's upper eyelid opening transitions from a state greater than a second opening threshold to a state less than a first opening threshold, as a parameter relating to the eye-closing transition, as described in claim 1.

5. The alertness estimation device according to claim 1, comprising: a face orientation detection unit for detecting the orientation of a subject's face; and a downward gaze determination unit for determining whether or not to perform a downward gaze determination in the downward gaze determination unit based on the orientation of the subject's face detected by the face orientation detection unit, wherein the downward gaze determination unit does not perform a downward gaze determination if the downward gaze determination unit determines that a downward gaze determination is unnecessary.

6. The facial image acquisition unit performs the step of acquiring a facial image of a subject; the upper eyelid opening degree calculation unit performs the step of calculating the opening degree of the subject's upper eyelids based on the facial image of the subject acquired by the facial image acquisition unit; the upper and lower eyelid distance parameter calculation unit performs the step of calculating a parameter relating to the distance between the subject's upper and lower eyelids based on the facial image of the subject acquired by the facial image acquisition unit, when the opening degree of the subject's upper eyelids calculated by the upper eyelid opening degree calculation unit is smaller than a first opening degree threshold; the eye-closing transition parameter calculation unit performs the step of calculating a parameter relating to the eye-closing transition from a state in which the opening degree of the subject's upper eyelids is greater than a second opening degree threshold (which is greater than the first opening degree threshold) to a state in which it is smaller than the first opening degree threshold, when the opening degree of the subject's upper eyelids calculated by the upper eyelid opening degree calculation unit is smaller than a first opening degree threshold. A method for estimating a state of

Images