In-vehicle monitoring device and in-vehicle monitoring method

Abstract

An in-vehicle monitoring device (10) comprises: an image acquisition unit (11) that acquires information indicating an image of an occupant in a vehicle captured by an imaging device; a feature amount detection unit (12) that detects a feature amount of a face of the occupant on the basis of the image indicated by the information acquired by the image acquisition unit; an abnormality determination unit (14) that determines, by using the feature amount detected by the feature amount detection unit, whether an abnormal state has occurred in the occupant; and a shielding detection unit (13) that detects a shielded state of the imaging device on the basis of the image indicated by the information acquired by the image acquisition unit. If the shielding detection unit detects, during the determination, the state in which the imaging device is shielded, the abnormality determination unit performs suppression control for suppressing the determination that the abnormal state has occurred in the occupant.

Description

In-vehicle monitoring device and in-vehicle monitoring method

[0001] The present disclosure relates to an in-vehicle monitoring device and an in-vehicle monitoring method.

[0002] The in-vehicle monitoring device is provided with a sensing function that determines whether an abnormality has occurred in an occupant based on the occupant's face image captured by an imaging device such as a camera installed in the vehicle, and issues an alarm when it is determined that an abnormality has occurred. Most of the conventional sensing functions are configured to stop the determination of whether an abnormality has occurred and not issue an alarm for the occupant's abnormality when the face of the occupant can no longer be captured by the imaging device. For example, in the face detection device disclosed in Patent Document 1, when a proximity object determination unit determines that a proximity object exists between the driver's face and the imaging device, a face orientation detection unit does not perform face orientation detection because there is an obstacle that hides the face.

[0003] Japanese Unexamined Patent Publication No. 2009-294781

[0004] In recent years, an increasing number of sensing functions in in-vehicle monitoring devices continue the determination of whether a predetermined abnormal state such as the occupant's posture collapse or inattentive driving has occurred even when the occupant's face is no longer captured by the imaging device. A problem that arises here is a case where a predetermined abnormal state such as the occupant's posture collapse or inattentive driving that has not actually occurred is erroneously determined to have occurred. For example, when the occupant lightly holds a hand over the imaging device or slightly covers the imaging device with an object such as luggage, the conventional in-vehicle monitoring device may erroneously detect the object as the occupant's face at the boundary between the hand or the object (hereinafter collectively referred to as "object") and the background. After that, even when the imaging device is shielded by the object, the conventional in-vehicle monitoring device continues the determination of whether an abnormality has occurred, so triggered by the aforementioned erroneous face detection, it may erroneously determine that an abnormal state such as the occupant's posture collapse or inattentive driving that has not actually occurred has occurred.

[0005] The present disclosure has been made to solve the above problems, and an object thereof is to provide an in-vehicle monitoring device capable of suppressing erroneous determination that an abnormal state of an occupant that has not actually occurred has occurred when an imaging device is shielded.

[0006] The in-vehicle monitoring device according to this disclosure comprises: an image acquisition unit that acquires information showing an image of a vehicle occupant captured by an imaging device; a feature detection unit that detects the facial feature quantities of the occupant based on the image shown by the information acquired by the image acquisition unit; an abnormality determination unit that determines whether or not an abnormal condition has occurred to the occupant using the feature quantities detected by the feature detection unit; and an occlusion detection unit that detects the occlusion state of the imaging device based on the image shown by the information acquired by the image acquisition unit. The abnormality determination unit is characterized in that, when determining the abnormality, if the occlusion detection unit detects that the imaging device is occupying, it performs suppression control to suppress the determination that an abnormal condition has occurred to the occupant.

[0007] According to this disclosure, the above configuration makes it possible to suppress the false determination that an abnormal condition of the occupant has occurred when the imaging device is shielded, even though such an abnormal condition has not actually occurred.

[0008] This figure shows an example of the configuration of the in-vehicle monitoring device according to Embodiment 1. This flowchart shows an example of the operation of the in-vehicle monitoring device according to Embodiment 1. This figure explains the reset in Embodiment 1. This flowchart shows other examples of the operation of the in-vehicle monitoring device according to Embodiment 1. Figures 5A and 5B show an example of the hardware configuration of the in-vehicle monitoring device according to Embodiment 1.

[0009] The embodiments will now be described in detail with reference to the drawings. Embodiment 1. Figure 1 is a diagram showing an example of the configuration of a vehicle in-vehicle monitoring device 10 according to Embodiment 1. The vehicle in-vehicle monitoring device 10 is configured to include, for example, an image acquisition unit 11, a feature quantity detection unit 12, an occlusion detection unit 13, and an abnormality determination unit 14, as shown in Figure 1. The vehicle in-vehicle monitoring device 10 is also communicated with an imaging device (not shown) installed outside the vehicle in-vehicle monitoring device 10. The imaging device may be provided inside the vehicle in-vehicle monitoring device 10.

[0010] The imaging device consists of, for example, a camera installed inside the vehicle. The imaging device captures still or moving images of the vehicle's occupants. Moving images consist of multiple still images (hereinafter simply referred to as "images" or "frames") arranged in chronological order, and information is provided from the imaging device to the in-vehicle monitoring device 10 on an image-by-image basis.

[0011] The image acquisition unit 11 acquires information indicating images of vehicle occupants captured by the imaging device (hereinafter referred to as "occupant image information"). The image acquisition unit 11 acquires occupant image information from the imaging device at unit time intervals (for example, every second). When the image acquisition unit 11 acquires occupant image information from the imaging device, it outputs the acquired occupant image information to the feature detection unit 12 and the occlusion detection unit 13.

[0012] The feature detection unit 12 acquires crew image information from the image acquisition unit 11. Upon acquiring crew image information from the image acquisition unit 11, the feature detection unit 12 detects the facial features of the crew based on the image indicated by the crew image information. Here, known methods may be used to detect the facial features of the crew. The feature detection unit 12 outputs information indicating the detected features (hereinafter referred to as "feature information") to the anomaly determination unit 14.

[0013] The occlusion detection unit 13 acquires occupant image information from the image acquisition unit 11. Upon acquiring occupant image information from the image acquisition unit 11, the occlusion detection unit 13 detects the occlusion state of the imaging device based on the image indicated by the occupant image information. The occlusion state of the imaging device is either a state in which the imaging device is occluded or a state in which the imaging device is not occluded. The occlusion detection unit 13 outputs information indicating the detected occlusion state of the imaging device (hereinafter referred to as "occlusion state information") to the abnormality determination unit 14.

[0014] The occlusion detection unit 13 can detect the occlusion state of the imaging device using, for example, a known blocking detection method. The blocking detection method scores the edge detection results in the image, and if the score is below a predetermined threshold, it is detected that the imaging device is occluded, while if the score is greater than the predetermined threshold, it is detected that the imaging device is not occluded, and the imaging device is not occluded.

[0015] Normal images without screen defects or filters exhibit strong edges (high-frequency components), while images with screen defects or filters exhibit weak edges. Therefore, blockage detection methods use a Laplacian filter to detect edges in the image, and the integral value of the detected edge intensity is used as a score (index value) to detect the occlusion state of the imaging device. Generally, the closer the image is to a normal image, the larger the score.

[0016] The abnormality determination unit 14 acquires feature information from the feature detection unit 12 and occlusion state information from the occlusion detection unit 13. Upon acquiring this information, the abnormality determination unit 14 uses the features indicated by the feature information acquired from the feature detection unit 12 to determine whether or not an abnormal state has occurred for the occupant. Furthermore, if the occlusion state information acquired from the occlusion detection unit 13 indicates that the imaging device is occupying, that is, if the occlusion detection unit 13 detects that the imaging device is occupying at the time of the determination, the abnormality determination unit 14 performs suppression control to prevent it from determining that an abnormal state has occurred for the occupant. Subsequently, if the abnormality determination unit 14 determines that an abnormal state has occurred for the occupant, it issues an alarm indicating that an abnormal state has occurred for the occupant. The manner of the alarm is not particularly limited, but for example, the abnormality determination unit 14 may display the fact that an abnormal state has occurred for the occupant on a monitor (not shown) installed in the vehicle and also output an audio message to the vehicle indicating that an abnormal state has occurred for the occupant.

[0017] The suppression control performed by the abnormality detection unit 14 may include the following, depending on the abnormality state determination method adopted by the abnormality detection unit 14. For example, the abnormality detection unit 14 can adopt a determination method based on the duration of the abnormal state as the abnormal state determination method. In this case, the abnormality detection unit 14 uses the feature quantities detected by the feature quantity detection unit 12 to detect an abnormal state of the occupant and starts timing the duration of the abnormal state from the time of detection. The abnormality detection unit 14 then determines that an abnormal state has occurred in the occupant when the duration exceeds a predetermined time threshold. In this case, if the imaging device is detected to be obscured by the occlusion detection unit 13 after the start of timing, the abnormality detection unit 14 resets the duration as a suppression control.

[0018] Furthermore, the abnormality detection unit 14 can employ a detection method using a learner (a pre-trained model) as a method for determining abnormal conditions. The learner is pre-trained to output an inference result of whether or not an abnormal condition has occurred for the occupant, based on inputs of facial features of the occupant and information indicating the occlusion state of the imaging device. Here, the information indicating the occlusion state of the imaging device, which is used as input when training the learner, may include not only information indicating whether the imaging device is occluded or not, but also a score (index value) calculated in the blocking detection method described above.

[0019] The abnormality determination unit 14 inputs the feature quantities detected by the feature quantity detection unit 12 and information indicating the occlusion status of the imaging device detected by the occlusion detection unit 13 to the learning device described above, and determines whether or not an abnormal condition has occurred for the occupants based on the results output from the learning device. Here, the information indicating the occlusion status of the imaging device that the abnormality determination unit 14 inputs to the learning device during the determination may not only indicate whether the imaging device is occluded or not, but may also include the score calculated in the blocking detection method described above.

[0020] When the abnormality determination unit 14 makes a determination using the learning device described above, if the occlusion detection unit 13 detects that the imaging device is occupying it, it adjusts the parameters of the learning device as a suppression control so that the learning device does not output an inference result that an abnormal condition has occurred for the occupant.

[0021] In addition, the in-vehicle monitoring device 10 may include a storage unit (not shown) for storing various types of information. The storage unit is composed of a storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive).

[0022] Next, an example of the operation of the in-vehicle monitoring device 10 shown in Figure 1 will be described. Figure 2 is a flowchart illustrating an example of the operation of the in-vehicle monitoring device 10.

[0023] As a prerequisite for explaining the operation example of the in-vehicle monitoring device 10, it is assumed that the abnormality detection unit 14 employs a detection method based on the duration of the abnormal state as its method for determining abnormal conditions. Furthermore, in this explanation, the in-vehicle monitoring device 10 will repeatedly execute the following flowchart process at each unit of time. Also, for the sake of clarity in the following explanation, it will be assumed that the unit of time is 1 second and the time threshold used by the abnormality detection unit 14 for determination is 5 seconds.

[0024] First, the image acquisition unit 11 acquires information indicating images of the vehicle's occupants (occupant image information) captured by the imaging device (step ST1). Once the image acquisition unit 11 acquires the occupant image information from the imaging device, it outputs the acquired occupant image information to the feature detection unit 12 and the occlusion detection unit 13.

[0025] Next, the feature detection unit 12 acquires occupant image information from the image acquisition unit 11. Once the feature detection unit 12 acquires occupant image information from the image acquisition unit 11, it detects the facial features of the occupants based on the images indicated by the occupant image information (step ST2-1). Here, known methods may be used to detect the facial features of the occupants. The feature detection unit 12 outputs information indicating the detected features (feature information) to the anomaly determination unit 14.

[0026] Next, the occlusion detection unit 13 acquires occupant image information from the image acquisition unit 11. Once the occlusion detection unit 13 acquires occupant image information from the image acquisition unit 11, it detects the occlusion state of the imaging device based on the image indicated by the occupant image information (step ST2-2). The occlusion detection unit 13 outputs information indicating the detected occlusion state (occlusion state information) to the abnormality determination unit 14. The occlusion detection unit 13 can detect the occlusion state of the imaging device using, for example, a known method such as a blockage detection method.

[0027] For the sake of explanation, the steps ST2-1 and ST2-2 are described in that order, but these two steps may be carried out in parallel.

[0028] Next, the abnormality detection unit 14 acquires feature information from the feature detection unit 12 and occlusion detection information from the occlusion detection unit 13. Based on the feature information obtained from the feature detection unit 12, the abnormality detection unit 14 detects an abnormal state of the occupant (step ST3). When the abnormality detection unit 14 detects an abnormal state of the occupant, it starts timing the duration of the abnormal state from the time of detection.

[0029] Next, the abnormality determination unit 14 refers to the occlusion status of the imaging device indicated by the occlusion detection information obtained from the occlusion detection unit 13 (step ST4). If the occlusion status of the imaging device indicates that it is "not occluded" (shown as "Detection result is OFF" in Figure 2), the process proceeds to step ST5. On the other hand, if the occlusion status of the imaging device indicates that it is "occluded" (shown as "Detection result is ON" in Figure 2), the process proceeds to step ST6.

[0030] In step ST5, the abnormality detection unit 14 updates the duration of the abnormal state described above (step ST5). For example, the abnormality detection unit 14 counts up the duration of the abnormal state described above in units of time (e.g., 1 second). After that, the process transitions to step ST7.

[0031] In step ST6, the abnormality detection unit 14 resets the duration of the abnormal state as described above as suppression control (step ST6). After that, the process transitions to step ST7.

[0032] In step ST7, the abnormality determination unit 14 determines whether or not an abnormal condition has occurred to the occupant based on the duration of the abnormal condition (step ST7). For example, the abnormality determination unit 14 compares the duration of the abnormal condition with a predetermined time threshold and determines that an abnormal condition has occurred to the occupant if the duration of the abnormal condition is equal to or greater than the predetermined time threshold. On the other hand, if the duration of the abnormal condition is less than the predetermined time threshold, the abnormality determination unit 14 determines that no abnormal condition has occurred to the occupant. Subsequently, if the abnormality determination unit 14 determines that an abnormal condition has occurred to the occupant, it issues an alarm indicating that an abnormal condition has occurred to the occupant.

[0033] As described above, in step ST6, the abnormality detection unit 14 resets the duration of the abnormal state. Here, the reset will be explained with reference to Figure 3. A reset means, for example as shown in Figure 3, returning the duration of the abnormal state to an initial value (for example, 0) and restarting the timer from that initial value.

[0034] In Figure 3, Δt represents a time threshold (5 seconds in this case). In other words, the abnormality detection unit 14 determines that an abnormal condition has occurred when the duration of the abnormal condition exceeds Δt. Also, in Figure 3, the "shielding period" includes not only the period during which the imaging device is completely shielded by an object, but also the period during which the imaging device is not completely shielded, such as when a crew member lightly places their hand over the imaging device or when a crew member slightly covers the imaging device with an object. During the former period, the shielding detection unit 13 detects that the imaging device is shielded, but during the latter period, the shielding detection unit 13 may detect that the imaging device is not shielded.

[0035] For example, at time t0 shown in Figure 3, the occlusion detection unit 13 detects that the imaging device is not occupying. This is because, at time t0, the imaging device is not completely occupying, such as when a crew member lightly holds their hand over the imaging device or when a crew member slightly covers the imaging device with an object such as luggage, and the score in the blocking detection method described above becomes greater than the threshold.

[0036] On the other hand, at time t0, the abnormality detection unit 14 may mistakenly detect a part of an object (hand or luggage) that is not actually the occupant's face as the occupant's face, based on the facial features of the occupant. This is because, for example, the shape or pattern on the surface of the object (e.g., wrinkles on a hand, wrinkles on clothing) is similar to the brightness gradient of a human face. Due to this erroneous detection of a face, the abnormality detection unit 14 detects an abnormal state of the occupant that it should not have detected, and starts timing the duration of the abnormal state from the moment of detection. In this case, conventional abnormality detection units, for example, at time t1, even if the occlusion detection unit 13 detects that the imaging device is occluded, continue to update the duration of the abnormal state, and when the duration exceeds a time threshold, mistakenly determine that an abnormal state of the occupant has occurred, even though it has not actually occurred.

[0037] On the other hand, in this disclosure, if the obstruction detection unit 13 detects that the imaging device is obstructed at time t1, for example, the abnormality determination unit 14 repeatedly performs the process of step ST6 in the flowchart shown in Figure 2, i.e., resets the duration, until the obstruction detection unit 13 detects that the imaging device is not obstructed. Here, the in-vehicle monitoring device 10 repeatedly executes the process of this flowchart every second, which is a unit of time, so it resets every second as shown by the triangle in Figure 3. As a result, as long as the obstruction detection unit 13 detects that the imaging device is obstructed, the duration of the abnormal state will not exceed the time threshold, and as a result, the abnormality determination unit 14 will not determine that an abnormal state has occurred for the occupant. In other words, the in-vehicle monitoring device 10 is prevented from mistakenly determining that an abnormal state has occurred for the occupant when it has not actually occurred.

[0038] Next, we will describe other examples of operation of the in-vehicle monitoring device 10 shown in Figure 1. Figure 4 is a flowchart illustrating other examples of operation of the in-vehicle monitoring device 10.

[0039] Furthermore, as a prerequisite for explaining other operational examples of the in-vehicle monitoring device 10, it should be assumed that the abnormality determination unit 14 employs a determination method using a learning device as the method for determining abnormal conditions. Also, steps ST1, ST2-1, ST2-2, and ST3 shown in Figure 4 are the same as steps ST1, ST2-1, ST2-2, and ST4 shown in Figure 2, so a further explanation will be omitted.

[0040] In step ST3 shown in Figure 4, the abnormality determination unit 14 refers to the occlusion status of the imaging device indicated by the occlusion detection information obtained from the occlusion detection unit 13. If the occlusion status of the imaging device indicates "not occluded" (indicated as "Detection result is OFF" in Figure 4), the process proceeds to step ST5. On the other hand, if the occlusion status of the imaging device indicates "occluded" (indicated as "Detection result is ON" in Figure 4), the process proceeds to step ST4.

[0041] In step ST4, the abnormality determination unit 14 adjusts the parameters of the learning device as a suppression control so that the learning device does not output an inference result that an abnormal condition has occurred for the occupant (step ST4).

[0042] For example, the anomaly determination unit 14 may use a method called online learning to adjust the parameters of the learner in real time so that when the occlusion detection unit 13 detects that the imaging device is occupying, the learner does not output an inference result that an abnormal condition has occurred for the occupant. Alternatively, the anomaly determination unit 14 may use a method called dynamic fine-tuning to adjust the weights or biases of the learner so that when the occlusion detection unit 13 detects that the imaging device is occupying, the learner does not output an inference result that an abnormal condition has occurred for the occupant.

[0043] Further, when performing determination using the above-described learner, even if the obstruction detection unit 13 detects a state where the imaging device is not obstructed, in other words, even if the score calculated by the above-described blockage detection method is greater than a threshold, the abnormality determination unit 14 may perform the above-described parameter adjustment of the learner as suppression control when the difference between the score and the threshold is very small. As described above, this is to prevent the learner from erroneously inferring that an abnormal state has occurred for the occupant, for example, when the occupant lightly holds a hand over the imaging device, or when the occupant slightly covers the imaging device with an object such as luggage.

[0044] In step ST5, the abnormality determination unit 14 inputs, to the learner, the feature amount information acquired from the feature amount detection unit 12 and the information indicating the obstruction state of the imaging device acquired from the obstruction detection unit 13, and determines whether or not an abnormal state has occurred for the occupant based on the result output from the learner (step ST5). At this time, when step ST5 is executed via step ST4, the abnormality determination unit 14 has performed predetermined suppression control in step ST4, so that determination that an abnormal state has occurred for the occupant is suppressed.

[0045] As described above, even when the abnormality determination unit 14 employs a determination method using a learner as the abnormal state determination method, the in-vehicle monitoring device 10 can suppress erroneous determination that an abnormal state of an occupant that does not actually occur has occurred.

[0046] <Effects of In-Vehicle Monitoring Device 10> Next, the effects of the in-vehicle monitoring device 10 according to Embodiment 1 will be described in comparison with a conventional in-vehicle monitoring device (hereinafter referred to as "conventional device").

[0047] In conventional devices, when an occupant lightly holds an object over the imaging device, or when the occupant slightly covers the imaging device with an object such as luggage, the object may be erroneously detected as the occupant's face at the boundary between the object and the background. After that, even when the imaging device is in a state of being shielded by the object, the conventional device continues to determine whether an abnormality has occurred, so triggered by the aforementioned erroneous face detection, it detects the occurrence of an abnormal state of the occupant, and starts counting the duration of the abnormal state from that point in time. Then, when the duration reaches or exceeds a preset time threshold, the conventional device may erroneously determine that an abnormal state of the occupant that has not actually occurred has occurred.

[0048] In this regard, for example, in the case of simple eye closure detection, or a glancing action to the extent that the occupant turns toward the side mirror, the aforementioned duration may be reset when the face can no longer be imaged by the imaging device. However, in the case of a posture collapse where the occupant's face cannot be imaged by the imaging device, or similarly a glance toward the rear seat direction where the occupant's face cannot be imaged, the aforementioned duration must be continuously updated even if the face cannot be imaged by the imaging device, otherwise the abnormality determination cannot be performed and an alarm cannot be issued. However, the problem in that case is the problem of erroneously determining that an abnormal state such as posture collapse or glancing that has not actually occurred has occurred.

[0049] In this regard, the in-vehicle monitoring device 10 according to Embodiment 1 is configured to include the image acquisition unit 11, the feature amount detection unit 12, the shielding detection unit 13, and the abnormality determination unit 14 as described above, and in particular, when the shielding detection unit 13 detects that the imaging device is shielded during determination, the abnormality determination unit 14 performs suppression control for suppressing the determination that an abnormal state has occurred in the occupant. The suppression control is, for example, resetting the duration of the abnormal state described above, or adjusting the parameters of the learning device. Thereby, the in-vehicle monitoring device 10 can suppress erroneous determination that an abnormal state of the occupant that has not actually occurred has occurred when the imaging device is shielded.

[0050] Next, with reference to Figure 5, an example of the hardware configuration of the in-vehicle monitoring device 10 according to Embodiment 1 will be described. The functions of the image acquisition unit 11, feature quantity detection unit 12, occlusion detection unit 13, and abnormality determination unit 14 in the in-vehicle monitoring device 10 are realized by a processing circuit. The processing circuit may be dedicated hardware as shown in Figure 5A, or it may be a CPU (Central Processing Unit, central processing unit, processing unit, arithmetic unit, microprocessor, microcomputer, processor, or DSP (Digital Signal Processor)) 52 that executes a program stored in memory 53, as shown in Figure 5B.

[0051] If the processing circuit is dedicated hardware, the processing circuit 51 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination thereof. The processing circuit 51 may implement the functions of each part, such as the image acquisition unit 11, the feature quantity detection unit 12, the occlusion detection unit 13, and the anomaly determination unit 14, or it may implement the functions of each part together in the processing circuit 51.

[0052] When the processing circuit is a CPU 52, the functions of the image acquisition unit 11, feature detection unit 12, occlusion detection unit 13, and anomaly determination unit 14 are realized by software, firmware, or a combination of software and firmware. The software and firmware are written as programs and stored in memory 53. The processing circuit realizes the functions of each unit by reading and executing the programs stored in memory 53. In other words, the in-vehicle monitoring device 10 is equipped with memory for storing programs that, when executed by the processing circuit, result in the execution of each step shown in Figures 2 and 4, for example. These programs can also be said to cause the computer to execute the procedures and methods of the image acquisition unit 11, feature detection unit 12, occlusion detection unit 13, and anomaly determination unit 14. Here, the memory 53 includes, for example, non-volatile or volatile semiconductor memory such as RAM (RAM Access Memory), ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable ROM), EEPROM (Electrically EPROM), magnetic disks, flexible disks, optical disks, compact disks, minidiscs, or DVDs (Digital Versatile Discs).

[0053] Furthermore, the functions of the image acquisition unit 11, feature detection unit 12, occlusion detection unit 13, and anomaly determination unit 14 may be partially implemented by dedicated hardware and partially by software or firmware. For example, the image acquisition unit 11 can be implemented by a processing circuit as dedicated hardware, while the feature detection unit 12, occlusion detection unit 13, and anomaly determination unit 14 can be implemented by a processing circuit reading and executing a program stored in memory 53.

[0054] Thus, the processing circuit can realize each of the above-mentioned functions through hardware, software, firmware, or a combination thereof.

[0055] As described above, according to this embodiment 1, the in-vehicle monitoring device 10 includes an image acquisition unit 11 that acquires information showing an image of the vehicle occupant captured by the imaging device, a feature detection unit 12 that detects the facial feature quantities of the occupant based on the image shown by the information acquired by the image acquisition unit 11, an abnormality determination unit 14 that determines whether or not an abnormal condition has occurred to the occupant using the feature quantities detected by the feature detection unit 12, and an obstruction detection unit 13 that detects the obstruction state of the imaging device based on the image shown by the information acquired by the image acquisition unit 11. The abnormality determination unit 14 performs suppression control to suppress the determination that an abnormal condition has occurred to the occupant if the obstruction detection unit 13 detects that the imaging device is obstructed during the determination. As a result, the in-vehicle monitoring device 10 according to embodiment 1 can suppress the erroneous determination that an abnormal condition has occurred to the occupant when the imaging device is obstructed, even though such an abnormal condition has not actually occurred.

[0056] Furthermore, the abnormality determination unit 14 uses the feature quantities detected by the feature quantity detection unit 12 to detect an abnormal state of the occupant, and starts timing the duration of the abnormal state from the time of detection. When the duration exceeds a predetermined time threshold, it determines that an abnormal state has occurred in the occupant. If the occupancy detection unit 13 detects that the imaging device is occupying after the start of timing, it resets the duration as a suppression control. As a result, even when the in-vehicle monitoring device 10 according to Embodiment 1 employs a determination method based on the duration of the abnormal state as the abnormal state determination method, it can suppress misdetermination of an abnormal state of the occupant when the imaging device is occupying.

[0057] Furthermore, the abnormality determination unit 14 receives the feature quantities detected by the feature quantity detection unit 12 and the information indicating the occupancy state of the imaging device detected by the occupancy detection unit 13 as input to a learner that outputs a result inferring whether or not an abnormal state has occurred to the occupant in response to input of feature quantities and information indicating the occupancy state of the imaging device. Based on the result output from the learner, it determines whether or not an abnormal state has occurred to the occupant. When the occupancy detection unit 13 detects that the imaging device is occupied, the learner's parameters are adjusted as a suppression control so that the learner does not output an inference result that an abnormal state has occurred to the occupant. As a result, the in-vehicle monitoring device 10 according to Embodiment 1 can suppress misjudgments of an abnormal state to the occupant when the imaging device is occupied, even when employing a learning-based determination method as the abnormal state determination method.

[0058] Furthermore, an abnormal state of the occupant includes the occupant looking away or losing their posture. As a result, the in-vehicle monitoring device 10 according to Embodiment 1 can prevent it from mistakenly determining that the occupant is looking away or losing their posture when the imaging device is obstructed, even though such an event did not actually occur.

[0059] Furthermore, the occlusion detection unit 13 performs edge detection on the image indicated by the information acquired by the image acquisition unit 11, scores the result of the edge detection, and detects that the imaging device is occluded if the score is below a predetermined threshold. As a result, the in-vehicle monitoring device 10 according to Embodiment 1 can accurately detect the occlusion state of the imaging device.

[0060] Furthermore, this disclosure allows for modifications of any component of the embodiment, or the omission of any component in the embodiment.

[0061] This disclosure makes it possible to suppress the false determination of an abnormal condition of the occupant that does not actually occur when the imaging device is obscured, and is suitable for use in in-vehicle monitoring devices and in-vehicle monitoring methods.

[0062] 10 In-vehicle monitoring device, 11 Image acquisition unit, 12 Feature detection unit, 13 Obstruction detection unit, 14 Anomaly determination unit, 51 Processing circuit, 52 CPU, 53 Memory.

Claims

1. A vehicle in-vehicle monitoring device comprising: an image acquisition unit that acquires information showing an image of a vehicle occupant captured by an imaging device; a feature detection unit that detects the facial feature quantities of the occupant based on the image shown by the information acquired by the image acquisition unit; an abnormality determination unit that determines whether or not an abnormal condition has occurred to the occupant using the feature quantities detected by the feature detection unit; and an obstruction detection unit that detects the obstruction state of the imaging device based on the image shown by the information acquired by the image acquisition unit, wherein the abnormality determination unit performs suppression control to suppress the determination that an abnormal condition has occurred to the occupant if the obstruction detection unit detects that the imaging device is obstructed during the determination.

2. The abnormality determination unit detects an abnormal state of the occupant using the feature quantity detected by the feature quantity detection unit, and starts timing the duration of the abnormal state from the time of detection. When the duration exceeds a predetermined time threshold, it determines that an abnormal state has occurred in the occupant. The in-vehicle monitoring device according to claim 1, characterized in that, after the start of timing, the obstruction detection unit detects that the imaging device is obstructed, and the suppression control resets the duration.

3. The abnormality determination unit inputs the feature quantity detected by the feature quantity detection unit and the information indicating the occupancy state of the imaging device detected by the occupancy detection unit to a learner that outputs a result of inference as to whether or not an abnormal state has occurred to the occupant, and determines whether or not an abnormal state has occurred to the occupant based on the result output from the learner, wherein, in the event of such determination, the occupancy detection unit detects that the imaging device is occupied, and the parameters of the learner are adjusted as suppression control so that the learner does not output an inference result that an abnormal state has occurred to the occupant.

4. The in-vehicle monitoring device according to any one of claims 1 to 3, characterized in that the abnormal condition of the occupant includes the occupant looking away or losing their posture.

5. The vehicle in-vehicle monitoring device according to any one of claims 1 to 3, characterized in that the occlusion detection unit performs edge detection of the image indicated by the information acquired by the image acquisition unit, scores the result of the edge detection, and detects that the imaging device is occluded if the score is below a predetermined threshold.

6. A vehicle in-vehicle monitoring method using a vehicle in-vehicle monitoring device, comprising: a step of an image acquisition unit acquiring information indicating an image of a vehicle occupant captured by an imaging device; a step of a feature detection unit detecting the facial feature quantities of the occupant based on the image indicated by the information acquired by the image acquisition unit; a step of an abnormality determination unit determining whether or not an abnormal state has occurred to the occupant using the feature quantities detected by the feature detection unit; and a step of an occlusion detection unit detecting the occlusion state of the imaging device based on the image indicated by the information acquired by the image acquisition unit, wherein the abnormality determination unit performs suppression control to suppress determining that an abnormal state has occurred to the occupant if the occlusion detection unit detects that the imaging device is occupied during the determination.

Images