Information processing device, information processing system, information processing method, and program

Abstract

This information processing device comprises an acquisition unit, a risk detection unit, an overlooking determination unit, an abnormality determination unit, and a notification control unit. The acquisition unit acquires a peripheral image of a vehicle and a face image of a driver of the vehicle. The risk detection unit detects a risk object in the peripheral image. The overlooking determination unit determines, on the basis of information pertaining to the position of the risk object and information pertaining to the face image, whether or not the driver has overlooked the risk object. On the basis of information pertaining to the position of the risk object, of whether or not the risk object has been overlooked, and of the line of sight detected from the face image, the abnormality determination unit determines whether or not an abnormality has occurred with the driver. The notification control unit causes a notification unit to provide a notification when it has been determined that an abnormality has occurred with the driver.

Description

Information processing apparatus, information processing system, information processing method, and program

[0001] The present invention relates to an information processing apparatus, an information processing system, an information processing method, and a program.

[0002] In recent years, efforts have been activated to provide access to sustainable transportation systems that also take into consideration vulnerable people among traffic participants, such as the elderly, people with disabilities, and children. Toward achieving this goal, research and development are being focused on further improving traffic safety and convenience, particularly through development related to means of transportation for the elderly and people with disabilities.

[0003] Japanese Patent Application Laid-Open No. 2020-71528, Japanese Patent Application Laid-Open No. 2017-16568

[0004] Incidentally, when a driver suffers from a disease such as visual field defect or dementia (that is, the driver has an abnormality), the driver's ability to avoid risky objects decreases. However, there are cases where the driver is not aware of such a disease. If the driver continues to drive without being aware of the disease and the resulting decline in ability, there is a possibility that traffic safety and convenience will be impaired.

[0005] The present application has been made in consideration of such circumstances, and one of the objects thereof is to provide an information processing apparatus, an information processing system, an information processing method, and a program capable of detecting an abnormality in a driver, and thereby contributes to the development of a sustainable transportation system.

[0006] The information processing device, information processing system, information processing method, and program according to this invention employ the following configuration: (1) An information processing device according to one aspect of this invention includes: an acquisition unit that acquires an image of the area surrounding a vehicle and an image of the face of the driver of the vehicle; a risk detection unit that detects a risk object in the surrounding image; an oversight determination unit that determines whether or not the driver has overlooked the risk object based on information relating to the location of the risk object and information relating to the face image; an abnormality determination unit that determines whether or not an abnormality has occurred in the driver based on information relating to the location of the risk object, whether or not an oversight has occurred, and the line of sight detected from the face image; and a notification control unit that causes a notification unit to send a notification when it is determined that an abnormality has occurred in the driver.

[0007] (2) In the embodiment of (1) above, the information processing device further comprises a gaze detection unit that detects the gaze from the face image.

[0008] (3) In the embodiment of (1) or (2) above, the information processing device further includes a storage control unit that stores in a storage unit judgment history information linking information relating to the location of the risk object, whether or not the object was overlooked, and the line of sight.

[0009] (4) In any of the embodiments described in (1) to (3) above, the information processing device further comprises a generation unit that generates field of view map information that links information relating to the frequency of the oversight with the direction within the driver's field of view, based on the judgment history information.

[0010] (5) In any one embodiment of (1) to (4) above, the abnormality determination unit determines that an abnormality has occurred in the driver in the first case in which the oversight has occurred in the entire field of view of the driver, and in the second case in which the oversight has occurred in a part of the field of view of the driver.

[0011] (6) In the embodiment of (5) above, in the first case, the notification control unit causes the notification unit to notify the driver that a visual field defect is occurring or may occur.

[0012] (7) In the embodiment of (5) or (6) above, in the second case, the notification control unit causes the notification unit to give a notification indicating that the driver has dementia or may have dementia.

[0013] (8) An information processing system according to one aspect of the present invention comprises an information processing device according to any of the above aspects (1) to (7), a first terminal device which is not the notification unit and is used by the driver and whose notifications are controlled by the notification control unit, and a second terminal device which is the notification unit and is used by a user other than the driver and whose notifications are controlled by the notification control unit, wherein the second terminal device causes the first terminal device to make a notification based on information input by the user's operation.

[0014] (9) An information processing method according to one aspect of the present invention involves a computer performing the following steps: a process of acquiring an image of the area around a vehicle and an image of the face of the driver of the vehicle; a process of detecting a risk object in the surrounding image; a process of determining whether the driver has overlooked the risk object based on information relating to the location of the risk object and information relating to the face image; a process of determining whether an abnormality has occurred in the driver based on information relating to the location of the risk object, whether or not the oversight occurred, and the line of sight detected from the face image; and a process of causing a notification unit to send a notification if it is determined that an abnormality has occurred in the driver.

[0015] (10): A program according to one aspect of the present invention is a program that causes a computer to perform the following: a process of acquiring an image of the area around a vehicle and an image of the face of the driver of the vehicle; a process of detecting a risk object in the surrounding image; a process of determining whether the driver has overlooked the risk object based on information relating to the location of the risk object and information relating to the face image; a process of determining whether an abnormality has occurred in the driver based on information relating to the location of the risk object, whether or not the oversight occurred, and the gaze detected from the face image; and a process of causing a notification unit to send a notification when it is determined that an abnormality has occurred in the driver.

[0016] According to (1)-(10), abnormalities in the driver can be detected.

[0017] This figure shows an example of the configuration and operating environment of the operation monitoring device 1, which includes the information processing device 2 according to the first embodiment. This figure shows an example of the operation monitoring device 1. This figure shows an example of an image acquired by the operation monitoring device 1. This figure is for explaining the oversight alarm processing performed by the operation monitoring device 1. This figure shows an example of information detected by the detection unit 30. This figure shows an example of an alarm performed by the alarm unit 82. This figure is for explaining the abnormality notification processing performed by the operation monitoring device 1. This figure shows an example of judgment history information 72. This figure shows an example of field of view map information 74. This flowchart shows an example of the processing flow executed by the abnormality judgment unit 48. This figure shows an example of field of view map information 74. This figure shows an example of field of view map information 74. This figure is for explaining the abnormality notification processing in the information processing system according to the second embodiment.

[0018] Hereinafter, embodiments of the information processing apparatus, information processing system, information processing method, and program of the present invention will be described with reference to the drawings. In the following description, the forward direction of the vehicle will be described as the positive X direction, the rear direction of the vehicle as the negative X direction, the width direction of the vehicle with respect to the positive X direction as the positive Y direction to the right, the left direction as the negative Y direction, and the direction perpendicular to the X and Y directions, which is the height direction of the vehicle as the positive Z direction.

[0019] <First Embodiment> [Overall Configuration] Figure 1 is a diagram showing an example of the configuration and operating environment of the driving monitoring device 1 including the information processing device 2 according to the first embodiment. The information processing device 2 is mounted on the driving monitoring device 1, which is retrofitted to the vehicle (hereinafter referred to as "vehicle M"), such as a drive recorder. That is, the driving monitoring device 1 (information processing device 2) is detachable from the vehicle M via a detachable part (hereinafter referred to as "detachable part 16"). The information processing device 2 is, for example, a device that is not connected to the vehicle's in-vehicle network. The in-vehicle network is an in-vehicle network using a communication standard such as CAN (Controller Area Network) to which ECUs (Electronic Control Units) related to vehicle control, such as driving, braking, steering, various vehicle sensors, driving controls, and operation buttons, are connected. The vehicle M's network for HMI (Human Machine Interface) may be excluded from the above-mentioned in-vehicle network.

[0020] Figure 2 shows an example of a driving monitoring device 1. The driving monitoring device 1 is installed, for example, in the vicinity of the front windshield of the vehicle M, in a location near the front of the driver's seat. The driving monitoring device 1 stores images (moving images) captured by the imaging unit 10 in, for example, a storage unit 70 (described later). The driving monitoring device 1 may also be configured to include a display device such as an LCD (Liquid Crystal Display) and to display the images (moving images) stored in the storage unit 70 on the display device.

[0021] The operation monitoring device 1 includes, for example, an imaging unit 10, a detachable unit 16, and an information processing device 2. In the illustrated example, the imaging unit 10 includes a front camera 12 and a rear camera 14.

[0022] Each of the front camera 12 and rear camera 14 is a digital camera utilizing a solid-state image sensor such as a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor). The front camera 12 repeatedly (periodically) images the area in front of the vehicle M from the position where the driving monitoring device 1 is mounted. The rear camera 14 repeatedly (periodically) images the area behind the vehicle M and the interior of the vehicle M from the position where the driving monitoring device 1 is mounted. The horizontal field of view captured by each of the front camera 12 and rear camera 14 is, for example, 180° or more. In other words, the driving monitoring device 1 (imaging unit 10) captures a 360° range around the vehicle M from the position where it is mounted. Each of the front camera 12 and rear camera 14 may be, for example, a fisheye camera including a fisheye lens. Alternatively, each of the front camera 12 and rear camera 14 may be a wide-angle camera, etc. In this embodiment, the system is described using a front camera 12 and a rear camera 14, but other cameras may be used instead or in addition to them. In other words, the imaging unit 10 should be configured to capture the images used in this embodiment (peripheral images and facial images, which will be described later).

[0023] The attachment / detachment part 16 is, for example, a component for attaching the driving monitoring device 1 to the vehicle M. The attachment / detachment part 16 is any component such as a suction cup, seal, bracket, or other support member. The attachment / detachment part 16 may also include a power supply cable (wiring, etc.) for supplying power to the driving monitoring device 1.

[0024] Figure 3 shows an example of an image acquired by the driving monitoring device 1. Specifically, Figure 3 shows an example of an image captured by the imaging unit 10. The front camera 12 captures the area in front of the vehicle M through the front windshield, the driver's side window, and the passenger's side window. The rear camera 14 captures the interior of the vehicle M (including the driver and other occupants), as well as the area behind the vehicle M through the driver's side window, the passenger's side window, and the rear window.

[0025] The images captured by the front camera 12 (hereinafter referred to as "front images") include objects such as other vehicles, pedestrians, bicycles, fixed objects, and road markings that are visible in front of the vehicle M through the front windshield, or to the left and right (in front of) the vehicle M through the left and right side windows.

[0026] The images captured by the rear camera 14 (hereinafter referred to as "rear images") show the interior of the vehicle M, the area behind the vehicle M as seen through the rear window, and landmarks located to the left and right (rear) of the vehicle M as seen through the left and right side windows. Therefore, the rear images captured by the rear camera 14 also include, as a subject, the driver of the vehicle M to which the driving monitoring device 1 is installed.

[0027] The front and rear images include images capturing the surroundings of vehicle M (hereinafter referred to as "surroundings images"). The rear image includes an image capturing the face of the driver of vehicle M (hereinafter referred to as "face image"). The driving monitoring device 1 may be mounted at any location as long as the surroundings images and face images can be captured by the imaging unit 10.

[0028] The information processing device 2 (see Figure 1) outputs alarms and notifications via the first terminal device T1 based on the results of referencing the surrounding image output by the imaging unit 10 and the face image. The operation monitoring device 1 and the first terminal device T1 together are sometimes referred to as the "information processing system 100".

[0029] The first terminal device T1 is a portable terminal device used by the driver operating the vehicle M to which the driving monitoring device 1 is installed, such as a smartphone or tablet terminal. The first terminal device T1 runs applications, for example, for receiving alarms and notifications from the information processing device 2. The applications display images on a display device or emit sounds through a speaker based on information transmitted by the information processing device 2 (for example, alarm information and notification information described later). The first terminal device T1 is also referred to as the alarm unit 82 and the notification unit 84. The first terminal device T1 is used, for example, in a state where it is detachably attached to the vehicle M. For example, a holder for the first terminal device T1 having a detachable part is provided on either the first terminal device T1 or the vehicle M, or both, and the first terminal device T1 is supported by the holder.

[0030] Instead of the first terminal device T1, the navigation system, display unit, or speaker of the vehicle M may be used. In other words, the navigation system, display unit, or speaker of the vehicle M may output alarms or notifications based on instructions from the information processing device 2. That is, the navigation system, display unit, or speaker of the vehicle M may be the alarm unit 82 or the notification unit 84. Alternatively, instead of the first terminal device T1, the driving monitoring device 1 may be equipped with a display unit (alarm unit 82, notification unit 84) or a speaker (alarm unit 82, notification unit 84), etc.

[0031] As shown in Figure 1, the information processing device 2 includes, for example, an acquisition unit 20, a detection unit 30, a processing unit 40, a control unit 50, a communication unit 60, and a storage unit 70. The detection unit 30 includes a risk detection unit 32, a gaze detection unit 34, and a face orientation detection unit 36. The processing unit 40 includes an oversight determination unit 42, a storage control unit 44, a generation unit 46, and an abnormality determination unit 48. The control unit 50 includes an alarm control unit 52 and a notification control unit 54.

[0032] The acquisition unit 20, the detection unit 30, the processing unit 40, and the control unit 50 each include, for example, a hardware processor such as a CPU (Central Processing Unit) and a storage device (a storage device equipped with a non-transient storage medium) that stores a program (software), and the functions of each component are realized when the processor executes the program. Some or all of these components may be realized by hardware (including circuitry) such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), and GPU (Graphics Processing Unit), or the functions of each component may be realized by the cooperation of software and hardware. Some or all of these components may also be realized by a dedicated LSI.

[0033] The program (software) may be stored in advance in the storage unit 70 (a storage device equipped with a non-transient storage medium) of the information processing device 2, such as ROM (Read Only Memory), RAM (Random Access Memory), or flash memory, or it may be stored in a removable storage medium (non-transient storage medium) such as a memory card, and installed in the storage device when the storage medium is attached to the operation monitoring device 1. The program (software) may also be downloaded in advance from another computer device via short-range communication or wide-area communication by an application running on the first terminal device T1, and installed in the storage device by being transmitted from the first terminal device T1.

[0034] The memory unit 70 stores, for example, judgment history information 72, field of view map information 74, a program (for example, the program described above), and various other information. The memory unit 70 may be implemented by the various storage devices described above, or by an EEPROM (Electrically Erasable Programmable Read Only Memory), etc. The communication unit 60 is an interface for wireless or wired communication with an information processing device (external device) such as the first terminal device T1.

[0035] The driving monitoring device 1 (information processing device 2) performs at least two functions: oversight alarm processing and abnormality notification processing. Oversight alarm processing is the process of detecting an oversight of a risk object by the driver based on surrounding images and facial images, and issuing an alarm to the alarm unit 82. Abnormality notification processing is the process of detecting an abnormality in the driver (for example, a disease such as dementia or visual field defects) based on surrounding images and facial images, and issuing a notification to the notification unit 84.

[0036] A risk object is, for example, an object that vehicle M should avoid. A risk object is, for example, an object that vehicle M will interfere with if it moves in its current state, or an object that has the potential to interfere with vehicle M. An object that may interfere is, for example, an object whose degree of proximity after a predetermined time exceeds a predetermined degree when considering the object's position, direction of movement, and speed of movement, and the position, method of movement, and speed of movement of vehicle M. The object may be a traffic participant such as a pedestrian, cyclist, or vehicle, or it may be an object other than a traffic participant, such as an object placed on the road or a fallen object.

[0037] [Oversight Alarm Processing] Figure 4 is a diagram illustrating the oversight alarm processing performed by the operation monitoring device 1 (information processing device 2).

[0038] The acquisition unit 20 acquires the front image and the rear image from the imaging unit 10. As a result, the acquisition unit 20 acquires the surrounding image and the face image in association. Since the imaging unit 10 repeatedly (periodically) captures the surrounding image and the face image, the acquisition unit 20 repeatedly (periodically) acquires the surrounding image and the face image. The acquisition unit 20 may also acquire the surrounding image and the face image in association with the time they were captured (imaging time).

[0039] The risk detection unit 32 detects risk objects in the surrounding image. "Detecting risk objects" means, for example, detecting the presence of a risk object and the position of that risk object relative to the vehicle M (driving monitoring device 1) (hereinafter referred to as the risk position). For example, reference information (not shown) indicating the relationship between the risk position in the surrounding image and its relative position to the vehicle M may be stored in the storage unit 70 in advance. The risk detection unit 32 may detect the position of the risk object relative to the vehicle M based on this reference information.

[0040] The risk detection unit 32 detects the relative position of the risk object with respect to the vehicle M, thereby also detecting the direction of the risk object's presence (approach direction) relative to the vehicle M. The "direction of the risk object's presence" refers, for example, to the direction of a risk object approaching the vehicle M (driving monitoring device 1) relative to the vehicle M (driving monitoring device 1). Hereinafter, the "direction of the risk object's presence" may be referred to as the "risk presence direction."

[0041] The gaze detection unit 34 detects the driver's gaze in the facial image. Specifically, the gaze detection unit 34 detects the driver's gaze by analyzing the facial image using a predetermined image analysis method. "Detecting the gaze" means, for example, detecting the position of the driver's eyeballs and the direction of their gaze (direction of the eyeballs).

[0042] The face orientation detection unit 36 ​​detects at least the orientation of the driver's face in the face image. Specifically, the face orientation detection unit 36 ​​detects the orientation of the driver's face by analyzing the face image using a predetermined image analysis method. In addition to the orientation of the driver's face, the face orientation detection unit 36 ​​may also detect the position of the driver's face. Hereinafter, the combined information of "face orientation" and "face position" may be referred to as "face information." The face orientation detection unit 36 ​​detects face information in the face image.

[0043] The detection unit 30 may be configured to be capable of detecting lanes, crosswalks, and the like in addition to the aforementioned risk objects, line of sight, and face information. FIG. 5 is a diagram showing an example of information detected by the detection unit 30. In the example of FIG. 5, pedestrians, other vehicles, lanes, and crosswalks detected by the detection unit 30 are shown with reference to the vehicle M, the direction of the line of sight, and the direction of the face. Pedestrians and other vehicles are examples of traffic participants (objects) that can be risk objects. Further, the information detected by the detection unit 30 may be such that traffic participants (objects) that can be risk objects detected by the detection unit 30 are shown based on the vehicle M, the orientation of the vehicle M, and / or the traveling direction of the vehicle M, instead of based on the vehicle M, the direction of the line of sight, and the direction of the face.

[0044] The oversight determination unit 42 (see FIG. 4) determines whether the driver overlooks a risk object based on the risk position and / or the direction in which the risk exists, the line of sight, and the face information. Each of the "risk position" and the "direction in which the risk exists" is an example of "information related to the position of a risk object". The line of sight and the face information are examples of "information related to a face image".

[0045] For example, reference information (not shown) indicating the relationship between the direction of the driver's eyes (gaze) and / or face in the facial image and the relative orientation of the vehicle M with respect to the direction of travel may be stored in the storage unit 70 in advance. The oversight determination unit 42 may refer to this reference information to determine whether the direction of the gaze and / or face matches the direction of the risk. If the direction of the gaze and / or face matches the direction of the risk, the oversight determination unit 42 may determine that the driver has seen the risk object. "When the direction of the gaze and / or face matches the direction of the risk" means, for example, that the risk location lies on a straight line along the direction of the gaze and / or face. Alternatively, "when the direction of the gaze and / or face matches the direction of the risk" means that the risk location lies within a conical region with the position of the viewpoint (eyes) and / or face as its apex, having a central axis along the direction of the gaze and / or face, and widening to the sides as it moves forward. If the direction of the driver's gaze and / or the direction of their face do not match the location of the hazardous object, the oversight determination unit 42 may determine that the driver has not seen the hazardous object.

[0046] The oversight detection unit 42 may determine that the driver has overlooked a risk object if the driver has not seen the risk object for a predetermined period of time or longer. The oversight detection unit 42 may also determine that the driver has not overlooked a risk object if the driver has seen the risk object, or if the state in which the driver has not seen the risk object is resolved before the predetermined period of time has elapsed. However, the method used by the oversight detection unit 42 to determine oversight is not limited to the above and can be changed as appropriate.

[0047] The oversight detection unit 42 generates oversight detection information based on the above detection result. The oversight detection information includes, for example, whether or not an oversight occurred and the direction in which the risk exists.

[0048] The alarm control unit 52 causes the first terminal device T1 (alarm unit 82) to output an alarm based on the oversight determination information. For example, the alarm control unit 52 transmits alarm information to the first terminal device T1 via the communication unit 60. The alarm information includes, for example, information indicating whether an alarm is necessary and a risk existing direction. For example, when the oversight determination information includes information indicating that "there is an oversight", the alarm control unit 52 generates alarm information including information indicating that "an alarm is necessary". For example, when the oversight determination information includes information indicating that "there is no oversight", the alarm control unit 52 generates alarm information including information indicating that "no alarm is necessary". The first terminal device T1 outputs an alarm based on the alarm information.

[0049] FIG. 6 is a diagram showing an example of an alarm performed by the first terminal device T1 (alarm unit 82). In the example of FIG. 6, the first terminal device T1 displays content by which the risk existing directions (D1 and D2 in the figure) can be recognized. The first terminal device T1 issues an alarm by making the display mode of the risk existing direction different between a case where there is an oversight of a risk object (that is, a case where the alarm information includes information indicating that "an alarm is necessary") and a case where there is no oversight of a risk object (that is, a case where the alarm information includes information indicating that "no alarm is necessary"). Specifically, when there is an oversight of a risk object, the risk existing direction is displayed in a more emphasized mode compared to the case where there is no oversight. The emphasized mode only needs to be any mode that makes it easier for the driver to visually recognize the risk existing direction, such as emphasis by color, or emphasis by blinking or lighting an icon or the like. In addition to or instead of the above, the first terminal device T1 may issue an alarm by sound output from a speaker, light emission from a lamp, or the like (A in the figure). The specific method of the alarm can be appropriately changed as long as it can be recognized by the driver.

[0050] The alarm issued by the first terminal device T1 (alarm unit 82) may, for example, be issued while the driver is operating the vehicle M (i.e., in real time). Examples of "timing while the driver is operating the vehicle M" include when the vehicle M is moving or when the vehicle M is temporarily stopped. For example, the alarm control unit 52 may transmit alarm information at the above timing. Alternatively, the alarm control unit 52 may transmit alarm information at any timing, and the first terminal device T1 (alarm unit 82) may control the timing of the alarm so that the alarm is issued at the above timing.

[0051] [Abnormality Notification Processing] Figure 7 is a diagram illustrating the abnormality notification processing performed by the operation monitoring device 1 (information processing device 2).

[0052] The memory control unit 44 stores the determination time, the risk location described above, whether or not the risk object was overlooked, and the line of sight in the memory unit 70, linking them together. The determination time corresponds, for example, to the time when the surrounding image and / or face image were captured, which was used for detecting the risk location, determining whether or not an object was overlooked, and detecting the line of sight. The risk location is an example of "information related to the location of the risk object." The above processing (storage processing) by the memory control unit 44 is performed, for example, each time the overlooked determination unit 42 generates overlooked determination information. Hereinafter, the information stored in the memory unit 70 in this manner will be referred to as determination history information 72.

[0053] Figure 8 shows an example of judgment history information 72. As the memory control unit 44 repeats the above storage process, multiple sets of information (hereinafter referred to as judgment information sets J) including the judgment time, risk location, whether or not an oversight occurred, and gaze direction are accumulated in the judgment history information 72. In other words, the memory control unit 44 generates a judgment information set J each time an oversight judgment information is generated by the oversight judgment unit 42. The memory control unit 44 then updates the judgment history information 72 by adding the generated judgment information set J to the judgment history information 72.

[0054] Returning to Figure 7, the generation unit 46 generates field map information 74 based on the judgment history information 72.

[0055] First, let's explain an example of the contents of the field of view map information 74. Figure 9 shows an example of the field of view map information 74. The field of view map information 74 is information that links information related to the frequency of drivers overlooking risk objects with the direction within the driver's field of view.

[0056] The field of view map information 74 includes, for example, a plurality of regions AR arranged in a two-dimensional (e.g., two-dimensional grid) manner. Each region AR corresponds to a direction within the driver's field of view (hereinafter referred to as the in-field of view direction). For example, the plurality of regions AR include a central region ARa that corresponds to the center (front) of the driver's field of view, and a plurality of non-central regions ARb that are regions AR other than the central region ARa. In the field of view map information 74, the relative position of each non-central region ARb with respect to the central region ARa corresponds to the relative position of each in-field of view direction with respect to the center of the field of view in the driver's field of view. The plurality of regions AR may be set such that, for example, the combined region of all of them includes the entire portion of the driver's field of view, when the driver is looking in the direction of travel of the vehicle M, that is reflected in the front windshield. In other words, the region AR located on the outermost periphery of the plurality of regions AR (non-central region ARb) may correspond to the outer edge of the front windshield, or the portion outside the outer edge, of the driver's field of view.

[0057] The numerical values ​​(0.8, 0.9, 1.0, etc.) listed for each area of ​​the AR indicate the driver's recognition rate of risk objects occurring in that area of ​​the AR (in the direction of the field of view). The recognition rate is defined, for example, as "1 - (oversight rate)". The oversight rate is defined as the number of times a risk object occurring in that area of ​​the AR was overlooked divided by the total number of times a risk object occurred in that area of ​​the AR. The recognition rate is an example of "information related to the frequency of oversight of risk objects". Hereinafter, "information related to the frequency of oversight of risk objects" may be referred to as "oversight frequency information".

[0058] Here, the visual field map information 74 shown in Figure 9 is information from a healthy person (i.e., a person without dementia, visual field defects, or other diseases). As shown in Figure 9, in the visual field map information 74 of a healthy person, the recognition rate is generally highest in the central region ARa, and decreases symmetrically as you move away from the central region ARa. This is because, generally, visual acuity and attention decrease as you move away from the center of the visual field.

[0059] Next, an example of a method for generating field of view map information 74 will be described. The generation unit 46 generates the field of view map information 74 as described above based on the judgment history information 72. For example, for each judgment information set J included in the field of view map information 74, the generation unit 46 identifies, based on the risk location and line of sight, which direction within the field of view the risk object originated (in other words, which region AR in the field of view map information 74 it corresponds to).

[0060] For example, the generation unit 46 may calculate a risk vector and a line-of-sight vector. The risk vector is a vector that starts at the position of the eyeball and ends at the risk location. The line-of-sight vector is a vector that starts at the position of the eyeball, points in the direction of the line of sight (eyeball), and has a predetermined magnitude. The generation unit 46 may then determine the direction of the risk object within the field of view by finding the angle between the risk vector and the line-of-sight vector.

[0061] The generation unit 46 may calculate the recognition rate (frequency of oversight) in each AR region by performing predetermined statistical processing based on the identified in-field direction and whether or not an oversight is included in the judgment information set J. The generation of the field map information 74 is completed by calculating the recognition rate for all AR regions. For example, the generation unit 46 may calculate the recognition rate (generate the field map information 74) based on the judgment information set J belonging to a predetermined calculation period (for example, the most recent year) from the judgment history information 72.

[0062] The generation unit 46 stores the generated field of view map information 74 in the storage unit 70, linking it with information indicating the calculation period. The generation of field of view map information 74 by the generation unit 46 may be repeated at a predetermined cycle. In this case, the cycle at which the field of view map information 74 generates the generation unit 46 may be set based on the calculation period. Alternatively, the field of view map information 74 may be generated at the instruction of the driver or the like. As the generation unit 46 repeatedly generates the field of view map information 74, the storage unit 70 stores (accumulates) multiple field of view map information 74 with different calculation periods.

[0063] The content and generation method of the field of view map information 74 are not limited to the above example and can be changed as appropriate. For example, the number of area ARs included in the field of view map information 74 and the width of the field of view corresponding to each area AR can be changed as appropriate. In addition, the oversight rate may be used as oversight frequency information, or the number of times the driver overlooked (or did not overlook) a risk object may be used. The content of the field of view map information 74 can be changed as appropriate as long as the information links the oversight frequency information with the direction within the driver's field of view. Furthermore, the generation method of the field of view map information 74 can be changed as appropriate depending on the content of the field of view map information 74.

[0064] Returning to Figure 7, the abnormality determination unit 48 determines whether or not an abnormality has occurred in the driver based on the field of view map information 74. As described above, the field of view map information 74 is information generated based on information relating to the location of a risk object (risk location), whether or not the risk object was overlooked, and the line of sight detected from the facial image. Therefore, it can be interpreted that the abnormality determination unit 48 is configured to determine whether or not an abnormality has occurred in the driver based on information relating to the location of a risk object, whether or not the risk object was overlooked, and the line of sight detected from the facial image. The abnormality determination unit 48 generates abnormality determination information based on the determination result. The abnormality determination information includes the presence or absence of an abnormality in the driver as described above, and information indicating the content of the abnormality.

[0065] Figure 10 is a flowchart showing an example of the processing flow performed by the abnormality detection unit 48. The processing shown in the flowchart in Figure 10 may be started repeatedly at a predetermined cycle, started each time the field of view map information 74 is generated by the generation unit 46, or started by instruction from the driver or the like.

[0066] First, the abnormality detection unit 48 reads the field of view map information 74 from the memory unit 70 (step S102). Specifically, the abnormality detection unit 48 reads the field of view map information 74 from the memory unit 70 that is to be used to determine whether or not there is an abnormality in the driver. For example, the abnormality detection unit 48 may read the most recent field of view map information 74 stored in the memory unit 70.

[0067] Next, the abnormality detection unit 48 determines, based on the read visual field map information 74, whether or not many oversights are occurring throughout the entire field of view of the driver (step S104). Figure 11 is a diagram showing an example of visual field map information 74, which shows an example where many oversights are occurring throughout the entire field of view. In the example shown in Figure 11, the recognition rate is lower in all areas of the AR compared to, for example, the visual field map information 74 of a healthy person shown in Figure 9. Details of the judgment process in step S104 (hereinafter referred to as the dementia judgment process) will be described later.

[0068] Returning to Figure 10, if it is determined that many oversights have occurred throughout the driver's field of vision (step S104; YES), the abnormality determination unit 48 generates abnormality determination information indicating an abnormality (dementia) (step S106). That is, the abnormality determination unit 48 generates abnormality determination information that includes information indicating that there is an abnormality in the driver and information indicating that the abnormality is dementia. This is because, if the driver suffers from dementia, it is thought that oversights of risk objects are more likely to occur regardless of the direction within the field of vision. After that, the abnormality determination unit 48 completes the processing of the flowchart shown in Figure 10. Note that the case in which many oversights have occurred throughout the driver's field of vision may be referred to as "the first case" below.

[0069] If it is not determined that many oversights are occurring throughout the entire field of view of the driver (step S104; NO), the abnormality determination unit 48 determines, based on the read field of view map information 74, whether or not many oversights are occurring in a part of the field of view of the driver (step S108). Figure 12 is a diagram showing an example of field of view map information 74, which shows an example where many oversights are occurring in a part of the field of view. In the example shown in Figure 12, compared with the field of view map information 74 of a healthy person, such as shown in Figure 9, for example, the recognition rate is reduced in some areas of the visual field (area P in the figure) and all areas of the visual field. On the other hand, in other areas of the visual field, there is no significant reduction in the recognition rate compared with the field of view map information 74 of a healthy person, such as shown in Figure 9. Details of the judgment process in step S108 (hereinafter referred to as the visual field defect judgment process) will be described later.

[0070] Returning to Figure 10, if the system determines that many oversights are occurring in a portion of the driver's field of view (step S108; YES), the abnormality determination unit 48 generates abnormality determination information indicating an abnormality (visual field defect) (step S110). That is, the abnormality determination unit 48 generates abnormality determination information that includes information indicating that there is an abnormality in the driver and information indicating that the abnormality is a visual field defect. This is because, if the driver suffers from a visual field defect, it is considered that oversights of risk objects are more likely to occur in a specific direction within the field of view. The abnormality determination unit 48 may also include information indicating the specific area (hereinafter referred to as the "defective area") where many oversights are occurring in the abnormality determination information it generates. After that, the abnormality determination unit 48 completes the processing of the flowchart shown in Figure 10. The case in which many oversights are occurring in a portion of the driver's field of view may be referred to as the "second case" below.

[0071] If the system does not determine that there are many oversights in a particular area of ​​the driver's field of vision (step S108; NO), the abnormality determination unit 48 generates abnormality determination information indicating that there is no abnormality (step S112). In other words, the abnormality determination unit 48 generates abnormality determination information that includes information indicating that there is no abnormality with the driver. After that, the abnormality determination unit 48 completes the processing shown in the flowchart in Figure 10.

[0072] Returning to Figure 7, the notification control unit 54 outputs a notification to the first terminal device T1 (notification unit 84) based on the abnormality determination information. For example, the notification control unit 54 transmits notification information to the first terminal device T1 via the communication unit 60. For example, the notification control unit 54 may transmit notification information to the first terminal device T1 if the abnormality determination information includes information indicating that there is an abnormality in the driver (i.e., the abnormality determination unit 48 has determined that an abnormality has occurred in the driver). The notification information may include, for example, information indicating the nature of the driver's abnormality (dementia, visual field defect). If the abnormality determination information includes information indicating a defect area, the notification information may also include information indicating that defect area. The notification control unit 54 does not need to transmit notification information to the first terminal device T1 if the abnormality determination information includes information indicating that there is no abnormality in the driver (i.e., the abnormality determination unit 48 has determined that no abnormality has occurred in the driver).

[0073] The first terminal device T1 (notification unit 84) outputs a notification based on the notification information. The notification may be made, for example, by displaying a string of characters, an image, etc., based on the notification information on a display device of the first terminal device T1 (notification unit 84). The first terminal device T1 (notification unit 84) may also provide a notification according to the nature of the driver's abnormality (dementia, visual field defect). For example, if the notification information includes information indicating dementia, the first terminal device T1 (notification unit 84) may provide a notification indicating that the driver has or may have dementia. Also, if the notification information includes information indicating visual field defect, the first terminal device T1 (notification unit 84) may provide a notification indicating that the driver has or may have a visual field defect. If the notification information includes information indicating a defective area, the first terminal device T1 (notification unit 84) may output content (for example, a string of characters, an image, etc.) that allows the driver to recognize the defective area.

[0074] The notification from the first terminal device T1 (notification unit 84) may include a recommendation to visit a medical institution (for example, the driver's primary care physician). In this case, the first terminal device T1 (notification unit 84) may prompt the driver to input information indicating whether or not they accept the visit to the medical institution. Specifically, the display device of the first terminal device T1 (notification unit 84) may display a virtual button or the like indicating acceptance of the visit. Such a button or the like functions as an acceptance information acquisition unit that acquires information indicating the driver's intention to accept the visit to the medical institution (hereinafter referred to as acceptance information). When the driver operates the first terminal device T1 (notification unit 84) and the acceptance information acquisition unit acquires the acceptance information, the first terminal device T1 (notification unit 84) may cause a predetermined information processing terminal to send a notification indicating that the driver has accepted the visit. The predetermined information processing terminal may be, for example, a terminal used in a medical institution (for example, the second terminal device T2 described later). Specifically, the designated information processing terminal may be, for example, a terminal used by personnel belonging to a medical institution (e.g., doctors, nurses, etc.).

[0075] However, the manner in which the notification is made by the first terminal device T1 (notification unit 84) is not particularly limited, and can be changed as appropriate, as long as it is in a manner that the driver can recognize.

[0076] The notification by the first terminal device T1 (notification unit 84) may be made at times other than when the driver is operating the vehicle M. Examples of "times other than when the driver is operating the vehicle M" include when the vehicle M is stopped, when the driver has finished operating the vehicle M, or when the driver has left the vehicle M and is at home, etc. For example, the notification control unit 54 may transmit notification information at the above timings. Alternatively, the notification control unit 54 may transmit notification information at any timing, and the first terminal device T1 (notification unit 84) may control the timing of the notification so that the notification is made at the above timings.

[0077] [Details of Dementia and Visual Field Defect Assessment Processes] The details of dementia and visual field defect assessment processes are explained below with an example. However, the details of dementia and visual field defect assessment processes are not limited to the example below and can be modified as appropriate.

[0078] The abnormality determination unit 48 reads, for example, the visual field map information 74 of a healthy person pre-stored in the memory unit 70 as reference map information. Alternatively, the abnormality determination unit 48 may read past visual field map information 74 generated by the generation unit 46 as reference map information. Subsequently, the abnormality determination unit 48 compares the reference map information with the visual field map information 74 read in the process of step S102 (i.e., the target of determining whether there is an abnormality in the driver) for each region AR, including the oversight frequency information (e.g., recognition rate). In other words, a process is performed for each region AR to calculate the difference from the reference value of the oversight frequency information included in the visual field map information 74, using the oversight frequency information included in the reference map information as the reference value. Subsequently, the abnormality determination unit 48 identifies region ARs (hereinafter referred to as oversight frequent regions) where the difference from the reference value is greater than or equal to a predetermined threshold. The predetermined threshold may be a value greater than 0 or may be 0.

[0079] Furthermore, if the identified areas where oversights frequently occur are distributed across the entire field of view map information 74, the abnormality determination unit 48 may determine that "oversights are occurring frequently across the entire field of view of the driver." In other words, the abnormality determination unit 48 may determine that "oversights are occurring frequently across the entire field of view of the driver" if oversights occur at or above the threshold value, exceeding the above-mentioned standard value, across the entire field of view of the driver. Additionally, if the identified areas where oversights frequently occur are concentrated in a portion of the field of view map information 74 (for example, the portion indicated by symbol P in Figure 12), the abnormality determination unit 48 may determine that "oversights are occurring frequently in a portion of the field of view of the driver." In other words, the abnormality determination unit 48 may determine that "oversights are occurring frequently in a portion of the field of view of the driver" if oversights occur at or above the threshold value, exceeding the above-mentioned standard value, in a portion of the driver's field of view. Furthermore, the field of view map information 74 may identify the area where areas where oversights frequently occur are concentrated as the aforementioned missing area. As described above, in this embodiment, the content of the generated abnormality judgment information is determined according to the distribution of frequently overlooked areas in the field of view map information 74, and the notification information and the content of the notification are determined accordingly. In other words, the abnormality judgment unit 48 compares the overlooked frequency information (e.g., recognition rate) for each area AR included in the field of view map information 74 with a reference value (e.g., recognition rate of a healthy person), and determines whether the driver is normal or abnormal based on the result of this comparison.

[0080] According to the first embodiment described above, the information processing device 2 includes an acquisition unit 20 that acquires an image of the area around the vehicle M and an image of the driver of the vehicle M; a risk detection unit 32 that detects risk objects in the area image; an oversight determination unit 42 that determines whether the driver has overlooked a risk object based on information relating to the location of the risk object and information relating to the face image; an abnormality determination unit 48 that determines whether an abnormality has occurred in the driver based on information relating to the location of the risk object, whether or not it was overlooked, and the gaze detected from the face image; and a notification control unit 54 that causes a notification unit 84 to send a notification when it is determined that an abnormality has occurred in the driver. This makes it possible to detect abnormalities in the driver.

[0081] <Second Embodiment> Next, a second embodiment will be described, but the basic configuration is the same as that of the first embodiment. For this reason, the same reference numerals are used for similar components, and their descriptions are omitted; only the differences will be described.

[0082] Figure 13 is a diagram illustrating the abnormality notification process in the information processing system 100A according to the second embodiment. As shown in Figure 13, the information processing system 100A according to the second embodiment includes a second terminal device T2 in addition to the operation monitoring device 1 and the first terminal device T1.

[0083] The second terminal device T2 is an information processing device used by a user different from the driver using the first terminal device T1 (for example, an employee of a hospital or insurance company). The second terminal device T2 may be, for example, a smartphone, a tablet device, or a personal computer.

[0084] In the second embodiment, the operation monitoring device 1 (notification control unit 54) transmits notification information to the second terminal device T2 instead of the first terminal device T1. That is, in the second embodiment, the notification unit 84 whose notifications are controlled by the notification control unit 54 is the second terminal device T2, not the first terminal device T1. The second terminal device T2 runs, for example, an application for receiving notifications from the information processing device 2. The application displays images, strings, etc., based on the information (notification information, etc.) transmitted by the information processing device 2 on a display device, or plays sounds through a speaker.

[0085] The second terminal device T2 causes the first terminal device T1 to send a notification based on information input by the user. The second terminal device T2 includes, for example, an input unit 92, a transmission control unit 94, and a communication unit 96.

[0086] The input unit 92 and the transmission control unit 94 each include, for example, a hardware processor such as a CPU and a storage device (a storage device equipped with a non-transient storage medium) that stores a program (software), and the functions of each component are realized when the processor executes the program. Some or all of these components may be realized by hardware (including circuit parts) such as LSIs, ASICs, FPGAs, and GPUs, or the functions of each component may be realized by the cooperation of software and hardware. Some or all of these components may be realized by dedicated LSIs. The communication unit 96 is an interface for wireless or wired communication with an information processing device such as a first terminal device T1.

[0087] The input unit 92 receives user operations and inputs them to the second terminal device T2. The input unit 92 is configured using existing input devices such as a keyboard, pointing device (mouse, tablet, etc.), buttons, or touch panel. The input unit 92 may also be an interface for connecting the input device to the second terminal device T2. In this case, the input unit 92 inputs the input signal generated in response to the user's operation in the input device to the second terminal device T2. The input unit 92 can be configured in any way that allows it to input instructions based on user operations to the second terminal device T2.

[0088] For example, the user checks the content of the notification sent by the driving monitoring device 1 (notification control unit 54) to the second terminal device T2 and considers what should be notified to the driver. The content to be notified to the driver may be, for example, the possibility of a disease (abnormality) such as dementia or visual field defects, or a recommendation to visit a medical institution for the purpose of detecting such a disease. The user inputs the information, including the notification content they considered, into the second terminal device T2 by operating the input unit 92. The transmission control unit 94 transmits the information input by the user's operation to the first terminal device T1 via the communication unit 96. The first terminal device T1 makes a notification to the driver based on the transmitted information.

[0089] Furthermore, the information processing system 100A may be configured so that the driver can operate the first terminal device T1 or the driving monitoring device 1 to restrict the transmission of notification information by the driving monitoring device 1 (notification control unit 54) to the second terminal device T2. For example, when the driver operates the first terminal device T1 or the driving monitoring device 1, restriction information indicating whether or not the transmission of the above notification information is restricted may be generated and stored in the storage unit 70. The alarm control unit 52 may then decide whether or not to transmit the notification information to the second terminal device T2 based on the restriction information stored in the storage unit 70. With such a configuration, whether or not to inform users using the second terminal device T2 of information related to the driver's abnormality is determined according to the driver's wishes. Therefore, it becomes easier to protect the driver's privacy.

[0090] According to the embodiment described above, the information processing system 100A comprises an information processing device 2, a first terminal device T1 used by the driver and not a notification unit 84 whose notifications are controlled by a notification control unit 54, and a second terminal device T2 used by a user other than the driver and being a notification unit 84 whose notifications are controlled by a notification control unit 54. The second terminal device T2 causes the first terminal device T1 to send a notification based on information input by the user's operation. This makes it possible to provide the driver using the first terminal device T1 with notification content considered by the user using the second terminal device T2.

[0091] <Modifications> The technical scope of the present invention is not limited to the embodiments described above, and various modifications can be made without departing from the spirit of the invention.

[0092] For example, the driving monitoring device 1 (information processing device 2) may perform other alarm processing (hereinafter referred to as auxiliary alarm processing) in addition to (or instead of) the oversight alarm processing described above. The auxiliary alarm processing may be, for example, collision alarm processing that detects the possibility of a risk object colliding with the vehicle M based on the position and relative speed of the risk object with respect to the vehicle M, and causes the alarm unit 82 to issue an alarm. In collision alarm processing, the relative speed of the risk object with respect to the vehicle M may be calculated, for example, by the processing unit 40 analyzing the time change of the risk position detected by the risk detection unit 32. If the processing unit 40 detects the possibility of a risk object colliding with the vehicle M, the alarm control unit 52 may cause the alarm unit 82 to issue an alarm. The auxiliary alarm processing may be performed at the timing while the driver is operating the vehicle M (i.e., in real time), similar to the oversight alarm processing. In a configuration in which the driving monitoring device 1 (information processing device 2) performs auxiliary alarm processing, the alarm unit 82 does not have to issue an alarm based on the result of the oversight determination by the oversight determination unit 42. Even in this case, the oversight detection unit 42 performs the oversight detection for the purpose of the abnormality notification process described above.

[0093] Furthermore, information relating to the results of the oversight determination by the oversight determination unit 42 may be accumulated in the storage unit 70 (for example, determination history information 72), and content for the driver to review their past driving (hereinafter referred to as "review content") generated based on the accumulated information may be provided to the driver by the first terminal device T1 (alarm unit 82). The review content may include, for example, information (such as text strings or images) that the driver can use to identify the risk object that the driver overlooked. The review content may be generated by the driving monitoring device 1 (information processing device 2) or by the first terminal device T1 (alarm unit 82).

[0094] Furthermore, in the information processing system 100A according to the second embodiment, the operation monitoring device 1 (notification control unit 54) may transmit notification information to both the first terminal device T1 and the second terminal device T2. In other words, both the first terminal device T1 and the second terminal device T2 may be notification units 84 whose notifications are controlled by the notification control unit 54.

[0095] Furthermore, although the driving monitoring device 1 (information processing device 2) and the first terminal device T1 are configured as separate devices in the first embodiment, they may be configured as an integrated device. For example, the driving monitoring device 1 may have all of the functions of the imaging unit 10, acquisition unit 20, detection unit 30, processing unit 40, control unit 50, storage unit 70, alarm unit 82, and notification unit 84. Alternatively, all of these functions may be implemented in the first terminal device T1 or the vehicle M. Similarly in the second embodiment, all of the functions of the imaging unit 10, acquisition unit 20, detection unit 30, processing unit 40, control unit 50, storage unit 70, and alarm unit 82 may be implemented in the driving monitoring device 1, or all of these functions may be implemented in the first terminal device T1. In addition, a camera other than the camera provided in the driving monitoring device 1 may be assigned to perform some or all of the functions of the imaging unit 10. For example, a driver monitor camera that images the driver may capture a facial image.

[0096] Furthermore, the driving monitoring device 1 may be implemented using multiple information processing devices. For example, the functions of the information processing device 2 may be implemented in the first terminal device T1 or the vehicle M (e.g., the ECU). Alternatively, the driving monitoring device 1 may be implemented using a device such as a cloud. For example, in the driving monitoring device 1, the detection unit 30, the processing unit 40 and the control unit 50 and the storage unit 70 may be implemented in different information processing devices. For example, the storage unit 70 may be distributed and implemented in multiple information processing devices.

[0097] The embodiments and modifications described above can be expressed as follows: An information processing device comprising: a storage medium for storing computer-readable instructions; a processor connected to the storage medium, wherein the processor, by executing the computer-readable instructions to: acquire an image of the area around the vehicle and an image of the driver of the vehicle; detect a risk object in the surrounding image; determine whether the driver overlooked the risk object based on information relating to the location of the risk object and information relating to the face image; determine whether an abnormality has occurred in the driver based on information relating to the location of the risk object, whether or not the oversight occurred, and the line of sight detected from the face image; and, if it is determined that an abnormality has occurred in the driver, causes a notification unit to send a notification.

[0098] Without departing from the spirit of the present invention, the components in the above-described embodiments may be replaced with well-known components as appropriate, and the above-described embodiments and modifications may be combined as appropriate.

[0099] 100, 100A... Information processing system 2... Information processing device 20... Acquisition unit 32... Risk detection unit 34... Gaze detection unit 42... Oversight detection unit 48... Anomaly detection unit 54... Notification control unit 70... Storage unit 72... Judgment history information 74... Field of view map information 84... Notification unit T1... First terminal device T2... Second terminal device M... Vehicle

Claims

1. Information processing device comprising: an acquisition unit that acquires an image of the area around a vehicle and an image of the driver of the vehicle; a risk detection unit that detects a risk object in the surrounding image; an oversight determination unit that determines whether the driver has overlooked the risk object based on information relating to the location of the risk object and information relating to the face image; an abnormality determination unit that determines whether an abnormality has occurred in the driver based on information relating to the location of the risk object, whether or not an oversight has occurred, and the gaze detected from the face image; and a notification control unit that causes a notification unit to send a notification when it is determined that an abnormality has occurred in the driver.

2. The information processing apparatus according to claim 1, further comprising a gaze detection unit for detecting the gaze from the facial image.

3. The information processing apparatus according to claim 1 or 2, further comprising a storage control unit that stores in a storage unit judgment history information linking information relating to the location of the risk object, whether or not the object was overlooked, and the line of sight.

4. The information processing apparatus according to claim 3, further comprising a generation unit that generates field of view map information linking information relating to the frequency of the oversight and the direction within the driver's field of view based on the judgment history information.

5. The information processing apparatus according to claim 1 or 2, wherein the abnormality determination unit determines that an abnormality has occurred in the driver in the first case in which the oversights occur at a predetermined threshold above a predetermined standard value in the entire field of view of the driver, and in the second case in which the oversights occur at a predetermined threshold above a predetermined standard value in a part of the field of view of the driver.

6. The information processing apparatus according to claim 5, wherein, in the first case, the notification control unit causes the notification unit to notify the driver that a visual field defect is occurring or is likely to occur.

7. In the second case, the notification control unit causes the notification unit to give a notification indicating that the driver is suffering from or may be suffering from dementia, as described in claim 5.

8. An information processing system comprising: an information processing device according to claim 1 or 2; a first terminal device, which is not a notification unit, used by the driver and whose notifications are controlled by the notification control unit; and a second terminal device, which is the notification unit, used by a user other than the driver and whose notifications are controlled by the notification control unit, wherein the second terminal device causes the first terminal device to make a notification based on information input by the user's operation.

9. An information processing method in which a computer performs the following steps: a process of acquiring an image of the area around a vehicle and an image of the driver of the vehicle; a process of detecting a risk object in the surrounding image; a process of determining whether the driver overlooked the risk object based on information relating to the location of the risk object and information relating to the face image; a process of determining whether an abnormality has occurred in the driver based on information relating to the location of the risk object, whether or not the oversight occurred, and the gaze detected from the face image; and a process of causing a notification unit to send a notification if it is determined that an abnormality has occurred in the driver.

10. A program for causing a computer to execute the following processes: acquiring an image of the area around a vehicle and an image of the driver of the vehicle; detecting a risk object in the surrounding image; determining whether the driver overlooked the risk object based on information relating to the location of the risk object and information relating to the face image; determining whether an abnormality has occurred in the driver based on information relating to the location of the risk object, whether or not the oversight occurred, and the gaze detected from the face image; and, if it is determined that an abnormality has occurred in the driver, causing the notification unit to send a notification.

Images