Gaze target estimation device, gaze target estimation method, program, and recording medium

Abstract

A gaze target estimation device for estimating a gaze target of a driver of a vehicle on the basis of face image data obtained by capturing the face of the driver with a camera, the gaze target estimation device comprising: an angle acquisition unit for acquiring a face orientation angle or a line-of-sight angle of the driver on the basis of the face image data of the driver; a first determiner for performing a first determination as to which of a plurality of gaze targets set for the vehicle the gaze target of the driver is, on the basis of the face orientation angle or the line-of-sight angle; a second determiner for performing a second determination as to which of a plurality of areas set for the vehicle the gaze target of the driver is in, on the basis of the face orientation angle or the line-of-sight angle; and an estimation result output unit for acquiring reliability of the first determination and outputting the determination result of one or both of the first determiner and the second determiner as an estimation result of the gaze target of the driver on the basis of the reliability.

Description

Visual recognition target estimation device, visual recognition target estimation method, program, and recording medium

[0001] The present invention relates to a visual recognition target estimation device, an information processing method, a program, and a recording medium, and more particularly, to a visual recognition target estimation device, a visual recognition target estimation method, a program, and a recording medium for estimating the visual recognition target of a vehicle driver, for example.

[0002] Estimation of the line of sight of a vehicle driver has been carried out. In line-of-sight estimation, the line-of-sight angle of the driver is obtained, and the visual recognition points inside and outside the vehicle are geometrically estimated from that angle (for example, Patent Document 1).

[0003] Japanese Patent Application Laid-Open No. 2023-066924

[0004] In order to obtain the line-of-sight angle with high accuracy as an absolute angle, it is necessary that the installation point of the camera is in the front, the distance is short, the resolution is high, and so on. However, when obtaining the line-of-sight angle at the drive recorder level, the accuracy that can be used as an absolute angle is about ±20° including the front, and in many cases, for angles larger than that, there is only accuracy that can be used as a relative angle. Thus, when using a camera of the drive recorder type, there is a problem that it is difficult to geometrically estimate the visual recognition target from the line-of-sight angle because the measurement accuracy is not high due to the constraints of the camera placement position and camera performance.

[0005] The present invention has been made in view of the above points, and one of its objects is to provide a visual recognition target estimation device capable of estimating the visual recognition target with high accuracy even when the camera does not have sufficient accuracy.

[0006] The invention described in claim 1 is a sighting target estimation device that estimates the sighting target of a vehicle driver based on face image data obtained by photographing the driver's face with a camera, and is characterized by comprising: an angle acquisition unit that acquires the driver's face orientation angle or gaze angle based on the driver's face image data; a first determination unit that makes a first determination based on the face orientation angle or gaze angle whether the sighting target of the driver is one of a plurality of sighting targets set for the vehicle; a second determination unit that makes a second determination based on the face orientation angle or gaze angle whether the sighting target of the driver is in one of a plurality of areas set for the vehicle; and an estimation result output unit that acquires the confidence level of the first determination and outputs the determination results of either or both of the first determination unit and the second determination unit as the estimated sighting target of the driver based on the confidence level.

[0007] The invention described in claim 7 is a method for estimating the target of a vehicle driver's gaze based on facial image data obtained by photographing the driver's face with a camera, comprising: an angle acquisition step of acquiring the driver's face orientation angle or gaze angle based on the driver's facial image data; a first determination step of making a first determination based on the face orientation angle or gaze angle of which of a plurality of target objects set for the vehicle the driver's gaze target is; a second determination step of making a second determination based on the face orientation angle or gaze angle of which of a plurality of areas set for the vehicle the driver's gaze target is located in; and an estimation result output step of acquiring the confidence level of the first determination and outputting the determination results of either or both of the first determination and the second determination as the estimated target of the driver's gaze based on the confidence level.

[0008] The invention described in claim 8 is an information processing program executed by an information processing device equipped with a computer, characterized in that the computer is caused to perform the following steps: an angle acquisition step of acquiring the face orientation angle or gaze angle of a vehicle driver based on face image data obtained by capturing the face of the vehicle driver with a camera; a first determination step of making a first determination based on the face orientation angle or gaze angle of which of a plurality of viewing targets set for the vehicle the driver is looking at is; a second determination step of making a second determination based on the face orientation angle or gaze angle of which of a plurality of areas set for the vehicle the driver is looking at is located in; and an estimation result output step of acquiring the confidence level of the first determination and outputting the determination results of either or both of the first determination and the second determination as an estimated result of the driver's viewing location based on the confidence level.

[0009] The invention described in claim 9 is a computer-readable recording medium, characterized in that it stores an information processing device equipped with a computer, which causes the device to perform the following steps: an angle acquisition step of acquiring the face orientation angle or gaze angle of a vehicle driver based on face image data obtained by photographing the driver's face with a camera; a first determination step of making a first determination based on the face orientation angle or gaze angle of which of a plurality of viewing targets set for the vehicle the driver is looking at is; a second determination step of making a second determination based on the face orientation angle or gaze angle of which of a plurality of areas set for the vehicle the driver is looking at is located in; and an estimation result output step of acquiring the confidence level of the first determination and outputting the determination result of either the first determination or the second determination, or both, as an estimated result of the driver's viewing location based on the confidence level.

[0010] This is a block diagram showing the configuration of a target viewing device according to an embodiment of the present invention. This is a block diagram showing the configuration of the angle acquisition unit. This is a diagram schematically showing the relationship between the angle between the camera direction and the direction in front of the driver's seat. This is a diagram showing the positions of multiple targets viewing together with the front seat area of ​​the vehicle. This is a block diagram showing the configuration of the determination unit. This is a diagram showing an example of the distribution of the first face-direction angle. This is a diagram showing an example of the probability distribution of the first face-direction angle. This is a diagram showing an example of the distribution of the second face-direction angle. This is a diagram showing an example of the combined probability distribution of face-direction angles. This is a diagram showing an example of the distribution of line-of-sight angles. This is a diagram showing an example of the combined probability distribution of line-of-sight angles. This is a diagram showing the calculation of probability coefficients based on face-direction angles. This is a diagram showing the calculation of probability coefficients based on line-of-sight angles. This is a diagram showing the distribution of face-direction angles and the distribution of line-of-sight angles side by side. This is a diagram showing the calculation of probability coefficients used for combined determination. This is a block diagram showing the configuration of a target viewing device according to Embodiment 2. This is a diagram showing an example of the arrangement of cameras inside the vehicle. This is a diagram conceptually showing the synthesis of probability coefficients by the determination unit. This is a block diagram showing the configuration of a target viewing device according to Embodiment 3. This is a diagram showing an example of multiple determination areas set in a vehicle. This figure shows an example of area determination performed by the second determination unit.

[0011] Preferred embodiments of the present invention are described in detail below. In the following descriptions and accompanying drawings, substantially identical or equivalent parts are denoted by the same reference numerals.

[0012] The sight target estimation device according to this embodiment is a device that estimates where a driver is looking based on image data of a face captured by a camera installed inside the vehicle's cabin.

[0013] Figure 1 is a block diagram showing the configuration of the sight target estimation device 100 according to this embodiment. The sight target estimation device 100 consists of an angle acquisition unit 11 and a determination unit 12. In the case where multiple cameras are installed in the vehicle, multiple determination units 12 are provided, one for each of the multiple cameras. In the following description, the configuration and operation of each part will be explained using a determination unit 12 provided in accordance with camera CA installed at the location of the drive recorder as an example.

[0014] The angle acquisition unit 11 acquires a first face orientation angle FA1, a second face orientation angle FA2, a first line of sight angle SA1, and a second line of sight angle SA2 based on face image data VD obtained by photographing the driver's face with a camera CA mounted on the vehicle M, and supplies these to the determination unit 12.

[0015] The first face orientation angle FA1 is angle data indicating the angle of the driver's face orientation, based on the direction the driver's face is facing when the driver is facing the camera CA. The second face orientation angle FA2 is angle data indicating the direction of the driver's face orientation, based on the direction the driver's face is facing the front of the driver's seat. In the following description, these will be collectively referred to simply as face orientation angle FA. In this embodiment, the camera CA is located in the upper center of the front seat area of ​​the vehicle M (for example, at the drive recorder position).

[0016] The first line of sight angle SA1 is angle data indicating the angle of the driver's line of sight, based on the driver's line of sight when facing the direction of camera CA. The second line of sight angle SA2 is angle data indicating the angle of the driver's line of sight, based on the driver's line of sight when facing straight ahead in the driver's seat. In the following explanation, these will be collectively referred to simply as line of sight angle SA.

[0017] The determination unit 12 receives inputs of the first face orientation angle FA1, the second face orientation angle FA2, the first line of sight angle SA1, and the second line of sight angle SA2, and estimates the driver's line of sight.

[0018] Figure 2 is a block diagram showing the processing block of the angle acquisition unit 11.

[0019] The face coordinate AI 21 consists of AI libraries such as MediaPipe that detect the coordinates of facial feature points based on a face image. The face coordinate AI 21 generates two-dimensional face coordinate data FCD based on the face image data VD.

[0020] The coordinate transformation unit 22 performs a process to convert the two-dimensional face coordinate data FCD into three-dimensional coordinates. The coordinate transformation unit 22 performs the coordinate transformation using, for example, the solvePnP function of OpenCV (registered trademark) to generate face orientation angle data consisting of three-dimensional coordinates. In this embodiment, a first face orientation angle FA1, which is face orientation angle data based on the direction of camera CA as seen from the driver's seat (hereinafter referred to as the camera direction), is generated as face orientation angle data.

[0021] The facial recognition AI 23 consists of an AI that recognizes human faces based on image data. The facial recognition AI 23 receives facial image data VD as input and performs facial recognition of the driver of vehicle M.

[0022] The Face Orientation AI 24 determines the driver's face orientation based on the results of face recognition by the Face Recognition AI 23. The Face Coordinate AI 25 detects the driver's face coordinates based on the results of face recognition by the Face Recognition AI 23.

[0023] The eye image extraction unit 26 extracts the driver's eye image from the face image data VD based on the face coordinate detection results from the face coordinate data FCD and face coordinate AI 25.

[0024] The gaze estimation AI 27 generates gaze angle data of the vehicle M driver based on the eye image extracted by the eye image extraction unit 26, the determination result by the face orientation AI 24, and the first face orientation angle FA1. In this embodiment, the first gaze angle SA1, which is gaze angle data based on the camera direction, is generated as gaze angle data.

[0025] The reference angle correction unit 28 corrects the reference angle of the first face-direction angle FA1 to generate a second face-direction angle FA2, which is the face-direction angle relative to the front direction of the driver's seat.

[0026] Figure 3 schematically shows the relationship between the angle between the camera direction and the direction in front of the driver's seat. The offset angle θ0 is set according to the angle between the camera direction as seen from the driver's seat (i.e., the direction connecting the driver's seat position and the camera position) and the direction in front of the driver's seat. The reference angle correction unit 28 generates the second face direction angle FA2 by shifting the first face direction angle FA1 by the offset angle θ0. The reference angle correction unit 28 also generates the second line of sight angle SA2 by shifting the first line of sight angle SA1 by the offset angle θ0.

[0027] The determination unit 12 receives inputs of a first face-direction angle FA1, a second face-direction angle FA2, a first line-of-sight angle SA1, and a second line-of-sight angle SA2, and estimates the driver's line of sight based on these. The determination unit 12 estimates the line of sight by calculating a coefficient (hereinafter referred to as the probability coefficient) that indicates the probability that the driver is seeing each of a set of pre-set line of sight targets.

[0028] Figure 4 shows the locations of multiple viewing targets along with the front seat area of ​​vehicle M. The side window next to the passenger seat is designated as the left window VP1, the left side mirror as the left mirror VP2, the camera installed in the front left of the interior as the left camera VP3, the rearview mirror at the front upper part of the interior as the rearview mirror VP4, the car navigation display screen as the navigation VP5, a predetermined area of ​​the windshield in front of the driver's seat as the front VP6, the speedometer as the meter VP7, the camera installed in the front right of the interior as the right camera VP8, the right side mirror as the right mirror VP9, ​​and the side window next to the driver's seat as the right window VP10, each of which is set as a viewing target.

[0029] Figure 5 is a block diagram showing the configuration of the determination unit 12. The determination unit 12 includes a probability distribution holding unit 31, a composite probability distribution generation unit 32, and an estimation unit 33.

[0030] The probability distribution holding unit 31 holds a probability distribution indicating the likelihood that the driver of vehicle M has seen each of the multiple sighted objects. In this embodiment, the probability distribution holding unit 31 holds a first probability distribution PD1, a second probability distribution PD2, a third probability distribution PD3, and a fourth probability distribution PD4.

[0031] The first probability distribution PD1 is a probability distribution that shows the probability (likelihood) of the driver viewing each of multiple viewing targets for each first face-direction angle FA1. The first probability distribution PD1 is pre-set based on the results of test drives by multiple subjects. Specifically, the first face-direction angle FA1 is acquired as data when the driver, as a subject, gazes at each viewing target multiple times during a test drive, and the first probability distribution PD1 is generated by creating an angle distribution (hereinafter referred to as the first face-direction angle distribution) based on the data of the first face-direction angle FA1 acquired for multiple subjects.

[0032] Figure 6 shows an example of the first face orientation angle distribution. Here, the camera direction is set as the reference angle (0°), and the horizontal and vertical angle distributions corresponding to each object being viewed are shown. In particular, a highly accurate angle distribution can be obtained for horizontal angles within ±20° of the reference angle.

[0033] For example, from the first face orientation angle distribution in Figure 6, it can be seen that when the horizontal angle is 0°, there is a high probability that the navigation VP5 is being viewed. Also, when the horizontal angle is 30°, there is a high probability that the right mirror VP9 is being viewed.

[0034] Figure 7 shows the first probability distribution PD1 obtained based on the first face orientation angle distribution in Figure 6. Here, only the probability distributions for the left mirror VP2, rearview mirror VP4, front VP6, and right mirror VP9 are extracted and shown from among the multiple viewing targets.

[0035] On the other hand, the second probability distribution PD2 is a probability distribution that shows the probability (likelihood) that the driver is viewing each of the multiple viewing targets for each second face-direction angle FA2. The second probability distribution PD2 is pre-set based on the results of test drives by multiple subjects. Specifically, the second face-direction angle FA2 is acquired as data when the driver, who is a subject, gazes at each viewing target multiple times during a test drive, and the second probability distribution PD2 is generated by creating an angle distribution (hereinafter referred to as the second face-direction angle distribution) based on the second face-direction angle FA2 data acquired for multiple subjects.

[0036] Figure 8 shows an example of the second face-direction angle distribution. Here, the forward direction of the driver's seat is used as the reference angle (0°), and the horizontal and vertical angle distributions corresponding to each object being viewed are shown. The lower part of the figure shows the second face-direction angle distribution, and the upper part shows the first face-direction angle distribution.

[0037] Figure 9 shows the second probability distribution PD2 obtained based on the second face orientation angle distribution shown in Figure 8. Similar to the first probability distribution PD1 in Figure 7, only the probability distributions for the left mirror VP2, rearview mirror VP4, front VP6, and right mirror VP9 are extracted and shown here.

[0038] For example, from the second face-direction angle distribution shown in Figure 9, it can be seen that when the horizontal angle of the second face-direction angle FA2 is 0°, there is a high probability that the driver is looking at the front VP6. Also, when the horizontal angle is +20°, there is a high probability that the driver is looking at the right mirror VP9.

[0039] Referring again to Figure 5, the composite probability distribution generation unit 32 generates a composite probability distribution for the face orientation angle FA by combining the first probability distribution PD1 and the second probability distribution PD2.

[0040] Figure 10 shows an example of a composite probability distribution. Since both the first probability distribution PD1 and the second probability distribution PD2 have a base of "1", a composite probability distribution can be generated by taking their product. By estimating the target of observation based on the composite probability distribution generated by taking the product of these two probability distributions, it is possible to strongly express highly reliable evoked and suppressive effects.

[0041] Furthermore, the composite probability distribution generation unit 32 may generate a new composite probability distribution each time the offset angle θ0 is changed. By generating a composite probability component according to the offset angle θ0 in this way, it is possible to achieve highly robust estimation of the target even when conditions change.

[0042] Furthermore, the composite probability distribution generation unit 32 generates a composite probability distribution for the line of sight angle SA by combining the third probability distribution PD3 and the fourth probability distribution PD4.

[0043] The third probability distribution PD3 is a probability distribution that shows the probability that a driver is visually recognizing each of a plurality of visual recognition targets for each first line-of-sight angle SA1. The third probability distribution PD3 is preset based on the results of test runs by a plurality of subjects. That is, the first line-of-sight angle SA1 when the driver, who is a subject, gazes at each visual recognition target a plurality of times during a test run is acquired as data, and an angle distribution (first line-of-sight angle distribution) is created based on the data of the first line-of-sight angle SA1 acquired for a plurality of subjects, whereby the third probability distribution PD3 is generated.

[0044] The fourth probability distribution PD4 is a probability distribution that shows the probability that a driver is visually recognizing each of a plurality of visual recognition targets for each second line-of-sight angle SA2. The fourth probability distribution PD4 is preset based on the results of test runs by a plurality of subjects. That is, the second line-of-sight angle SA2 when the driver, who is a subject, gazes at each visual recognition target a plurality of times during a test run is acquired as data, and an angle distribution (second line-of-sight angle distribution) is created based on the data of the first line-of-sight angle SA1 acquired for a plurality of subjects, whereby the fourth probability distribution PD4 is generated.

[0045] FIG. 11 is a diagram showing an example of the second line-of-sight angle distribution. Here, with the front direction of the driver's seat as the reference angle (0°), the angle distributions in the horizontal and vertical directions corresponding to each visual recognition target are shown. Note that the lower part in the figure is the second line-of-sight angle distribution, and the upper part shows the first line-of-sight angle distribution.

[0046] The combined probability distribution generation unit 32 generates a combined probability distribution for the line-of-sight angle SA by taking the product of and combining the third probability distribution PD3 obtained based on the first line-of-sight angle distribution and the fourth probability distribution PD4 obtained based on the second line-of-sight angle distribution.

[0047] FIG. 12 is a diagram showing an example of a combined probability distribution generated by combining the third probability distribution PD3 and the fourth probability distribution PD4. Here, only the probability distributions of the left mirror VP2, the room mirror VP4, the front VP6, and the right mirror VP9 are extracted and shown. Similar to the combined probability distribution for the face orientation, since both the third probability distribution PD3 and the fourth probability distribution PD4 use the reference as "1", they can be combined by taking their product, and reliable induction effects and suppression effects can be strongly exhibited.

[0048] The estimation unit 33 determines the viewing target of the driver of the vehicle M based on the probability distribution (PD1 to PD4) held in the probability distribution holding unit 31 or the combined probability distribution generated by the combined probability distribution generation unit 32. In this embodiment, the estimation unit 33 takes the face orientation angle FA (FA1, FA2) or the line-of-sight angle SA (SA1, SA2) acquired by the angle acquisition unit 11 as an input, and calculates a probability coefficient using the probability distributions PD1 to PD4 or the combined probability distribution. The calculated probability coefficient indicates the probability that the driver is viewing each of a plurality of preset viewing targets (VP1 to VP10). That is, the probability coefficient calculated by the estimation unit 33 becomes the estimation result of the viewing target of the driver of the vehicle M.

[0049] For example, when the estimation unit 33 determines the viewing target based on the face orientation angle FA, as shown in FIG. 13, the first face orientation angle FA1 or the second face orientation angle FA2 is applied to the combined probability distribution for the face orientation angle FA to calculate the probability coefficient P1. The estimation unit 33 outputs the calculated probability coefficient P1 as the estimation result of the viewing target of the driver of the vehicle M.

[0050] Also, when the estimation unit 33 determines the viewing target based on the line-of-sight angle SA, as shown in FIG. 14, the first line-of-sight angle SA1 or the second line-of-sight angle SA2 is applied to the combined probability distribution for the line-of-sight angle SA to calculate the probability coefficient P2. The estimation unit 33 outputs the calculated probability coefficient P2 as the estimation result of the viewing target of the driver of the vehicle M.

[0051] Furthermore, the estimation unit 33 determines the viewing target using both the face orientation angle FA (FA1 or FA2) and the line of sight angle SA (SA1 or SA2). This improves the estimation accuracy compared to estimating the viewing target using only one of the face orientation angle FA or line of sight angle SA.

[0052] Figure 15 is a diagram that shows, side by side, the second face-direction angle FA2, which is angle data of the face direction relative to the front direction of the driver's seat, and the second line-of-sight angle SA2, which is angle data of the line of sight relative to the front direction of the driver's seat.

[0053] As can be seen from the vertical distribution in the figure, gaze direction changes significantly in the vertical direction, while face orientation changes less in the vertical direction. This is because the accuracy of vertical angle determination using facial feature points is low when recording from the position of the dashcam inside the car.

[0054] On the other hand, regarding the horizontal direction of the diagram, in ranges with large angles, such as when viewing left and right mirrors, the eyes may be hidden from the camera CA, making it impossible to determine the angle of the line of sight or resulting in misrecognition of the line of sight.

[0055] Thus, since both estimating the target of gaze based on the face orientation angle (FA) and estimating the target of gaze based on the line of sight angle (SA) have their advantages and disadvantages, it is possible to achieve highly accurate estimation of the target of gaze by using these methods in combination.

[0056] For example, as shown in Figure 16, the estimation unit 33 generates a composite probability coefficient P3 by multiplying (i.e., multiplying together) the probability coefficient P1 calculated based on the face angle FA and the probability coefficient P2 calculated based on the line of sight angle SA. The estimation unit 33 outputs the generated composite probability coefficient P3 as the estimated result of the vehicle M driver's line of sight. By multiplying the probability coefficients in this way, highly reliable judgment results are emphasized and less reliable judgment results are suppressed.

[0057] As described above, the sight target estimation device 100 of this embodiment estimates the driver's sight target by using a first face direction angle FA1, which is the face direction angle based on the direction the driver's face is facing when facing the camera, or a second face direction angle FA2, which is the face direction angle based on the direction in front of the driver's seat, and calculating a probability coefficient P1 based on a composite probability distribution obtained by combining a first probability distribution PD1 corresponding to the first face direction angle FA1 and a second probability distribution PD2 corresponding to the second face direction angle FA2.

[0058] Furthermore, the sight target estimation device 100 of this embodiment estimates the driver's sight target by using a first line of sight angle SA1, which is the line of sight angle when the driver is facing the camera, or a second line of sight angle SA2, which is the line of sight angle based on the direction in front of the driver's seat, and calculating a probability coefficient P2 based on a composite probability distribution obtained by combining a third probability distribution PD3 corresponding to the first line of sight angle SA1 and a fourth probability distribution PD4 corresponding to the second line of sight angle SA2.

[0059] Furthermore, the sight target estimation device 100 of this embodiment estimates the driver's sight target by calculating a composite probability coefficient P3 based on a probability coefficient obtained based on a first face orientation angle FA1 or a second face orientation angle FA2 and the probability distribution corresponding to each, and a probability coefficient obtained based on a first line of sight angle SA1 or a second line of sight angle SA2 and the probability distribution corresponding to each.

[0060] With this configuration, even if the first face orientation angle FA1, the second face orientation angle FA2, the first line of sight angle SA1, and the second line of sight angle SA2 cannot obtain highly accurate angles individually due to constraints such as the camera resolution and the distance to the subject, it becomes possible to estimate the target of the line of sight with high accuracy by combining them.

[0061] Next, Embodiment 2 of the present invention will be described. In addition to the functions of the visibility target estimation device 100 of Embodiment 1, the visibility target estimation device of this embodiment has the function of selecting a determination device (corresponding to the determination unit 12 of Embodiment 1), combining determination devices, etc., based on the angle (offset angle) between the camera direction and the direction in front of the driver's seat. In this embodiment, n cameras (n is an integer of 2 or more) are installed inside the vehicle M, and n determination devices corresponding to each camera are provided in the visibility target estimation device.

[0062] Figure 17 is a block diagram showing the configuration of the viewing target estimation device 200 in this embodiment. The viewing target estimation device 200 includes an angle acquisition unit 11, a judgment unit selection unit 40, and an estimation unit 50. The configuration and operation of the angle acquisition unit 11 are the same as in Embodiment 1, so a description is omitted here.

[0063] The determination unit 40 includes an offset calculation unit 41, a camera position determination unit 42, a selection processing unit 43, and a difference angle calculation unit 44.

[0064] The offset calculation unit 41 calculates the offset angle θ as the angle between the direction of the driver's face or line of sight when the driver of the vehicle M is facing the camera and the direction in front of the driver's seat, based on the face direction angle FA or line of sight angle SA acquired by the angle acquisition unit 11.

[0065] The camera position determination unit 42 determines the position of the corresponding camera based on the offset angle θ calculated by the offset calculation unit 41.

[0066] The selection processing unit 43 identifies a camera corresponding to the camera position determined by the camera position determination unit 42 from among the n cameras, and selects a determination device corresponding to the identified camera from among the n determination devices.

[0067] Figure 18 shows an example of camera placement inside the vehicle M. In this example, camera CA1 is located at the location of the drive recorder, camera CA2 is located above the driver's seat, camera CA3 is located at the location of the car navigation system's display panel, camera CA4 is located directly in front of the driver's seat, camera CA5 is located on the right pillar, and camera CA6 is located on the left pillar.

[0068] Cameras CA1 to CA6 each have different angles between their direction as seen from the driver's seat and the direction directly in front of the driver's seat. Therefore, the camera position determination unit 42 can determine the position of a camera based on the offset angle θ and identify the camera corresponding to that position.

[0069] The difference angle calculation unit 44 calculates the difference between the offset angle θ calculated by the offset calculation unit 41 and the default offset angle pre-set in the determination unit selected by the selection processing unit 43, and uses this difference angle φ as the difference angle.

[0070] In other words, cameras CA1 to CA6 may be installed at a position different from where they should be installed. In such cases, there may be a difference between the actual offset angle θ and the predetermined offset angle (default offset angle). For this reason, the sight target estimation device 200 in this embodiment adjusts the angle using the difference angle φ, which corresponds to that difference, and then estimates the driver's sight target.

[0071] The estimation unit 50 includes a determination unit 51-1 to 51-n, an angle adjustment unit 52, a warning notification control unit 53, and a determination unit generation unit 54.

[0072] The detectors 51-1 to 51-n are provided in accordance with the n cameras installed inside the vehicle M. Each detector 51-1 to 51-n has a predetermined offset angle set according to the position of the corresponding camera and the direction in front of the driver's seat.

[0073] Each of the determination devices 51-1 to 51-n is provided as a default determination device that estimates the target of the vehicle M driver's gaze by taking either or both of the face direction angle FA and the gaze angle SA as input and calculating a probability coefficient using a probability distribution set for multiple viewing targets.

[0074] The angle adjustment unit 52 adjusts the angle of the face direction angle FA or gaze direction angle SA input to the determination units 51-1 to 51-n based on the difference angle φ calculated by the difference angle calculation unit 44. Specifically, if the difference angle φ is less than a predetermined threshold, the angle adjustment unit 52 generates an angle in which the face direction angle FA or gaze direction angle SA is shifted by the difference angle φ. The determination units 51-1 to 51-n use the face direction angle FA or gaze direction angle SA adjusted by the angle adjustment unit 52 to determine the target of the gaze.

[0075] The warning control unit 53, when the difference angle φ calculated by the difference angle calculation unit 44 is greater than or equal to a predetermined threshold, causes the navigation system to notify the user, via a display on the screen or by sound, that the camera's position has deviated from its default position.

[0076] Furthermore, if the camera position determined based on the offset angle θ calculated by the offset calculation unit 41 corresponds to a position near the midpoint of multiple cameras, it is difficult to identify a single camera that corresponds to that position. In such cases, in this embodiment, the selection processing unit 43 identifies multiple cameras from among the n cameras. The selection processing unit 43 then selects multiple determination units corresponding to these cameras.

[0077] The classifier generation unit 54 generates a classifier to be actually used for estimating the target of visual observation, based on the classifier selection unit 40's selection of classifiers, as a practical classifier. For example, if multiple classifiers are selected by the classifier selection unit 40, the classifier generation unit 54 generates a practical classifier by combining the selected multiple classifiers. Specifically, the classifier generation unit 54 generates a combined probability distribution by combining the probability distributions set for each of the classifiers selected by the classifier selection unit 40, and uses this combined probability distribution as the probability distribution of the practical classifier.

[0078] Furthermore, if one determination device is selected by the determination device selection unit 40, the determination device generation unit 54 will use that one determination device as the practical determination device.

[0079] Figure 19 conceptually illustrates the synthesis of probability distributions when the classifier generation unit 54 generates a practical classifier based on multiple classifiers. For example, if the camera position determined based on the offset angle θ calculated by the offset calculation unit 41 is located near the midpoint between the camera corresponding to classifier 1 and the camera corresponding to classifier 2, the classifier selection unit 40 selects both classifier 1 and classifier 2.

[0080] The classifier generation unit 54 synthesizes probability distributions while weighting them according to the distance between the camera position determined by the camera position determination unit 42 and each of the multiple cameras corresponding to the selected multiple classifiers.

[0081] For example, in the example shown in Figure 19, the probability distribution obtained by classifier 1 is multiplied by the coefficient "a", and the probability distribution obtained by classifier 2 is multiplied by the coefficient "b", thereby weighting the two probability distributions and combining them.

[0082] The estimation unit 50 applies the face orientation angle FA and gaze angle SA acquired by the angle acquisition unit 11 to the probability distribution of a practical judge (i.e., a judge selected from judges 51-1 to 51-n or a judge obtained by combining multiple judges) generated by the judge generation unit 54, and outputs the probability coefficient obtained as the estimated target of the driver's gaze.

[0083] As described above, the sight target estimation device 200 of this embodiment calculates an offset angle θ based on the face orientation angle FA or the line of sight angle SA, and determines the camera position based on the calculated offset angle θ. This allows for the selection of an appropriate determination device according to the camera position, and the estimation of the driver's sight target.

[0084] Furthermore, the target viewing device 200 in this embodiment calculates the difference angle φ between the calculated offset angle θ and a predetermined offset angle, and adjusts the face orientation angle FA or line of sight angle SA. This enables highly accurate estimation of the target viewing location in response to the shift in camera position.

[0085] Furthermore, in this embodiment, the sight target estimation device 200 estimates the driver's sight target using a combined determination device when multiple predetermined determination devices are selected based on the offset angle θ. This allows for flexible and highly accurate estimation of the sight target.

[0086] Next, Embodiment 3 of the present invention will be described. The sight target estimation device of this embodiment differs from the sight target estimation device 100 of Embodiment 1 in that, in addition to having the same function as Embodiment 1 in determining which of a plurality of preset sight targets the driver of vehicle M is looking at, it also has the function of determining which of a plurality of preset areas the driver is looking at.

[0087] Figure 20 is a block diagram showing the configuration of the viewing target estimation device 300 in this embodiment. The viewing target estimation device 300 includes an angle acquisition unit 11 and an estimation unit 60. The configuration and operation of the angle acquisition unit 11 are the same as in Embodiment 1, so a description is omitted here.

[0088] The estimation unit 60 includes a first determination unit 61, a second determination unit 62, a reliability determination unit 63, and an estimation result output unit 64.

[0089] The first determination device 61, similar to the determination unit 12 in Embodiment 1, receives input of either or both of the face direction angle FA and the gaze angle SA from the angle acquisition unit 11, and performs a determination (hereinafter referred to as detailed determination) of which of a plurality of pre-set viewing targets the driver of the vehicle M is viewing.

[0090] The second determination device 62 receives input of face direction angle FA or gaze direction angle SA (either one or both) from the angle acquisition unit 11 and determines which of the pre-set multiple viewing areas the driver of the vehicle M is viewing (hereinafter referred to as area determination).

[0091] Figure 21 shows an example of multiple viewing areas set in vehicle M. Here, the central area of ​​the front seat area of ​​vehicle M is set as the center area A1, the area directly in front of the driver's seat is set as the front area A2, the area on the right is set as the right side area A3, the area including the right window is set as the right window area A4, the area on the left is set as the left side area A5, and the area including the left window is set as the left window area A6.

[0092] These viewing areas are set to include viewing objects that are prone to misidentification in detailed judgment among the multiple viewing objects shown in Figure 4. For example, the center area A1 includes the rearview mirror VP4 and navigation VP5 among the multiple viewing objects shown in Figure 4. The front area A2 includes the front VP and meter VP7. The right side area A3 includes the right camera VP8 and right mirror VP9. The right window area A4 includes the right window VP10. The left side area A5 includes the left mirror VP2 and left camera VP3. The left window area A6 includes the left window VP1.

[0093] The second determination unit 62 maintains a probability distribution for each of the first face orientation angle FA1, the second face orientation angle FA2, the first line of sight angle SA1, and the second line of sight angle SA2, indicating that each determination area is the target of the gaze. The determination of the target of the gaze is performed by calculating a probability coefficient using these probability distributions. Note that the probability distribution for area determination by the second determination unit 62 is set separately from the probability distribution for detailed determination by the first determination unit 61.

[0094] Figure 22 shows an example of area determination performed by the second determination device 62. When determination is made using four parameters: the first face orientation angle FA1, the second face orientation angle FA2, the first line of sight angle SA1, and the second line of sight angle SA2, the probability coefficient for the final determination result is obtained by calculating the product of the probability coefficients obtained for each of these parameters.

[0095] For example, in the example shown in Figure 22, for the left side area A5, values ​​of 0.75 are obtained for the first face orientation angle FA1, 1 for the second face orientation angle FA2, and 0.9 for the second line of sight angle SA2, but the value for the first line of sight angle SA1 is 0, so the probability coefficient obtained by multiplying these values ​​is 0. Ultimately, the probability coefficient for the center area A1 is 1.4, the probability coefficient for the front area A2 is 0.8, and the probability coefficients for the other viewing areas are 0. When normalized, the probability of viewing the center area A1 is 63%, and the probability of viewing the front area A2 is 37 percent.

[0096] Referring again to Figure 20, the reliability determination unit 63 determines the reliability of the detailed determination as an estimated result of the sighted object based on the determination result of the first determination unit 61. In the detailed determination by the first determination unit 61, a probability coefficient is obtained for all sighted objects, and by normalizing this, the reliability of the estimated sighted object can be determined.

[0097] For example, if the detailed determination by the first determination device 61 shows that the probability of viewing the rearview mirror VP4 and the probability of viewing the navigation VP5 are both high and of similar magnitude, then, after normalizing the probability coefficients, the reliability of the detailed determination is judged to be lower than a predetermined level. In other words, if it is determined that either the rearview mirror VP4 or the navigation VP5 is being viewed in this state, there is a certain degree of possibility of misjudgment, and therefore the reliability of the detailed determination is judged to be low.

[0098] The estimation result output unit 64 outputs the estimated target as the visual target, prioritizing either the detailed determination result by the first determination unit 61 or the area determination result by the second determination unit 62, based on the reliability level determined by the reliability determination unit 63. Specifically, the estimation result output unit 64 prioritizes outputting the detailed determination result if the reliability level of the detailed determination is above a predetermined level, and prioritizes outputting the area determination result if the reliability level of the detailed determination is below a predetermined level.

[0099] For example, if the confidence level of the detailed determination is lower than a predetermined level, the estimation result output unit 64 outputs only the determination result of the area determination as the estimated target of the sighting. Also, if the confidence level of the detailed determination is at or above a predetermined level, the estimation result output unit 64 outputs only the determination result of the detailed determination as the estimated target of the sighting.

[0100] Furthermore, the estimation result output unit 64 only needs to prioritize outputting one of the results according to the confidence level of the detailed determination. Therefore, it is not necessary to output only one of the results as the estimation result. For example, the determination results of both the detailed determination and the area determination may be output in a manner that prioritizes one of them, such as by changing the size of the display.

[0101] As described above, the sight target estimation device 300 of this embodiment performs detailed determination, which pinpoints the sight target for a plurality of pre-set sight targets, as well as area determination, which determines the sight target for a wider area. The sight target estimation device 300 then determines the reliability of the detailed determination, and if the reliability is above a predetermined level, it prioritizes the determination result of the detailed determination and outputs it as the estimated sight target. If the reliability is below the predetermined level, it prioritizes the determination result of the area determination and outputs it as the estimated sight target. This makes it possible to provide information about roughly where the driver is looking, for example, when it is difficult to pinpoint the driver's sight target.

[0102] It should be noted that the present invention is not limited to those shown in the above embodiments. For example, in Embodiment 1, the estimation unit 33 was described as generating a composite probability coefficient P3 by taking the product of a probability coefficient P1 calculated based on a first face direction angle FA1 and a second face direction angle FA2, and a probability coefficient P2 calculated based on a first line of sight angle SA1 and a second line of sight angle SA2. However, even when estimating the target of view using both face direction and line of sight angles, it is not necessarily required to use all four angles. It is sufficient to use at least one of the first face direction angle FA1 and the second face direction angle FA2, and at least one of the first line of sight angle SA1 and the second line of sight angle SA2. For example, it is possible to calculate a composite probability coefficient by taking the product of a probability coefficient obtained based only on the first face direction angle FA1 and a probability coefficient obtained based only on the second line of sight angle SA2, and then estimate the target of view.

[0103] Furthermore, in the above embodiment 1, when estimating the target of gaze using face direction angle FA (either one or both of FA1 and FA2) and gaze direction angle SA (either one or both of SA1 and SA2) as input, the case where the target of gaze is estimated is described as being estimated based on the product of probability coefficients calculated based on each probability distribution. However, even when face direction angle FA and gaze direction angle SA are used as input, if the reliability of either angle is low, the determination may be made using only the probability coefficient calculated based on the probability distribution of the other angle.

[0104] For example, if the driver's face is completely turned towards the right window, the driver's right eye will be obscured from the camera, significantly reducing the reliability of the gaze angle SA. Also, if the driver blinks frequently, the gaze will waver before and after blinking, resulting in a lot of noise in the distribution of the gaze angle SA, greatly reducing its reliability. In such cases where the reliability of the gaze angle SA is low, the gaze angle SA element can be excluded, and the target of the gaze can be estimated with high accuracy using only the probability distribution and probability coefficient of the face orientation angle FA.

[0105] Furthermore, in the above-described embodiment 1, a case was explained in which a composite probability coefficient P3 is generated by taking the product of a probability coefficient P1 obtained based on the face direction angle FA and a probability coefficient P2 obtained based on the gaze direction angle SA. However, the method of composing probability coefficients is not limited to this, and if the estimation accuracy of one of the face direction angle FA and gaze direction angle SA is expected to be considerably higher than the estimation accuracy of the other, the combination ratio may be varied to perform the combination of probability coefficients.

[0106] For example, if the estimation accuracy using the gaze angle SA is expected to be sufficiently high, the probability coefficient of the face orientation angle FA can be adjusted by changing the synthesis ratio of the face orientation angle FA during synthesis. Let β1 be the coefficient of variation, B be the probability coefficient P1 of the face orientation angle FA, and B' be the corrected face orientation probability coefficient. Then, the probability coefficient of the face orientation angle FA can be corrected using the formula B' = 1 + (B - 1) × β1. Furthermore, if γ1 is the estimation accuracy of the target of gaze, then β1 can be written as β1 = β1(γ1).

[0107] Similarly, if D is the probability coefficient P2 of the line of sight angle SA, D' is the probability coefficient P2 of the corrected line of sight angle SA, β2 is the coefficient of variation with respect to the line of sight angle, and γ2 is the estimation accuracy of the target of view, then D' can be written as D' = 1 + (D - 1) × β2, and β2 = β1(γ2).

[0108] As a result, the composite probability coefficient P3 is B' × D' = (1 + (B - 1) × β1(γ1)) × (1 + (D - 1) × β2(γ2)).

[0109] Furthermore, in the above embodiment 2, an example was described in which, if the difference angle φ calculated by the difference angle calculation unit 44 is greater than or equal to a predetermined threshold, the system notifies the user that the camera position has deviated from the default position through a display on the navigation screen or by sound. However, contrary to this, even if the difference angle φ calculated by the difference angle calculation unit 44 is greater than or equal to a predetermined threshold, the system generator generation unit 54 may generate a practical system generator by combining system generators, similar to the case where multiple system generators are selected. For example, if only system generator 1 is selected by the system generator selection unit 40 and the difference angle φ is large, a combined system generator is generated by combining system generator 1 and system generator 2, making it possible to perform a more accurate judgment than if system generator 1 were used as the practical system generator. Alternatively, instead of generating a new system generator, another system generator that can obtain a smaller difference angle φ may be selected from the existing system generators.

[0110] Furthermore, in the above embodiment 3, we described a case where the determination result of area determination is prioritized when the reliability of detailed determination is low. However, the manner of estimating the viewing target when reliability is low is not limited to this. For example, if two viewing targets are estimated with roughly the same probability, it is possible to reduce the possibility of misdetermination while maintaining detailed determination by pre-determining that one of them be prioritized as the viewing target (for example, prioritizing the mirror if it is a window and a mirror).

[0111] Furthermore, the estimation of the gaze target may be performed at predetermined time intervals, or only when the face orientation angle FA or gaze angle SA changes significantly. It is assumed that the driver looks straight ahead for most of the time while driving, and that changes in gaze are not significant. Also, estimating the gaze target requires a certain amount of time for the estimation operation (in other words, the analysis process of the gaze target) because it involves comparing the acquired face orientation angle FA or gaze angle SA with a probability distribution. For example, estimating the gaze target using all four angles—first face orientation angle FA1, second face orientation angle FA2, first gaze angle SA1, and second gaze angle SA2—requires approximately 50 msec for estimation based on one face image data VD. Therefore, by estimating the gaze target only when the face orientation angle FA or gaze angle SA changes significantly, the estimation time (analysis processing time) for estimating the gaze target can be shortened, and a decrease in operating speed can be prevented. For example, by acquiring the face orientation angle FA and gaze direction angle SA at predetermined time intervals, and performing gaze direction estimation only when at least one of the face orientation angle FA or gaze direction angle SA has changed by 5° or more from the previously acquired angle, the analysis processing time can be shortened.

[0112] Furthermore, while the target of observation can be estimated at a frequency of approximately 10 Hz, a change in the target of observation requires about 0.2 seconds, so a change occurring in 0.1 seconds is likely to be noise due to blinking or other factors. For this reason, it may be advisable to perform processing such as excluding short-term judgments or smoothing the judgment results for several frames in order to estimate the target of observation.

[0113] 100 Target Estimation Device 11 Angle Acquisition Unit 12 Judgment Unit 21 Face Coordinate AI 22 Coordinate Transformation Unit 23 Face Recognition AI 24 Face Orientation AI 25 Face Coordinate AI 26 Eye Image Extraction Unit 27 Gaze Estimation AI 28 Reference Angle Correction Unit 31 Probability Distribution Holding Unit 32 Composite Probability Distribution Generation Unit 33 Estimation Unit 41 Offset Calculation Unit 42 Camera Position Determination Unit 43 Selection Processing Unit 44 Difference Angle Calculation Unit 50 Estimation Unit 51-1 to 51-n Determinants 52 Angle Adjustment Unit 53 Warning Notification Control Unit 54 Determinant Generation Unit 60 Estimation Unit 61 First Determinant 62 Second Determinant 63 Confidence Determination Unit 64 Estimation Result Output Unit

Claims

1. A sighting target estimation device that estimates the target of a vehicle driver based on facial image data obtained by photographing the driver's face with a camera, comprising: an angle acquisition unit that acquires the driver's face orientation angle or gaze angle based on the driver's facial image data; a first determination unit that makes a first determination based on the face orientation angle or gaze angle whether the target of the driver's gaze is one of a plurality of sighting targets set for the vehicle; a second determination unit that makes a second determination based on the face orientation angle or gaze angle whether the target of the driver's gaze is in one of a plurality of areas set for the vehicle; and an estimation result output unit that acquires the confidence level of the first determination and outputs the determination results of either or both of the first and second determination units as the estimated target of the driver's gaze based on the confidence level.

2. The sight target estimation device according to claim 1, further comprising a reliability determination unit that determines the reliability of the first determination based on the determination result of the first determination unit, wherein the estimation result output unit outputs the determination result of the second determination unit with priority over the determination result of the first determination unit when the reliability is below a predetermined level.

3. The sight destination estimation device according to claim 2, characterized in that, when the confidence level is below a predetermined level, the estimation result output unit outputs only the determination result of the second determination device as the estimated sight destination of the driver.

4. The sight target estimation device according to claim 1, characterized in that the estimation result output unit outputs the determination result of the first determination device with priority over the determination result of the second determination device when the reliability is above a predetermined level.

5. The sight destination estimation device according to claim 4, characterized in that the estimation result output unit outputs only the determination result of the first determination device as the estimated sight destination of the driver when the reliability is above a predetermined level.

6. The sight target estimation device according to claim 1, characterized in that each of the plurality of areas is configured to include one or more sight targets.

7. A method for estimating the target of a vehicle driver's gaze based on facial image data obtained by photographing the driver's face with a camera, comprising: an angle acquisition step of acquiring the driver's face orientation angle or gaze angle based on the driver's facial image data; a first determination step of making a first determination based on the face orientation angle or gaze angle to determine which of a plurality of target objects set for the vehicle the driver is looking at; a second determination step of making a second determination based on the face orientation angle or gaze angle to determine which of a plurality of areas set for the vehicle the driver is looking at; and an estimation result output step of acquiring the confidence level of the first determination and outputting the determination results of either the first determination or the second determination, or both, as the estimated target of the driver's gaze based on the confidence level.

8. An information processing program executed by an information processing device equipped with a computer, characterized in that the computer is caused to execute: an angle acquisition step of acquiring the face orientation angle or gaze angle of a vehicle driver based on face image data obtained by photographing the face of the vehicle driver with a camera; a first determination step of making a first determination based on the face orientation angle or gaze angle of which of a plurality of viewing targets set for the vehicle the driver is looking at; a second determination step of making a second determination based on the face orientation angle or gaze angle of which of a plurality of areas set for the vehicle the driver is looking at; and an estimation result output step of acquiring the confidence level of the first determination and outputting the determination result of either or both of the first determination and the second determination as an estimated result of the driver's viewing location based on the confidence level.

9. A computer-readable recording medium storing an information processing program for an information processing device equipped with a computer, which includes: an angle acquisition step of acquiring the angle of the driver's face or gaze direction based on face image data obtained by photographing the driver's face with a camera; a first determination step of making a first determination based on the angle of the face or gaze direction to determine which of a plurality of targets set for the vehicle the driver is looking at is located where the driver is looking at is located; a second determination step of making a second determination based on the angle of the face or gaze direction to determine which of a plurality of areas set for the vehicle the driver is looking at is located where the driver is looking at is located; and an estimation result output step of acquiring the confidence level of the first determination and outputting the determination result of either the first determination or the second determination, or both, as an estimated result of the driver's looking at location based on the confidence level.

Images