Line of sight calibration device, line of sight calibration method, and line of sight calibration program

Abstract

A line of sight calibration device (100) comprises a correlation determination unit (130) and an expression parameter generation unit (140). The correlation determination unit (130) selects elements that correspond to line-of-sight information that is based on images in accordance with the level of correlation between the elements that correspond to the line-of-sight information and elements included in fixation-related information that indicates physical quantities related to fixation positions for persons shown in the images and generates a fixation-related information estimation model that is made up of the selected elements and expression parameters. The expression parameter generation unit (140) calculates values for the expression parameters on the basis of the line-of-sight information, the fixation-related information estimation model, and the fixation-related information.

Description

Eye-line calibration device, eye-line calibration method, and eye-line calibration program

[0001] This disclosure relates to an eye-line calibration device, an eye-line calibration method, and an eye-line calibration program.

[0002] A person's gaze is one of the important pieces of information for understanding their intentions, and gaze estimation technology is used in various situations. Conventional technology generally estimates the direction of gaze or the position of fixation based on the movement of the eyeballs, etc. In gaze estimation, there is generally an error between the estimated position of fixation and the actual position of fixation due to individual differences in eyeball shape or the positional relationship between the sensor that acquires the gaze and the person. Therefore, calibration methods are widely used to correct for this error. As an example, there is a method that uses a calibration matrix to project the estimated position of fixation to the appropriate position of fixation. In this method, multiple calibrations are required depending on various relative posture states in order to accommodate the camera posture and how the person is captured (for example, if the person is seen looking from an angle). Here, if optimal calibration data does not exist in advance, it is not possible to perform appropriate calibration, and the accuracy decreases. Therefore, as a specific example, Patent Document 1 discloses a technology that aims to enable the handling of various relative posture states using data obtained from a single calibration. This technology involves capturing an eye image, detecting the roll angle, which is the relative angle change of the eyes, from the captured eye image, and performing a correction process based on the detected roll angle. This correction process corrects the calibration data according to the direction of the user's eyes. To realize this technology, the calibration data is stored linked to information correlated with the line of sight, such as angle information.

[0003] International Publication No. 2022 / 196093 Pamphlet

[0004] Conventional techniques perform calibration when the positional relationship between the sensor and the person, or the positional relationship between the person and the object being gazed upon, is known, or when the changes in these relationships are relatively small. Here, the technique disclosed in Patent Document 1 is intended for use in situations where information correlated with the line of sight is known. Furthermore, in this technique, only the relative posture between the sensor and the person is considered, and the positional relationship between the two is not considered. If a constant calibration matrix is ​​always used while ignoring changes in these relationships, the error in line of sight estimation increases. Here are the problems with conventional techniques (calibration methods) that address changes in these relationships: [Problem 1] When using conventional techniques, it is necessary to prepare a large number of calibration matrices according to the conditions. Therefore, the data acquisition cost becomes very high in order to increase comprehensiveness. [Problem 2] When learning a dynamic calibration matrix when using conventional techniques, it is difficult to apply changes in conditions, etc., because it is not known which physical quantities are correlated with each condition. [Problem 3] When using conventional techniques, it is necessary to prepare calibration data in combination with parameters that have a high correlation with the line of sight, such as the roll angle of the face. However, conventional techniques do not address how to handle situations where the correlation is unknown. [Problem 4] When using conventional techniques, it is necessary to create a calibration model that can be applied to situations where a person is looking at an arbitrary location. However, conventional techniques do not address situations where the positional and angular relationships between the sensor and the person change.

[0005] This disclosure aims to realize a calibration method applicable to a technology that performs calibration for an estimated gaze position, even when the positional relationship between the sensor and the person is unknown or changes.

[0006] The gaze calibration device according to this disclosure comprises: a correlation determination unit that calculates correlations between elements of gaze-related information, which indicates physical quantities related to the gaze position of each person at each time point, estimated based on images taken at each time point of each person who is gazing at a target gaze position defined by the arrangement of three or more calibration points, and elements of gaze-related information, which indicates physical quantities related to the gaze of a person shown in each image estimated based on the images corresponding to the gaze-related information, and selects one or more elements from the elements corresponding to the gaze-related information according to the height of the calculated correlations, and generates a gaze-related information estimation model, which is a model consisting of the selected elements and each expression parameter, and which estimates physical quantities related to the estimated gaze position of each person as estimated gaze-related information values ​​based on gaze-related information corresponding to each person; and an expression parameter generation unit that calculates the value of each expression parameter based on physical quantities corresponding to each gaze-related information estimated value estimated based on gaze-related information corresponding to each image corresponding to the gaze-related information and the gaze-related information estimation model.

[0007] According to this disclosure, the correlation determination unit generates a gaze-related information estimation model based on the level of correlation between each element included in the gaze-related information and each element corresponding to the gaze information. The representation parameter generation unit generates representation parameters based on the gaze information, the gaze-related information estimation model, and the gaze-related information. Here, the gaze-related information estimation model can be generated even when the positional relationship between the sensor and the person is unknown or changes. The gaze-related information estimation model and each representation parameter may be used to calculate a calibration matrix. Therefore, according to this disclosure, in a technique for performing calibration on an estimated gaze position, a calibration method applicable even when the positional relationship between the sensor and the person is unknown or changes can be realized.

[0008] A diagram showing an example configuration of the gaze calibration system 90 according to Embodiment 1. A diagram illustrating each calibration point and each estimated gaze position according to Embodiment 1. Gaze condition i and estimated gaze position X according to Embodiment 1.ｉ A diagram illustrating the relationship with '. A diagram illustrating the method for calculating the calibration matrix according to Embodiment 1. A diagram illustrating the calibration matrix according to Embodiment 1. A diagram illustrating the gaze position according to Embodiment 1. Gaze condition i and estimated gaze position X according to Embodiment 1. ｉ A diagram illustrating the relationship with '. A diagram showing an example of the hardware configuration of the gaze calibration device 100 according to Embodiment 1. A flowchart showing the operation of the gaze calibration device 100 according to Embodiment 1 in calculating each expression parameter. A flowchart showing the operation of the gaze calibration device 100 according to Embodiment 1 in performing gaze calibration. A diagram showing an example of the hardware configuration of the gaze calibration device 100 according to a modified example of Embodiment 1.

[0009] In the descriptions and drawings of the embodiments, the same elements and corresponding elements are denoted by the same reference numerals. The descriptions of elements denoted by the same reference numerals are omitted or simplified as appropriate. The arrows in the figures mainly indicate the flow of data or processing. Also, "part" may be read as "circuit," "device," "equipment," "process," "step," "procedure," "processing," or "circuitry" as appropriate. The functions of each part of each device may be realized by firmware, software, hardware, or a combination thereof.

[0010] Embodiment 1. This embodiment will be described in detail below with reference to the drawings.

[0011] ***Configuration Description*** Figure 1 shows an example configuration of the gaze calibration system 90 according to this embodiment. As shown in Figure 1, the gaze calibration system 90 consists of a sensor 20, a device 30, and a gaze calibration device 100. The sensor 20 has the function of capturing images of people. Specific examples of the sensor 20 include an RGB (Red, Green, Blue) camera, an RGB-D (Depth) camera, or a stereo camera. The device 30 is a device that utilizes the output of the gaze calibration device 100. As shown in Figure 1, the gaze calibration device 100 includes a person capturing unit 110, a gaze information acquisition unit 120, a correlation determination unit 130, a representation parameter generation unit 140, a virtual gaze position estimation unit 150, a calibration matrix calculation unit 160, and a gaze calibration execution unit 170. Calibration is a technique for estimating true values ​​and is a concept that includes the meanings of calibration and adjustment.

[0012] The person-capture unit 110 has the function of capturing images of each person who is the target of gaze detection using the sensor 20 and generating person-capture data. Instead of generating person-capture data, the person-capture unit 110 may accept input of pre-prepared person-capture data. The person-capture data is data that shows images of each person captured at each point in time so that the eyes of each person who is gazing at the area containing all the calibration points included in the calibration point cloud are visible. Each image shown in the person-capture data may be a still image or a video. The person-capture data may include multiple capture data for one person, or multiple capture data for multiple people. The person shown in each image shown in the person-capture data may be called the subject. The calibration point cloud consists of three or more calibration points. The calibration point cloud corresponds to a predetermined known gaze target set for calibration and corresponds to the correct gaze position. There may be multiple calibration point clouds. The arrangement of each calibration point among multiple calibration point clouds is predetermined. Also, the positional relationship between each calibration point cloud and the sensor 20 is constant. The correct gaze position corresponds to the correct values ​​of the estimated gaze position and virtual gaze position described later. The calibration point is the point that the person gazes at. The coordinates of the calibration point may be two-dimensional or three-dimensional. The calibration point is also called the gaze point.

[0013] The gaze information acquisition unit 120 detects the gaze of each person from the person shooting data generated by the person shooting unit 110 and acquires gaze information corresponding to each person based on the detected gaze of each person. Any method may be used as the gaze detection method. The gaze information acquisition unit 120 also acquires ambient information corresponding to each person, estimates the gaze position corresponding to each person based on the acquired gaze information and ambient information, and generates gaze-related information based on the estimated gaze position. Each gaze information is information corresponding to each image taken at each point in time of a person gazing at a target gaze position defined by the arrangement of three or more calibration points, and is information indicating each physical quantity related to the gaze of the person shown in each image estimated based on each image corresponding to the gaze-related information, and is information indicating each physical quantity related to the situation in which the person shown in each image is gazing at the gaze position. The gaze information basically indicates each physical quantity. Each physical quantity indicated by the gaze information corresponding to each image corresponding to the gaze-related information is basically a physical quantity that may have a correlation with each element included in the gaze-related information. The parameters corresponding to the physical quantities indicated by the gaze information are explanatory variables of the gaze-related information estimation model. The gaze information corresponding to each image corresponding to the gaze-related information may include information indicating the physical relationship between the sensor 20 that captured each image corresponding to the gaze-related information and the person indicated in each image corresponding to the gaze-related information. The gaze information corresponding to a person is information about the gaze of that person and indicates each physical quantity that may be correlated with the estimated gaze position of that person. The gaze information indicates specific examples of each physical quantity, and specific examples of the physical relationship between the sensor 20 and the person, such as the orientation of the person's face (3D), the horizontal position of the eyes, the height of the eyes, the gaze vector (3D), and the vector starting point (3D). The surrounding information corresponding to a person is information indicating the surroundings of the person at the time the person was photographed. The surrounding information indicates, as a specific example, the field of view seen by the person and the environment surrounding the person. The surrounding information may be a 3D (3-dimension) model. The estimated gaze position is the estimated gaze position.A person's gaze position is the location that the person is looking at. A gaze position is an area defined by a point cloud. In this specification, point clouds and gaze positions may be treated as equivalent. Specifically, each point cloud may mean a gaze position corresponding to each point cloud, and each gaze position may mean a point cloud corresponding to each gaze position. Gaze-related information is information indicating a group of estimated gaze positions, and is information indicating each physical quantity related to the gaze position of each person at each point in time, estimated based on each image taken of each person looking at the target gaze position at each point in time. Gaze-related information may include information indicating the gaze position of a person shown in each image, estimated based on the gaze information corresponding to each image corresponding to the gaze-related information. Gaze-related information may also indicate at least a part of each gaze information. Each gaze information may be gaze information acquired by the gaze information acquisition unit 120. Each estimated gaze position indicated by the gaze-related information is defined by three or more estimated gaze points. The number of estimated gaze points defining each estimated gaze position indicated by the gaze-related information is basically the same as the number of calibration points defining the target gaze position. Each estimated gaze point defining each estimated gaze position indicated by the gaze-related information corresponds to each calibration point. Each estimated gaze position indicated by the gaze-related information may also be information generated based on ambient information that shows the situation around the person corresponding to each estimated gaze position at the time the image corresponding to each estimated gaze position was taken. The gaze-related information corresponding to a person is information that shows each physical quantity related to the object that the person is looking at. As a specific example, the gaze-related information is information that shows the gaze position and gaze direction of the person. As a specific example, the gaze direction consists of the heading angle and the pitch angle. The estimated gaze position group consists of multiple gaze positions estimated based on multiple gaze information. The estimated gaze position group may consist of estimated gaze positions corresponding to one or more people other than the subject. Not all estimated gaze positions included in the estimated gaze position group correspond to the same person. Each of the estimated gaze positions in the estimated gaze position group may be a gaze position estimated by the gaze information acquisition unit 120, or it may be an estimated gaze position prepared separately.

[0014] The correlation determination unit 130 calculates the correlation between each element included in the gaze-related information and each element corresponding to the gaze information, and selects one or more elements from each element corresponding to the gaze information according to the calculated correlation level. Subsequently, the correlation determination unit 130 generates a gaze-related information estimation model, which is a model consisting of the selected elements and each representation parameter, and which estimates each physical quantity related to the estimated gaze position corresponding to each person as a gaze-related information estimate, based on the gaze information corresponding to each person. The correlation determination unit 130 may utilize the correct gaze position when generating the gaze-related information estimation model. Each element corresponding to the gaze information may be a relational expression relating to parameters corresponding to each physical quantity indicated by the gaze information. Each element included in the gaze-related information may include each coordinate component of the estimated gaze point corresponding to each calibration point, and may also include each physical quantity indicated by the gaze information.

[0015] Specifically, the correlation determination unit 130 performs a process to verify the information necessary to represent the gaze-related information estimation model, with the aim of defining a gaze-related information estimation model in order to handle calibration under various conditions. Correlation determination data is used in this process. The correlation determination unit 130 is also called the gaze-line information correlation determination unit. In this process, the correlation determination unit 130 determines from the gaze-line information corresponding to the situation in which each person is gazing at the calibration point cloud that the correlation between the gaze-related information and each person is sufficiently high. The criterion value for the degree of correlation can be determined in any way. In other words, when the target variable of the gaze-related information estimation model is the gaze target information estimate, the correlation determination unit 130 identifies appropriate values ​​as explanatory variables that can represent the target variable. Each value is a physical quantity that has a sufficiently high correlation with the estimated gaze position. The correlation determination unit 130 uses values ​​that have a sufficiently high correlation with the estimated gaze position (for example, the position of the eyes) as explanatory variables to explain the calibration matrix, and uses the explanatory variables to represent the gaze position information estimate. The correlation determination data consists of gaze-related information and line-of-sight information corresponding to each estimated gaze position indicated by the gaze-related information. The correlation determination data may be calibration data acquired in advance or data acquired in a real environment. The gaze-related information estimation model is a model that expresses the gaze-related information estimate using "gaze conditions," and is a model consisting of each explanatory variable and each expression parameter described below. Each explanatory variable of the gaze-related information estimation model is a parameter relating to each physical quantity indicated by the line-of-sight information, and is a parameter used in the gaze-related information estimation model. The gaze-related information estimation model is also called a matrix calculation model or a calibration model. The number of explanatory variables in the gaze-related information estimation model may be one or multiple. The gaze conditions are conditions relating to each physical quantity in the situation in which each person is gazing at the gaze position. The gaze conditions include conditions relating to the positional relationship between the sensor 20 and each person. The gaze conditions may include at least one of the following: conditions relating to the positional relationship between each person and the object being gazed at, and conditions relating to the angular relationship between the sensor 20 and each person.

[0016] As a specific example, first, the correlation determination unit 130 aims to generate a gaze-related information estimation model capable of representing "estimated gaze-related information values ​​during gaze-gazing of a calibration point cloud" using values ​​acquired by the gaze-line information acquisition unit 120. To this end, it calculates the correlation between each candidate explanatory variable and the relationship between each candidate explanatory variable and each coordinate component of each estimated gaze position indicated by the gaze-related information. This correlation serves as reference information for determining each explanatory variable of the gaze-related information estimation model and the relationship between each explanatory variable. In this case, the correlation determination unit 130 may also calculate the correlation between each physical quantity other than each coordinate component of the estimated gaze position indicated by the gaze-related information. Next, the correlation determination unit 130 generates the gaze-related information estimation model by determining each explanatory variable of the gaze-related information estimation model according to the level of correlation and determining the relationship between each explanatory variable according to the correlation. Note that the correlation between each physical quantity may differ for each coordinate component. Therefore, it may be determined that a certain physical quantity has a sufficiently high correlation with a certain coordinate component of the estimated gaze position shown by the gaze-related information estimate, but does not have a sufficiently high correlation with other coordinate components of the estimated gaze position shown by the gaze-related information estimate. In this case, in the gaze-related information estimation model, the parameter relating to that physical quantity is used as an explanatory variable for that coordinate component, but the parameter relating to that physical quantity is not used as an explanatory variable for the other coordinate components. When the objective variable of the gaze-related information estimation model includes variables relating to physical quantities other than each coordinate component of the estimated gaze position, the same applies to each physical quantity other than each coordinate component of the estimated gaze position. Coordinate components are each element that constitutes the coordinates of each point. The relational expression relating to explanatory variables is a calculation formula used in the gaze-related information estimation model, and is a calculation formula that uses explanatory variables. Specific examples of such relational expressions include the explanatory variable itself, an n-th degree polynomial (where n is an integer greater than or equal to 2) using the explanatory variable, a trigonometric function with the explanatory variable as a parameter, or a polynomial using the explanatory variable.

[0017] The representation parameter generation unit 140 calculates the value of each representation parameter based on the gaze information corresponding to each image indicated by the gaze-related information, the gaze-related information estimation model, the physical quantities corresponding to each gaze-related information estimation value, and the physical quantities corresponding to each image corresponding to the gaze-related information. The representation parameter generation unit 140 may also calculate the value of each representation parameter based on regression analysis of the physical quantities corresponding to the gaze information corresponding to each image corresponding to the gaze-related information, the gaze-related information estimation model, and the physical quantities corresponding to each image corresponding to the gaze-related information. The representation parameter generation unit 140 may also utilize the correct gaze position when calculating the value of each representation parameter. Specifically, the representation parameter generation unit 140 calculates each representation parameter corresponding to the virtual gaze-related information under a certain gaze condition, with the aim of obtaining a calibration matrix from a known calibration point cloud that is used when performing calibration on an unknown gaze point corresponding to gaze information acquired under a certain gaze condition. In other words, the expression parameter generation unit 140 calculates each parameter to represent the relationship between the dependent variable and the independent variable based on the data acquired for calibration. The expression parameter generation unit 140 is also called the virtual gaze-related information expression parameter generation unit. More specifically, when the expression parameter generation unit 140 applies actual observation data to the gaze-related information estimation model, it uses regression analysis or the like to calculate each expression parameter so that the error between each actual estimated gaze position and each estimated gaze position represented by the gaze-related information estimation model is as small as possible. This is because it is usually impossible to find a formula for the gaze-related information estimation model that holds true without error for all data. Therefore, as the gaze-related information estimation model, a formula obtained by changing the coefficients and order, etc., is used that minimizes the error for each data. Note that the definition of error can be anything. Each actual estimated gaze position is a gaze position estimated by known technology.Representation parameters are parameters used to represent the virtual gaze position in the gaze-related information estimation model. They are used to represent the estimated gaze-related information (= dependent variable) using explanatory variables in a situation where a person is gazing at a predetermined known gaze target set up for calibration. Representation parameters are, in concrete terms, coefficients or constant terms within the gaze-related information estimation model. Representation parameters are also called calibration parameters. In the gaze-related information estimation model, each representation parameter is treated as a constant. The virtual gaze-related information corresponding to a person is information indicating the gaze position and line of sight of that person, assuming that the person is gazing at a known calibration point cloud.

[0018] If a calibration matrix corresponding to a gaze condition, where the orientation of the face and the height of the eyes are each specific values, is known, then calibration becomes possible for all gaze information under the same gaze condition as the said gaze condition. Here, if the relationship between three or more points whose positions are known and the gaze-related information estimate corresponding to the situation when those three or more points are gazed upon under a certain gaze condition is known, then a calibration matrix corresponding to that certain gaze condition can be obtained. Then, the representation parameter generation unit 140 defines a gaze-related information estimation model that can represent the gaze-related information when gazing on a calibration point cloud, using each explanatory variable that has a sufficiently high correlation with the gaze-related information, based on each explanatory variable and the relationship expression related to each explanatory variable determined by the correlation determination unit 130. After that, the representation parameter generation unit 140 calculates the value of each representation parameter of the defined gaze-related information estimation model. Here, the gaze-related information estimate corresponds to the target variable of the gaze-related information estimation model.

[0019] As an example, if both the first and second correlation coefficients are sufficiently high, the gaze-related information estimate W' can be defined as shown in [Equation 1]. The first correlation coefficient is the correlation coefficient between the value of the person's eye height h raised to the power of 1 and each coordinate component indicated by the gaze-related information W. The second correlation coefficient is the correlation coefficient between the value of the square of the person's standing position p in the left-right direction and each coordinate component indicated by the gaze-related information W.

[0020]

[0021] In [Equation 1], a, b, and c are each representation parameters. h to the power of 1 is a relational expression relating to h. p squared is a relational expression relating to p. The representation parameter generation unit 140 calculates the value of each representation parameter using the gaze information acquired by the gaze information acquisition unit 120, thereby defining a gaze-related information estimation model that can represent the estimated gaze position using information with a sufficiently high correlation with the gaze. h and p are each explanatory variables determined by the correlation determination unit 130.

[0022] As another example, if the correlation coefficient between each coordinate component of the correct gaze position X and each value of the gaze-related information W is sufficiently high, and if the correlation coefficient between the square of the person's standing position p in the left-right direction and each value of the gaze-related information W is sufficiently high, the gaze-related information estimate W' can be defined as shown in [Equation 2].

[0023]

[0024] The virtual gaze position estimation unit 150 aims to calculate a calibration matrix from the positional relationship between the subject and the gaze position (virtual gaze position) assuming that the subject is gazing at a calibration point cloud. To this end, it estimates the virtual gaze position using the subject's gaze information and a gaze-related information estimation model defined by the representation parameter generation unit 140. Here, the gaze position is unknown in the data actually acquired as the target of gaze detection. Therefore, the virtual gaze position estimation unit 150 calculates the estimated gaze position of the subject as the virtual gaze position, assuming that the subject was gazing at a predetermined position under the conditions under which the data was acquired. The virtual gaze position is a hypothetical estimated gaze position for calibration purposes. Note that the correct value of the gaze position in the actual data is unknown. Therefore, the virtual gaze position estimation unit 150 estimates where the estimated gaze position would be if the subject were gazing at a predetermined position under the conditions corresponding to the data, using representation parameters.

[0025] The calibration matrix calculation unit 160 calculates a calibration matrix corresponding to the actual gaze position corresponding to data acquired under the same gaze conditions, based on the relationship between the virtual gaze position and the calibration point cloud. Here, the calibration point cloud corresponds to the true value of the virtual gaze position. The calibration matrix is ​​a matrix for estimating the actual gaze position based on the estimated gaze position. The calibration matrix corresponding to the conditions corresponding to the virtual gaze position calculated by the virtual gaze position estimation unit 150 is a projection transformation matrix that transforms the virtual gaze position to a predetermined set position.

[0026] The gaze calibration execution unit 170 performs calibration to estimate the actual gaze position using the calibration matrix calculated by the calibration matrix calculation unit 160. The gaze calibration execution unit 170 outputs the calibration result to the device 30.

[0027] Here, we will explain the processing of each part when gaze information corresponding to the subject is acquired as acquired gaze information based on an image of the subject, and the estimated gaze position of the subject is estimated as the calibration target gaze position based on the acquired gaze information. Alternatively, the gaze information acquisition unit 120 may acquire the acquired gaze information based on an image of the subject and estimate the calibration target gaze position based on the acquired gaze information. The virtual gaze position estimation unit 150 estimates the subject's estimated gaze position, assuming that the subject is gazing at a target gaze position defined by the arrangement of three or more calibration points, as a virtual gaze position, based on the acquired gaze information, the gaze-related information estimation model generated by the correlation determination unit 130, and the values ​​of each expression parameter calculated by the expression parameter generation unit 140. The calibration matrix calculation unit 160 calculates a calibration matrix for performing calibration on the calibration target gaze position based on the target gaze position and the virtual gaze position. The gaze calibration execution unit 170 performs calibration on the calibration target gaze position using the calibration matrix. The target gaze position for calibration is defined by three or more estimated gaze points. The number of estimated gaze points defining the target gaze position is basically the same as the number of calibration points defining the target gaze position. Each estimated gaze point defining the target gaze position for calibration corresponds to each calibration point.

[0028] In the inventor's verification of data in a real environment, a trend was observed in the discrepancy between each estimated gaze position and the correct value for each estimated gaze position when the positional relationship between the person and the sensor 20 changed. Therefore, it is considered that when the positional relationship between the person and the sensor 20 changes, the estimated gaze position is correlated with the positional relationship between the person and the sensor 20, or the orientation of the person's face as seen from the sensor 20. Accordingly, in this embodiment, a gaze-related information estimation model is defined that can express the gaze-related information estimate value, which is the objective variable, using parameters corresponding to each physical quantity that has a sufficiently high correlation with the estimated gaze position as explanatory variables.

[0029] Here, the notation in this specification will be explained. Consider the estimation of the fixation position when a calibration point group consisting of three or more known calibration points is set. As a specific example, as shown in FIG. 2, when setting calibration points t １ to t ４ on a plane as the calibration point group. Also, let the fixation-related information when each person fixes on the calibration point group under a certain fixation condition be represented by t １ ’ to t ４ ’. It is assumed that the fixation-related information is generated based on the data captured by the sensor 20 for each person. Note that t １ corresponds to t １ ’, t ２ corresponds to t ２ ’, t ３ corresponds to t ３ ’, t ４ corresponds to t ４ ’, and t １ : (x １ , y １ ), t ２ : (x ２ , y ２ ), t ３ : (x ３ , y ３ ), t ４ : (x ４ , y ４ ), t １ ’: (x １ ’, y １ ’), t ２ ’: (x ２ ’, y ２ ’), t ３ ’: (x ３ ’, y ３ ’), t ４ ’: (x ４ ’, y ４ ’) are each represented as such. Hereinafter, the coordinates of each calibration point of the known calibration point group such as t １ to t ４ are collectively expressed as X, and from t １ ’ to t ４The coordinates of each point that defines each estimated gaze position corresponding to a known calibration point cloud such as ' will be collectively represented as X'. Here, if the gaze conditions are different, the estimated gaze position may differ even if each person is gazing at the same calibration point cloud. Therefore, the coordinates of the estimated gaze position corresponding to a certain gaze condition i will be collectively represented as X'. ｉ Let's represent it as '. ｉ ' is, t １ｉ : (x １ｉ ', y １ｉ ') and, t ２ｉ : (x ２ｉ ', y ２ｉ ') and, t ３ｉ : (x ３ｉ ', y ３ｉ ') and, t ４ｉ : (x ４ｉ ', y ４ｉ ') and consists of X ｉ ' represents each estimated gaze position indicated by gaze-related information. ｉ This is a calibration point. ｉ ' indicates an estimated point of focus.

[0030] As a concrete example, consider the case shown in Figure 3, where the estimated gaze position X' corresponding to the calibration point cloud X, which is the gaze position of each person, has a sufficient correlation with the first power of each person's eye height and also with the square of each person's standing position in the left-right direction. The gaze position defined by the calibration point cloud X is also called the gaze target position. In this case, the gaze conditions are conditions relating to the person's eye height and the person's position. X corresponds to the target gaze position defined by the four calibration points. In this case, the estimated gaze position X is when each person is gazing at the calibration point cloud X under gaze condition i (i = 1, 2, ..., N). ｉ ' is assumed to be known data. Here, N is the number of data points. The physical quantity of the gaze condition i is height h. ｉ and position p ｉ And so it is. Height h ｉ is the eye level of the person corresponding to gaze condition i. Position p ｉis the left-right position of the person corresponding to gaze condition i. Position p ｉ The coordinate system is, in specific examples, a coordinate system with the front of sensor 20 as the origin. The left-right direction is, in specific examples, the direction parallel to the plane corresponding to the calibration point cloud X, and is also horizontal. The front of sensor 20 is, in specific examples, the direction passing through sensor 20, and each point is on the perpendicular direction to the plane corresponding to the calibration point cloud X. In this case, the estimated gaze position P is when each person is gazing at an arbitrary unknown point cloud under gaze condition N+1. Ｎ＋１ ' = (p Ｎ＋１ ', q Ｎ＋１ ',1) is assumed to be unknown data. Also, the estimated gaze position X is assumed to be when the subject is gazing at the calibration point cloud X under gaze condition N+1. Ｎ＋１ ' is assumed to be unknown data. Here, the physical quantity of the gaze condition N+1 is height h Ｎ＋１ and position p Ｎ＋１ Therefore, the estimated gaze position P is as follows: Ｎ＋１ The coordinates of ' are extended to three dimensions. Also, it is unknown to the gaze calibration device 100 whether the person corresponding to gaze condition N+1 is gazing at the calibration point cloud X. Therefore, in the gaze position estimation process corresponding to gaze condition N+1, the calibration point cloud X can be said to be a virtual gaze point group. At this time, gaze condition i and estimated gaze position X ｉ The relationship with ' is shown in Figure 3. x ｊｉ ' and y ｊｉ Each of these is a coordinate component of each estimated point of focus. １ｊ and a ２ｊ and b １ｊ and b ２ｊ and c １ｊ and c ２ｊ Each of these is a representation parameter. The gaze-related information estimation model f is a regression model, etc. Note that in Figure 3, the correlation between height and position for all coordinate components is assumed to be sufficiently high. However, for some coordinate components, the correlation with at least one of height and position does not need to be sufficiently high. In other words, the values ​​of some elements of the matrix corresponding to the gaze-related information estimation model f may be 0.

[0031] When it is assumed that the actual fixation position of the subject is the calibration point group X under the fixation condition N+1, the estimated fixation position X Ｎ＋１ ’ can be calculated as shown in [Equation 3] using the fixation-related information estimation model f as described above.

[0032]

[0033] Here, when considering the relationship between the calibration point group X and the estimated fixation position X Ｎ＋１ ’ as a projective transformation, the calibration point group X and the estimated fixation position X Ｎ＋１ ’ can be expressed as in [Equation 4]. Here, j is each integer from 1 to the total number of calibration points included in the calibration point group X. The calibration point group X assumed to be fixated by the subject may be different from the calibration point group X when generating the fixation-related information estimation model f. However, it is assumed that the positional relationship between each calibration point group X and the sensor 20 is constant. Also, when summarizing [Equation 4], it becomes as shown in [Equation 5]. Note that the coordinates of each calibration point of the calibration point group X are extended to three dimensions.

[0034]

[0035] The matrix H Ｎ＋１ is the first matrix on the right side of [Equation 4] and is a calibration matrix for performing calibration on the estimated fixation position under the fixation condition N+1. The matrix H Ｎ＋１ can be calculated based on the method for calculating the projective transformation matrix as shown in FIG. 4. Since the fixation condition corresponds to the explanatory variable of the estimated fixation position X Ｎ＋１ ’, the fixation condition corresponds to the explanatory variable of the matrix H Ｎ＋１ ’. Therefore, the fixation position P Ｎ＋１ ’ obtained as a result of performing calibration on the estimated fixation position P Ｎ＋１It can be obtained as shown in [Equation 6].

[0036]

[0037] FIG. 5 is a diagram for explaining the matrix H Ｎ＋１ The matrix H Ｎ＋１ converts X Ｎ＋１ ' to X, and P Ｎ＋１ ' to P Ｎ＋１ ''. By using the matrix H Ｎ＋１ as shown in FIG. 6, even when the fixation position is unknown and the fixation position is not a calibration point group, calibration can be performed for the estimated fixation position.

[0038] As another specific example, as shown in FIG. 7, consider the case where there is a sufficient correlation between the estimated fixation position X' corresponding to the calibration point group X and the face angle θ ｉ of each person. In this case, the fixation condition is a condition related to the face angle. In this case, assume that the estimated fixation position X ｉ ' in the situation where each person is fixing the calibration point group X under the fixation condition i (i = 1, 2,..., N) is known data. Here, the physical quantity of the fixation condition i is the face angle θ ｉ . The angle θ ｉ is, as a specific example, the angle of the roll angle with the vertical direction as the axis. The coordinate system of the angle θ ｉ is, as a specific example, the perpendicular direction of the plane corresponding to the calibration point group X, and the coordinate system in which the direction from the face to the calibration point group X is set to 0 degrees when the calibration point group X and the face face each other. In this case, assume that the estimated fixation position P Ｎ＋１ ' = (p Ｎ＋１ ', q Ｎ＋１ ', 1) in the situation where the subject is fixing an arbitrary unknown point group under the fixation condition N + 1 is unknown data. Also, assume that the estimated fixation position X Ｎ＋１ ' in the case where the subject is assumed to be fixing the calibration point group X under the fixation condition N + 1 is unknown data. Here, the physical quantity of the fixation condition N + 1 is the face angle θ Ｎ＋１This is the case. At this time, the gaze condition i and the estimated gaze position X are as follows: ｉ The relationship with ' is shown in Figure 7. Also, matrix H corresponding to the gaze condition N+1. Ｎ＋１ This is calculated in the same way as in the example above.

[0039] Figure 8 shows an example of the hardware configuration of the eye-line calibration device 100 according to this embodiment. The eye-line calibration device 100 consists of a computer. The eye-line calibration device 100 may consist of multiple computers.

[0040] As shown in this figure, the eye-line calibration device 100 is a computer equipped with hardware such as a processor 11, memory 12, auxiliary storage device 13, input / output interface (IF) 14, and communication device 15. These hardware components are appropriately connected via signal lines 19.

[0041] The processor 11 is an IC (Integrated Circuit) that performs arithmetic processing and controls the hardware of the computer. Specific examples of the processor 11 include a CPU (Central Processing Unit), a DSP (Digital Signal Processor), or a GPU (Graphics Processing Unit). The eye-line calibration device 100 may have multiple processors that replace the processor 11. The multiple processors share the role of the processor 11.

[0042] Memory 12 is typically a volatile storage device, specifically RAM (Random Access Memory). Memory 12 is also called main memory. Data stored in memory 12 is saved to auxiliary storage device 13 as needed.

[0043] The auxiliary storage device 13 is typically a non-volatile storage device, specifically a ROM (Read Only Memory), HDD (Hard Disk Drive), or flash memory. Data stored in the auxiliary storage device 13 is loaded into the memory 12 as needed. The memory 12 and the auxiliary storage device 13 may be configured as an integrated unit.

[0044] Input / Output IF14 is a port to which input and output devices are connected. A specific example of an input / output IF14 is a USB (Universal Serial Bus) terminal. Specific examples of input devices include a keyboard and mouse. Specific examples of output devices include a display.

[0045] The communication device 15 consists of a receiver and a transmitter. A specific example of the communication device 15 is a communication chip or a NIC (Network Interface Card).

[0046] Each part of the eye-line calibration device 100 may use the input / output IF 14 and the communication device 15 as appropriate when communicating with other devices.

[0047] The auxiliary storage device 13 stores the gaze calibration program. The gaze calibration program is a program that enables the computer to implement the functions of each part of the gaze calibration device 100. The gaze calibration program is loaded into memory 12 and executed by the processor 11.

[0048] Data used when executing the eye-tracking calibration program, and data obtained by executing the eye-tracking calibration program, are appropriately stored in a storage device. Each part of the eye-tracking calibration device 100 makes appropriate use of the storage device. The storage device consists, specifically, of memory 12, auxiliary storage device 13, registers in the processor 11, and at least one of the cache memory in the processor 11. Note that the terms data and information may have the same meaning. The storage device may be independent of the computer. The functions of memory 12 and auxiliary storage device 13 may be implemented by other storage devices.

[0049] The gaze calibration program may be recorded on a computer-readable non-volatile recording medium. Examples of non-volatile recording media include optical discs or flash memory. The gaze calibration program may also be provided as a program product.

[0050] ***Explanation of Operation*** The operation procedure of the eye-line calibration device 100 corresponds to the eye-line calibration method. The program that implements the operation of the eye-line calibration device 100 corresponds to the eye-line calibration program.

[0051] Figure 9 is a flowchart illustrating an example of how the gaze calibration device 100 calculates each expression parameter when calibration data is acquired. This operation will be explained using Figure 9.

[0052] (Step S101) The gaze information acquisition unit 120 receives person shooting data, acquires gaze information based on each of the received person shooting data, and outputs each of the acquired gaze information. The person shooting data is assumed to include multiple shooting data for one or more people.

[0053] (Step S102) The gaze information acquisition unit 120 acquires ambient information corresponding to each of the person shooting data and outputs each of the acquired ambient information.

[0054] (Step S103) The gaze information acquisition unit 120 calculates each estimated gaze position, etc., based on the outputted gaze information and each surrounding information, and outputs gaze-related information indicating each calculated estimated gaze position, etc.

[0055] (Step S104) The correlation determination unit 130 calculates the correlation between each coordinate component of the gaze position indicated by the gaze-related information and the gaze-related information, based on the output gaze-line information and gaze-related information, and generates a gaze-related information estimation model based on the calculated correlation.

[0056] (Step S105) The expression parameter generation unit 140 generates each expression parameter based on the output gaze information, gaze-related information, and gaze-related information estimation model.

[0057] Figure 10 is a flowchart illustrating an example of the operation in which the gaze calibration device 100 performs gaze calibration when real data related to the gaze detection target is acquired. This operation will be explained using Figure 10. Note that this operation is performed after the operation shown in Figure 9 is executed.

[0058] (Step S111) This step is the same as step S101. However, the person image data is information that indicates one image data related to the subject.

[0059] (Step S112) This step is the same as step S102.

[0060] (Step S113) This step is the same as step S103.

[0061] (Step S114) The virtual gaze position estimation unit 150 estimates the virtual gaze position based on the gaze information, the gaze-related information estimation model, and each representation parameter, and outputs the estimated virtual gaze position.

[0062] (Step S115) The calibration matrix calculation unit 160 acquires the correct gaze position relative to the virtual gaze position, calculates a calibration matrix based on the acquired correct gaze position and the output virtual gaze position, and outputs the calculated calibration matrix. The correct gaze position corresponds to a virtual calibration point cloud set based on the position of the sensor 20.

[0063] (Step S116) The gaze calibration execution unit 170 calculates the post-calibration gaze position by performing calibration on the estimated gaze position based on the output estimated gaze position and calibration matrix, and outputs the calculated post-calibration gaze position.

[0064] ***Explanation of the Effects of Embodiment 1*** In the prior art, calibration methods have been proposed that are applicable when the angular relationship between the sensor and the person, and the positional relationship between the person and the object of gaze change. However, the prior art had the problem that it could not properly perform gaze calibration in cases where the positional relationship between the sensor and the person changes relatively large, such as when the person moves. On the other hand, in this embodiment, one or more parameters that are correlated with gaze information are extracted, and a gaze-related information estimation model and a calibration matrix are calculated using each extracted parameter. Specifically, each parameter is a parameter corresponding to eye height or a parameter corresponding to eye position. Therefore, this embodiment has the following effects: Appropriate calibration can be performed at a relatively low cost when the positional relationship between the person and the sensor is unknown or changes relatively large. Since humans can identify parameters with a sufficiently high correlation, it is relatively easy to change gaze conditions or applications. Calibration can be performed even if parameters with a sufficiently high correlation are unknown in advance. In other words, even in environments or gaze conditions where values ​​with a sufficiently high correlation with gaze are not known in advance, it is possible to determine appropriate values ​​for gaze representation. In this embodiment, the purpose is to extract elements that have a sufficiently high correlation with the line of sight when acquiring calibration data before processing actual data.

[0065] Here, the parameters that have a sufficiently high correlation with the gaze condition are not constant; for example, if the distance between the person and the sensor changes, the parameters are related to distance, and if the orientation of the person's face changes, the parameters are related to face orientation. Also, if the data includes relatively low-accuracy data due to sensor performance and surrounding conditions, the appropriate parameters for representing gaze may change. On the other hand, in this embodiment, even when the gaze conditions are unknown, only the appropriate parameters are used according to the gaze conditions or the surrounding conditions of the person. A model capable of representing gaze is created using elements that have a sufficiently high correlation with gaze. Therefore, according to this embodiment, calibration can be performed in response to changes in the position and angle relationship between the sensor and the person. From this, it can be said that appropriate calibration can be performed for moving subjects. Furthermore, according to this embodiment, gaze estimation and calibration can be performed even in situations where eye information and other data cannot be properly acquired due to sensor performance or surrounding conditions. In other words, it becomes relatively easy to change parameters, such as excluding them from the calculation process when the acquisition accuracy of specific parameters is low. Furthermore, according to this embodiment, it can be said that calibration can be performed with relatively high accuracy while reducing the computational load.

[0066] ***Other Configurations*** <Modification 1> Figure 11 shows an example of the hardware configuration of the eye-tracking calibration device 100 according to this modification. The eye-tracking calibration device 100 includes a processing circuit 18 instead of a processor 11, a processor 11 and memory 12, a processor 11 and auxiliary storage device 13, or a processor 11, memory 12 and auxiliary storage device 13. The processing circuit 18 is hardware that realizes at least a part of each part of the eye-tracking calibration device 100. The processing circuit 18 may be dedicated hardware, or it may be a processor that executes a program stored in memory 12.

[0067] If the processing circuit 18 is dedicated hardware, the processing circuit 18 may, in specific examples, be a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination thereof. The gaze calibration device 100 may also include multiple processing circuits that substitute for the processing circuit 18. The multiple processing circuits share the role of the processing circuit 18.

[0068] In the eye-tracking calibration device 100, some functions may be implemented by dedicated hardware, while the remaining functions may be implemented by software or firmware.

[0069] The processing circuit 18 can be implemented, in specific examples, by hardware, software, firmware, or a combination thereof. The processor 11, memory 12, auxiliary storage device 13, and processing circuit 18 are collectively referred to as the "processing circuitry." In other words, the functions of each functional component of the eye-tracking calibration device 100 are implemented by the processing circuitry.

[0070] ***Other Embodiments*** Although Embodiment 1 has been described, multiple parts of this embodiment may be combined and implemented. Alternatively, this embodiment may be implemented partially. Furthermore, this embodiment may be modified in various ways as needed, and may be implemented as a whole or in any combination. Note that the embodiments described above are essentially preferred examples and are not intended to limit this disclosure, its applications, or the scope of use. Procedures described using flowcharts, etc., may be modified as appropriate.

[0071] 11 Processor, 12 Memory, 13 Auxiliary storage device, 14 Input / Output IF, 15 Communication device, 18 Processing circuit, 19 Signal line, 20 Sensor, 30 Equipment, 90 Eye-line calibration system, 100 Eye-line calibration device, 110 Person shooting unit, 120 Eye-line information acquisition unit, 130 Correlation determination unit, 140 Expression parameter generation unit, 150 Virtual gaze position estimation unit, 160 Calibration matrix calculation unit, 170 Eye-line calibration execution unit.

Claims

1. A gaze calibration device comprising: a correlation determination unit that calculates correlations between elements of gaze-related information, which indicates physical quantities related to the gaze position of each person at each time point, estimated based on images taken at each time point of each person who is gazing at a target gaze position defined by the arrangement of three or more calibration points, and elements corresponding to gaze information, which indicates physical quantities related to the gaze of each person shown in each image estimated based on the images corresponding to the gaze-related information, and selects one or more elements from the elements corresponding to the gaze information according to the height of the calculated correlations, and generates a gaze-related information estimation model, which is a model consisting of the selected elements and each expression parameter, and which estimates physical quantities related to the estimated gaze position of each person as estimated gaze-related information values ​​based on the gaze information corresponding to each person; and an expression parameter generation unit that calculates the value of each expression parameter based on the physical quantities corresponding to each gaze-related information estimated value estimated based on the gaze information corresponding to each image corresponding to the gaze-related information and the gaze-related information estimation model.

2. The gaze calibration device according to claim 1, wherein the expression parameter generation unit calculates the value of each expression parameter based on a regression analysis of each physical quantity corresponding to each gaze-related information estimate value estimated based on gaze-line information corresponding to each image corresponding to the gaze-related information and the gaze-related information estimation model, and each physical quantity corresponding to each image corresponding to the gaze-related information.

3. The gaze calibration device according to claim 1 or 2, wherein each physical quantity indicated by the gaze information corresponding to each image corresponding to the gaze-related information is a physical quantity that may be correlated with each element included in the gaze-related information.

4. The gaze calibration device according to any one of claims 1 to 3, wherein the gaze information corresponding to each image corresponding to the gaze-related information includes information indicating the physical relationship between the sensor that captured each image corresponding to the gaze-related information and the person shown in each image corresponding to the gaze-related information.

5. The gaze calibration device according to any one of claims 1 to 4, wherein the gaze-related information includes, as each estimated gaze position, information indicating the gaze position of a person shown in each image, estimated based on gaze information corresponding to each image corresponding to the gaze-related information.

6. Each estimated gaze position indicated by the gaze-related information is defined by three or more estimated gaze points, the number of estimated gaze points defining each estimated gaze position indicated by the gaze-related information is the same as the number of calibration points defining the target gaze position, each estimated gaze point defining each estimated gaze position indicated by the gaze-related information corresponds to each calibration point, each element included in the gaze-related information includes each coordinate component of each estimated gaze point, each element corresponding to the gaze information is a relational expression relating to parameters corresponding to each physical quantity indicated by the gaze information, and the parameters corresponding to each physical quantity indicated by the gaze information are explanatory variables of the gaze-related information estimation model, as described in claim 5.

7. The gaze calibration device according to claim 5 or 6, wherein each estimated gaze position indicated by the gaze-related information is further generated based on ambient information indicating the surrounding conditions of the person corresponding to each estimated gaze position at the time the image corresponding to each estimated gaze position was taken.

8. The gaze calibration device further comprises: a virtual gaze position estimation unit that estimates the subject's estimated gaze position as a virtual gaze position, assuming the subject is gazing at the target gaze position, based on the acquired gaze information, the gaze-related information estimation model, and the calculated values ​​of each representation parameter, based on the acquired gaze information; and a calibration matrix calculation unit that calculates a calibration matrix for performing calibration on the target gaze position, based on the target gaze position and the virtual gaze position, wherein the target gaze position is defined by three or more estimated gaze points, the number of estimated gaze points defining the target gaze position is the same as the number of calibration points defining the target gaze position, and each estimated gaze point defining the target gaze position corresponds to each calibration point.

9. The gaze calibration device according to claim 8, further comprising: a gaze information acquisition unit that acquires the acquired gaze information based on an image of the subject and estimates the gaze position of the calibration target based on the acquired gaze information; and a gaze calibration execution unit that performs calibration for the gaze position of the calibration target using the calibration matrix.

10. A gaze calibration method comprising: a computer calculating correlations between elements of gaze-related information, which represents physical quantities related to the gaze position of each person at each point in time, estimated based on images taken at each point in time of each person who is gazing at a target gaze position defined by the arrangement of three or more calibration points, and elements of gaze information, which represents physical quantities related to the gaze of each person shown in each image estimated based on the images corresponding to the gaze-related information; selecting one or more elements from the elements corresponding to the gaze information according to the calculated correlations; generating a gaze-related information estimation model, which is a model consisting of the selected elements and each expression parameter, and which estimates physical quantities related to the estimated gaze position of each person as each gaze-related information estimate based on the gaze information corresponding to each person; and calculating the value of each expression parameter based on each physical quantity corresponding to each gaze-related information estimate estimated based on the gaze information corresponding to each image corresponding to the gaze-related information and the gaze-related information estimation model; and calculating the value of each expression parameter based on each physical quantity corresponding to each expression parameter based on the gaze information corresponding to each image corresponding to the gaze-related information.

11. A gaze calibration program that causes a computer gaze calibration device to execute a correlation determination process that generates a gaze-related information estimation model which is a model consisting of the selected elements and expression parameters, and which estimates each physical quantity related to the gaze position of each person as an estimated gaze-related information value based on the gaze information of each person; and an expression parameter generation process which calculates the value of each expression parameter based on the physical quantity related to the gaze of each person shown in each image estimated based on the image corresponding to the gaze-related information. The program calculates the correlation between each element of gaze-related information which is a model consisting of each element of gaze-related information which is estimated based on the gaze information of each person, and selects one or more elements from the elements corresponding to the expression parameters according to the height of the calculated correlations, and generates a gaze-related information estimation model which is a model consisting of each selected element and each expression parameter, and which estimates each physical quantity related to the estimated gaze position of each person as an estimated gaze-related information value based on the gaze information of each person; and an expression parameter generation process which calculates the value of each expression parameter based on the physical quantity corresponding to each expression parameter based on the physical quantity corresponding to each image corresponding to the gaze-related information.

Images