Computer-implemented method for reconstructing a circular shape of a circular object in three-dimensional space and devices therefor

Abstract

The invention relates to a computer-implemented method for reconstructing a circular shape of an at least substantially circular object in three-dimensional space from at least two at least substantially mutually synchronized partial images which comprise at least substantially elliptical two-dimensional images of the object and which have been captured by respective different optical sensors from respective different perspectives or which have been captured by a single optical sensor from respective different sensor perspectives, comprising at least the method steps of: a) extracting an outline contour of the object from each of the partial images, the outline contours being at least substantially elliptical, and b) determining an ellipse center point for each of the extracted at least substantially elliptical outline contours. It is proposed that, in at least one method step, in each case a plurality of mutually spaced discrete ellipse contour points are placed in a distributed arrangement on the outline contours extracted from the partial images and a subsequent computer-implemented back-projection of the two-dimensional images of the object into three-dimensional space for reconstruction of the circular shape in three-dimensional space is performed at least using the determined ellipse center points and using the ellipse contour points placed in a distributed arrangement.

Description

[0001] R.415917

[0002] - 1 -

[0003] Description

[0004] Computer-implemented method for reconstructing the circular shape of a circular object in three-dimensional space and devices for this purpose.

[0005] State of the art

[0006] Determining 3D information from 2D image data is a significant challenge in the field of computer vision. One specific application is eye tracking, where, for example, a stereoscopic image (comprising two stereoscopic sub-images) uses the ellipses within it as projections of the approximately circular pupil or iris to obtain information about the current gaze direction. Depending on the application, information such as pupil position and / or pupil diameter may be of interest, either in addition to or instead of gaze direction. For this purpose, the complete computational 3D reconstruction of the pupil is particularly attractive because it allows for the simultaneous calculation of its position, orientation, and diameter.

[0007] In this context, the determination of 3D information is typically divided into two essential steps. In a first step, the ellipses or discrete elliptical points in the stereoscopic image that result from the projections of a circular object onto the two image planes must be detected. In the case of eye tracking, this step is also referred to as pupil detection, for example. In a second step, the ellipses or elliptical points can be used to computationally reconstruct the circular object in 3D space based on the detected ellipses or elliptical points. The present invention addresses this second step. R.415917

[0008] - 2 -

[0009] There are already various approaches to determining 3D information for circular objects in three dimensions. See, for example: “Three-dimensional location estimation of circular features for machine vision” [IEEE Transactions on Robotics and Automation: R. Safaee-Rad; I. Tchoukanov; KC Smith; B. Benhabib], or “Conics-Based Stereo, Motion Estimation, and Pose Determination” [International Journal of Computer Vision: SONG DE MA] or “Calibration-free eye tracking by reconstruction of the pupil ellipse in 3D space” [Proceedings of the 2008 symposium on Eye tracking research & applications: S. Kohlbecher et al.]. Thus, there are various closed-form analytical solutions as well as iterative solution methods.A disadvantage of the known methods can be, in particular, that with many of these methods the radius / diameter of the circular object must be known in advance, that some of these methods have ambiguities in the solutions, that some of these methods very easily produce unusable results and / or converge poorly with real (error-prone) input variables, that some of these methods are very computationally intensive and / or that some of these methods do not readily allow for easy influence on the trade-off between accuracy and computation time.

[0010] Disclosure of the invention

[0011] The invention is based on a computer-implemented method, preferably a computer-implemented eye-tracking method or a computer-implemented object and / or self-localization method, e.g., of an AR headset or a VR headset or another camera system, for reconstructing the circular shape of an at least substantially circular object, in particular an at least substantially circular pupil disc or a circular marking applied to any body, in three-dimensional space, from at least two at least substantially elliptical two-dimensional images of the object, in particular stereoscopic, at least substantially synchronized partial images, which are each captured from different perspectives by different optical sensors, in particular camera sensors, preferably spatially calibrated to each other, or which are from R.415917

[0012] - 3 - a single optical sensor, in particular a camera sensor, from different, preferably spatially calibrated, sensor perspectives, in particular camera perspectives, comprising at least the following process steps: a) extracting an outline contour of the object from each of the partial images, wherein the outline contours are at least substantially elliptical, and b) determining an ellipse center point for each of the extracted at least substantially elliptical outline contours.

[0013] It is proposed that, in at least one process step, several spaced-apart discrete ellipse contour points are distributed across the extracted outline contours of the partial images, and that a subsequent computer-implemented backprojection of the two-dimensional images of the object into three-dimensional space is performed to reconstruct the circular shape in three-dimensional space, at least using the determined ellipse centers and the distributed ellipse contour points. This advantageously allows for a very efficient reconstruction method. A compromise between accuracy and computation time can be easily set, particularly by selecting a specific number of ellipse contour points.Furthermore, a reconstruction method can be advantageously obtained in which the radius and / or diameter of the circle to be reconstructed does not need to be known in advance, but can be calculated. The same advantageously applies to the center coordinates and the normal vector of the circular object in three-dimensional space.

[0014] The computer-implemented method can be carried out, in particular, by / on a computing unit, by / on a computing infrastructure, or by / on a control unit integrated into a device, in particular one comprising optical sensor(s), especially camera sensor(s). A "computing unit" is understood to mean, in particular, a unit with an information input, information processing, and information output. The computing unit can be spatially separated from a device comprising optical sensor(s), in particular camera sensor(s), or connected to the device comprising optical sensor(s), in particular camera sensor(s). Advantageously, the computing unit comprises at least one processor, an R.415917

[0015] - 4 -

[0016] The computing unit comprises storage, input and output devices, other electrical components, an operating program, rule routines, control routines, and / or calculation routines. The components of the computing unit can be arranged on a common circuit board and / or in a common housing. The computing infrastructure, however, can be distributed. For example, the computing infrastructure could be designed as a cloud computing infrastructure. In an eye-tracking method, the circular object to be reconstructed can be a person's pupil (pupil contour) or iris (iris contour). In an object localization method, the circular object to be reconstructed can be a circular marker or a circular target, which is applied to or placed on an object to be localized.In a self-localization method, the circular object to be reconstructed can be a circular marker or target placed in an environment where self-localization is to take place, for example, on a wall forming part of the environment. The circular object itself can be a physical object with a circular contour or a circular marking or drawing on a (not necessarily circular) body. The reconstruction using the described computer-implemented method should then preferably determine the actual circular shape of the object, i.e., the circular contour of the body or the marking on the body, and preferably also its orientation in three-dimensional space.

[0017] In the illustrations, the actually circular object appears elliptical, particularly due to an angled perspective view. "Essentially elliptical" is meant to mean, in particular, that a shape should also be considered an ellipse if it does not exactly meet the mathematical definition of a perfect ellipse, but visually resembles one. Specifically, a shape that exhibits deviations from the ideal elliptical shape, but which are so slight that the shape can still be considered elliptical, should also be understood as essentially elliptical. For example, in the case of a pupil, everything that falls within the range of individual differences in the pupils of different healthy individuals (R.415917) should be considered essentially elliptical.

[0018] - 5 - can occur as circular or, when viewed from an angled perspective, elliptical. A "stereoscopic partial image" is, in particular, one of two images that are used together to create a stereoscopic effect. In particular, the partial images used in the method according to the invention are synchronized to each other in such a way that the object with the circular shape to be reconstructed does not move, or only moves negligibly, between the recording of the two partial images. Preferably, when using two optical sensors / camera sensors, the partial images with the two (different) elliptical representations of the object are recorded simultaneously, preferably completely synchronously.Preferably, when using a moving optical sensor / camera sensor, the partial images with the two (different) elliptical representations of the object are generated so rapidly in succession that the object cannot have moved in the meantime, or only minimally, i.e., within a few milliseconds. The optical sensor(s) is / are preferably designed as (conventional) camera sensors (black and white, color, infrared, etc.). Alternatively, the optical sensors can also be designed as other optical systems capable of capturing images of an object, for example, as laser feedback interferometry (LFI) sensors. LFI sensors can scan an object, e.g., an eye, using an infrared laser beam and record the reflected signal using a photodiode, from which images of the object can then be generated.

[0019] Preferably, the two different optical sensors and / or the two different sensor perspectives of the single optical sensor are spatially calibrated relative to each other such that at least the relative positions and / or the relative orientations of the two different optical sensors relative to each other / the partial image capture settings of the single optical sensor relative to each other are known. It is also conceivable that the absolute spatial positions and / or orientations are known in each case. In particular, if the optical sensors are configured as camera sensors, the extrinsic camera parameters of the camera sensor(s) are known exactly through the spatial calibration. The orientations and / or positions relative to each other can, for example, be mathematically described by a rotation matrix and a translation vector. Furthermore, preferably R.415917

[0020] - 6 - Intrinsic sensor parameters of the optical sensor are also known, in particular through calibration, e.g., camera calibration. Intrinsic sensor parameters can be a focal length, a sensor orientation in three-dimensional space, a position of an image plane in three-dimensional space, a coordinate of an associated principal point (i.e., in particular a point defining the image center of an image, at which an optical axis of the respective optical sensor intersects an image plane of the respective optical sensor), and / or a pixel size of a camera or a photodiode. In particular, the optical-physical model of a pinhole camera can be used as the basis for a computational 3D reconstruction in the described method.The determination of the intrinsic sensor parameters and / or the extrinsic sensor parameters, in particular the camera calibration, is preferably carried out in advance of carrying out the method according to the invention.

[0021] The determination / extraction of the object's outline from the partial images is preferably performed using known outline recognition algorithms (e.g., Canny, Sobel, Prewitt, Laplace, etc.). The ellipse's center point is preferably also determined using known ellipse center determination algorithms (ellipse fitting, Fitzgibbon, Ransac, etc.) based on the extracted ellipse's outline. Ellipse contour points are preferably (zero-dimensional) points arranged on the (one-dimensional) extracted ellipse's outline in the two-dimensional imaging plane of the optical sensor, which lies in three-dimensional space. The ellipse contour points are preferably countable. The ellipse contour points are preferably separated from each other. The ellipse contour points are preferably independent of each other.In computer-implemented backprojection, information is preferably transferred back into three-dimensional space from two-dimensional images (or projections) of an object. This technique is used in particular to reconstruct the three-dimensional structure of an object (here, its circular shape) from multiple two-dimensional views or projections of the object. Specifically, computer-implemented backprojection is performed independently of contour lines. In particular, instead of lines or uncountable numbers of R.415917.

[0022] - 7 -

[0023] The limited, countable set of ellipse contour points and the ellipse centers are used for the computer-implemented backprojection. No further ellipse parameters besides the ellipse contour points and the ellipse centers are required for the procedure to work.

[0024] If the ellipse contour points are evenly spaced and arranged with the same direction of rotation on the extracted outline contours of each of the sub-images / each of the outline contours extracted from the sub-images, and / or if, alternatively or additionally, the ellipse contour points are arranged in equal numbers on each of the extracted outline contours of the sub-images, a particularly good, reliable, and / or efficient reconstruction of the desired circular shape can be advantageously achieved. The numerical convergence of a solution method used can be positively influenced. Furthermore, the uniqueness of a resulting reconstruction solution can be advantageously ensured.Furthermore, this is advantageous because it makes it easier to ensure that the back-projected points along the desired circumference of the circle are largely equidistant and alternately arranged with respect to their assignment to the back-projected ellipse points. The additional conditions mentioned in this paragraph do not necessarily have to be met for the proposed method to be generally successful. System equations could also be formulated in such a way that they are compatible with other point distributions, but this could be more cumbersome and / or less efficient and would not offer any other advantages.

[0025] Furthermore, if the ellipse contour points on the two outline contours, which were determined from the partial images of the single optical sensor or the two different optical sensors, are arranged alternately offset from each other, in particular so that the ellipse contour points of the two outline contours alternate, preferably as uniformly as possible, when the outline contours are superimposed along a traversal direction around the outline contours, and / or so that all points that can be assigned to each of the ellipse contour points of the two outline contours on the reconstructed circular shape along a traversal direction around the outline contour of the reconstructed circular shape R.415917

[0026] - 8 - By alternating the points, preferably as evenly as possible, the advantages listed in the preceding paragraph can also be achieved and / or further enhanced. Each back-projected point of one of the elliptical outline contours is then followed by a back-projected point of the other elliptical outline contour. The suitable / optimal distribution depends on the respective arrangements of the camera sensors / camera perspectives. The transmission direction, for example, runs with a mathematically positive rotation direction relative to the respective z-axis of the partial images. However, a reverse rotation direction is also conceivable.

[0027] Furthermore, it is proposed that, in addition to the computer-implemented backprojection of the two-dimensional images of the object into three-dimensional space for the reconstruction of the circular shape in three-dimensional space, a system of equations comprising a plurality of system equations is solved by means of a computer-implemented numerical solution method, in particular a Gauss-Newton solution method, which solves a minimization problem, wherein at least a majority of all, preferably all, system equations include at least one of the ellipse centers and / or at least one of the ellipse contour points and / or wherein at least a majority of all, preferably each, of the ellipse points, preferably of the ellipse contour points and / or of the ellipse centers, are covered by a plurality of system equations.This, and in particular the formulation as a numerically solvable minimization problem that always determines the best solution in terms of the least squared error, advantageously allows for a computer-implemented backprojection that is particularly robust against disturbances in the input variables. Furthermore, the proposed method advantageously avoids ambiguities in the solutions. Alternatively, the application of other computer-implemented numerical solution methods known to those skilled in the art, in addition to the Gauss-Newton method, is also conceivable.

[0028] An example of a system of equations suitable for the Gauss-Newton solution method, with system equations fi to fen+sm+4, is shown below. Here, f represents the system equation, and its index denotes the "numbering" of the system equation, where m represents the number of ellipse contour points on one of the outline contours and n represents the number of ellipse contour points on the other of the outline contours. Advantageously, as mentioned previously, n = m

[0029] The first partial derivatives of the system equations are entered into a Jacobian matrix, where p represents coordinates in three-dimensional space and the index C stands for circle.

[0030] Furthermore, it applies and the termination criterion

[0031] Correction step size || p z c || < £ ; Number of iterations < N max

[0032] If required, a damping strategy can be implemented for adaptive control of the correction step size. Often, to save computation time, it can be advantageous to use the calculated solutions as new starting values ​​for the next iteration of the procedure. This can significantly reduce the number of required iterations. The ability to select the number of points per ellipse used for the calculations allows for a conveniently simple adjustment of the trade-off between computation time and accuracy.

[0033] The following parameters are particularly needed to solve the problem:

[0034] Normal vector n of the circular object R.415917

[0035] - 10 - nc

[0036] Center point of the circular object. The index 0 stands for "center".

[0037] PC,o x c,o>yc,o> z c,o)

[0038] Backprojections of the ellipse contour points k of one of the outline contours onto the circular object

[0039] PC, KX c ,k>yc,k>Zc,kk = 1 ...n

[0040] Backprojections of the ellipse contour points q of the other outline contours onto the circular object

[0041] PC, q (xc, q >yc, q >zc, q q = n + l ... (n + m)

[0042] From these calculated points, the radius and diameter of the circular object can subsequently be calculated.

[0043] In this context, it is proposed that a subgroup, in particular a first subgroup, of system equations (6 to fc) of the system of equations requires that normalized center-direction vectors, each extending from a sensor center / camera center of one of the two optical sensors or one of the two sensor perspectives of the individual optical sensor to the center of the ellipse of the respective image planes of this optical sensor (known in particular through calibration), correspond to normalized center-direction vectors from the respective center of the ellipse to a circle center of the circular shape to be reconstructed. This advantageously allows the position of the circle center to be back-projected.

[0044] This condition can be expressed mathematically as below. R.415917

[0045] - 11 -

[0046] For the first sub-image with the index E1,0 for "center of ellipse #1", the index C1 for "coordinates of the camera center (sensor center) of camera (sensor) #1" and the index C,0 for "center of the reconstructed circle":

[0047] For the second sub-image with the index E2,0 for "center of ellipse #2" and the index C2 for "coordinates of the camera center (sensor center) of camera (sensor) #2":

[0048] The vector representation, transformed into individual equations, yields the first six system equations fi to fe (three per ellipse center) of the system of equations:

[0049] Furthermore, it is proposed that one, in particular a second, subgroup of system equations of the system of equations requires that normalized contour point- R.415917

[0050] - 12 -

[0051] Direction vectors, each originating from a sensor center / camera center of one of the two optical sensors or one of the two sensor perspectives of the individual optical sensor (known, in particular, through calibration), to one of the elliptical contour points of the respective image planes of this optical sensor, correspond to normalized contour point direction vectors from the respective elliptical contour point to a corresponding circle contour point on an outline contour of the circular shape to be reconstructed. This allows the positions of the circle contour points to be advantageously back-projected.

[0052] This condition can be expressed mathematically as below.

[0053] For the first sub-image with the index E1,i for "ellipse contour point #i of ellipse #1" and the index C,k for "circle contour point #k":

[0054] For the second sub-image with the index E2,j for "ellipse contour point #j of the ellipse"

[0055] #2" and the index C,q for "circle contour point #q":

[0056] Transformed into individual equations, this yields the further 3(n+m) system equations fv to f3(n+m)+6 (three per ellipse contour point) of the system of equations: R.415917

[0057] - 13 - f 3n+6 ^c,^ yc,k^ Cik )

[0058] Furthermore, it is proposed that a subgroup, in particular a third subgroup, of the system equations requires that all circle radius vectors, or all of an arbitrary or previously defined selection of all circle radius vectors, in particular those pointing from a common center of the circle to be reconstructed to a circle contour point on an outline of the circle to be reconstructed, are perpendicular to a mean normal vector of the circle to be reconstructed. This mean normal vector is formed, for example, by calculating several cross products of two circle radius vectors, preferably consecutive ones, in a direction of travel around an outline of the circle to be reconstructed, followed by a mean calculation. This advantageously allows a planar circle to be back-projected in three dimensions.

[0059] This condition can be met using the mean normal vector n. Cn mathematically expressed as below.

[0060] Using the back-projected points of the first partial image:

[0061] (Pc,t - Pc,o) ■ rä Cn = 0

[0062] Using the back-projected points of the second sub-image: R.415917

[0063] - 14 -

[0064] (Pc,q - Pc,o) ■ n Cn = 0

[0065] The individual normal vectors are each derived from the corresponding

[0066] Cross products and

[0067] Here, k* and q* represent different points on the circle's outline (following points in the direction of travel), distinct from points k and q.

[0068] The mean normal vector can then be calculated by averaging the individual normal vectors, for example according to and standardization then takes place with

[0069] Using a substitution for the individual terms in the equations above to calculate the individual normal vectors n Cni and n Cn; according to the scheme with and R.415917

[0070] - 15 - Finally, further system equations can be formulated compactly for the conditions described.

[0071] The number of additional system equations can be limited in this case by a clever selection of equations. For example, by selecting only enough of all possible combinations of radius vectors so that each radius vector appears at least once in the system equations. In the example below, this results in the equations f3(n+m>+7) to f4( n +m)+6 of the system of equations: f 4n+3m+7 '•= ​​0 = C lx n Cn ,x + c ly^Cn,y + clz^Cn,zf 4n+4m+6 0 ^mx^Cn,x + ^my^Cn,y + i'z^Cn,z

[0072] Furthermore, it is proposed that a subgroup, in particular a fourth, of the system equations requires that all circle contour points, or all of an arbitrary or previously determined selection of all circle contour points, in particular those belonging to each of the ellipse contour points, along an outline contour of the circular shape to be reconstructed, have an identical distance to a common center of the circular shape to be reconstructed. This advantageously ensures that the backprojection actually has a circular shape.

[0073] This condition can be expressed mathematically as below. R.415917

[0074] - 16 -

[0075] The number of additional system equations can also be limited in this case by a clever selection of equations.

[0076] For example, the following restriction selection could be made for m = n = 4:

[0077] Pc,k of E1 : 1,2; 2,3; 3,4 p c q from E2: 1, 2; 2, 3; 3, 4

[0078] PC, K & PC, q from E1 & E2: 1,1; 2,2; 3,3; 4,4

[0079] Transforming this into individual equations yields a further number of system equations f4n+4m+7 to fen+5m+4 of the system of equations: R.415917

[0080] - 17 -

[0081] As an alternative to the aforementioned subgroup of system equations, also referred to as the third subgroup of system equations, this (third) subgroup of system equations of the system of equations can also require that all circle normal vectors of the circular shape to be reconstructed, or all of an arbitrary or previously defined selection of all circle normal vectors, which are formed by calculating a cross product of each pair of circle radius vectors, preferably consecutive ones lying one after the other in a direction of travel around an outline contour of the circular shape to be reconstructed, are aligned in the same direction. This allows a planar circle to be advantageously back-projected in three dimensions.

[0082] This condition can be expressed mathematically as below.

[0083] Using a substitution for the individual terms according to the scheme with R.415917

[0084] - 18 - and For the conditions described, further system equations can be formulated compactly.

[0085] The number of additional system equations can also be limited in this case by a clever selection of equations.

[0086] For example, the following restriction selection could be made for m = n = 4:

[0087] PC, FC & PC, q von E1 & E2: 1 x2k = 1 x2q, 2x3k = 2x3q, 3x4k = 3x4 q

[0088] Transformed into individual equations, this yields a further alternative

[0089] Condition a further number of system equations fen+sm+s to fan+sm+i of the

[0090] System of equations: R.415917

[0091] - 19 -

[0092] It is particularly conceivable that one of the two conditions for this (third) subgroup is taken into account, or that the system equations of the two different conditions for this (third) subgroup are mixed, or that even all system equations of both possible conditions for this (third) subgroup are used together.

[0093] Furthermore, a computing unit, computing infrastructure, or a system-integrated

[0094] Control unit, e.g. AR (Augmented Reality) or VR (Virtual Reality) headset - R.415917

[0095] - 20 -

[0096] A control unit comprising at least one processor and at least one data storage device, which includes at least program instructions for carrying out the aforementioned computer-implemented method, is proposed. This advantageously allows for a very efficient circular shape reconstruction.

[0097] Furthermore, an eye-tracking device is proposed, comprising a computing unit, a system-integrated control unit, or a network connection to the computing infrastructure, and a single optical sensor or at least two optical sensors that can be oriented or aligned differently to one eye from different perspectives. This advantageously enables reliable and efficient eye tracking, particularly for smart glasses or other electronic devices. The eye-tracking device is specifically designed to determine and output the position and / or gaze direction of a person's eye, especially that of a user of a device equipped with the eye-tracking device. "Designed" is understood to mean specifically programmed, designed, and / or equipped.The fact that an object is intended for a specific function should be understood in particular to mean that the object fulfills and / or performs this specific function in at least one application and / or operating state.

[0098] Furthermore, an object and / or self-localization device is proposed, comprising the computing unit, the system-integrated control unit, or a network connection to the computing infrastructure, and a single optical sensor or at least two optical sensors that can be oriented or aligned from different perspectives to an object and / or a marker. This advantageously enables reliable and efficient localization of objects in space or of a device in space that incorporates the self-localization device, e.g., for smart glasses. The object localization device is specifically designed to determine and output the position and / or orientation of an object, either absolutely or relative to a device containing the object localization device, such as smart glasses. The self-localization device is specifically designed for this purpose (R.415917).

[0099] - 21 - designed to determine and output the position and / or orientation of a device which has the self-localization device, such as data glasses or other electronic devices, absolutely or relative to a coordinate system.

[0100] The inventive method, the inventive computing unit, the inventive computing infrastructure, the inventive system-integrated control unit, the inventive eye-tracking device, the inventive object and / or self-localization device, and / or the inventive smart glasses are not to be limited to the application and embodiment described above. In particular, the inventive method, the inventive computing unit, the inventive computing infrastructure, the inventive system-integrated control unit, the inventive eye-tracking device, the inventive object and / or self-localization device, and / or the inventive smart glasses may, to achieve a functionality described herein, comprise a different number of individual elements, components, units, and process steps than specified herein.Furthermore, values ​​within the specified ranges of values ​​in this disclosure shall also be considered disclosed and freely usable.

[0101] drawing

[0102] Further advantages will become apparent from the following description of the drawing. The drawing illustrates an embodiment of the invention. The drawing, the description, and the claims contain numerous features in combination. A person skilled in the art will expediently consider the features individually and combine them into meaningful further combinations.

[0103] It shows: R.415917

[0104] - 22 -

[0105] Fig. 1 shows, by way of example, data glasses with an eye tracking device and / or with an object and / or self-localization device, each based on the same computer-implemented method,

[0106] Fig. 2 shows a schematic flowchart of the computer-implemented procedure,

[0107] Fig. 3 shows two exemplary partial images of the same eye, each with a different perspective and captured by two different optical sensors of the eye tracking device and / or the object and / or self-localization device, with elliptical outline contours of a pupil forming a circular object and with elliptical centers and elliptical contour points of the outline contours.

[0108] Fig. 4 shows an exemplary schematic arrangement of the optical sensors in three-dimensional space with the circular object to be reconstructed and exemplary optical axes of the two optical sensors marked by dashed lines.

[0109] Fig. 5a shows an enlarged schematic representation of a sensor area of ​​one of the optical sensors, which forms a first of the partial images with the associated elliptical outline contour from a first perspective and with the optical axis of this optical sensor shown again as a dashed line.

[0110] Fig. 5b shows an enlarged schematic representation of a sensor area of ​​the other optical sensor, which forms a second of the partial images, with the associated elliptical outline contour from a second perspective and with the optical axis of this other optical sensor again shown as a dashed line.

[0111] Fig. 6 shows a schematic representation of the circular shape of the reconstructed circular object in three-dimensional space, with an associated center point and corresponding contour points. R.415917

[0112] - 23 -

[0113] Description of the exemplary embodiment

[0114] Figure 1 shows an example of a pair of smart glasses 60. Alternatively, instead of smart glasses 60, another electronic device with an eye-tracking function and / or with an optical object localization function and / or with an optical self-localization function could also be shown. The smart glasses 60 are designed as an example AR headset, but could also be a VR headset. AR headsets and their operation are known to those skilled in the art. Therefore, they will not be discussed in more detail here. The smart glasses 60 have a head mounting unit 62. The head mounting unit 62 is designed as a pair of temples and a nose pad. However, alternative head mounting units 62 known from the prior art are of course also conceivable. The smart glasses 60 have an eye-tracking device 56 for carrying out a computer-implemented method described herein.Alternatively or additionally, the data glasses 60 could also include an object and / or self-localization device 58, which is intended for carrying out the computer-implemented procedure described herein.

[0115] The eye-tracking device 56 and / or the object and / or self-localization device 58 includes, by way of example, a first optical sensor 20. The eye-tracking device 56 and / or the object and / or self-localization device 58 also includes, by way of example, a second optical sensor 22. The two optical sensors 20, 22 are oriented from different perspectives toward an area of ​​the data glasses 60, which is intended for positioning one eye 64 of a user. Alternatively, the eye-tracking device 56 and / or the object and / or self-localization device 58 could also have only a single optical sensor 20, which is intended for very rapid position and / or perspective changes. In addition, the sensors 20, 22 of the object and / or self-localization device 58 could also be oriented from different perspectives toward objects other than one eye 64, e.g., toward external objects and / or markers.Optical sensors 20 and 22 are designed as camera sensors by way of example. Other types / Type R.415917.

[0116] - 24 - However, the use of optical sensors with an image acquisition capacity is also conceivable. The eye-tracking device 56 and / or the object and / or self-localization device 58 has a computing unit 48. The computing unit 48 is integrated into the data glasses 60. Alternatively, the computing unit 48 could also be externalized, e.g., in a smartphone connected to the data glasses 60, as schematically shown in Fig. 1, or in an external computing infrastructure. The computing unit 48 of the data glasses 60 is designed, by way of example, as a system-integrated control unit (here: AR headset control unit). The computing unit 48 has a processor 50. The computing unit 48 has a data memory 54. The data memory 54 contains program instructions for carrying out the computer-implemented procedure. The processor 50 executes the program instructions for carrying out the computer-implemented procedure.

[0117] Figure 2 shows a schematic flowchart of the computer-implemented method. The reference symbols used below are illustrated in Figures 3 to 6, to which reference is hereby made. The computer-implemented method is, by way of example, a computer-implemented eye-tracking method. Alternatively, however, it could also be a computer-implemented object and / or self-localization method based on the same fundamental principle or the same computational methods. The method is for the reconstruction of a circular shape 10 (see Fig. 6) of an at least substantially circular object 66 (see Fig. 4), in particular an at least substantially circular pupil disk or a circular marking applied to any body, in three-dimensional space, from at least two at least substantially elliptical two-dimensional images 12, 14 (see Fig. 3 or Figs. 4).5a and 5b) of object 66, stereoscopic partial images 16, 18, at least substantially synchronized to each other (see Fig. 3), are provided. The partial images 16, 18 used in the method are each acquired from different perspectives by the two spatially calibrated optical sensors 20, 22. Alternatively, the partial images 16, 18 used in the method could also have been acquired from different spatially calibrated sensor perspectives by the single optical sensor 20, 22. R.415917.

[0118] In at least one process step 90, the two images 12, 14 of the same object 66 are recorded from different perspectives. This is illustrated in Figure 3 for one eye 64. In at least one process step 100, an outline contour 24, 26 (see Fig. 3) of the object 66 is extracted from each of the partial images 16, 18 using the processing unit 48. Due to the non-frontal perspective, the two outline contours 24, 26 are elliptical. In at least one process step 110, an ellipse center 28, 30 (see Fig. 3) is determined for each previously extracted elliptical outline contour 24, 26 using the processing unit 48. In at least a first sub-step 121 of a further process step 120, several mutually spaced discrete ellipse contour points 32, 34, 36, 38 (see Fig. 3 and Figs. 5a and 5b) are arranged distributed on the extracted outline contours 24, 26 of the partial images 16, 18.The elliptical contour points 32, 34, 36, 38 are spaced evenly and arranged with the same direction of rotation on the extracted outline contours 24, 26. The elliptical contour points 32, 34, 36, 38 are also arranged in equal numbers on each of the extracted outline contours 24, 26. Furthermore, the elliptical contour points 32, 34, 36, 38 are arranged alternately offset from each other on the two outline contours 24, 26, which were determined from the partial images 16, 18 of the single optical sensor 20, 22. This means that the ellipse contour points 32, 34, 36, 38 are arranged on the outline contours 24, 26 such that the ellipse contour points 32, 34, 36, 38 of the two outline contours 24, 26 alternate as evenly as possible when the outline contours 24, 26 are superimposed along a transit direction 40 around the outline contours 24, 26 (see Fig. 3).Furthermore, the ellipse contour points 32, 34, 36, 38 are arranged on the outline contours 24, 26 such that all points (circle contour points 32', 34', 36', 38', cf. Fig. 6) that can be assigned to each of the ellipse contour points 32, 34, 36, 38 of the two outline contours 24, 26 alternate as evenly as possible on the reconstructed circular shape 10 along a traverse direction 40 (cf. Fig. 6) that runs around an outline contour 52 of the reconstructed circular shape 10. R.415917.

[0119] - 26 -

[0120] In at least a second sub-step 122 of process step 120, a computer-implemented back-projection of the two-dimensional images 12, 14 of object 66 into three-dimensional space is carried out for the reconstruction of the circular shape 10 in three-dimensional space. The computer-implemented back-projection is carried out at least by means of the ellipse centers 28, 30 determined in the preceding process step 110 and by means of the ellipse contour points 32, 34, 36, 38 distributed in the preceding sub-step 121. For the computer-implemented back-projection of the two-dimensional images 12, 14 of object 66 into three-dimensional space for the reconstruction of the circular shape 10 in three-dimensional space, a system of equations comprising a multitude of system equations is solved by means of a computer-implemented numerical solution method that solves a minimization problem, e.g.The system equations were solved using a Gauss-Newton solution method. These equations have already been presented and described above. Repetition will be avoided here. Therefore, a detailed description is omitted, and the preceding text provides further information regarding the system equations and the solution method.

[0121] All system equations include at least one of the ellipse centers 28, 30 and / or at least one of the ellipse contour points 32, 34, 36, 38. Each of the ellipse points 28, 30, 32, 34, 36, 38 is covered by a plurality of system equations. A first subgroup of system equations of the system of equations requires that normalized center-direction vectors, each extending from a sensor center / camera center 42, 44 (see Fig. 4) of one of the two optical sensors 20, 22 known through calibration to the ellipse center 28, 30 of the respective image planes of this optical sensor 20, 22, correspond to normalized center-direction vectors from the respective ellipse center 28, 30 to a circle center 46 (see Fig. 6) of the circular shape 10 to be reconstructed.A second subgroup of system equations of the system of equations requires that normalized contour point direction vectors, each extending from the sensor center / camera center 42, 44 of one of the two optical sensors 20, 22 known through calibration to one of the elliptical contour points 32, 34, 36, 38 of the respective associated image area R.415917.

[0122] - 27 - of this optical sensor 20, 22 each correspond to normalized contour point direction vectors from the respective ellipse contour point 32, 34, 36, 38 to the respective associated circle contour point 32', 34', 36', 38' on the outline contour 52 of the circular shape 10 to be reconstructed. A third subgroup of system equations of the system of equations requires that all circle radius vectors, or all of an arbitrary or previously determined selection of all circle radius vectors pointing from the common circle center 46 of the circular shape 10 to one of the circle contour points 32', 34', 36', 38' on the outline contour 52 of the circular shape 10 to be reconstructed, are perpendicular to a mean normal vector of the circular shape 10 to be reconstructed.This mean normal vector is formed by calculating several cross products of two consecutive circle radius vectors arranged in a direction 40 around an outline 52 of the circular shape 10 to be reconstructed, followed by averaging. Alternatively or additionally, the third subgroup of system equations could require that all circle normal vectors of an arbitrary or previously defined selection of the circular shape 10 to be reconstructed, which are formed by calculating a cross product of two consecutive circle radius vectors arranged in a direction 40 around the outline 52 of the circular shape 10 to be reconstructed, are in the same direction.A fourth subset of system equations requires that all circle contour points 32', 34', 36', 38' belonging to any or a previously defined selection of each of the ellipse contour points 32, 34, 36, 38 along the outline 52 of the circular shape 10 to be reconstructed have an identical distance to the common center 46 of the circular shape 10 to be reconstructed. In at least one further process step 130, the determined reconstructed circular shape 10 is output, in particular in three-dimensional coordinates. The output can then be used for eye tracking, object localization, self-localization, or other applications.

Claims

R.415917 - 28 - Claims 1. Computer-implemented method, preferably a computer-implemented eye-tracking method or a computer-implemented object and / or self-localization method, e.g.of an AR headset or a VR headset or another camera system, for a reconstruction of a circular shape (10) of an at least substantially circular object (66), in particular an at least substantially circular pupil disk or a circular marking applied to any body, in three-dimensional space, from at least two at least substantially elliptical two-dimensional images (12, 14) of the object (66), in particular stereoscopic, at least substantially synchronized partial images (16, 18), which each were captured from different perspectives by different, preferably spatially calibrated, optical sensors (20, 22) or which were captured from different, preferably spatially calibrated, sensor perspectives by a single optical sensor (20, 22), comprising at least the process steps (100, 110):. - Extracting one outline contour (24, 26) of the object (66) from each of the partial images (16, 18), wherein the outline contours (24, 26) are at least substantially elliptical, and - Determining an ellipse center point (28, 30) for each of the extracted at least substantially elliptical outline contours (24, 26), characterized in that in at least one process step (120) several mutually spaced discrete ellipse contour points (32, 34, 36, 38) are arranged distributed on the extracted outline contours (24, 26) of the partial images (16, 18) and a subsequent computer-implemented backprojection of the two-dimensional images (12, 14) of the object (66) into three-dimensional space to reconstruct the circular shape (10) in three-dimensional space at least by means of R.415917 - 29 - the determined ellipse centers (28, 30) and by means of the distributed ellipse contour points (32, 34, 36, 38).

2. Computer-implemented method according to claim 1, characterized in that the ellipse contour points (32, 34, 36, 38) are each evenly spaced and arranged with the same direction of rotation on the extracted outline contours (24, 26).

3. Computer-implemented method according to claim 1 or 2, characterized in that the ellipse contour points (32, 34, 36, 38) are arranged in equal numbers on each of the extracted outline contours (24, 26).

4. Computer-implemented method according to one of the preceding claims, characterized in that the elliptical contour points (32, 34, 36, 38) on the two outline contours (24, 26), which were determined from the partial images (16, 18) of the single optical sensor (20, 22) or of the two different optical sensors (20, 22), are arranged alternately offset from one another, in particular such that the elliptical contour points (32, 34, 36, 38) of the two outline contours (24, 26) alternate, preferably as uniformly as possible, when the outline contours (24, 26) are superimposed along a traverse direction (40) around the outline contours (24, 26), and / or such that all of the elliptical contour points (32, 34, 36, 38) correspond to one of the elliptical contour points (32, 34, 36, 38) 38) of the two outline contours (24,26) assignable points on the reconstructed circular shape (10) along a transit direction (40) circumscribing around an outline contour (52) of the reconstructed circular shape (10), each time as regularly as possible.

5. Computer-implemented method according to one of the preceding claims, characterized in that, for the computer-implemented back-projection of the two-dimensional images (12, 14) of the object (66) into three-dimensional space, a plurality of system equations are used for the reconstruction of the circular shape (10) in three-dimensional space. R.415917 - 30 - system of equations comprising a computer-implemented numerical solution method, in particular a Gauss-Newton solution method, which solves a minimization problem, wherein at least a majority of all, preferably all, system equations comprise at least one of the ellipse centers (28, 30) and / or at least one of the ellipse contour points (32, 34, 36, 38) and / or wherein at least a majority of all, preferably each, of the ellipse points (28, 30, 32, 34, 36, 38) are covered by a plurality of system equations.

6. Computer-implemented method according to claim 5, characterized in that a, in particular a first, subgroup of system equations of the system of equations requires that normalized center-direction vectors, which each extend from a sensor center / camera center (42, 44) of one of the two optical sensors (20, 22) or one of the two sensor perspectives of the individual optical sensor (20, 22) to the center of the ellipse (28, 30) of the respective associated image planes of this optical sensor (20, 22), respectively, correspond to normalized center-direction vectors from the respective center of the ellipse (28, 30) to a center of the circle (46) of the circular shape (10) to be reconstructed.

7. Computer-implemented method according to claim 5 or 6, characterized in that a, in particular a second, subgroup of system equations of the system of equations requires that normalized contour point direction vectors, each extending from a sensor center / camera center (42, 44) of one of the two optical sensors (20, 22) or one of the two sensor perspectives of the individual optical sensor (20, 22) to one of the elliptical contour points (32, 34, 36, 38) of the respective associated image planes of this optical sensor (20, 22), are each coupled with normalized contour point direction vectors from the respective elliptical contour point (32, 34, 36, 38) to an associated circular contour point (32', 34', 36', 38') on an outline contour (52) of the circular shape (10) to be reconstructed. agree. R.415917 - 31 - 8. Computer-implemented method according to one of claims 5 to 7, characterized in that a, in particular a third, subgroup of system equations of the system of equations requires that all or all of an arbitrary or previously determined selection of all, in particular from a common center point (46) of the circular shape (10) to be reconstructed to a circular contour point (32', 34', 36', 38') on an outline contour (52) of the circular shape (10) to be reconstructed, circle radius vectors run perpendicular to a mean normal vector of the circular shape (10), wherein this mean normal vector is formed, for example, by calculating several cross products of each pair of, in particular in a traversal direction (40) around an outline contour (52) of the circular shape (10) to be reconstructed, preferably successive, circle radius vectors, with subsequent averaging.and / or that all or all of an arbitrary or previously determined selection of all circle normal vectors of the circular shape (10) to be reconstructed, which are formed by calculating a cross product of each pair of circle radius vectors, in particular in a traversal direction (40) around an outline contour (52) of the circular shape (10) to be reconstructed, are in the same direction.

9. Computer-implemented method according to one of claims 5 to 8, characterized in that a, in particular a fourth, subgroup of system equations of the system of equations requires that all or all of an arbitrary or previously determined selection of all, in particular each belonging to one of the ellipse contour points (32, 34, 36, 38), circle contour points (32', 34', 36', 38') along an outline contour (52) of the circular shape (10) to be reconstructed have an identical distance to a common circle center (46) of the circular shape (10).

10. Computing unit (48), computing infrastructure or system-integrated control unit, e.g. AR or VR headset control unit, comprising at least one R.415917 - 32 - processor (50) and at least one data storage device (54) which includes at least program instructions for carrying out the computer-implemented method according to one of the preceding claims.

11. Eye tracking device (56) with the computing unit (48) or with the system-integrated control unit according to claim 10 or with a network connection to the computing infrastructure according to claim 10 and with the single optical sensor (20, 22) or the at least two optical sensors (20, 22) which can be aligned or oriented differently perspectively to an eye (64).

12. Object and / or self-localization device (58) with the computing unit (48) or with the system-integrated control unit according to claim 10 or with a network connection to the computing infrastructure according to claim 10 and with the single optical sensor (20, 22) or the at least two optical sensors (20, 22) which can be aligned or are aligned with different perspectives on an object (66) and / or a marking.

13. Data glasses (60), in particular AR headset or VR headset, with a head mounting unit (62) and with the eye tracking device (56) according to claim 11 and / or with an object and / or self-localization device (58) according to claim 12.

Images