Method and control unit for gaze detection for smart glasses, and smart glasses

Abstract

The invention relates to a method for gaze detection for smart glasses. The method comprises a step of reading in a sequential data stream of measurement data (418) with a sequence of data portions. Each data portion is generated by one pass of a scanning process scanning at least one sub-region of an eye by means of an optical sensor system (110) of the smart glasses. A data feature indicating a pupil of the eye is contained in at least one subset of the data portions at gaze direction-dependent occurrence positions. The method also comprises a step of sequentially carrying out an evaluation process on the plurality of data portions. For gaze detection, presence of the data feature is ascertained in each data portion and, in the event of such presence, at least one occurrence position (425a) of the data feature is ascertained and pupil information (425b) of the eye is determined for each data portion using the at least one ascertained occurrence position (425a). The method further comprises a step of modifying the scanning process and / or the evaluation process as a function of the at least one ascertained occurrence position (425a). By means of this modification, the scanning process is shortened and / or the evaluation process is limited to sub-portions of the data portions comprising the occurrence positions.

Description

[0001] R. 416813

[0002] - 1 -

[0003] Description

[0004] title

[0005] Method and control unit for eye tracking for smart glasses and smart glasses

[0006] State of the art

[0007] The invention relates to a method for eye tracking for smart glasses, a corresponding control unit, and smart glasses according to the preamble of the independent claims. The present invention also relates to a computer program.

[0008] When displaying images to the eye with smart glasses, it is often necessary to detect the precise direction of the eye's gaze in order to accurately position the displayed image. This is often achieved using eye tracking to determine the position of the eye's pupil. However, eye-tracking methods can be complex and time-consuming.

[0009] Disclosure of the invention

[0010] Against this background, the approach presented here introduces a method for eye-tracking for smart glasses, a control unit that uses this method, smart glasses, and finally a corresponding computer program according to the main claims. Advantageous further developments and improvements of the device specified in the independent claim are possible through the measures listed in the dependent claims. R. 416813

[0011] - 2 -

[0012] This document presents a method for eye tracking for smart glasses, the method comprising the following steps:

[0013] Reading a sequential data stream of measurement data with a sequence of data segments, wherein each data segment is generated by a pass of a scanning process of at least a sub-area of ​​an eye by an optical sensor system of the data glasses, wherein at least one subset of the data segments contains a data feature indicating a pupil of the eye at gaze direction-dependent appearance positions;

[0014] Performing an evaluation process sequentially on the majority of data sections, wherein, for eye-tracking purposes, the presence of the data feature is determined in each data section, and, in the case of presence, at least one occurrence position of the data feature is determined, and, for each data section, pupil information of the eye is determined using the at least one determined occurrence position; and

[0015] Effecting a modification of the sampling process and / or the evaluation process that depends on at least one determined performance position, wherein the modification shortens the sampling process and / or restricts the evaluation process to subsections of the data sections encompassing the performance positions.

[0016] The optical sensor system can be configured to enable, for example, a camera-based eye-tracking method, such as video oculography. It can include camera sensors and means for IR LED illumination. It can also be configured to determine the position and shape of a pupil using classical image processing techniques, such as thresholding, Canny edge detection, ellipse fitting, histogram equalization, or similar methods; deep learning, such as CNN models (CNN = Convolutional Neural Network); determining the gaze vector from the pupil; and model-based (eye model), marker-based (calibration of pupil positions on a display), or glint-based (using IR LED glints and the pupil center). The optical sensor system can also be configured, for example, as a laser-based system for determining eye position, such as a scanned laser system. R.416813

[0017] - 3 -

[0018] The system can be implemented with an external photodiode, a scanned laser system with an integrated photodiode or LFI system (LFI = Laser Feedback Interferometry), a stereo system or similar, or a static laser system for determining eye movements. It can also employ neuromorphic cameras to track the pupil of an eye, in particular using modulated pulsed IR LEDs, which can be integrated into the spectacle lens or frame to generate glints on the cornea that can be registered as events in the event camera. By synchronizing the pulse frequency with the events, the glint positions on the cornea can be captured, and the cornea can be fitted as a sphere into the glints.

[0019] According to embodiments, dynamic optimizations can be enabled, in particular, to increase the efficiency of methods for parameter estimation of geometric objects in sequential data streams. A sequential data stream consisting of data segments with known start and end times, and thus defined lengths, can be processed. Within at least one subset of data segments of this data stream, an event can be contained, which represents only a small time interval of the data stream or the respective data segment. In general, multiple events are conceivable, but their number should be known. This event can occur at different times in the respective data segments of the data stream—for example, early or late. According to embodiments, it is particularly possible to extract information associated with the event with minimal computational effort and low latency.In this way, measurement and evaluation can be advantageously limited to the time frame of the event.

[0020] In other words, according to embodiments, a dynamic reduction of the operations required for parameter estimation of geometric objects in sequential data streams, as well as a temporal restriction of the area to be measured, can be enabled. This can be achieved, for example, using data-based methods in the form of neural networks with additionally learned termination conditions, but also classical methods. R. 416813

[0021] - 4 -

[0022] These designs can be used not only to reduce computational operations and measurement effort, but also, depending on the data, to dynamically reduce latency and are therefore particularly suitable for time-critical applications.

[0023] Particularly with regard to smart glasses and their application in the context of eye tracking for the optical sensor system of the smart glasses, the pupil signal for a single frame or data segment typically represents only a small portion of the total signal; that is, a large part of the data usually contains no information about the pupil. According to embodiments, parameters of an ellipse that best describes the pupil can be estimated for gaze detection, where the parameters can include center, axes, and orientation. This allows overhead to be avoided during both recording and processing: For example, during the recording of this data, power consumption caused by storage, data transfer, sampling of the analog-to-digital converter, driving the laser, energizing the input stage, and the analog front end of the photodiode can be animated.Furthermore, the computational effort required for parameter estimation, which usually scales with the number of time steps or data segments within the data stream, can be reduced. According to some embodiments, it is particularly advantageous to limit the entire evaluation to the time steps containing informative samples in order to minimize power consumption and processing latency.

[0024] According to one embodiment, a modification instruction can be generated for use in the execution step. Additionally or alternatively, a modification sampling instruction can be generated and output to the sensor system during the modification step. This allows eye-tracking to be implemented advantageously with regard to energy consumption and / or computational effort by concentrating on data containing the desired content during measurement acquisition and / or data processing.

[0025] In this step, the implementation rule and, additionally or alternatively, the scanning rule may be used, in conjunction with R. 416813.

[0026] - 5 -

[0027] Predictive values ​​are generated through machine learning. In particular, a model trained using machine learning methods can be used. This allows for early and reliable predictions about potential sensor-related and / or processing-related limitations.

[0028] In the "causing" step, an execution rule can also be generated that terminates the evaluation process for a currently processed data segment as soon as a occurrence position of the data feature is determined within that segment. Additionally or alternatively, a sampling rule can be generated in the "causing" step that terminates the current sampling process as soon as a occurrence position of the data feature is determined within the currently processed data segment. This allows for minimizing the area of ​​the eye to be scanned and / or the amount of data to be processed or evaluated.

[0029] The execution rule and / or the sampling rule generated in the execution step can include a termination condition, which can be predefined or adjustable through machine learning. Depending on the specific design, this allows for a safe, robust, and / or early termination.

[0030] Furthermore, in the activation step, depending on the at least one determined occurrence position of at least one previously processed data segment, an implementation rule can be generated that delays the start of the evaluation process for a subsequently processed data segment by a time interval dependent on the at least one determined occurrence position. Additionally or alternatively, in the activation step, depending on the at least one determined occurrence position of at least one previously processed data segment, a sampling rule can be generated that delays the start of a subsequent iteration of the sampling process by a time interval dependent on the at least one determined occurrence position. This is also possible in R. 416813.

[0031] - 6 -

[0032] This can minimize the area of ​​the eye that needs to be scanned and / or the amount of data that needs to be processed or evaluated.

[0033] This process can be implemented, for example, in software or hardware, or in a hybrid form of software and hardware, for example in a control unit or control device.

[0034] The approach presented here further creates a control unit designed to execute, control, and implement the steps of a variant of the method presented here in appropriate devices. This embodiment of the invention, in the form of a control unit, also allows the underlying problem to be solved quickly and efficiently.

[0035] For this purpose, the control unit can have at least one processing unit for processing signals or data, at least one storage unit for storing signals or data, at least one interface to a sensor or actuator for reading sensor signals from the sensor or for outputting data or control signals to the actuator, and / or at least one communication interface for reading or outputting data embedded in a communication protocol. The processing unit can be, for example, a signal processor, a microcontroller, or the like, and the storage unit can be flash memory or a magnetic storage device.The communication interface can be configured to read or output data wirelessly and / or via wired connections, whereby a communication interface that can read or output wired data can, for example, read this data electrically or optically from or output it into a corresponding data transmission line.

[0036] In this context, a control unit can be understood as an electrical device that processes sensor signals and outputs control and / or data signals accordingly. The control unit may have an interface, which can be implemented in hardware and / or software. In the case of a hardware-based implementation, the interfaces can, for example, be part of a [R. 416813]

[0037] - 7 -

[0038] These interfaces can be so-called system ASICs, which incorporate a wide variety of device functions. However, it is also possible that the interfaces are separate integrated circuits or at least partially comprised of discrete components. In software-based implementations, the interfaces can be software modules, such as those found on a microcontroller alongside other software modules.

[0039] Also advantageous is a computer program product or computer program with program code that can be stored on a machine-readable carrier or storage medium such as a semiconductor memory, a hard disk memory or an optical memory and is used to carry out, implement and / or control the steps of the method according to one of the embodiments described above, in particular if the program product or program is executed on a computer, a control unit or a device.

[0040] A particularly advantageous embodiment of the approach presented here is that of smart glasses, incorporating an optical sensor system and a control unit according to a variant presented herein, wherein the sensor system and the control unit are interconnected for data transmission. Such an embodiment allows for simple, inexpensive, fast, and reliable eye tracking, enabling the precise display of information to the user of the smart glasses.

[0041] Examples of the approach presented here are shown in the drawings and explained in more detail in the following description. It shows:

[0042] Fig. 1 shows a schematic representation of a data glasses according to an exemplary embodiment;

[0043] Fig. 2 shows a schematic representation of a sensor system of data glasses according to an exemplary embodiment;

[0044] Fig. 3 shows a schematic representation of a sensor system of smart glasses according to an exemplary embodiment; R. 416813

[0045] - 8 -

[0046] Fig. 4 shows a schematic representation of a control unit according to an exemplary embodiment;

[0047] Fig. 5 shows a flowchart of an exemplary embodiment of a method for eye tracking for data glasses;

[0048] Fig. 6 shows a schematic representation of a scanning process by a sensor system of smart glasses according to an exemplary embodiment; and

[0049] Fig. 7 shows a schematic representation of pupil information determined by an embodiment of a control unit and / or a method.

[0050] In the following description of favorable embodiments of the present invention, the same or similar reference numerals are used for the elements shown in the various figures and acting similarly, without repeating these elements.

[0051] Figure 1 shows a schematic representation of an embodiment of a pair of smart glasses 100. The smart glasses 100 are designed, for example, as AR glasses (AR = Augmented Reality) or as VR glasses (VR = Virtual Reality). The smart glasses 100 comprise an optical sensor system 110 and a control unit 120. The sensor system 110 and the control unit 120 are connected to each other in a way that enables data transmission. The control unit 120 is configured to perform and / or control a method for eye tracking for the smart glasses 100. The sensor system 110 and the control unit 120 will be discussed in more detail below. A frame with temples 102 and lenses 104 of the smart glasses 100 are also shown.

[0052] Fig. 2 shows a schematic representation of a sensor system 110 of a pair of smart glasses according to an exemplary embodiment. The sensor system 110 corresponds to or is similar to the sensor system from Fig. 1. According to the exemplary embodiment shown here, the sensor system 110 is a so-called scanned sensor. 416813

[0053] - 9 -

[0054] The laser system is designed with an external photodiode. The sensor system 110 comprises an external photodiode 212 as a single-pixel detector, a deflection device 214 or a scanning module or MEMS scanning module with deflection units, in particular mirrors, MEMS mirrors or the like, and a laser module 215. Furthermore, one of the temples 102 or temples and one of the lenses 104 of the data glasses are shown, in or on which a holographic-optical element (HOE) 203 or the like is arranged. An eye 201 is also shown.

[0055] During operation of the sensor system 110, a laser beam or infrared laser beam from the laser module 215 is moved in two dimensions, for example sinusoidally, across the eye region by means of the deflection device 214, so that the eye region is scanned in two dimensions. This is illustrated in a scan partial representation 230 with line synchronization 232 and frame synchronization 234. Depending on the reflectivity of the eye surface, light is scattered from the surface of the eye 201, as illustrated in a raster partial representation 240. The external photodiode 212 on the temple of the spectacles 102 is scanned at equidistant intervals with a fixed sampling rate. Based on the mirror position of the deflection device 214 or the measurement point on the eye 201, a 2D image in the form of a reflectivity map 250 is generated.The pupil appears dark because it acts as an aperture for the IR light from the laser module 215, allowing the IR light beam to penetrate the pupil and thus preventing much light from reaching the external off-axis photodetector or the external photodiode 212. Image processing algorithms, such as Canny Edge Detector, detect the dark pupil outline in the image and subsequently determine the pupil ellipse.

[0056] Fig. 3 shows a schematic representation of a sensor system 110 of a pair of smart glasses according to an exemplary embodiment. The sensor system 110 corresponds to or is similar to the sensor system from Fig. 1 and to the sensor system from Fig. 2. According to the exemplary embodiment shown here, the sensor system 110 is designed as a so-called scanned laser system with an integrated photodiode. The representation of the sensor system 110, the part of the smart glasses, and the eye 101 in Fig. 3 corresponds to the representation of the same in Fig. 2 with R. 416813

[0057] - 10 -

[0058] The exception is that the sensor system 110 has a laser module 315 with integrated photodiode or a laser feedback interferometry sensor (LFI sensor) instead of the laser module with external photodiode.

[0059] In operation of the sensor system 110, an IR laser beam, generated by the IR laser of the laser feedback interferometry sensor or laser module 315, is moved in two dimensions, for example sinusoidally, by means of the deflection device 214 and an infrared-compatible beam combiner, for example a holographic beam combiner such as the holographic-optical element (HOE) 203, over the eye region so that the eye region is scanned in two dimensions. Depending on the reflectivity of the retina, light is scattered back from the retina of the eye 201. The photodiode integrated in the LFI sensor or laser module 315 is scanned at equidistant intervals with a fixed sampling rate. Based on the mirror position of the deflection device 214 or the measurement point on the eye 101, a 2D image or a reflectivity map is generated.The pupil appears bright because the high reflectivity of the retina causes light to be reflected back from the retina into the LFI sensor 315, where amplitude modulation occurs. This modulation is measured by the integrated photodiode. Image processing algorithms, such as Canny Edge Detector, detect the bright pupil outline in the image and subsequently determine the pupillary ellipse.

[0060] With reference to Figures 1, 2, and 3, the following summary statements can be made, particularly concerning the sensor system 110 or an eye-tracking sensor unit of the data glasses 100, regarding a purely exemplary, general sequence of gaze detection or eye tracking: First, the entire eye 201 or a partial region thereof is scanned using an infrared laser beam. Subsequently, scattered light from a surface of the eye 201 is captured using the remote or external photodiode 212 or the internal photodiode integrated into the laser module 315 or the LFI sensor. Following this, a 2D image with either the light or dark pupil as a feature is generated from synchronization signals of the rows and columns of the deflection device 214 or the MEMS micromirror system. Finally, the pupil is processed using classical image processing methods, such as R. 416813.

[0061] - 11 -

[0062] Edge detection, histogram operations, morphological filters, etc., or machine learning methods, such as Convolutional Neural Networks (CNN), applied to the image data determine or detect.

[0063] Fig. 4 shows a schematic representation of a control unit 120 according to an exemplary embodiment. The control unit 120 corresponds to or is similar to the control unit from Fig. 1. The control unit 120 is capable of data transmission and can be connected to, or is connected to, the optical sensor system 110, wherein the sensor system 110 corresponds to or is similar to the sensor system from one of the figures described above. The control unit 120 is configured to implement, control, and / or execute eye-tracking for smart glasses. In other words, the control unit 120 is configured to execute, implement, and / or control an eye-tracking method for smart glasses. The control unit 120 comprises a reading device 422, a transmission device 424, and an action device 426.

[0064] The reading device 422 of the control unit 120 is configured to read a sequential data stream of measurement data 418, consisting of a sequence of data segments, from the sensor system 110. Each data segment of the measurement data 418 is generated by a single pass of a scanning process of at least a partial area of ​​an eye by the sensor system 110 of the smart glasses. In at least one subset of the data segments, a data feature indicating a pupil of the eye is included at gaze-direction-dependent positions. The reading device 422 is also configured to transmit the measurement data 418, or the sequential data stream with the sequence of data segments, to the transmission device 424.

[0065] The implementation device 424 is configured to perform an evaluation process on the majority of data sections of the sequential data stream of the measurement data 418. The implementation device 424 is configured to determine, for eye tracking purposes, the presence of the data feature in each data section and, if present, at least one occurrence position 425a of the data feature in the respective data section. Furthermore, the implementation device 424 is configured to... 416813

[0066] - 12 -

[0067] The data section uses at least one determined tactile position 425a to determine pupil information 425b of the eye. Furthermore, the implementation device 424 is designed to output or make available for output the tactile position 425a and / or the pupil information 425b.

[0068] The action device 426 is designed to effect a modification of the scanning process of the sensor system 110 and / or the evaluation process of the feedthrough device 424, depending on the at least one determined entry position 425a. This modification shortens the scanning process of the sensor system 110. Additionally or alternatively, the modification limits or restricts the evaluation process of the feedthrough device 424 to subsections of the data sections of the sequential data stream of the measurement data 418 that comprise the entry positions 425a.

[0069] According to one embodiment, the effecting device 426 is designed to generate an implementation instruction 427 that effects the modification for use by the implementation device 424.

[0070] Additionally or alternatively, the action device 426 is configured to generate a sampling instruction 428 that effects the modification and output it to the sensor system 110. Optionally, the action device 426 is also configured to generate the implementation instruction 427 and / or the sampling instruction 428 using prediction values ​​generated by machine learning.

[0071] According to one embodiment, the action device 426 is configured to generate an implementation instruction 427 that terminates the evaluation process of the implementation device 424 for a currently processed data segment. Additionally or alternatively, the action device 426 is configured to generate a sampling instruction 428 that terminates a current pass of the sampling process of the sensor system 110 as soon as the implementation device 424 detects an occurrence position 425a of the data feature in the currently processed data segment. Optionally, this additionally includes or includes R. 416813

[0072] - 13 -

[0073] The implementation rule 427 and / or sampling rule 428 generated by the action device 426 for the termination is a termination condition that is predefined or adjustable by machine learning.

[0074] According to one embodiment, the action device 426 is configured to generate an implementation instruction 427, depending on the at least one determined occurrence position 425a of at least one previously processed data section. This instruction causes the start of the evaluation process for a subsequently processed data section to be delayed by a time interval dependent on the at least one determined occurrence position 425a. Additionally or alternatively, the action device 426 is configured to generate a sampling instruction 428, depending on the at least one determined occurrence position 425a of at least one previously processed data section. This sampling instruction causes the start of a subsequent iteration of the sampling process to be delayed by a time interval dependent on the at least one determined occurrence position 425a.

[0075] Fig. 5 shows a flowchart of an embodiment of method 520 for eye-tracking for smart glasses. Method 520 for eye-tracking can be implemented using the control unit from one of the figures described above. Method 520 for eye-tracking can be implemented for the smart glasses from one of the figures described above or for similar smart glasses. Method 520 for eye-tracking comprises a reading step 522, an execution step 524, and an effect step 526.

[0076] In step 522 of the reading process, a sequential data stream of measurement data is read in, consisting of a sequence of data segments. Each data segment is generated by a single pass of a scanning process of at least a sub-area of ​​an eye by an optical sensor system of the smart glasses. At least one subset of the data segments contains a data feature indicating a pupil of the eye at gaze-direction-dependent positions. R. 416813

[0077] - 14 -

[0078] In step 524 of the execution, an evaluation process is carried out sequentially on the majority of data segments. For eye-tracking purposes, the presence of the data feature is determined in each data segment, and if present, at least one appearance position of the data feature is identified. For each data segment, pupil information is then determined using the at least one identified appearance position.

[0079] In step 526 of the process, a modification of the sampling process and / or the evaluation process is effected, depending on the at least one determined occurrence position. This modification shortens the sampling process and / or restricts the evaluation process to subsections of the data segments encompassing the occurrence positions.

[0080] Method 520 can generally be combined with both classical algorithms and neural networks. The only requirement is that the information is processed sequentially over time. Processing on individual time intervals, which do not necessarily correspond to a single magazine, is also conceivable; for example, data windows that combine several time steps, image lines in the context of eye tracking, etc. The goal is, in particular, a dynamic restriction to the informative portion of the data stream, achieved, among other things, by feeding the predictions back to the sensor system 110. This can make a significant contribution, especially with regard to resource-limited hardware. Increases or...Improvements are achieved in the overall system, including a reduction in power consumption in the analog front end of the photodiode amplifier, since no further signals need to be recorded; in the laser driver, since the IR laser driver can be switched off more frequently; in the MEMS scanner, since the scanning process can be aborted; in the digital section of an ASIC, since less data needs to be digitized (ADC) and stored (SRAM); and in data transmission and in the digital section of the computing unit or control unit 120, as well as a reduction in latency, as shown below with reference to Fig.

[0081] 6 explained.R. 416813

[0082] - 15 -

[0083] Fig. 6 shows a schematic representation of a scanning process by a sensor system of smart glasses according to an exemplary embodiment. The scanning process can be carried out by a sensor system that corresponds to or is similar to the sensor system from one of the figures described above. Furthermore, the scanning process can be modified by a control unit that corresponds to or is similar to the control unit from one of the figures described above. A first partial representation I illustrates a regular scanning process, and a second partial representation II illustrates a modified scanning process.

[0084] In the regular scanning process, a raster scan 661 is initiated, and the user's eye region is scanned. This captures the user's pupil 601. The pupil's position is then calculated, and the result is sent to a rendering pipeline of the data glasses' graphics unit, as shown in section 662. The new image is then calculated, as shown in section 663. The scanning process is then repeated.

[0085] For the modified scanning process, two approaches or a combination thereof are possible according to the exemplary embodiments: a so-called late-starting approach and / or a so-called early-stopping approach. In the late-starting approach, the scan area is restricted based on the position determined from the previous scan, using the physical knowledge of the system. In the early-stopping approach, the raster scan 661 is aborted after the detection of the pupil 601, and the pupil calculation is performed on the significantly reduced data. This allows the transmission to take place faster and for a shorter time, as illustrated by a time span 665, and the rendering of the image content can also begin much earlier. This results in a significantly shorter system latency At 664.

[0086] Fig. 7 shows a schematic representation of pupil information 425b determined by an embodiment of a control unit and / or a method. The pupil information 425b is determined in particular by the control unit and / or by means of the method from one of the above R. 416813

[0087] - 16 -

[0088] The described figures are determined. In each of the four partial representations I, II, III and IV, pupil information 425b is shown.

[0089] In other words, Fig. 7 shows four exemplary pupil positions for the procedure. Several realizations are shown for each position: an ellipse describing the pupil and a model prediction (one ellipse for each realization). For the representation, the data stream was reconstructed into a two-dimensional image. The scanning is performed as shown in Fig. 2 (starting at the bottom of the image). The pupil (dark areas) occupies a small portion of the image. The trained termination mechanism of the modification makes a prediction for each image row. For the examples shown, termination is predicted after the pupil is observed, which in turn can be used to dynamically reduce the computational and measurement effort, depending on the current pupil position—the earlier the position, the greater the reduction.

[0090] With reference to the figures described above, exemplary embodiments are summarized and briefly explained below in other words.

[0091] Exemplary implementations make it possible to dynamically increase the efficiency of eye-tracking methods, i.e., depending on the input data or measurement data 418. Since the actual signal component—in this case, the pupil signal—represents only a very small portion of the input data or measurement data 418, it is advantageous to minimize the scan and calculations on all other data segments. Two approaches are possible for this:

[0092] Early stopping: If a signal or data feature is detected in the data stream and corresponding parameters are estimated, the model evaluation or analysis process can be terminated. This can be done classically using a termination criterion, e.g., the prediction no longer changes after processing several image lines, or by extending the model, as in the field of dynamic neural networks and the associated early stopping approaches, which can be used here. R. 416813

[0093] - 17 -

[0094] The extension for predicting a termination criterion includes, for example, at least one additional layer parallel to the actual model, but can also incorporate a complete neural network. The task of this extension is to provide a prediction of the termination condition after processing each unit, where, for example, 0 represents processing of the next row and 1 represents termination. Since this extension can be trained with established loss functions, it offers the possibility of incorporating knowledge about the characteristics of the measuring apparatus or sensor system 110 into the termination condition. Furthermore, the solution found can be influenced by additional regularization terms, allowing the area to be measured and the number of required computations to be weighed against the accuracy of the model.For training, established loss functions and regularization terms can be used in combination with well-known surrogate gradient approaches to handle the binary nature of the termination mechanism during training. In the case of eye tracking, the regularization term can be chosen such that predicted early stops occurring before or after the actual pupil signal cause a large loss, e.g., a generalized normal distribution with the mean set to the pupil center. The training comprises at least one step: The network can be trained directly with the termination condition, or a pre-trained network, trained to provide a parameter estimate as soon as the pupil is included in the data stream, can be extended to include the termination functionality in a subsequent step by optimizing only the additional gate / network or its parameters.

[0095] Late-Starting: By observing the natural movement of the pupil and its maximum velocity, and by predicting the parameters from the last scan, a temporal range can be defined in the input data or measurement data (418) that can accommodate the signal or data feature. This allows a starting position for the model evaluation to be determined, and all previous parts of the sequence do not need to be measured or processed. This starting point can be determined either with classical models (definition of a region of interest) or via data-driven methods (416813).

[0096] - 18 -

[0097] can be learned (predicting a late start for the next scan, similar to early stopping).

[0098] Modifying the evaluation process by restricting the area to be processed, as described in both of the aforementioned approaches, can not only increase the efficiency of the model. Feedback to the optical system or sensor system 110 can also be implemented, so that, for example, the sampling process or scan only starts and / or ends at a specific line, or at least the laser is only activated for the relevant area. This feedback significantly increases the efficiency of the overall system. An indicator of the dynamic behavior according to the exemplary embodiments is, for example, a variable time interval between the start of a frame and the receipt of a prediction, which is also reflected in a variable number of computational operations. Furthermore, the laser can be switched off after a pupil appears in the image, or dynamically switched on.With the eye closed, the IR laser is not switched off or only briefly, whereas with the eye open, the laser is switched off earlier or for a shorter time.

[0099] If an embodiment includes an “and / or” connection between a first feature and a second feature, this is to be read as meaning that the embodiment according to one embodiment has both the first feature and the second feature, and according to another embodiment has either only the first feature or only the second feature.

Claims

R. 416813 - 19 - Claims 1. Method (520) for eye tracking for a data glasses (100), wherein the method (520) comprises the following steps: Reading (522) a sequential data stream of measurement data (418) with a sequence of data segments, wherein each data segment is generated by a pass of a scanning process of at least a sub-area of ​​an eye (201) by an optical sensor system (110) of the data glasses (100), wherein at least one subset of the data segments contains a data feature indicating a pupil (601) of the eye (201) at gaze direction-dependent appearance positions; Performing (524) an evaluation process sequentially on the majority of data sections, wherein, for eye-tracking purposes, the presence of the data feature is determined in each data section, and, in the case of presence, at least one appearance position (425a) of the data feature is determined, and, for each data section, pupil information (425b) of the eye (201) is determined using the at least one appearance position (425a) determined; and Effect (526) a modification of the sampling process and / or the evaluation process dependent on the at least one determined performance position (425a), wherein the modification shortens the sampling process and / or restricts the evaluation process to subsections of the data sections comprising the performance positions.

2. Method (520) according to claim 1, wherein in step (526) of effecting, an implementing instruction (427) effecting the modification is generated for use in step (524) of carrying out. R. 416813 - 20 - and / or a scanning instruction (428) that effects the modification is generated and output to the sensor system (110).

3. Method (520) according to claim 2, wherein in step (526) of effecting the implementation rule (427) and / or the sampling rule (428) is generated using predictive values ​​generated by machine learning.

4. Method (520) according to one of claims 2 to 3, in which, in step (526) of effecting, an implementation instruction (427) is generated which causes the evaluation process to be terminated for a currently processed data section, and / or a sampling instruction (428) is generated which causes a current pass of the sampling process to be terminated as soon as, in step (524) of execution, an occurrence position (425a) of the data feature is determined in the currently processed data section.

5. Method (520) according to claim 4, wherein the implementation rule (427) generated in step (526) of causing and / or the sampling rule (428) generated in step (526) of causing comprises a termination condition which is predefined or adjustable by machine learning.

6. Method (520) according to any one of claims 2 to 5, wherein in step (526) of effecting, depending on the at least one determined occurrence position (425a) of at least one previously processed data section, an implementation instruction (427) is generated which causes a start of the evaluation process for a subsequently processed data section to be delayed by a time interval dependent on the at least one determined occurrence position (425a), and / or a sampling instruction (428) is generated which causes a start of a subsequent pass of the sampling process to be delayed by a time interval dependent on the at least one determined occurrence position (425a). R. 416813 - 21 - 7. Control unit (120) configured to perform and / or control the steps (522, 524, 526) of the method (520) according to any of the preceding claims in corresponding units (422, 424, 426).

8. Data glasses (100) with an optical sensor system (110) and a control unit (120) according to claim 7, wherein the sensor system (110) and the control unit (120) are connected to each other in a data-transmitting manner.

9. Computer program configured to execute and / or control the steps (522, 524, 526) of the method (520) according to any one of claims 1 to 6.

10. Machine-readable storage medium on which the computer program according to claim 9 is stored.

Images