Method for detecting viewing directions of a wearer of smart glasses, and computing unit
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- ROBERT BOSCH GMBH
- Filing Date
- 2026-01-09
- Publication Date
- 2026-07-30
Smart Images
Figure EP2026050411_30072026_PF_FP_ABST
Abstract
Description
[0001] R.416477
[0002] - 1 -
[0003] Description
[0004] Method for detecting the gaze direction of a wearer of data glasses and a computing unit
[0005] State of the art
[0006] A method for detecting the gaze direction of a wearer of smart glasses, which includes laser feedback interferometry (LFI) multipoint sensors, has already been proposed. This method involves generating a plurality of LFI measurement points in an area of the smart glasses intended for positioning the wearer's eye, and capturing back-reflection signals from the LFI measurement points reflected by the wearer's eye.
[0007] Disclosure of the invention
[0008] To reliably track eye movements using LFI multipoint sensors, several LFI measurement points should be positioned within the eye. For robust performance in as many people as possible, smart glasses with this sensor technology should incorporate a large number of distributed LFI measurement points, as individuals can have significantly different free eye and head parameters, such as sclera size, pupillary distance, iris diameter, head width, and vertex distance. For example, a two-dimensional array of individual, separate LFI sensors could be used to cover a sufficiently large area of the eye. However, arrays of individual, separate LFI sensors have increased space requirements, weight, optical power (impacting eye safety), power consumption, and complexity (cost). One way to mitigate these potential drawbacks is through the replication of R.416477.
[0009] - 2 -
[0010] Light signals emitted by individual LFI sensors can be multiplied, for example, using an optical element (diffractive optical element, holographic optical element, meta-optic element, segmented lens, biconical lens, etc.) to generate multiple LFI measurement points from the individual light signals. This approach necessitates that the back-reflected signals of the individual LFI measurement points created by multiplication must be separable from one another, and thus, in particular, uniquely identifiable.
[0011] The invention relates to a method for detecting gaze directions, in particular of eyes, of a wearer of smart glasses, in particular an AR headset or a VR headset, which comprises a laser feedback interferometry (LFI) multipoint sensor, wherein the LFI multipoint sensor generates a plurality of LFI measuring points in an area of the smart glasses provided for positioning an eye of the wearer, and wherein the LFI multipoint sensor detects back reflection signals of the LFI measuring points reflected back from the wearer's eye.
[0012] It is proposed that a characteristic distance signal pattern, which differs for different gaze directions of the wearer's eye, be read from the back-reflection signals and evaluated for gaze direction determination. This allows for the advantageous use of LFI multipoint sensors based on the previously described duplication, whose individual LFI measurement points can nevertheless be reliably back-identified. This enables the advantageous creation of a lightweight, cost-effective, compact, energy-efficient, and / or user-compatible LFI multipoint sensor, particularly in the form of smart glasses, that still possesses high quality and / or reliability for gaze direction detection.
[0013] Data glasses, such as AR headsets or VR headsets, are also frequently called smart glasses. Alternatively, gaze direction detection can also be used in other systems, for example, head-up displays, ophthalmic examination devices, telescopes, microscopes, etc. AR headsets and / or VR headsets are, in particular, head-worn smart devices that project artificially generated image content into a user's field of vision. The LFI multipoint sensor technology includes, in particular, several LFI-R.416477 sensors.
[0014] - 3 -
[0015] Sensors. For example, it is conceivable that the LFI multipoint sensor system forms an LFI sensor array with two, three, or more than three LFI sensors. In particular, each LFI sensor of the LFI multipoint sensor system generates a plurality of LFI measurement points, especially two, three, or more than three LFI measurement points. The laser beams associated with the LFI measurement points are directed at least substantially perpendicular to a pupil plane of the smart glasses, in particular to the eye of the wearer of the smart glasses. Preferably, the laser beams associated with the LFI measurement points are aligned at least substantially parallel to each other. Preferably, the laser beams associated with the LFI measurement points are free from overlap and / or intersection with each other. "Provided for" is to be understood in particular as being specifically programmed, designed, and / or equipped.The fact that an object is intended for a specific function should be understood in particular to mean that the object fulfills and / or performs this specific function in at least one application and / or operating state.
[0016] The LFI sensor(s) of the LFI multipoint sensor system can be configured, for example, as a VCSEL, preferably a ViP-VCSEL (vertical-cavity surface-emitting laser with integrated photodiode). The LFI sensor(s) is / are integrated, in particular, into the AR headset and / or the VR headset, preferably the smart glasses, e.g., into a frame, lens, or temple of the smart glasses. The LFI sensor(s) can also be part of a laser projector unit that generates the artificial image content of the AR and / or VR headset, i.e., integrated, for example, together with (RGB) laser diodes. Preferably, however, the LFI sensor(s) is / are configured separately from a laser projector unit that generates the artificial image content of the AR and / or VR headset. Preferably, the LFI sensor(s) are static and illuminate the eye independently of a scanner.Preferably, the infrared laser beam(s) of the LFI sensor unit is / are free from interference by a scanner, such as a controlled MEMS mirror system. The LFI sensor is based on an interferometric measurement method, also known as laser self-mixing. In particular, the LFI sensor emits a laser beam in the infrared spectrum (the infrared laser signal), which then strikes a surface with reflectivity (e.g., the retina). From this surface, the light of the laser beam is then backscattered, so that it re-enters a spectrum.
[0017] - 4 -
[0018] The laser cavity of the LFI sensor is entered. Within the laser cavity of the LFI sensor, the backscattered light interferes with a locally oscillating field of the LFI sensor. This leads, in particular, to a modulation of the laser power of the laser source, which can be detected either by a photodiode integrated into a back reflector of the laser cavity or by a direct measurement of a voltage of the laser source. The infrared laser source of the LFI sensor is, in particular, an infrared laser diode. The photodetector of the LFI sensor is, in particular, formed by the photodiode.
[0019] The distance signal pattern is primarily formed by measured intensity maxima of back-reflection signals, each having traveled different distances between emission and detection. These distance differences originate from different points of impact on the (non-planar) eye of the wearer and / or different points of reflection within the wearer's eye (outside the pupil: on the outer surface of the eye; inside the pupil: on the retinal surface). With different eye directions, the eye surface is oriented differently relative to the laser beams of the LFI multipoint sensor. With different eye directions, the pupil is struck / passed through by different laser beams / LFI measurement points of the LFI multipoint sensor. Specifically, the LFI multipoint sensor detects a signal peak in the distance signal pattern for each LFI measurement point / laser beam of the LFI multipoint sensor.The signal peaks in the distance signal pattern oscillate between higher and lower values as the eye moves. These oscillations are preferably continuous / without discontinuities. The characteristic distance signal patterns are preferably distinguished by characteristic sequences and / or characteristic relative distances between signal peaks. Each direction of gaze preferably possesses at least one, and preferably exactly one, characteristic distance signal pattern. The evaluation of the back-reflection signals and / or the distance signal pattern is performed by a processing unit. The processing unit can be assigned to or integrated into the smart glasses, or it can be separate from the smart glasses and communicate with them via a communication interface.A "computing unit" is understood to be, in particular, a unit with an information input, information processing, and information output. Advantageously, the computing unit has at least one R.416477.
[0020] - 5 -
[0021] processor, memory, input and output devices, other electrical components, an operating program, control routines, regulation routines and / or calculation routines.
[0022] If several of the LFI measurement points of the LFI multipoint sensor are generated by identical laser sources, a lightweight, cost-effective, compact, energy-efficient, and / or user-compatible LFI multipoint sensor, particularly smart glasses, can be advantageously created, which nevertheless possesses high quality and / or reliability for gaze direction detection. In principle, however, the described method is also applicable to LFI multipoint sensors where each LFI measurement point is generated by a different laser source. Specifically, the LFI multipoint sensor generates an LFI measurement point pattern in the area of the smart glasses intended for positioning the wearer's eye, particularly in the pupil plane. Various LFI measurement point patterns are conceivable. For example, the LFI measurement point patterns can be at least essentially matrix-shaped or quincunx-shaped.Adjacent LFI measurement points can originate from the same laser source of the LFI multipoint sensor, or LFI measurement points from different laser sources of the LFI multipoint sensor can be arranged adjacent to each other in the matrix or quincunx pattern. For example, the LFI measurement points in the LFI measurement point pattern can form concentric or non-concentric (intersecting) circles. Each circle can originate from a replicated laser source, or parts of the circles can originate from different laser sources. For example, the LFI measurement points in the LFI measurement point pattern can form concentric or non-concentric (intersecting) triangles or other polygons. Each triangle or polygon can originate from a replicated laser source, or parts of the triangles or polygons can originate from different laser sources.The examples from the EM are all described in the application text. Preferably, these are superimposed. For example, the LFI measurement points in the LFI measurement point pattern can form parallel or intersecting linear patterns. Each line can originate from a replicated laser source, or sections of the lines can originate from different laser sources. In particular, the light from multiple R.416477.
[0023] - 6 -
[0024] LFI measurement points generated by laser sources can be multiplied using diffractive optical elements, holographic optical elements, meta-optical elements and / or lenses or lens arrangements.
[0025] In particular, each back-reflection signal from each LFI measurement point contains at least the following information: the Doppler velocity of the reflecting object at the location of the respective LFI measurement point, the distance between the associated laser source and the reflecting object at the location of the respective LFI measurement point, and / or an intensity that depends on the output intensity of the associated laser beam and the (infrared) reflectivity of the reflecting object at the location of the respective LFI measurement point. In particular, the LFI method is based on a Frequency-Modulated Continuous Wave (F-MCW) approach known to those skilled in the art.
[0026] Furthermore, it is proposed that the beam directions of at least two of the laser sources of the LFI multipoint sensor system, preferably all laser sources of the LFI multipoint sensor system, are at least mirror-symmetry-free with respect to each other. This advantageously avoids ambiguities in the distance signal pattern. In particular, at least two of the LFI sensors / laser sources of the LFI multipoint sensor system, preferably all LFI sensors / laser sources of the LFI multipoint sensor system, are arranged relative to each other such that the horizontal and vertical symmetry of their beam directions is broken.
[0027] Furthermore, it is proposed that a back-reflection signal calibration be performed prior to gaze direction detection. This calibration would record characteristic distance signal patterns for different gaze directions, specifically tailored to each individual wearer of the smart glasses. This would advantageously ensure that the lightweight, cost-effective, compact, and / or energy-efficient gaze direction detection system functions with exceptional accuracy and individualization. Specifically, during the back-reflection signal calibration, the wearer of the smart glasses would be asked to look at various points in the room or on a screen, such as a smartphone. For each gaze direction, which the wearer can confirm through input, a uniquely identifiable and characteristic distance signal pattern would then be recorded during the back-reflection signal calibration.This characteristic distance signal pattern then acts in particular like a kind of fingerprint, which reveals the corresponding direction of gaze, should the same distance signal pattern R.416477 be present.
[0028] - 7 -
[0029] These patterns reappear later during normal operation of the smart glasses. During normal operation, the currently determined distance signal patterns are preferably compared with distance signal patterns known from calibration, and the current gaze directions are then determined based on similarities and / or matches. In particular, the back-reflection signal calibration serves to assign characteristic, wearer-specific distance signal patterns to gaze directions. Specifically, recording the calibration distance signal patterns includes storing the distance signal patterns captured during the back-reflection signal calibration in a storage unit, for example, a storage unit within the smart glasses or an external storage unit connected to the smart glasses (e.g., cloud storage).
[0030] Furthermore, it is proposed that a calibration map, particularly a two-dimensional one, be generated from the characteristic distance signal patterns recorded during back-reflection signal calibration. This map is then used to evaluate the currently detected back-reflection signals during gaze direction determination. This advantageously creates a particularly comprehensive basis for gaze direction detection. Specifically, the calibration map includes characteristic distance signal patterns for different gaze directions in which the pupil is deflected vertically and horizontally. Each entry on the two-dimensional calibration map corresponds to a gaze direction with a horizontal and a vertical deflection value relative to a neutral / resting position of the eye.
[0031] If gaps in the calibration map, for which no explicit calibration measurement exists, are filled by interpolation of neighboring calibration measurements, a particularly accurate gaze direction detection can be advantageously achieved, which can be generated with the least possible calibration effort for the user.
[0032] Furthermore, it is proposed that, prior to gaze direction detection, the wearer's head and / or eye dimensions be recorded, based on which characteristic distance signal patterns, particularly those specific to the wearer of the smart glasses, can be calculated and / or adapted for different gaze directions. This advantageously allows for a particularly simple calibration. R.416477
[0033] - 8 -
[0034] This could be achieved, in particular, without the previously described handling of different calibration gaze directions. Alternatively, the previously described calibration method could be advantageously optimized and / or improved by individually designing and / or adapting calibration gaze direction patterns according to the user's head and / or eye dimensions / ethnicity. Advantageously, in LFI multipoint sensor configurations with symmetries, i.e., with ambiguous measurement signals, the head and / or eye measurement information can be used to exclude the unsuitable measurement signal from the ambiguous signals. Head and / or eye measurement preferably also includes recording answers to questions posed to the wearer. Head and / or eye measurements can be directly determined or queried.Head and / or eye dimensions can be precisely queried, or at least some of the precise queries can be replaced by ethnicity queries. Specifically, the wearer is asked targeted questions to capture their head and / or eye dimensions, for example, specifically about their individual head and / or eye measurements or their ethnicity. Since the distribution of certain head and / or eye dimensions across the world population is known, such as free eye parameters (ratio of interpupillary distance to head width, etc.), an algorithm can use this information to exclude certain ambiguous measurement signals. For example, if the wearer chooses smart glasses with a temple width of 140 mm (matching their head width), measures an interpupillary distance of 60 mm, and indicates European ethnicity, the algorithm, which, for example,The system, which may be stored on a processing unit of the smart glasses, determines which of the ambiguous measurement signals can be excluded from the distance signal pattern to determine the gaze direction. Adjusting the distance signal patterns can therefore be understood as eliminating false / ambiguous measurement signals during the evaluation of the distance signal patterns. The back-reflection signal calibration can vary depending on ethnicity or specific head parameters. Therefore, head and / or eye measurement is preferably performed before the back-reflection signal calibration.
[0035] Furthermore, it is proposed that a distance signal (R.416477) currently captured by the LFI multipoint sensor system be used to evaluate the gaze direction determination.
[0036] - 9 -
[0037] The pattern is compared with the characteristic distance signal patterns known from the wearer-specific calibration, each representing a specific gaze direction of the wearer, in particular with the calibration card that assigns a characteristic distance signal pattern to each gaze direction. Alternatively, for the evaluation of the gaze direction determination, the distance signal pattern currently detected by the LFI multipoint sensor is compared with characteristic distance signal patterns calculated for the wearer based on the head and / or eye dimensions, each representing a specific gaze direction of the wearer. This advantageously allows for a particularly precise gaze direction determination that is individually tailored to the wearer of the data glasses.
[0038] If the distance signal pattern comprises at least one pattern in a distance-intensity space, in which distances traveled by laser light from the LFI measurement points to the wearer's eye are plotted against their respective intensities detected by the LFI multipoint sensor, a simple and / or unambiguous evaluation of the distance signal patterns can advantageously be enabled. Advantageously, the distance signal pattern in the distance-intensity space has a particularly simple and therefore also simple and quick-to-evaluate pattern shape. The distance-intensity space is spanned in particular by a distance axis (e.g., x-direction) and an intensity axis (e.g., y-direction). The measurement signals associated with the individual LFI measurement points are preferably sharp, in particular nearly linear, intensity signal peaks.
[0039] Alternatively or additionally, it is proposed that the distance signal pattern includes at least one pattern in a distance-velocity space, in which distances traveled by laser light from the LFI measurement points to the wearer's eye are plotted against their respective corresponding Doppler velocities detected by the LFI multipoint sensor. This advantageously enables a simple and / or unambiguous evaluation of the distance signal patterns. The distance signal pattern in the distance-velocity space advantageously exhibits a particularly simple and therefore also simple and quick-to-evaluate pattern shape. The distance-velocity space is spanned in particular by a distance axis (e.g., y-direction) and a velocity magnitude axis (e.g., x-direction). The R.416477 associated with the individual LFI measurement points
[0040] - 10 -
[0041] The measurement signals are preferably velocity values. Advantageously, the individual measurement signals can be separated particularly clearly and cleanly within the distance-velocity space. Over time, due to eye movements and / or shifting of the smart glasses, the distance signal pattern also shifts within the distance-velocity space. Based on the eye's geometry and known movement patterns (eye movement is always continuous, no jumps are possible, etc.), the individual measurement signals of the distance signal pattern can be advantageously tracked over time within the distance-velocity space and unambiguously assigned to one of the LFI measurement points or the associated laser beams. For the continuous tracking of the measurement signals in the distance-velocity space, a model-based approach such as a Kalman filter approach or a particle filter approach can be used.Alternatively, a statistical approach such as a hidden Markov model could be used, which, for example, assigns the LFI measurement points based on changes in velocities and / or distances in the distance-velocity space. For an initial assignment of the measurement signals in the distance-velocity space to the respective LFI measurement points, an additional initial calibration step may be required, in which, for example, a change in the LFI measurement signals is observed when a modulation ramp of the individual LFI sensors of the LFI multipoint sensor system is changed.
[0042] If the individual LFI measurement points of the LFI multipoint sensor system are continuously tracked, so that each distance signal of a currently measured distance signal pattern can be uniquely assigned to a specific LFI measurement point, a change in the depth measurement of the measurement points can be advantageously used to determine a change in the direction of gaze. As already mentioned, the individual LFI measurement points to be tracked should first be identified. This can be done, for example, through an initial calibration in which the wearer looks straight ahead and at various predefined points, and the respective depth signals are recorded. Alternatively or additionally, the initial identification and / or continuous identification can be achieved and / or at least supported by recognizing the behavior of measurement signals from individual LFI measurement points during a consistently similar movement, such as when putting on the smart glasses.R.416477.
[0043] - 11 -
[0044] Additionally, it is proposed that the continuous tracking of the individual LFI measurement points of the LFI multipoint sensor system includes monitoring continuous changes in the distance signals in the distance-intensity space and / or the distance-velocity space. This allows for the advantageous detection of changes in the distance signal corresponding to the pupil being hit by an LFI measurement point. This, in turn, allows for the advantageous identification of the LFI measurement point at which the pupil is currently positioned.
[0045] If, for the purpose of determining gaze direction, a characteristic distance signal identifying the pupil is extracted from a momentarily acquired distance signal pattern, a specific eye property can be advantageously utilized to determine the current gaze direction. This property allows the laser light from the LFI multipoint sensor to penetrate the eye only at the location of the pupil. This enables a reliable and, in particular, error-resistant measurement of gaze direction.
[0046] Therefore, if the characteristic distance signal of a distance signal pattern, which identifies the pupil, is distinguished by a significantly higher distance value compared to the other distance signals of the distance signal pattern, the pupil position can advantageously be reliably extracted from an LFI measurement point array generated on the eye by the LFI multipoint sensor.
[0047] Furthermore, a computing unit, in particular smart glasses with laser feedback interferometry (LFI) multipoint sensors and a computing unit or a data communication link to a computing unit, is proposed, wherein the computing unit comprises at least one processor and at least one data memory with an operating program designed to execute the aforementioned method when run by the processor. This advantageously allows the use of LFI multipoint sensors based on the aforementioned duplication, whose individual LFI measurement points can nevertheless be reliably re-identified. This advantageously enables the creation of a lightweight, cost-effective, compact, energy-saving, and / or user-compatible LFI multipoint sensor, in particular smart glasses, which nevertheless possesses high quality and / or reliability for gaze direction detection.416477.
[0048] - 12 -
[0049] The method and computing unit according to the invention are not to be limited to the application and embodiment described above. In particular, the method and computing unit according to the invention may, to fulfill a function described herein, have a different number of individual elements, components, units, and process steps than specified herein. Furthermore, values within the specified limits of the value ranges stated in this disclosure are also to be considered disclosed and freely usable.
[0050] drawing
[0051] Further advantages will become apparent from the following description of the drawing. The drawing illustrates an embodiment of the invention. The drawing, the description, and the claims contain numerous features in combination. A person skilled in the art will expediently consider the features individually and combine them into meaningful further combinations.
[0052] They show:
[0053] Fig. 1a schematically shows a pair of smart glasses with a computing unit and with exemplary head and / or eye dimensions of a person wearing the smart glasses,
[0054] Fig. 1b schematically shows the person with the data glasses and with further head and / or eye dimensions,
[0055] Fig. 1c schematically shows one eye of the person with further head and / or eye dimensions,
[0056] Fig. 2 schematically shows an LFI multipoint sensor system of the data glasses, Fig. 3 schematically shows exemplary beam directions of the LFI multipoint sensor system,
[0057] Fig. 4a shows a first example of a measurement point pattern of LFI measurement points of the LFI multipoint sensor system.
[0058] Fig. 4b shows a second example of a measurement point pattern of LFI measurement points of the LFI multipoint sensor, R.416477
[0059] - 13 -
[0060] Fig. 4c shows a third example of a measurement point pattern of LFI measurement points of the LFI multipoint sensor system.
[0061] Fig. 4d shows a fourth example of a measurement point pattern of LFI measurement points of the LFI multipoint sensor system.
[0062] Fig. 5 shows a schematic flowchart of a procedure carried out by the computing unit for detecting the gaze direction of the eye of the wearer of the data glasses.
[0063] Fig. 6a shows a schematic representation of the eye in a first viewing direction with a measurement point pattern,
[0064] Fig. 6b shows a schematic representation of a calibration distance signal pattern of the first viewing direction in a distance-intensity space.
[0065] Fig. 7a shows a schematic representation of the eye in a second viewing direction with the measurement point pattern,
[0066] Fig. 7b shows a schematic representation of a calibration distance signal pattern of the second viewing direction in the distance-intensity space.
[0067] Fig. 8a shows a schematic representation of the eye with three gaze directions at different times, showing the measurement point pattern.
[0068] Fig. 8b shows a schematic representation of different calibration distance signal patterns of the three viewing directions in the distance-intensity space.
[0069] Fig. 9 shows an example calibration chart with a variety of calibration distance signal patterns.
[0070] Fig. 10a is a schematic representation of an exemplary current distance signal pattern in the distance-intensity space, Fig. 10b is a schematic representation of an enlarged section with two distance signals of the current distance signal pattern in the distance-intensity space,
[0071] Fig. 11a is a schematic representation of the eye in a first viewing direction with a further exemplary measurement point pattern; Fig. 11b is a schematic representation of the current distance signal pattern in the distance-intensity space for the viewing direction of Fig. 11a, R.416477
[0072] - 14 -
[0073] Fig. 12a shows a schematic representation of the eye in a second viewing direction with the further exemplary measurement point pattern,
[0074] Fig. 12b shows a schematic representation of the current distance signal pattern in the distance-intensity space in the viewing direction of Fig. 12a.
[0075] Fig. 13 shows a schematic example of a distance-velocity space, which can also be used to evaluate the current direction of view, and Fig. 14 shows a schematic setup of an LFI sensor of the LFI multipoint sensor system.
[0076] Description of the exemplary embodiment
[0077] Figures 1a and 1b each schematically depict a person wearing smart glasses 10. The person is a wearer of the smart glasses 10. Figure 1c schematically shows one eye 18 of the wearer of the smart glasses 10. The smart glasses 10 represent an example of an AR headset. The smart glasses 10 include a processing unit 54. In the depicted case, the processing unit 54 is an external processing unit (e.g., a paired smartphone or a paired cloud) that is connected to the smart glasses 10 via a data communication link. Alternatively, the processing unit 54 could also be integrated into the smart glasses 10. The processing unit 54 includes a processor 56. The processing unit 54 includes a data memory 58. An operating program is stored on the data memory 58. The operating program is designed, when executed by the processor 56, to perform a method for detecting the gaze direction of the wearer of the smart glasses 10.The wearer of the data glasses 10 has individual head and / or eye dimensions 60. Several examples of these individual head and / or eye dimensions 60 are shown in Figures 1a to 1c. The abbreviations in the figures mean: IDT: Inter-Distance of Temples (distance between the temples of the data glasses 10); IPD: Interpupillary Distance (pupillary distance); rPD: Right Pupillary Distance (the distance of a right pupil 48 to a central line of the data glasses 10); IPD: Left Pupillary Distance (the distance of a left pupil 48 to the central line of the data glasses 10); W: Width (maximum width of the data glasses 10); DECP: Distance to ear contact point.
[0078] - 15 -
[0079] (The distance of an eye 18 to an ear contact point 62 of the data glasses 10); DCN: Distance to nose contact point (The distance of the eye 18 to a nose contact point 64 of the data glasses 10); D (The distance of the eye 18 from a lens of the data glasses 10); aN (Nasal angle); H (Vertical height of a sclera of the eye 18); L (Horizontal width of the sclera of the eye 18); PS: Pupil size; IS: Iris size; aPS (Angle of scleral vertices to the horizontal).
[0080] The data glasses 10 feature a laser feedback interferometry (LFI) multipoint sensor system 12, as schematically depicted in Figure 2. The LFI multipoint sensor system 12 comprises a plurality of laser sources 24, 26. The laser sources 24, 26 are each formed by LFI sensors. The laser sources 24, 26 are each infrared laser sources. The laser sources 24, 26 are each formed by ViP (VCSEL with integrated photodiode). The LFI multipoint sensor 12 generates a plurality of LFI measurement points 14, 16. The LFI multipoint sensor 12 generates the LFI measurement points 14, 16 all within an area of the data glasses 10 approximately intended for positioning the eye 18 of the wearer. Each of the laser sources 24, 26 generates several of the LFI measurement points 14, 16 of the LFI multipoint sensor 12.More precisely, each of the laser sources 24, 26 generates a laser beam which is split by an optical element of an optical unit 66 into partial beams generating one of the LFI measurement points 14, 16. The beam directions 28, 30 of at least two of the laser sources 24, 26 of the LFI multipoint sensor system 12 are free of mirror symmetry with respect to each other (see Fig. 3).
[0081] The LFI measurement points 14, 16 can form different measurement point patterns in the area of the data glasses 10 intended for positioning the wearer's eye 18. Figure 2 shows an example of a possible measurement point pattern consisting of several concentric circles formed by the LFI measurement points 14, 16. Figure 4a shows an example of another possible measurement point pattern formed by the LFI measurement points 14, 16, consisting of several non-concentric circles distributed in two dimensions. Figure 4b shows an example of another possible measurement point pattern formed by the LFI measurement points 14, 16, consisting of several non-concentric semicircles distributed in two dimensions. Figure 4c shows an example of yet another possible measurement point pattern formed by the LFI measurement points 14, 16.
[0082] - 16 -
[0083] The pattern is formed from several non-concentric circles arranged along a common line. Figure 4d shows another possible measurement point pattern, which is formed as a regular matrix, by means of the LFI measurement points 14, 16.
[0084] The eye 18 reflects the light signals from the LFI measuring points 14, 16. The resulting back-reflection signals 20 are detected by the LFI sensors of the LFI multipoint sensor system 12. The processing unit 54 is designed to evaluate the detected back-reflection signals 20. Based on the evaluation of the detected back-reflection signals 20, the processing unit 54 is designed to determine the gaze direction of the eye 18 of the wearer of the data glasses 10.
[0085] Figure 5 shows a schematic flowchart of a method, performed in particular by the processing unit 54, for detecting the gaze direction of the eye 18 of the wearer of the data glasses 10. In at least one first process part 100 of the method, a back-reflection signal calibration is performed. The back-reflection signal calibration takes place prior to normal gaze direction detection operation of the data glasses 10. In at least one process step 110 of the first process part 100, which takes place prior to active gaze direction detection, the head and / or eye dimensions 60 of the wearer are recorded. The recording of the head and / or eye dimensions 60 is carried out partially or completely by automated detection of the head and / or eye dimensions 60 of the wearer and / or partially or completely by input / response to a query by the wearer of the data glasses 10.
[0086] In at least one process step 120 of the first process part 100, carrier-specific, characteristic distance signal patterns 32 (see Figs. 6b, 7b and 8b) for different calibration gaze directions (see Figs. 6a, 7a and 8a, which each correspond to the distance signal patterns 32 of Figures 6b, 7b and 8b) of the eye 18 are recorded. The carrier-specific, characteristic distance signal patterns 32 recorded in this way are stored in the data storage device 58. Alternatively or additionally to this process step 120, in a further process step 130 of the first process part 100, the recorded and / or recognized head and / or eye dimensions are used to determine the distance signal patterns 32 of the eye 18.
[0087] - 17 -
[0088] In step 60, wearer-specific, characteristic distance signal patterns 42 are calculated for different gaze directions. The head and / or eye dimensions 60 can also be taken into account in the preceding process step 120, in which the wearer-specific, characteristic distance signal patterns 32 for different gaze directions are recorded from calibration measurements, e.g., for selecting the sequence and / or locations of the different calibration gaze directions.
[0089] Figures 6b, 7b, 10a, 11b, and 12b each depict different distance signal patterns 22, 32, and 42. The distance signal patterns 32 shown in Figures 6b, 7b, 10a, 11b, and 12b are each represented in a distance-intensity space 36. In this distance-intensity space 36, distances 38, traveled by laser light from the LFI measurement points 14 and 16 to the eye 18 of the wearer and back to the emitting LFI sensor, are plotted against corresponding light signal intensities 40 detected by the LFI multipoint sensor system 12. The calibration viewing directions shown as examples in Figures 6a and 7a each generate the different distance signal patterns characteristic of the carrier for which the calibration is carried out, as shown in the respective Figures 6b and 7b 32.The characteristic distance signal patterns 32 are unique for each gaze direction and each forms a kind of fingerprint for the corresponding gaze direction, by which the corresponding gaze direction can be re-identified. If different characteristic distance signal patterns 32 are recorded at three different times t = 0, t = 1, and t = 2 of an eye movement, each of these three positions can be re-identified by tracking subsequently recorded distance signal patterns 22. See Figures 8a and 8b.
[0090] In a further process step 140 of the first process part 100, a two-dimensional calibration map 34 is generated based on the characteristic distance signal patterns 32 recorded during the back-reflection signal calibration (see Fig. 9). In a sub-step 141 of process step 140, gaps in the calibration map 34 for which no explicit calibration measurement exists are filled by interpolation from adjacent existing calibration measurements. The calibration map 34 comprises a multitude of characteristic distance signal patterns 32, which can be used in a future viewing direction determination with R.416477.
[0091] - 18 -
[0092] The currently detected back-reflection signals 20 can be compared. Each of the characteristic distance signal patterns 32 of the calibration chart 34 is assigned a specific x-position (x-axis of the calibration chart 34) and a specific y-position (y-axis of the calibration chart 34) of the eye 18 through calibration. From a match between the currently determined distance signal pattern 22 and one of the distance signal patterns 32 of the calibration chart 34, an x- and y-position of the eye 18 can be determined.
[0093] In at least one second process part 200 of the procedure, a gaze direction determination is performed. In at least one process step 210 of the second process part 200, during normal operation of the data glasses 10, the LFI measurement points 14, 16 are generated by the LFI multipoint sensor 12 and projected onto the eye 18. In at least one process step 220 of the second process part 200, the eye 18 generates the back-reflection signals 20 and reflects them back to the LFI multipoint sensor 12. In at least one process step 230 of the second process part 200, the back-reflection signals 20 of the respective LFI measurement points 14, 16 are detected by the LFI multipoint sensor 12. The detected back-reflection signals 20 form the distance signal pattern 22 corresponding to the current gaze direction of the eye 18. These distance signal patterns 22 are different for different gaze directions of the wearer's eye 18. This distance signal pattern 22 can, if necessary,Ambiguities are included (see Fig. 10a). These ambiguities can be detected in substep 231 of process step 230 based on the previously recorded and / or entered head and / or eye dimensions 60, so that the non-physical of the ambiguous signals can be eliminated. Furthermore, in substep 232 of process step 230, a distinction can be made based on an evaluation of the intensities of the back-reflect signals 20, identifying which partial beam of a light signal from a single laser source 24, 26 one of the back-reflect signals 20 originates from. For this purpose, the optical element of the optical unit 66 can be provided, by a suitable selection of an integrated optical function, to equip each partial beam generated by the multiplication with a significantly different intensity.If, as shown in Figure 10b, two back-reflection signals 20 have an identical distance value, they can still be distinguished by their different intensities. R.416477.
[0094] - 19 -
[0095] In at least one process step 240 of the second process part 200, the currently detected distance signal pattern 22 is evaluated to determine the current gaze direction. Two methods can be used for this purpose, which can be performed separately or in combination. In a first method, which constitutes a sub-step 241 of process step 240, the currently detected distance signal pattern 22 is compared with the characteristic distance signal patterns 32 known from the previously performed back-reflection signal calibration and / or with distance signal patterns 42 calculated from the head and / or eye dimensions 60. Based on a match between the currently detected distance signal pattern 22 and one of the known characteristic distance signal patterns 32, 42, the current gaze direction is then determined.
[0096] In a second method, which forms a sub-step 242 of process step 240, the individual LFI measurement points 14, 16 of the LFI multipoint sensor system 12 are continuously tracked, so that each distance signal 52 of a currently measured distance signal pattern 22 can be uniquely assigned to one of the LFI measurement points 14, 16 of the multipoint sensor system 12 (see Figures 11a to 12b). Since the movement of the eye 18 is stepless, the movements of the distance signals 52 in the distance-intensity space 36 are also stepless. The continuous tracking of the individual LFI measurement points 14, 16 of the LFI multipoint sensor system 12 includes monitoring of continuous changes in the distance signals 52 in the distance-intensity space 36. For the evaluation of the gaze direction determination, a characteristic distance signal 50, which identifies the pupil 48, is then extracted from the currently recorded distance signal pattern 22.The characteristic distance signal 50 of a distance signal pattern 22, which identifies the pupil 48, is distinguished by a significantly higher distance value compared to the other distance signals 52 of the distance signal pattern 22, since in this case the light signal is reflected not from an ocular surface of the eye 18, but from a retina inside the eye 18. Figure 11a shows, by way of example, a straight-looking eye 18 and the LFI measurement points 14, 16 located on the eye 18. In this case, a central LFI measurement point 14 hits the pupil 48, while an adjacent LFI measurement point 16 hits the eye 18 next to the pupil 48. Figure 11b shows the corresponding current distance signal pattern 22. The distance-R.416477.
[0097] - 20 -
[0098] The distance signal 52 of the LFI measurement point 14, which lies on the pupil 48, is clearly the distance signal 52 with the greatest distance. The distance signal 52' of the LFI measurement point 16, which lies next to the pupil 48, is clearly at a smaller distance than the distance signal 52 located on the pupil 48. Based on knowledge of the position of the corresponding LFI measurement point 14 (through continuous tracking of all LFI measurement points 14, 16), the gaze direction of the eye 18 can now be determined. Figure 12a shows, by way of example, a sideways-looking eye 18 and the same LFI measurement points 14, 16 located on the eye 18 as in Figure 11a. In this case, a mean LFI measurement point 14 no longer coincides with the pupil 48, while the adjacent LFI measurement point 16 lies on the pupil 48 of the laterally looking eye 18. Figure 12b shows the corresponding current distance signal pattern 22 of the laterally looking eye 18.The distance signal 52' of the LFI measurement point 16, now located on pupil 48, which was previously far to the left (small distances) in the distance-intensity space 36, has now shifted completely to the right in the distance-intensity space 36 (large distances). The distance signal 52, which is no longer located on pupil 48, has, however, shifted significantly to the left in the distance-intensity space 36 (smaller distances). This makes it clear that a different distance signal 52', namely that of measurement point 16, which was previously not located on pupil 48, now originates from pupil 48, and thus the gaze direction has changed. Based on knowledge of the position of the corresponding LFI measurement point 16 (achieved through the continuous tracking of all LFI measurement points 14 and 16), the new gaze direction of eye 18 can now be determined. In at least one further process step 250, the currently determined viewing direction is used by the computing unit 54 for further use, e.g.output by software of the data glasses 10.
[0099] Alternatively or additionally to the distance-intensity space 36, the distance-velocity space 44 shown in Figure 13 can also be used to evaluate the gaze direction. In this case, the distance signal pattern 22, 32, 42 comprises patterns in the distance-velocity space 44, in which distances 38, traveled by the laser light from the LFI measurement points 14, 16 to the eye 18 of the wearer, are plotted against the corresponding Doppler velocities 46 detected by the LFI multipoint sensor 12.
[0100] - 21 -
[0101] Figure 14 schematically depicts an exemplary single LFI sensor of the LFI multipoint sensor system 12. The LFI sensor comprises the laser source 24, which generates and emits the laser signal. At the eye 18, the emitted laser signal is reflected and partially backscattered to the LFI sensor as the back-reflection signal 20. At the LFI sensor, the back-reflection signal 20 re-enters a laser cavity of the laser source 24 and modulates the resonating laser signal therein. This modulation can be detected by a photodiode 68 of the LFI sensor or by electronics of the LFI sensor, generating the distance signal pattern 22.
Claims
R.416477 - 22 - Claims 1. A method for detecting the gaze direction of a wearer of smart glasses (10), in particular an AR headset or a VR headset, which comprises a laser feedback interferometry (LFI) multipoint sensor (12), wherein the LFI multipoint sensor (12) generates a plurality of LFI measurement points (14, 16) in an area of the smart glasses (10) provided for positioning an eye (18) of the wearer, and wherein the LFI multipoint sensor (12) detects back-reflected signals (20) from the LFI measurement points (14, 16) reflected by the eye (18) of the wearer, characterized in that a characteristic distance signal pattern (22), which is different for different gaze directions of the eye (18) of the wearer, is read out from the back-reflected signals (20) and evaluated for gaze direction determination.
2. Method according to claim 1, characterized in that several of the LFI measuring points (14, 16) of the LFI multipoint sensor (12) are generated by identical laser sources (24, 26) of the LFI multipoint sensor (12).
3. Method according to claim 2, characterized in that beam directions (28, 30) of at least two of the laser sources (24, 26) of the LFI multipoint sensor system (12) are at least free of mirror symmetry relative to each other.
4. Method according to one of the preceding claims, characterized in that a back-reflection signal calibration is performed prior to the gaze direction detection, in which, in particular carrier-specific, characteristic distance signal patterns (32) are recorded for different gaze directions.
5. Method according to claim 4, characterized in that the characteristic R.416477 recorded during the back-reflection signal calibration - 23 - Distance signal patterns (32) generate a calibration map (34), in particular a two-dimensional one, which is used to evaluate the currently recorded back reflection signals (20) when determining the direction of view.
6. Method according to claim 5, characterized in that gaps in the calibration card (34), for which no explicit calibration measurement exists, are filled by means of interpolation from neighboring calibration measurements.
7. Method according to one of the preceding claims, characterized in that, prior to gaze direction detection, head and / or eye dimensions (60) of the wearer are recorded, on the basis of which, in particular wearer-specific, characteristic distance signal patterns (42) for different gaze directions are calculated and / or adapted.
8. Method according to one of claims 4 to 7, characterized in that, for the evaluation of the gaze direction determination, a distance signal pattern (22) currently detected by the LFI multipoint sensor (12) is compared with the characteristic distance signal patterns (32) known from the wearer-specific calibration, each of which represents a specific gaze direction of the wearer, in particular with the calibration card (34), which assigns a characteristic distance signal pattern (32) to each gaze direction, or that, for the evaluation of the gaze direction determination, the distance signal pattern (22) currently detected by the LFI multipoint sensor (12) is compared with characteristic distance signal patterns (42) calculated for the wearer on the basis of the head and / or eye dimensions (60), each of which represents a specific gaze direction of the wearer.
9. Method according to one of the preceding claims, characterized in that the distance signal pattern (22, 32, 42) comprises at least one pattern in a distance-intensity space (36), wherein in the distance-intensity space (36) distances (38) traveled by a laser light from the LFI measuring points (14, 16) to the eye (18) of the wearer are detected by the respective LFI multipoint sensor (12). - 24 - Intensities (40) are plotted.
10. Method according to one of the preceding claims, characterized in that the distance signal pattern (22, 32, 42) comprises at least one pattern in a distance-velocity space (44), wherein in the distance-velocity space (44) distances (38) traveled by a laser light from the LFI measuring points (14, 16) to the eye (18) of the wearer are plotted against the respective Doppler velocities (46) detected by the LFI multipoint sensor (12).
11. Method according to one of the preceding claims, characterized in that the individual LFI measuring points (14, 16) of the LFI multipoint sensor system (12) are continuously tracked, so that each distance signal (52) of a currently measured distance signal pattern (22) can be uniquely assigned to an LFI measuring point (14, 16) of the multipoint sensor system (12).
12. Method according to claim 9 and / or claim 10 and claim 11, characterized in that the continuous tracking of the individual LFI measuring points (14, 16) of the LFI multipoint sensor system (12) comprises monitoring of continuous changes in the distance signals (52) in the distance-intensity space (36) and / or in the distance-velocity space (44).
13. Method according to claim 12, characterized in that, for the evaluation of the gaze direction determination, a characteristic distance signal (50) characterizing the pupil (48) is extracted from a momentarily detected distance signal pattern (22).
14. Method according to claim 13, characterized in that the characteristic distance signal (50) of a distance signal pattern (22), which characterizes the pupil (48), is distinguished by a significantly higher distance value compared to the remaining distance signals (52) of the distance signal pattern (22). R.416477 - 25 - 15. Computing unit (54), in particular data glasses (10) with a laser feedback interferometry (LFI) multipoint sensor (12) and with a computing unit (54) or with a data communication link to a computing unit (54), wherein the computing unit (54) comprises at least one processor (56) and at least one data storage device (58) with an operating program which, when executed by the processor (56), is designed to execute a method according to one of the preceding claims.