Method for detecting a viewing direction, and vehicle

Abstract

The invention relates to a method for detecting a viewing direction, wherein at least one eye (1) of a user (2) is actively illuminated by a light source (3), the at least one eye (1) is captured by a camera (4), and the viewing direction (10) is determined while taking into account the known relative position of the at least one eye (1) with respect to the camera (4) and the geometric relationships of the reflections of the light source (3) detected on the eyeball in respective camera images. The method according to the invention is characterized by the following method steps: casting and capturing a light pattern (5) extending horizontally over the width of the eyeball, the light pattern (5) being composed of a plurality of strip-shaped segments (S), and the ratio of the widths (b) of the segments (S) to each other in the circumferential direction (U) being known; determining the width (b) of the captured segments (S); and determining the viewing direction (10) while taking into account the captured width (b) of the segments (S) and the fact that the eyeball is curved to a lesser extent in the region of the pupil than in the edge region.

Description

[0001] Mercedes-Benz Group AG

[0002] Methods for gaze direction detection and vehicle

[0003] The invention relates to a method for detecting the direction of gaze according to the type defined in more detail in the preamble of claim 1 and to a vehicle for carrying out the method.

[0004] Modern vehicles are equipped with a wide variety of driver assistance functions to enhance safety and comfort. These systems may require information about the head position and gaze direction of a vehicle occupant to function correctly. This is necessary, for example, for displaying augmented reality information on a head-up display (HUD) or for detecting fatigue and / or inattention. Furthermore, tracking the driver's gaze direction allows the system to determine whether they are recognizing relevant traffic objects in time.

[0005] To detect the direction of gaze, the person's eye area is typically captured by a camera located inside the vehicle. This camera is usually integrated into the instrument cluster, the headliner, or the rearview mirror. Since lighting conditions can change rapidly while driving, particularly due to shadows or light rays from the sun or streetlights entering the interior, the eye area can be actively illuminated, especially with an infrared light source. Various approaches exist for calculating the direction of gaze. One example is the so-called Purkinje projection. In this method, reflections are created on the cornea of ​​the user using a light source. The light emitted by the light source is reflected off specific areas of the retina. These reflections are then captured by the camera.The relative positions of the various reflections are recorded, and the gaze direction is calculated from this using a suitable algorithm. A disadvantage of this method is the requirement for a dedicated light source to illuminate the eyes. Furthermore, the area where reflections can be generated on the cornea is spatially limited, making it difficult to accurately determine the gaze direction during larger head rotations. Data-driven approaches are also known, in which the gaze direction is determined using appropriately trained artificial neural networks (ANNs).

[0006] The corresponding KNNs (Knowledge Neighborhood Networks) are fed 2D images of the vehicle occupant's eyes. Convolutional Neural Networks (CNNs) are most commonly used for this purpose. These artificial neural networks are usually trained using the results of Purkinje detection as their baseline. A disadvantage of this approach is the large amount of training data required and the high risk of inaccurate gaze direction determination under fluctuating operating conditions.

[0007] Therefore, there is a need to specify methods and means for reliably and easily determining the direction of gaze in vehicles.

[0008] A system and method for driver monitoring are disclosed in US patent 2019 / 0318181 A1. Data generated by an image sensor is processed by a processor in a computing unit. The driver's gaze direction is determined from this data. For this purpose, characteristics of the driver's eyes are extracted from the corresponding image data. To correctly calculate the gaze direction, the driver's head position within the vehicle interior must be determined. Relevant parameters include the radius of corneal curvature and the distance of the pupil plane to the corneal center of curvature. The optical axis of the eye is determined, from which the gaze direction is then inferred. This determined gaze direction is then fed as input data to downstream driver assistance systems.

[0009] Furthermore, WO 2004 / 034905 A1 discloses a method and arrangement for interpreting a subject's head and eye activity. This involves processing information characterizing the orientation of the eyes and / or head. First, the position of a driver's head relative to a reference base position is determined. A normalization step is then performed, which allows for the identification of positions of interest to the driver. These positions represent relevant objects viewed by the driver. Objects considered relevant are those with a high visual density. In this way, spatial regions are identified into which the driver looks most frequently.

[0010] Furthermore, DE 101 18314 A1 discloses a method for spatially aligning images taken at different times. This method can be used to monitor the progress of laser eye surgery. For this purpose, a portion of the corneal surface is cut with a blade, and the resulting flap is folded to the side to create a flat cut edge. A fluorescent material is applied to the eye and excited by UV light to emit light. The UV light has the form of a striped pattern. This fluorescent light is captured by a camera. By analyzing changes in the spacing of the stripes in successive camera images, it is possible to calculate which part of the cut edge has risen or fallen and to what extent. From this, the change in the eye's position can be deduced.

[0011] Furthermore, WO 2019 / 117000 A1 discloses an image processing device and method. The device enables the detection of a driver's gaze direction. Reflections from a light source on the driver's eye are captured by a camera.

[0012] Furthermore, WO 2017 / 079172 A1 discloses a gaze direction detection method using structured light in the context of virtual reality glasses. Infrared light is used for the projection of the light.

[0013] The present invention is based on the objective of providing an improved method for gaze direction detection, which is characterized by reliable and robust gaze direction detection while simultaneously being easy to integrate into a vehicle.

[0014] According to the invention, this problem is solved by a method for gaze direction detection with the features of claim 1. Advantageous embodiments and further developments, as well as a vehicle for carrying out the method, are described in the dependent claims. A generic method for gaze direction detection, wherein at least one eye of a user is actively illuminated by a light source, the at least one eye is detected by a camera, and the gaze direction is determined taking into account the known relative position of the at least one eye with respect to the camera and the geometric relationships of the reflections of the light source detected on the eyeball in the respective camera images, is further developed according to the invention by the following method steps:

[0015] Projecting and capturing a light pattern extending horizontally across the width of the eyeball, wherein the light pattern is composed of a multitude of strip-shaped segments, the ratio of the widths of the segments to each other in the circumferential direction being known;

[0016] Determining the width of the captured segments; and

[0017] Determining the direction of gaze, taking into account the measured width of the segments and the fact that the eyeball is less curved in the area of ​​the pupil than in the peripheral area.

[0018] The human eyeball is approximately spherical. In the area of ​​the pupil, the eyeball, or at least the part of the eye that reflects light particularly well, is less curved or flattened. This fact can be used to determine the direction of gaze. The core idea of ​​the invention is to project a light pattern onto the eyeball, the geometric design of which follows a defined and therefore known pattern. The reflection of the light pattern on the eyeball is then captured by a camera, and it is determined in which regions of space the light pattern is distorted and to what extent by the curvature of the eyeball. Due to the comparatively high curvature in the peripheral region of the eyeball, the individual segments of the light pattern are more strongly distorted there and thus wider, while in the area of ​​the pupil they are less distorted or not distorted at all, resulting in them hardly widening or not widening at all.By algorithmically analyzing corresponding camera images, it is possible to determine the region in the image where the pupil of the viewed eye is located. Since the relative position of the eyes to the camera is known, particularly because the user's head is in a known area, the direction of gaze can then be reconstructed, at least in the horizontal plane.

[0019] According to the invention, a strip-shaped light pattern is used, which is divided into individual segments. To detect distortion of the light pattern, both the width of the respective segments and the distance between the segments can be taken into account. In other words, the width of the unilluminated areas between the light segments can also be used for analysis. The ratio of the widths of the segments to each other, as well as their respective circumferential distances, is known. This means that information is available about how the circumferential extent changes from segment to segment. "Circular direction" refers to the direction in which the eyeball extends laterally, i.e., horizontally.

[0020] The width of each segment, i.e. its extent in the circumferential direction, can continuously increase or decrease from segment to segment, or follow a chaotic but known pattern.

[0021] However, an advantageous further development of the method according to the invention provides that segments of the same width are projected onto the eye and the angular position at which the shortest segment is detected is assumed to be the horizontal angle of the gaze direction. In the simplest case, the width of the segments thus does not change from segment to segment, but remains constant. This allows for a simpler design of the light pattern and also makes evaluation particularly easy and reliable. Since, as already mentioned, the eyeball is more curved at the periphery than in the pupil area, the light pattern is thus more distorted at the periphery. Accordingly, the segments of the light pattern are particularly wide in the reflection at the periphery and particularly narrow in the pupil area. The detected width of the reflections of the segments of the light pattern decreases continuously from the periphery towards the pupil and increases continuously from the pupil towards the periphery.This also applies to the distance between the reflected segments if the corresponding segments of equal width also have the same distance to each other in the circumferential direction.

[0022] Preferably, a length gradient is calculated circumferentially between the segments, describing the change in width for successive segments. The angular position at which the minimum of the length gradient is detected is then assumed to be the horizontal angle of the viewing direction. As previously described, the width of the segments on the eyeball, or rather their distance from each other, changes continuously from the periphery towards the pupil. By determining the length gradient circumferentially, the angular position of the shortest segment can be determined even more reliably. The difference in length between the detected reflections of segments is particularly large at the periphery of the eyeball and particularly small near the pupil. This is because a large number of segments lie close together in this area and are therefore less distorted.The longitude gradient determined for each segment can be interpolated to generate a continuous latitude or longitude gradient curve. This makes it possible to determine the position of the pupil of the eyeball relatively accurately, even when the pupil is not directly superimposed on a segment of the light pattern.

[0023] A further advantageous embodiment of the method according to the invention provides that adjacent segments differ in their color, brightness, and / or contrast. This makes it easier to distinguish adjacent segments from one another in corresponding camera images. This reduces the risk of adjacent segments being misinterpreted as a single, continuous segment. For example, two, three, four, or even more brightness and / or contrast levels can be present in the light pattern. The segments can be designed in one, two, three, or even more colors. For example, all segments of the light pattern can be blue. The segments can also be designed to alternate, for example, red and blue. The sequence of the element colors could also be, for example, red, green, blue. Other colors such as orange, pink, turquoise, and the like are also possible.

[0024] Typically, each photodiode of a corresponding image sensor is assigned a color filter. For example, these could be red, green, and blue color filters. Other colors such as yellow, cyan, magenta, and the like are also known. Preferably, the colors used to create the light pattern are matched to the respective colors of the image sensor's color filter elements. This allows individual segments of the light pattern to be detected by individual photodiodes of the image sensor or color channels. Thus, a segment of the light pattern in a specific color elicits a particularly strong signal in one of the three color channels. This allows the corresponding segments to be distinguished from one another even more reliably.

[0025] In this embodiment, the light pattern can also be designed such that segments lie seamlessly against each other in the circumferential direction, i.e., without any gap. In this case, however, at least the color, brightness, and / or contrast of two directly adjacent segments must be changed. Corresponding segments can also be spaced apart from each other, so that dark areas lie between the segments in the circumferential direction.

[0026] In this process, at least one segment is preferably generated using infrared light.

[0027] Infrared light is invisible to humans. This allows for a particularly discreet projection of the light pattern, making it imperceptible to the human eye. This increases comfort and safety when using the method according to the invention. Firstly, the user is not dazzled by the projection of the light pattern. Secondly, the light pattern is not perceptible, which could be distracting.

[0028] In particular, only a small proportion of infrared light passes through the windows of a vehicle into its interior. Therefore, when applying the method according to the invention in a vehicle, segments implemented in the infrared spectrum are easier to detect than segments implemented in the visible spectrum, since the risk of being overwhelmed by external light sources is particularly low.

[0029] A further advantageous embodiment of the method according to the invention further provides that all segments of the light pattern are projected simultaneously and depicted in a single camera image; or

[0030] At least a subset of the segments are projected sequentially, with the projection frequency of the light source and the recording frequency of the camera being synchronized so that the light pattern is composed of the reflections captured in at least two consecutive camera images. This allows for the use of differently configured systems. The light pattern can thus be projected onto the user's eyes, or into the vehicle interior, for a longer period than the camera's image capture duration, for example, for 100 ms, one second, or continuously. The camera's image capture frequency can be, for example, 30 frames per second, 60 frames per second, or fractions or multiples thereof. This ensures that the light pattern is present in each camera image.

[0031] The light pattern can also be projected into the vehicle interior using pulse-width modulation. In this case, the projection frequency is synchronized with the recording frequency. This means that the light pattern is projected into the vehicle interior precisely when the camera is recording corresponding images. This allows for differentiation between immediately adjacent segments not only based on visual characteristics such as differing color, brightness, and / or contrast, but also by the fact that adjacent segments are located in different camera images. For example, only one segment of the light pattern can be projected per camera image. This single segment then travels around the entire width of the light pattern in a circumferential direction. However, it would also be possible to project multiple segments simultaneously, such as two, three, or even more.Even with multiple projections of segments, these can rotate circumferentially in successive camera images.

[0032] According to a further advantageous embodiment of the method according to the invention, it is further provided that the light pattern is projected in a manner imperceptible to humans. This is particularly easy to achieve using infrared light as the light source. However, a projection of the light pattern imperceptible to humans can also be achieved using light in the visible spectrum. In this case, the projection of the light pattern occurs for such a short period that it is imperceptible to humans. In particular, the projection frequency is higher than the critical flicker fusion frequency of the human eye.

[0033] A further advantageous embodiment of the method according to the invention further provides that, in order to determine a vertical angle of the viewing direction:

[0034] the course of the distorted light pattern on the eyeball is determined for various predetermined vertical angles, whereby the courses thus known as a function of the respective vertical angles are stored, and the vertical angle to be determined is determined by comparing the currently detected course with the known courses of the light pattern; or

[0035] a second light pattern extending vertically over the height of the eyeball is projected onto the eyeball, and the vertical angle is determined taking into account the known length ratio of corresponding segments in the vertical direction to each other.

[0036] To determine the horizontal angle, a strip-shaped light pattern extending horizontally across the eyeball is projected. To also determine a vertical angle, a second light pattern, designed to match the first but oriented vertically, could be projected onto the eyeball. The projection directions of the first and second light patterns are then orthogonal to each other. Due to the near-spherical shape of the eye, the segments further outward in the second light pattern—that is, the segments higher and lower—are more distorted than those located near the pupil. This necessitates the projection of a corresponding second light pattern, which increases the complexity of the underlying system. However, this method allows for a particularly precise measurement of the vertical angle of gaze.

[0037] It is also advantageously possible to determine the vertical angle solely by projecting the horizontally oriented light pattern. According to the invention, the light pattern is projected onto the eyeball based on the known location of the user's eyes, such that when looking straight ahead, it runs horizontally or centrally, i.e., virtually "near the equator," across the eyes. If the user looks up or down, the horizontal light pattern runs above or below the "equator" of the eyeball, respectively. This results in a vertical displacement of the light pattern relative to the pupil. Since the eyeball becomes more curved with increasing vertical distance from the "equator," the change in the width of the segments also increases. The effect of the segments being hardly distorted, or not distorted at all, due to the flattened pupil, therefore decreases with increasing distance from the "equator." This means that the segments have a greater width at this distance.The longitudinal gradient assumes a characteristic shape. For different vertical angles, the corresponding profiles of the segment widths or the respective longitudinal gradients can be recorded and stored. By comparing the currently measured profile with the stored profiles, the vertical angle can then be determined.

[0038] Preferably, the progression of the distorted light pattern on the eyeball for different vertical angles can be collected through a large number of calibration measurements; and / or

[0039] The vertical angle can be determined using an alternative method for determining the direction of gaze, and this determined vertical angle can then be correlated with the corresponding path of the distorted light pattern. During such a calibration measurement, the user looks at a specific target located within a predefined region of space. This causes the gaze to assume a predetermined vertical angle. Simultaneously, the light pattern is projected onto the user's eye(s). Thus, for each predefined vertical angle, a characteristic path of the light pattern can be recorded, allowing a correlation between the vertical angle and the characteristic path of the light pattern on the eyeball to be established.

[0040] When alternative gaze direction determination methods are used, it is not necessarily required that the user views a specific target at a fixed, predetermined vertical angle. The vertical angle present in the respective recording situation can be determined using the alternative gaze direction determination method and correlated with the detected path. Combinations are also possible, in which the corresponding vertical angle is both predetermined and subsequently determined using the alternative gaze direction determination method for verification.

[0041] The method according to the invention for gaze direction detection can be implemented in a vehicle to determine the gaze direction of a vehicle occupant, such as the driver. The calibration measurements described above can be carried out in extensive measurement campaigns, for example, initiated by a vehicle manufacturer. Such measurement campaigns can be model-specific or cross-series. Calibration can also be performed individually for each vehicle, particularly immediately after production. Alternatively, the vehicle can be purchased by a user and then individually calibrated before, during, or after its initial use by that user. In this case, the user can be guided through the calibration measurement process by being provided with suitable information via an appropriate user interface, such as a graphical display.

[0042] To ensure the correct vertical angle is maintained, visual markers can be placed in the surroundings or inside the vehicle, or projected as needed. For example, a laser can project a point into the environment for the user to look at. This point is then tilted vertically up or down to capture the light pattern for different vertical angles. Alternatively, the user could be asked to look at components mounted in fixed locations within the vehicle, such as the rearview mirror, the instrument cluster, a rotary push-button controller in the armrest area, and so on. Optionally, the horizontal angle can also be calibrated.

[0043] All proven methods are suitable as alternative methods for determining gaze direction, in particular the gaze direction detection described at the beginning, based on the Purkinje projection, and / or the gaze direction detection based on the use of appropriately trained artificial neural networks.

[0044] This approach is particularly advantageous because it allows for the simultaneous execution of various gaze direction determination methods. In other words, different sensor systems are available for capturing the gaze direction, with corresponding sensor fusion taking place. This enables even more precise gaze direction detection.

[0045] In a vehicle of this type, comprising a light source and camera directed towards the area occupied by a person driving the vehicle, and a computing unit provided for controlling the light source and for processing camera images generated by the camera, the light source, the camera, and the computing unit are configured, according to the invention, to carry out a method described above. This means, in particular, that the computing unit has at least read access to a computer-readable storage medium comprising machine-interpretable instructions which, when executed by the processor of the computing unit, cause it to provide or execute the corresponding method according to the invention. The vehicle can be any road vehicle such as a car, truck, van, bus, or the like. Generally, it could also be a rail vehicle, watercraft, or aircraft.

[0046] The light source and the camera are installed at known positions and in a known orientation relative to each other. Therefore, the corresponding geometric relationships are known, which is a necessary prerequisite for correctly calculating the viewing direction. It is also essential to consider the user's eye position within the vehicle interior. Several methods are possible for this. The user's eye position can be estimated or assumed. For example, the so-called 95% occupancy zone or occupancy box can be used as a reference. This refers to the volume of space within the vehicle interior where the head of 95% of occupants is located. Alternatively, the head or eye position within the vehicle interior can be detected using sensors.Proven sensor systems can be used for this purpose, in particular sensor systems based on camera-based head position determination and / or depth information. Distance sensors, such as radar sensors, laser scanners, and the like, can be used to generate the corresponding depth information.

[0047] According to an advantageous embodiment of the vehicle according to the invention, the light source extends horizontally across the vehicle interior in the form of a light band, particularly as ambient lighting. If the light source extends across the vehicle interior in the form of a light band, it is possible to illuminate the user's eyeball, or both eyeballs, from different regions of the interior. In previously known systems, only a single point light source is provided, usually integrated into the instrument cluster. As mentioned at the outset, light reflections on the user's cornea are only inadequately produced during relatively large head movements.

[0048] According to the invention, however, the eyeball can be illuminated from different regions of space, so that even with correspondingly large head rotations, detectable light patterns are ensured on the eyeball. This improves the reliability of the inventive method.

[0049] Modern vehicles typically already feature ambient lighting. Therefore, if such ambient lighting is used according to the invention, the integration of dedicated light sources can be dispensed with. This allows for easy integration into a wide variety of vehicles. Furthermore, the vehicle's manufacturing costs are reduced, as such dedicated lighting elements can be eliminated. The main purpose of such ambient lighting is to create a specific lighting atmosphere in the vehicle interior or to emit warning light signals. As already described, the projection of the light pattern should preferably be imperceptible to humans. Thus, the projection of the light pattern is so brief that the vehicle occupants do not perceive any change in the ambient lighting.

[0050] Ideally, the light source in the vehicle interior extends within an angle of at least 180° around the driver. This covers a particularly large angular range, allowing for even more reliably detectable projections of the light pattern onto the eyeball.

[0051] According to a further advantageous embodiment of the vehicle according to the invention, the processing unit is also configured to supply the gaze direction as an input variable to a downstream driver assistance system. Using the method according to the invention, the gaze direction can be determined even more reliably than with conventional systems. This increases the robustness of the corresponding downstream driver assistance systems. In particular, the gaze direction can be determined with increased accuracy in a wide variety of operating situations according to the invention. Thus, the corresponding driver assistance systems can also be operated more reliably in the respective situations. In particular, the detection capability of the light pattern, and therefore also the determination of the gaze direction, is improved in darkness, for example at night, as well as in adverse lighting and weather conditions.

[0052] Further advantageous embodiments of the inventive method for detecting the direction of gaze and of the inventive vehicle also result from the exemplary embodiments, which are described in more detail below with reference to the figures.

[0053] This shows:

[0054] Fig. 1 is a schematic top view of a vehicle according to the invention;

[0055] Fig. 2 shows a flowchart of a method according to the invention for detecting the direction of gaze;

[0056] Fig. 3 shows a schematic sequence of the individual process steps in greater detail; and Fig. 4 shows a schematic representation of different embodiments of a light pattern projected onto the eyeball of a user in the course of the method according to the invention.

[0057] Figure 1 shows a top view of a vehicle 7 according to the invention. The vehicle incorporates a method according to the invention for detecting the direction of gaze. For this purpose, the vehicle 7 comprises a camera 4 for detecting at least the eye area of ​​a user 2, a light source 3 for projecting a light pattern 5, shown in the following figures, onto the eyes 1 or one eye 1 of the user 2, and a processing unit 8 for controlling the light source 3 and for processing camera images generated by the camera 4. The user 2 assumes a typical position in the vehicle interior. In particular, the head of the user 2 is located in a typical area, also referred to as the head compartment. The head position can also be detected using a suitable sensor system (not shown), which allows for a particularly precise determination of the position of the eyes 1.Based on the known position of the eyes 1, the camera 4 and their detection range in the vehicle interior, as well as the direction of propagation of the light emitted by the light source 3, it is possible to determine the direction of gaze depending on the light pattern 5 thrown by the light source 3 onto the eyes 1.

[0058] The light source 3 is preferably implemented as so-called ambient lighting. This ambient lighting preferably extends 180° around the user 2, making it possible to project a corresponding light pattern 5 onto the eyes 1 even during large head rotations by the user 2. Such ambient lighting comprises a multitude of individual light sources 9, for example in the form of LEDs, OLEDs, or the like, which are arranged at the same height in the vehicle interior around the user 2. Sections of such an LED strip can, for example, be connected to or integrated into the dashboard, attached to a door panel, and so on. Figure 1 shows the beam path of some exemplary individual light sources 9.

[0059] The procedure is explained with reference to Figure 2. In step 201, camera 4 can be calibrated if this has not already been done. In this calibration step, the position and detection range of camera 4 relative to the vehicle interior are checked and recorded. This allows camera 4 to be located relative to vehicle 7, particularly within a common vehicle-fixed reference coordinate system.

[0060] In a subsequent step 202, the computing unit 8 controls the light source 3 to project the light pattern 5 into the vehicle interior.

[0061] In step 203, which can be performed in parallel with step 202, camera 4 records images. Preferably, the recording frequency of camera 4 and the projection frequency of light source 3 are coordinated such that camera 4 records an image precisely when the light pattern 5 is projected into the vehicle interior. However, the light pattern could also be projected continuously.

[0062] In the subsequent step 204, the light pattern 5 on the eyeball, i.e., the reflection, is detected in the respective camera images, and its geometric relationships are determined. According to the invention, the light pattern 5 is a strip-shaped light pattern that extends horizontally across the width of the respective eyeball of the user 2. The light pattern 5 comprises several segments S, shown in the following figures, each having a defined width b. The circumferential distance U between the segments S is also defined.

[0063] In particular, all segments S in the projection have the same width b.

[0064] It is also particularly preferred that the distance between the segments S be of equal size, especially preferably at a height of the width b.

[0065] The core idea of ​​the invention is that, due to the near-spherical shape of the eyeball, the reflection of the light pattern 5 is more distorted at its periphery than in the pupil area. Thus, the direction of gaze, or the horizontal angle at which the user 2 looks into the environment, can be determined by identifying the region in corresponding camera images where the reflection of the light pattern 5 on the eyeball is least distorted. The segments S of the reflection have the smallest width b in the pupil area and a larger width b in the periphery. In the embodiment shown in Figure 2, a length gradient is determined in step 205 to identify the shortest segment S. The length gradient is the degree to which the width of the individual segments S of the reflection changes from segment to segment in the circumferential direction U. The shortest segment S lies at the minimum of the length gradient.

[0066] In step 206, this minimum of the length gradient is determined, and in the subsequent step 207, the viewing direction is calculated from it. In this case, the viewing direction is the horizontal angle, i.e., a solid angle measured in a plane parallel to the ground, with which user 2 looks into the surroundings.

[0067] In a subsequent step 208, a vertical angle of the gaze direction can also be determined. For this purpose, a corresponding second light pattern (not shown) running vertically over the eye(s) of user 2 can be projected, and an analogous procedure for angle determination can be carried out.

[0068] Alternatively or additionally, the vertical angle can be deduced from a characteristic course of the horizontally running light pattern 5, which will be discussed later.

[0069] In step 209, the gaze direction determined in this way is fed as input data to downstream driver assistance systems. Steps 202 to 209 and 201 to 209, respectively, can be repeated cyclically to track the gaze direction.

[0070] The individual process steps are shown in detail in Figure 3 in sub-figures a) to h).

[0071] Figure 3a) shows step 202, i.e., controlling the light source 3 or...

[0072] Ambient lighting is used to project the light pattern 5 onto the eyeball of user 2. The projection can be onto one eye 1 or both eyes 1. Each individual light source 9 of the light source 3 projects light into a defined area of ​​the vehicle interior. In the illustrated embodiment, the width b of all individual light sources 9 is the same in the projection. The reflection of the light pattern 5 on the eye 1 of user 2 is shown in Figure 3b). This reflection is captured in step 203. The individual segments S of the reflection are longer, i.e., wider, at the edges than in the pupil area. The light pattern 5 is projected onto the eyes 1 in such a way that it extends horizontally across the center of each eye, preferably near the equatorial position when user 2 is looking straight ahead.In Figure 3a), only a few alternating individual light sources 9 are operated to project the light pattern 5, while in Figure 3b), several immediately adjacent individual light sources 9 are operated as an example.

[0073] Figure 3c) shows step 204, in which the individual widths b of the segments S of the light pattern 5 are determined. The viewing direction 10, which is set in a horizontal plane, is indicated by a dashed arrow.

[0074] Figure 3d) shows step 205 in detail. Two diagrams are shown, with the upper diagram plotting the length gradient 6 over the respective segment S. The axis label LG stands for "length gradient," while the lower diagram, and all subsequent diagrams, show the magnitude of the length gradient 6. The axis label S / P stands for "segment / pixel." If the width b is determined in the circumferential direction U, the width b decreases continuously towards the pupil, reaching its minimum in the pupil area. The width b then increases again. The length gradient 6 describes the degree to which the width b changes from segment to segment. Thus, the sign changes in the graph, as the change in length is initially negative (width b decreases) and then positive (width b increases).In the pupil area, due to the low distortion, more segments S with similar width b are present in a smaller space, so that the length gradient 6 here approaches the value "0". By considering the length gradient 6, the "shortest" segment 6 can be found in a robust manner.

[0075] Figure 3e) shows step 206 in detail. The horizontal viewing direction of user 2 is assumed to be at the point where the minimum of the length gradient 6 is located. This solid angle corresponds to the area where the segment S with the smallest width b is located. Figure 3f) shows the execution of step 207. Taking into account the assumed or determined position of user 2's head or eyes 1 and the known installation position of the camera 4, the viewing direction 10 in the vehicle interior is now determined.

[0076] Figure 3g) shows the execution of step 208. As shown in the lower part of Figure 3g), the length gradient 6 assumes different paths for different vertical viewing angles. If the user 2 looks up or down, the light pattern 5 is not projected onto the "equator" Ä of the eyeball and therefore does not pass through the pupil region itself. For example, the reflection of the light pattern 5 on the eyeball can, as indicated by a northern line N or a southern line S, lie above or below a central projection. The eyeball is also more curved above and below the pupil than in the pupil region, so that the segments S located horizontally in the pupil region are also more deformed. Accordingly, the shape of the curve changes.

[0077] Based on calibration measurements, specific curve profiles for defined vertical angles can now be determined and stored in a memory of the processing unit 8. The inventive method can then be carried out during the use of the vehicle 7, and by accessing the memory and the known curve profiles stored therein, the actual vertical angle can be determined.

[0078] In such a calibration measurement, the respective vertical angle can be predefined and / or determined in parallel using a known method for determining the viewing direction. This allows the respective vertical angles to be assigned to the recorded curve profiles.

[0079] Figure 3h) shows the possible application of the viewing direction 10 determined in this way for downstream driver assistance systems. Figure 3h) shows, for example, a camera image taken by a surround-view camera of the vehicle 7, where the respective objects viewed by the user 2 are superimposed with circles in a contact-like manner.

[0080] Figure 4 shows exemplary embodiments of the projection of the light pattern 5. Figure 4a) shows the simultaneous control of the individual light sources 9 of the light source 3. Hatched sections correspond to illuminated regions of the room or activated individual light sources 9. Segments shown in white correspond to unilluminated regions of the room or deactivated individual light sources 9.

[0081] For example, segments S could be monochromatic, especially in blue.

[0082] As Figure 4b) shows, the light pattern 5 can also be multicolored. This is indicated by different hatching patterns, which could, for example, be assigned to the colors red, green, and blue. Additionally or alternatively, the differently executed hatching could also indicate differing brightness and / or contrast. In Figure 4b), the segments S are directly adjacent to each other, in contrast to the representation in Figure 4a). However, it would also be conceivable that, in a multicolored design, the segments S are spaced apart from each other. Shadowy or dark sections of the light pattern 5 can also be used to determine the viewing direction.

[0083] The light pattern 5 can also be animated. Individual segments S can rotate in the circumferential direction U during projection. This is illustrated in Figure 4c). Only a single rotating segment S is shown as an example. A right-pointing arrow at the end of the illustration indicates that the segment S continues to rotate. However, several segments S can rotate simultaneously, particularly in different colors, brightness levels, and / or contrasts. In such a case, the light pattern 5 is composed of several camera images recorded sequentially. The recording frequency of the camera 4 is synchronized with the projection frequency of the light source 3. Preferably, the projection of the light pattern 5 is imperceptible to the user 2, especially when using infrared light.

Claims

Mercedes-Benz Group AG Patent claims 1. Method for detecting the direction of gaze, wherein at least one eye (1) of a user (2) is actively illuminated by a light source (3), the at least one eye (1) is detected by a camera (4), and the direction of gaze (10) is determined taking into account the known relative position of the at least one eye (1) with respect to the camera (4) and the geometric relationships of the reflections of the light source (3) detected on the eyeball in the respective camera images, comprising the following method steps: Projecting and capturing a light pattern (5) extending horizontally across the width of the eyeball, wherein the light pattern (5) is composed of a plurality of strip-shaped segments (S), wherein the ratio of the widths (b) of the segments (S) to each other in the circumferential direction (U) is known; Determining the width (b) of the captured segments (S); and Determining the direction of gaze (10) taking into account the measured width (b) of the segments (S) and the fact that the eyeball is less curved in the area of ​​the pupil than in the periphery.

2. Method according to claim 1, characterized by the fact that Segments (S) of equal width (b) are projected onto the eye and the angular position at which the shortest segment (S) is detected is assumed to be the horizontal angle of the viewing direction (10).

3. Method according to claim 2, characterized by the fact that in the circumferential direction (U) a length gradient (6) is formed between the segments (S), describing the change in length of the width (b) for successive segments (S), and the angular position at which the minimum of the length gradient (6) is detected is assumed to be the horizontal angle of the viewing direction (10).

4. Method according to any one of claims 1 to 3, characterized by the fact that Adjacent segments (S) differ in their color, brightness and / or contrast.

5. Method according to any one of claims 1 to 4, characterized by the fact that at least one segment (S) is generated by infrared light.

6. Method according to any one of claims 1 to 5, characterized by the fact that all segments (S) of the light pattern (5) are projected simultaneously and captured in a single camera image; or at least a subset of the segments (S) are projected successively in time, wherein the projection frequency of the light source (3) and the recording frequency of the camera (4) are coordinated, so that the light pattern (5) is composed of the reflections shown in at least two successive camera images.

7. Method according to any one of claims 1 to 6, characterized by a projection of the light pattern that is not perceptible to humans (5).

8. Method according to any one of claims 1 to 7, characterized by the fact that to determine a vertical angle of the viewing direction (10): the course of the distorted light pattern (5) on the eyeball is determined for various predetermined vertical angles, whereby the courses thus known as a function of the respective vertical angles are stored, and the current The vertical angles to be determined are determined by comparing the currently detected trajectory with the known trajectories of the light pattern (5); or a second light pattern extending vertically over the height of the eyeball is projected onto the eyeball, and the vertical angle is determined taking into account the known length ratio of corresponding segments (S) in the vertical direction to each other.

9. Method according to claim 8, characterized by the fact that the course of the distorted light pattern (5) on the eyeball for different vertical angles is collected by a large number of calibration measurements; and / or the vertical angle is determined by an alternative gaze direction determination method and the vertical angle determined in this way is assigned to the corresponding course of the distorted light pattern (5).

10. Vehicle (7) comprising a light source (3) and camera (4) directed towards the area where a person is operating the vehicle, and a computing unit (8) provided for controlling the light source (3) and for processing camera images generated by the camera (4), characterized by the fact that the light source (3), the camera (4) and the computing unit (8) are arranged to carry out a method according to one of claims 1 to 9.

11. Vehicle (7) according to claim 10, characterized by the fact that the light source (3) extends horizontally across the vehicle interior in the form of a light band, in particular designed as ambient lighting.

12. Vehicle (7) according to claim 11, characterized by the fact that the light source (3) extends in the vehicle interior in an angular range of at least 180° around the person driving the vehicle.

13. Vehicle (7) according to one of claims 10 to 12, characterized by the fact that the computing unit (8) is set up to supply the direction of gaze (10) as an input variable to a downstream driver assistance system.

Images