Method for visualizing an assistance graphic

Abstract

The invention relates to a method for visualizing an assistance graphic on a display device in a motor vehicle, the assistance graphic being displayed as a sequence of individual images which are lined up next to one another temporally at an image refresh frequency. The method according to the invention is characterized in that the image refresh frequency is dynamically adaptively adapted, for which purpose a person using the assistance graphic is recorded by a driver observation camera and their viewing direction is determined, after which the resulting field of view (14) of the person is assigned to an object (12) displayed in the assistance graphic, with it then being detected whether the field of view (14) follows a trajectory (11) of the displayed object (12), to which the field of view (14) has been assigned, wherein, in the event that the field of view (14) follows the trajectory (11), the fixation of the object (12) along the trajectory (11) is detected, after which the scatter of the fixation is evaluated, and then the image refresh frequency is adapted depending on whether the scatter exceeds or falls below a specified threshold value.

Description

[0001] Mercedes-Benz Group AG

[0002] Method for visualizing an assistance graphic

[0003] The invention relates to a method for visualizing an assistance graphic according to the type defined in more detail in the preamble of claim 1.

[0004] Display devices in motor vehicles are known from the prior art. It is also known to monitor a person driving a motor vehicle using one or more cameras, for example, to determine their gaze direction. DE 102015 015 136 A1 describes a head-up display as a display device that adapts the information displayed to the actual use of the vehicle by the person driving. Essentially, this addresses the fact that some people prefer more information, while others prefer less. By recording the person's gaze direction, it is possible to analyze which information the person regularly looks at and uses, and which information they do not. The display device can then be controlled accordingly, based on the individual's preferences.

[0005] DE 102011 114 059 A1 describes a device for displaying moving images in a motor vehicle comprising a screen and a control unit which switches between two modes depending on the vehicle speed and the distance to objects ahead, wherein the frame rate is reduced in the second mode and the thresholds for the modes are adjusted by an isotonic function of the distance.

[0006] Furthermore, JP 2015-101189A describes a vehicle display that detects when the driver's gaze is not directed at the display and subsequently reduces the flicker component to improve the visibility of the vehicle display. It is well known that displaying constantly changing content on a display device is comparatively resource-intensive. This can be particularly problematic in a vehicle, which inherently has certain limitations in its data processing resources. This is especially true for so-called assistance graphics, which, by definition, depict moving objects, such as other road users, and stationary objects, such as roads and buildings, in the vehicle's surroundings.The individual objects are detected, in whole or in part, by the vehicle's environmental sensors and displayed to the driver, for example, on an instrument cluster or a head-up display. The display consists of several consecutive frames, similar to a video display. The initial refresh rate of the frames is typically specified at 60 Hz. However, in practice, particularly in highly dynamic scenarios with many moving objects, such as in city traffic, the display device may be unable to show some objects due to resource limitations. In such cases, either the number of visualized objects must be reduced to display the remaining objects, or the refresh rate must be lowered.

[0007] Reducing the refresh rate can, however, lead to a jerky display for the person viewing or using the assistive graphics. This is due to the so-called flicker fusion frequency of the affected person, above which they perceive the sequence of images not as a continuous film but as successive individual frames. In practice, this flicker fusion frequency varies considerably, as it is both an individual characteristic of each person and also depends on the time of day, fatigue, and other factors, even for the same person.

[0008] The object of the present invention is to provide an improved method for visualizing an assistance graphic, in which the refresh rate can be reduced as needed, but always remains above the current critical flicker fusion frequency of the person viewing or using the assistance graphic. According to the invention, this object is achieved by a method with the features of claim 1, and in particular in the characterizing part of claim 1. Advantageous embodiments and further developments are described in the dependent claims.

[0009] The inventive method provides for the dynamic adaptive adjustment of the frame rate to visualize an assistance graphic as defined above. To prevent a jerky image impression for the person viewing the display, the person using the assistance graphic is detected by a driver observation camera, and their gaze direction is determined. Based on this gaze direction, a viewing area of ​​the person can then be assigned to an object depicted in the assistance graphic. This can also be referred to as a static correspondence. Subsequently, it is detected whether the viewing area follows a trajectory of the depicted object to which the viewing area has been assigned. This would then be the dynamic correspondence. If this is the case, the fixation, i.e., the targeted viewing of the object along this trajectory, is detected. The variance of the fixation is then evaluated.The variation in fixation allows conclusions to be drawn about whether the person using the assistive graphic perceives the current image sequence as a smooth flow, or, with a sufficiently high variation, as a series of individual images. This leads to a jerky appearance in the perceived image sequence. Since the person cannot always directly follow the object's movements, but rather their gaze is sometimes still in the previous position when the object has already moved further along its trajectory, this results in a greater variation in fixation.

[0010] According to the invention, the frame rate can now be adjusted depending on whether the variation exceeds or falls below a predetermined threshold. This ensures that the user always sees the assistance graphic in a smooth sequence of images, while simultaneously reducing the frame rate as much as possible, at least for resource-intensive displays, to save resources and to be able to depict as many captured objects as possible with sufficient dynamic range.

[0011] The advantage of such adaptive adjustment of the refresh rate to the needs and current situation of the individual user enables exceptionally efficient resource utilization. In particular, resources can be freed up when a low flicker fusion frequency is detected. No human intervention is required; the adaptive dynamic adjustment occurs without any active participation from the person using the assistive graphics.

[0012] This method allows for highly realistic assistance graphics, as typically no objects need to be deleted due to resource constraints. The method is independent of the existing scene and automatically takes lighting and weather conditions into account if these affect the current flicker fusion frequency of the person.

[0013] The process can be repeated cyclically within a control loop and can be applied to virtually all dynamic display devices, such as screens, head-up displays, and the like.

[0014] A particularly advantageous embodiment of the method according to the invention provides that, prior to the method, the driver observation camera is calibrated in a higher-level coordinate system, so that the coordinate systems of the driver observation camera and the vehicle are calibrated relative to each other. If the location of the display device in this higher-level coordinate system is known, e.g., that of the vehicle, it is then possible to determine simply and precisely which of the objects, whose position on the display device and thus in the space of the vehicle is known, the person's gaze is directed at.

[0015] Another very advantageous design option involves displaying the assistance graphics at the start of the procedure in an instrument cluster or head-up display using a predefined refresh rate. This predefined refresh rate could, for example, be set at a typical refresh rate of 60 Hz, at which most people perceive the image sequence as smooth.

[0016] A further highly advantageous embodiment of the method according to the invention provides that the person's gaze direction is projected onto the display device in order to assign the person's field of vision to one of the displayed objects. Particularly when, due to the calibrated driver observation camera and the matching coordinate systems of the vehicle and the display device on the one hand, and the driver observation camera on the other, the spatial position of the objects on the surface of the display device is known, such a projection of the gaze direction onto the display device, which can be visualized but does not necessarily have to be, allows for the very simple and efficient identification of the object to which the person's gaze is directed.

[0017] Another highly advantageous approach involves determining the standard deviation between the object's position and the person's field of vision as a measure of dispersion. According to a particularly beneficial further development, a threshold value on the order of 1 can be used for the standard deviation, allowing for appropriate action depending on whether the determined dispersion value is above or below this threshold.

[0018] According to a particularly advantageous embodiment of the inventive method, it is now provided that, in the event that the variation lies below the predetermined threshold, the frame rate is maintained or reduced. With regard to the particularly advantageous embodiment using the standard deviation with a threshold of 1, this would mean that the described case, in which the frame rate is maintained or reduced, always occurs when the standard deviation is less than 1.

[0019] Maintaining the frame rate is correspondingly simple. However, to free up resources, possibly for subsequent operations, or in case of a current resource shortage, the frame rate can also be reduced. In particular, such a reduction can be performed iteratively, according to a very advantageous advanced technique, in order to gradually approach the threshold and determine the currently minimum possible frame rate in several iterative steps.

[0020] Conversely, a highly advantageous embodiment provides that if the dispersion exceeds a predetermined threshold, or, in the described embodiment with the standard deviation, is greater than 1, the frame rate is increased. In principle, it would of course be conceivable to increase it directly to the initially predetermined value or a maximum value to be on the safe side in any case. However, it is more resource-efficient if, according to a highly advantageous embodiment of the inventive method, the increase is carried out iteratively in order to approach the minimum required frame rate for displaying a smooth motion sequence, as already explained above – but now from the other direction.

[0021] In particular, but not exclusively, in the case of iterative adjustment, a highly advantageous further development may entail repeating the process with the newly adjusted frame rate as the new initial frame rate. The process is thus repeated to ensure continuous adaptation to the current conditions. What does not need to be repeated is the upstream step of calibrating the driver monitoring camera.

[0022] Further advantageous embodiments and developments of the method according to the invention also become apparent from the exemplary embodiment, which is described in more detail below with reference to the figures.

[0023] This shows:

[0024] Fig. 1 shows a schematic flowchart of a possible embodiment of the method according to the invention;

[0025] Fig. 2 shows a representation according to process step VS 7 from the representation in Fig. 1; and

[0026] Fig. 3 shows a representation of the detection according to process step VS 8 from the representation according to Fig. 1.

[0027] The core of this description is to adapt the visualization of an assistance graphic, as mentioned above, to the resource load in such a way that a comfortable visualization is still guaranteed, taking into account the individual critical flicker fusion frequency.

[0028] As illustrated in Figure 1 using an exemplary flow diagram, in a first upstream process step VS 1 a driver observation camera is calibrated in a higher-level coordinate system, in particular the coordinate system of the vehicle, whereby the intrinsic and extrinsic position in relation to a coordinate origin within the vehicle is known.

[0029] In a further process step, VS 2, the assistance graphic is generated based on data from the environmental sensors and map data and displayed on a display device. This makes it known which objects, with which position and movement trajectory, are visualized on the display device, for example, an instrument cluster, a head-up display, or the like.

[0030] In the subsequent third process step, VS 3, the gaze direction of a person using the assistance graphic is recorded and analyzed. This person is typically the driver of the vehicle. The recording and analysis of gaze directions is a well-established technique, so the specific eye-tracking algorithms used (Purkinje Reflection, Iris Silhouette Tracking, etc.) need not be discussed further. This allows for the detection of saccades (gaze movements) and fixations (gaze fixations). Based on the three-dimensional relationship between the driver observation camera and the vehicle's coordinate system, established during calibration, the person's field of view can be projected onto the display device in a fifth process step, VS 5. The position and trajectory of the objects in this display were previously stored in a fourth process step, VS 4.

[0031] This allows the saccade, and in particular the person's fixation, to be assigned to a specific visualized object, which, as mentioned above, is also referred to as static correspondence. This is illustrated in Figure 1 in process step VS 6. In the subsequent process step VS 7, the object's movement, i.e., its trajectory, is compared with the gaze saccades for a predetermined number of frames. If the object's movement vectors and the person's gaze saccades are similar, a dynamic correspondence is assumed. The person is thus viewing the identified visualized object and following its trajectory with their gaze. If no similar movement vectors are detected, the assignment according to process step VS 6 is checked again and performed using a different object.Ultimately, the procedural steps VS 6 and VS 7 are repeated until sufficiently similar movement vectors have been detected between the trajectory of the object and the eye movements of the person.

[0032] In the subsequent crucial step of the procedure, VS 8, the dispersion of the object's fixation is evaluated along its trajectory in relation to the object's trajectory. A high dispersion suggests that the person—as described above—does not perceive the visualization as a continuous sequence of images, but rather at least partially resolves individual frames, resulting in a jerky display.

[0033] If this variation exceeds a defined threshold, a control unit for the display device is instructed to increase the refresh rate in iterative steps, as the current refresh rate setting is potentially below the flicker fusion frequency currently present in the individual. This allows for adaptive adjustment of the refresh rate to the individual and highly variable flicker fusion frequency of each person at any given time. This enables resource-efficient visualization that is perceived as good by the individual. This approach also allows for the visualization of highly realistic assistive graphics, as increased resource usage does not necessarily lead to a reduction in the number of objects displayed.As explained above, resources can be freed up by reducing the refresh rate until the minimum refresh rate at which the person still perceives the process as smooth is reached.

[0034] The threshold comparison mentioned above is visualized in the representation of Figure 1 by process step VS 9, while process step VS 10 shows the reduction or increase of the frame rate.

[0035] The entire sequence between process steps VS 2 and VS 10 is then repeated cyclically, ensuring continuous adjustment. For this purpose, process step VS 2 is started with the previously determined new frame rate as the initial frame rate for the next process run. Figure 2 illustrates, using two examples, how the fixation variation can behave. Figure 2a) shows the trajectory 11 of an object labeled 12. The object 12 is shown multiple times along its trajectory 11. Various frames of the representation are schematically indicated in the vertical direction and labeled with the corresponding times T0 to T4.Within the multiple representations of object 12, a filled dot labeled 13 is visible at the bottom of each object 12. This dot represents the center of object 12 on its trajectory 11. Additionally, an unfilled circle marks the person's gaze area 14, i.e., the area on which the person is currently looking in the current frame or subframe. The distance represented by the thin line between the center 13 of object 12 and the gaze area 14 symbolizes the dispersion of the gaze area 14 relative to the actual position of object 12. This is illustrated again in Figure 2b) on the right.

[0036] Figure 2a) shows that the gaze areas 14 largely follow the objects 12. The dispersion of fixation is low. In contrast, Figure 2b) shows that several gaze areas 14 are assigned to one object position, while no gaze area 14 is assigned to other positions of the object 12. One of the two gaze areas 14 at the second object position from the bottom therefore actually belongs to the third object position above it. The dispersion between the gaze area 14 on the one hand and the center 13 of the object 12 on the other is thus significantly higher here.

[0037] Figure 3 illustrates this accordingly. The unfilled bars on the x-axis, representing the time axis, indicate the case shown in Figure 2a), where the dispersion, represented by the distance between the center 13 of object 12 and the viewpoint 14 on the y-axis, is comparatively small. In each of the individual frames, the hatched bars also depict the case shown in Figure 2b), revealing that, analogous to the representation in Figure 2b), the deviation between the viewpoint 14 and the center 13 of object 12 is very large in the third and fifth images. This graphic thus clearly demonstrates a relatively low dispersion for the first case shown in Figure 2a) and a very high dispersion for the second case in Figure 2b).In the embodiment shown here, the standard deviation for the case according to Figure 2a) is approximately 0.4, while the standard deviation for the case according to Figure 2b) is 1.9.

[0038] In the case shown in Figure 2a), the refresh rate can be reduced, while in the case shown in Figure 2b), it must be increased to provide the person with a smooth-looking representation of the assistance graphics.

Claims

Mercedes-Benz Group AG Patent claims 1. A method for visualizing an assistance graphic on a display device in a motor vehicle, wherein the assistance graphic is displayed as a sequence of individual images which follow one another at a specific frame rate, characterized in that the frame rate is dynamically adaptively adjusted, for this purpose a person using the assistance graphic is detected by a driver observation camera and their gaze direction is determined, after which the resulting field of view (14) of the person is assigned to an object (12) displayed in the assistance graphic, after which it is detected whether the field of view (14) follows a trajectory (11) of the displayed object (12) to which the field of view (14) has been assigned, wherein, in the case that the field of view (14) follows the trajectory (11), the fixation of the object (12) along the trajectory (11) is detected, after which an evaluation of the scatter of the fixation is carried out.and according to which the frame rate is adjusted depending on whether the variation exceeds or falls below a predetermined threshold.

2. Method according to claim 1, characterized in that the method is preceded by a calibration of the driver observation camera in a higher-level coordinate system of the vehicle.

3. Method according to claim 1 or 2, characterized in that a visualization of the assistance graphic is performed at the beginning of the method in an instrument cluster or a head display as a display device using an initially predetermined refresh rate.

4. Method according to claim 3, characterized in that the initial refresh rate is set to approximately 60 Hz.

5. Method according to one of claims 1 to 4, characterized in that a projection of the person's gaze direction into the visualization of the assistance graphic on the display device is carried out in order to assign the person's field of vision (14) to one of the depicted objects (12).

6. Method according to one of claims 1 to 5, characterized in that the standard deviation between the position of the object (12) and the viewing area (14) of the person is determined as a measure of dispersion.

7. Method according to claim 6, characterized in that the threshold for the standard deviation is specified at approximately 1.

8. Method according to one of claims 1 to 7, characterized in that, in the event that the variation is below the predetermined threshold, the frame rate is maintained or reduced.

9. Method according to claim 8, characterized in that the reduction only takes place in the event of the utilization of the available resources exceeding a predetermined limit.

10. Method according to claim 8 or 9, characterized in that the reduction is carried out in iterative steps.

11. Method according to any one of claims 1 to 10, characterized in that In the event that the variation is above the specified threshold, the refresh rate is increased.

12. Method according to claim 11, characterized in that the increase is carried out in iterative steps.

13. Method according to one of claims 1 to 12, characterized in that the method is repeated with the adapted frame rate as the new initial frame rate.

Images