Method and system for a human-machine interface using eye gaze for engagement and interaction

Abstract

A method includes determining that an eye gaze of a user is directed to an object, determining that the user is interested in the object, and, in response to the eye gaze being directed to the object and the user being interested in the object, emitting an output related to the object from an output device. A controller is programmed to perform the method. A system includes the controller.

Description

Atty. Doc. No. HARM0997PCTMETHOD AND SYSTEM FOR A HUMAN-MACHINE INTERFACE USING EYE GAZE FOR ENGAGEMENT AND INTERACTION TECHNICAL FIELD

[0001] Aspects disclosed herein generally relate to a method and system for a human-machine interface using eye gaze for engagement and interaction. These aspects and others will be discussed in more detail herein.CROSS-REFERENCE TO RELATED APPLICATION

[0002] This application claims priority to provisional U.S. Patent Appl. No. 63 / 743,522, filed on January 9, 2025; provisional U.S. Patent Appl. No. 63 / 756,620, filed on February 10, 2025; and provisional U.S. Patent Appl. No. 63 / 814,104, filed on May 29, 2025, which are hereby incorporated by reference in their entirety.BACKGROUND

[0003] In the modern world, rapid development of technology and its increasing complexity requires people to deal with vast amounts of information integrated into almost all aspects of life. Computers, smartphones, digital devices and even cars are capable of solving multiple tasks, but interactions with them are time- and energy-consuming. Each user-initiated action currently requires a physical move or touch or a voice command, which is not always possible, safe, or convenient. In the automotive domain, smart infotainment systems integrated into vehicles include a variety of functions, from monitoring and safety features to route planning, delivering informative content, and various entertainment options. Increasing capabilities of such systems also mean that their use becomes more and more complicated, which may create discomfort and safety concerns for drivers and passengers. Thus, a demand for more convenient human-like ways of communication with digital systems is increasing.SUMMARY

[0004] Using eye contact for interacting with the world and social environment is natural and easy for humans. This disclosure aims to implement and enhance a human-machine interface to make digital interactions more seamless and effortless in everyday life, thereby providing an efficient and comfortable way of communication with a digital system. A controller (e.g., on board a vehicle) may be programmed to perform a method including determining that an eye gaze of aAtty. Doc. No. HARM0997PCTuser is directed to an object, determining that the user is interested in the object, and, in response to the eye gaze being directed to the object and the user being interested in the object, emitting an output related to the object from an output device.

[0005] Human-machine interfaces utilizing eye gaze may significantly improve user experience. Using eye movements and gaze for achieving various adaptive outcomes is natural and easy for humans as vision provides our main source of information about the world. The disclosed system and method provide for a human-machine interface that utilizes eye gaze (and possibly other biosignals) from a user to establish attention to distinct objects and to provide the user with information related to these objects. The objects may include virtual objects generated by a user interface or physical objects in the environment.

[0006] For example, the object may be virtual object such as an avatar. The human-machine interface is able to provide a natural -feeling interaction with the avatar by detecting that the user is looking at and is interested in interacting with the avatar, and then responding with an output such as having the avatar look at the user, thereby returning the user’s gaze.

[0007] For another example, the disclosed system and method may enhance the user experience by automating the process of generating information about objects that interest the user. On one hand, a user may be able to utilize this functionality intentionally when trying to gather or obtain information about objects in the environment. By simply looking at an object for a short duration of time may result in information about the viewed object being collected from digital sources and becoming easily available to the user. On the other hand, the disclosed system and method may be able to detect naturally occurring interest and curiosity about the world around the user and provide information that can be useful or educational.

[0008] A method includes determining that an eye gaze of a user is directed to an object, determining that the user is interested in the object, and, in response to the eye gaze being directed to the object and the user being interested in the object, emitting an output related to the object from an output device. A controller is programmed to perform the method. A system includes the controller.

[0009] In an example, the object may be a virtual object generated by a user interface. In a further example, the virtual object may be an anthropomorphic avatar with eyes. In a still further example, the output related to the object may include changing a gaze direction of theAtty. Doc. No. HARM0997PCTanthropomorphic avatar. In a yet still further example, the gaze direction of the anthropomorphic avatar resulting from the output may be directed toward eyes of the user.

[0010] In an example, determining that the user is interested in the object may include determining that the user is interested in the object based on a voice command from the user.

[0011] In an example, determining that the user is interested in the object may include determining that the eye gaze of the user is directed to the object for a duration exceeding a time threshold.

[0012] In an example, determining that the eye gaze of the user is directed to the object may include determining that the eye gaze of the user is directed to the object based on a position of eyes of the user, a gaze direction of the eyes of the user, and a position of the object, and the position of the object may be represented in a same coordinate system as the position of the eyes of the user. In a further example, the object may be a virtual object generated by a user interface, and the position of the object may be prestored.

[0013] In another further example, the object may be a physical object in an environment around the user, and the method may further include determining the position of the object based on data from sensors indicating the environment.

[0014] In an example, determining that the user is interested in the object may include determining that the user is interested in the object based on a psychophysiological state of the user. In a further example, the method may further include determining the psychophysiological state of the user based on at least one biometric characteristic of the user.

[0015] In an example, emitting the output may include activating a voice assistant, and the voice assistant may be programmed to receive verbal commands from the user.

[0016] In an example, the object may be a physical object in an environment around the user. In a further example, emitting the output may include conveying information about the physical object to the user. In a still further example, the method may further include performing object recognition on the object, and the information conveyed about the object may be based on the object recognition.

[0017] A controller may include at least one memory device, and the controller may be programmed to perform the method of one of the foregoing examples.

[0018] A system may include the controller.Atty. Doc. No. HARM0997PCT

[0019] In an example, the system may further include a camera having a field of view encompassing the user, and determining that the eye gaze of the user is directed to the object may be based on data from the camera. In a further example, the camera may be located in a passenger compartment of a vehicle.BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The system may be better understood with reference to the following drawings and description. The components in the figures are not necessarily to scale, emphasis instead being placed upon illustrating the principles of the invention. Moreover, in the figures, like-referenced numerals designate corresponding parts throughout the different views.

[0021] Figure 1 is a block diagram of an example system for human-machine interface using eye gaze.

[0022] Figure 2 is a diagrammatic side view of a user using the human-machine interface.

[0023] Figure 3 is a side view of an example of a user with an eye gaze directed to a screen of a user interface.

[0024] Figure 4 is a rear view of the system with an avatar having eye contact with the user.

[0025] Figure 5 is a rear view of the system with the avatar maintaining eye contact with the driver while verbally interacting with the user irrespective of the user’s eye gaze.

[0026] Figure 6 is a rear view of the system with the avatar at a default gaze direction.

[0027] Figure 7 is a flowchart of an example process for controlling the human-machine interface.DETAILED DESCRIPTION

[0028] With reference to the Figures, wherein like numerals indicate like parts throughout the several views, a method includes determining that an eye gaze of a user is directed to an object 200, determining that the user is interested in the object 200, and, in response to the eye gaze being directed to the obj ect 200 and the user being interested in the obj ect 200, emitting an output related to the object 200 from an output device. A system 100 may include a controller 105. The controller 105 is programmed to perform the method.

[0029] With reference to Figure 1, the system 100 may include the controller 105, at least one front-facing camera 110, at least one user-facing camera 115, at least one microphone 120,Atty. Doc. No. HARM0997PCTbiometric sensors 125, a user interface 130, at least one speaker 135, a transceiver 140, and possibly other components.

[0030] The controller 105 is a microprocessor-based computing device such as a generic computing device including a processor and a memory device, an electronic controller or the like, a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a combination of the foregoing, etc. Typically, a hardware description language such as VHDL (VHSIC (Very High Speed Integrated Circuit) Hardware Description Language) is used in electronic design to describe digital and mixed-signal systems such as FPGA and ASIC. For example, an ASIC is manufactured based on VHDL programming provided pre-manufacturing, whereas logical components inside an FPGA may be configured based on VHDL programming (e.g., stored in a memory device electrically connected to the FPGA circuit). The controller 105 can thus include a processor, a memory device, etc. The memory device of the controller 105 can include media for storing instructions executable by the processor as well as for electronically storing data and / or databases, and / or the controller 105 can include structures such as the foregoing by which programming is provided. The controller 105 can include multiple processors and memories coupled together.

[0031] The front-facing cameras 110 and / or user-facing cameras 115 can detect electromagnetic radiation in some range of wavelengths. For example, the front-facing cameras 110 and / or user-facing cameras 115 may detect visible light, infrared radiation, ultraviolet light, or some range of wavelengths including visible, infrared, and / or ultraviolet light. For example, the front-facing cameras 110 and / or user-facing cameras 115 can be a charge-coupled devices (CCD), complementary metal oxide semiconductors (CMOS), or any other suitable type.

[0032] The front-facing cameras 110 may be positioned to have a field of view in a same direction as the user. The direction of the field of view of the user may be defined by a structure of the human-machine interface. For example, as shown in Figure 2, the user may be an occupant seated in a vehicle 205 with seats facing in a vehicle-forward direction, and the front-facing camera 110 may be aimed in the vehicle-forward direction. For another example (e.g., alternatively), the user may be wearing smart glasses 210 including the front-facing camera 110, and the front-facing camera 110 may be oriented straight ahead relative to the smart glasses 210. The front-facing camera 110 may thus be positioned to record image data of the same objects 200 viewed by the user.Atty. Doc. No. HARM0997PCT

[0033] The user-facing camera(s) 115 has a field of view encompassing the user. The userfacing cameras 115 may be oriented toward the user. For example, the user-facing camera 115 may be located in a passenger compartment 215 of the vehicle 205. The user-facing camera 115 may be positioned in a vehicle-forward direction from the user and may be aimed in a vehiclerearward direction. For another example, the smart glasses 210 may include the user-facing cameras 115, and the user-facing cameras 115 may be aimed at one or both of the eyes of the user. The user-facing camera 115 may be configured to capture the eye gaze or eye direction of the user while the user is positioned at a specific location (e.g., in a seat of the vehicle 205, wear the smart glasses 210, etc.) and transmit this information to the controller 105.

[0034] The microphone 120 is a transducer that converts sound to an electrical signal. The microphone 120 can be any suitable type, such as a dynamic microphone, which includes a coil of wire suspended in a magnetic field; a condenser microphone, which uses a vibrating diaphragm as a capacitor plate; a contact microphone, which uses a piezoelectric crystal; etc. The microphone 120 may be positioned to detect speech by the user, for example, in the passenger compartment 215 of the vehicle 205, mounted to the smart glasses 210, etc.

[0035] The biometric sensors 125 may be any sensors capable of measuring a biometric characteristic, such as heart rate, heart rate variability, breathing rate, skin conductance, etc. The biometric sensors 125 may include radar positioned in the passenger compartment 215, wearable devices such as smartwatches, pulse oximeters, galvanic sensors, etc. The biometric sensors 125 may further include the user-facing camera 115. For example, the user-facing camera 115 may detect facial expression, pupil dilation, and / or or eye gaze direction. The radar may detect breathing rate by tracking chest movement. The wearable device may detect heartrate and / or electrodermal activity. The wearable device may be connected to the communications network via the transceiver 140.

[0036] The user interface 130 presents information to and receives information from the user, who may be a driver of the vehicle 205. The user interface 130 may be located on an instrument panel in a passenger compartment 215 of the vehicle 205, and / or wherever may be readily seen by the user. The user interface 130 may include dials, digital readouts, screens 305, speakers 135, and so on for providing information to the user, such as human-machine interface (HMI) elements such as are known. The user interface 130 may include buttons, knobs, keypads, the microphone 120, and so on for receiving information from the user. A screen 305 of the user interface 130 may beAtty. Doc. No. HARM0997PCTany suitable type for displaying content legible to the respective occupants, such as light-emitting diode (LED), organic light-emitting diode (OLED), liquid crystal display (LCD), plasma, digital light processing technology (DLPT), etc. The screen 305 is positioned to be visible to the user. For example, the screen 305 can be a standalone device in the form of a tablet, part of the vehicle 205 such as mounted to a dashboard in the passenger compartment 215, etc. Alternatively or additionally to the screen 305, the user interface 130 may use a heads-up display that projects onto the windshield or onto the lenses of the smart glasses 210, and the description below referencing the screen 305 also applies to a projection surface of a heads-up display of the user interface 130.

[0037] The speaker 135 is electroacoustic transducer that converts an electrical signal into sound. The speaker 135 can be any suitable type for producing sound audible to the user (e.g., dynamic). The speaker 135 is positioned to emit sound audible to the user, for example, as part of the vehicle 205 in the passenger compartment 215, mounted to the smart glasses 210, etc.

[0038] The transceiver 140 may wirelessly receive information from devices such as the smart glasses 210, a mobile device of the user, a wearable device such as a smartwatch, etc. The transceiver 140 may transmit the information received from the smart glasses 210 to the controller 105. It is recognized that the smart glasses 210 and the controller 105 may engage in bi-directional wireless communication via WiFi, Bluetooth, or other suitable wireless communication protocol via the transceiver 140. Similarly, the mobile device of the user may engage in bi-directional communication with the controller 105 via the transceiver 140.

[0039] The system 100 described herein may be part of the vehicle 205, part of the smart glasses 210, or distributed across the vehicle 205, the smart glasses 210, and / or one or more other devices such as the mobile device of the user. The controller 105 may be part of the vehicle 205, part of the smart glasses 210, part of the mobile device, etc. One nonlimiting example may include the implementation of system 100 and method that is installed in the vehicle 205. It is further recognized that the disclosed system 100 and method may be applicable to any number of nonvehicle implementations such as, but not limited to, virtual reality (VR) / gaming implementations, lifestyle audio applications such as those involving home entertainment systems, etc. Another nonlimiting example may involve a system 100 that utilizes outputs from the smart glasses 210. The smart glasses 210 may include the user-facing camera 115 and / or the front-facing camera 110 that may automatically detect objects 200 that the user is looking at or viewing. The mobile device may obtain data about the objects 200 that the user is looking at from the smartAtty. Doc. No. HARM0997PCTglasses 210 and / or the vehicle 205, and the mobile device may emit output related to the objects 200 on the screen 305 or heads-up display, through speakers or headphones, etc., which may be part of the mobile device, the vehicle 205, or the smart glasses 210.

[0040] The controller 105 may be programmed to receive sensor data, e.g., from the userfacing camera 115, the front-facing camera 110, the microphone 120, and / or the biometric sensors 125. The data from the user-facing camera 115 may be image data depicting the user, e.g., data indicating the eye gaze of the user. The data from the front-facing camera 110 may include data indicating the environment around the user, e.g., image data depicting a physical object 200 in the environment that the user is looking at. The data from the microphone 120 may include audio data indicating a voice command by the user. The data from the biometric sensors 125 may indicate biometric characteristics of the user.

[0041] The controller 105 may be programmed to determine the positions of objects 200 in the environment based on data from sensors indicating the environment (e.g., the front-facing camera 110, ranging sensors of the vehicle 205 such as radar or lidar, etc.). The objects 200 may be physical objects 200 in the environment around the user (e.g., around the vehicle 205). The controller 105 may determine the positions of the objects 200 such that the positions are in a same coordinate system as used for the user-facing camera 115. For example, the controller 105 may synchronize or merge the information provided by the front-facing camera 110 and the user-facing camera 115 such that the information is brought on to the same coordinate system. The eye gaze data and the 3D representation of object data may be merged into the same coordinate system by applying one or more of the following procedures:

[0042] (1 ) The controller 105 assesses or finds intrinsic parameters for the front-facing camera 110 and the user-facing camera 115, for example, by applying standard camera calibration via a chess board as set forth in https: / / docs.opencv.Org / 4.x / dc / dbb / tutorial_py_calibration.html and / or in “AUTOMATIC CALIBRATION OF DIGITAL CAMERAS USING PLANAR CHESSBOARD PATTERNS, DOUSKOS et al., Department of Surveying, National Technical University of Athens, GR- 15780 Athens, Greece, pgs. 1 - 7, 2007. The intrinsic parameters may be prestored in the controller 105.

[0043] (2) The controller 105 finds or locates the relative position (e.g., rotation and translation matrices) of the front-facing camera 110 coordinate system in the user-facing camera 115 coordinate system, which can be done through special calibration for nonoverlapping cameras.Atty. Doc. No. HARM0997PCTAn example of locating the relative position of the front-facing camera 110 coordinate system may be found in, for example, “Simple Calibration of Non-overlapping Cameras with a Mirror”, KUMAR et al., Department of Computer Science University of North Carolina, Chapel Hill, USA, pgs. 1 - 7, IEEE 2008.

[0044] (3) The controller 105 solves a Metric Depth Estimation problem for each frame data of the video information provided by the front-facing camera 110. One example of the manner in which the Metric Depth Estimation problem may be solved is set forth in “Depth Anything V2”, YANG et al., Cornell University, 20 Oct. 2024. The controller 105 may utilize any metric depth estimation neural network or any simple logic (e.g., that the distance of each pixel is equal to constant D meters) to solve this problem.

[0045] (4) The controller 105 extracts the 3D representation of the environment. The depth of each pixel is known after the third step or item (3) above and the intrinsic parameters are known after the first step or item (1) above. The controller 105 extracts a colored 3D point cloud in the front-facing camera 110 coordinate system using this information. After that, the transformation (the result of the second step) is applied by the controller 105 to obtain a final 3D point cloud in the user-facing camera 115 coordinate system.

[0046] The controller 105 may be programmed to perform object recognition on the objects 200. In one example, the object recognition may be implemented using a model, such as, for example, a visual language model (VLM). Using VLM facilitates integration of object recognition and its description within a unified model. However, to attain optimal recognition and description quality, it may be required to employ resource-intensive models, which escalates the hardware prerequisites and query processing duration for the controller 105. An alternative approach involves using a distinct image recognition model that may be executed by the controller 105, with the recognized object 200 subsequently serving as input to a more extensive language model. This approach facilitates the management of resources allocated for object recognition and object description tasks separately.

[0047] The controller 105 is programmed to determine that the eye gaze of the user is directed to the object 200 is based on data from the user-facing camera 115. To do so, first, the controller 105 tracks the eye gaze of the user. Eye tracking systems can be categorized into, for example, several types: using special contact lenses or other devices to mechanically track gaze direction movement; using electrodes around the eye to measure electrical potentials during eye movements;Atty. Doc. No. HARM0997PCTusing optical tracking without direct contact to the eye (e.g., with the user-facing camera 115). Optical tracking employs images from multiple cameras and gaze direction detection algorithms to perform gaze tracking. This category may be further subdivided into devices in the form of glasses where cameras for eye tracking are located close enough to provide high-quality images, versus devices with cameras positioned at a certain distance from the user and designed to track the user’s gaze. These solutions have found wide application in the automotive industry.

[0048] An industrial human tracking system can be utilized to determine the mechanism of a driver’s attention. Such systems are installed in cars and provide monitoring of the state of the driver, passengers, the internal situation inside the vehicle 205 and the external environment outside the vehicle 205. The industrial driver monitoring system is capable of determining the direction of a person’s gaze, the degree of eye openness, the direction of the head, the position of landmarks, reference points on the person’s face, particularly the landmarks of the eyes, nose, ears, mouth, and facial oval. Cameras in such systems can be either RGB or infrared. RGB cameras provide more information due to the color image, but infrared cameras work better in low light conditions inside the vehicle 205. To obtain information from all angles and eliminate blind spots, tracking systems often use from one to four cameras. As one example, the system 100 may include two infrared user-facing cameras 115 as a data source.

[0049] The output of this tracking system may include the information in the following table:<>< >< >< >< > <> <>< ><> <>Atty. Doc. No. HARM0997PCT< >< >

[0050] An arithmetic mean of both directions of gaze is taken as a consensus gaze direction.

[0051] The controller 105 is programmed to determine that the eye gaze of the user is directed to the object 200. As a general overview, the controller 105 may determine that an intersection occurs between a ray 310 representing the eye gaze and position data representing the object 200, based on a position of eyes of the user, a gaze direction of the eyes of the user, and a position of the object 200, in which the position of the object 200 is represented in a same coordinate system as the position of the eyes of the user. The object 200 may be a virtual object on the screen 305 of the user interface 130 or a physical object in the environment, as will be explained in turn.

[0052] With reference to Figure 3, the controller 105 may determine that the eye gaze of the user is directed to a virtual object 200 based on a position of eyes of the user, a gaze direction of the eyes of the user, and a position of the virtual object 200. The position of the object 200 is represented in a same coordinate system as the position of the eyes of the user. The position of the object 200 may be prestored, e.g., as a polygon 315 in space circumscribing the screen 305. In this implementation, in order to interact with the virtual object 200, the virtual object 200 needs to be activated by eye contact. Eye contact occurs when a driver looks at the tablet where the virtual object 200 is displayed. As a response to the user’s eye contact with the screen 305, the virtual object 200 changes its head position and looks back at the user, as will be described below.

[0053] To detect the event when the user looks at the screen 305, the eye tracking system data is used such as consensus gaze ray direction and consensus gaze ray starting point for a ray 310 in the coordinate system of the user-facing camera 115. Also, the position of the screen 305 is known in the coordinate system of the camera. To detect the event when the user looks at the screen 305, the following algorithm is applied:

[0054] (1) Find an intersection point 320 of consensus gaze ray 310 and a display plane of the tablet.

[0055] (2) Check if the intersection point 320 exists and is positioned inside a polygon 315 of the screen 305. If the intersection point 320 lay inside the polygon 315, itmeans that the eye gaze is directed to the virtual object 200.Atty. Doc. No. HARM0997PCT

[0056] The first step of the algorithm is the solution of the ray-plane intersection problem. To find the solution, the equation of the plane and the equation of the ray 310 should be known for a fixed coordinate system. A plane is defined by the equation:Ax + By + Cz + D = 0in which A, B, C, and D are constants and x, y, and z are spatial coordinates. The ray 310 is defined by:>in which Ro = (Ao, To, Zo) is the ray starting point, Rd = (X, Yd, Zd) is the ray direction vector, and t is the magnitude of the ray 310. If the value t is a positive real number (t > 0), the ray-plane intersection problem has the following solution:_ _ AX0+ BY0+ CZ0+ DAXd+ BYd+ CZd

[0057] In this example, the coordinate system is a coordinate system of the user-facing camera 115, the ray equation is given by the eye tracking system, and the plane is a display plane of the screen 305 for which the position in coordinate system is known.

[0058] The second step of the algorithm checks whether the user is looking at the screen 305. If there is not a ray-plane intersection point 320 during the previous step, the user is not currently looking at the screen 305. Otherwise, if there is a ray-plane intersection point 320, the controller 105 may be programmed to determine whether the intersection point 320 lays inside of the screen 305. The screen 305 can be represented by the three-dimensional polygon 315 that lays on the display plane of the screen 305. So, this additional checking can be reformulated as the point-in-polygon (PIP) problem and solved by a geometric algorithm, as is known.

[0059] In the example shown in Figure 3, the arrow as illustrated generally corresponds to a consensus gaze ray 310. The polygon 315 circumscribes the screen 305. The dot on the display and positioned in the screen 305 corresponds to the intersection point 320 of consensus gaze ray 310 and a display plane of the screen 305. The intersection point 320 generally lies inside of the polygon 315. Thus, the algorithm as executed by the controller 105 detects that the eye gaze of the user is directed to the screen 305.

[0060] Returning to Figure 2, the object 200 may be a physical object in the environment around the user. The controller 105 may be programmed to determine the points of intersection between the gaze direction and the three-dimensional representation of the environment over time.Atty. Doc. No. HARM0997PCTTo achieve that, gaze direction vectors are usually filtered by the controller 105, and only valid vectors may be taken by the controller 105 into analysis. The controller 105 employs filtering which can be done by evaluating the internal self-quality of vectors. To determine points of intersection, the controller 105 determines distances between a gaze vector and each point within the 3D cloud. The controller 105 determines a nearest 3D point which may be assumed to serve as the intersection between the gaze and the 3D representation of the environment. In general, the controller 105 includes an object recognition module, as described above. The controller 105 may execute the object recognition module to obtain or provide an identification of distinct objects 200 in the environment.

[0061] The controller 105 is programmed to determine whether the user is interested in the object 200. The controller 105 may execute one or more algorithms to determine whether the user is interested in the object 200, for example, sustained eye gaze directed to the object 200, voice activation, or an attention model based on a psychophysiological state of the user, as will be described in turn below. The controller 105 may execute one of the algorithms. Alternatively, the controller 105 may execute more than one and determine that the user is interested in the object 200 if, e.g., at least one algorithm registers interest.

[0062] For example, the controller 105 may be programmed to determine that the eye gaze of the user is directed to the object 200 for a sustained period. The controller 105 may determine whether the user is looking at the object 200 for a total time duration that exceeds a threshold duration. For example, if the data indicates that a user has been looking at the object 200 for a total duration exceeding the threshold duration (for example, 1 second), then the user may be interested in the object 200, or the object 200 has caught the user’s attention.

[0063] As another example of sustained gaze, the controller 105 executes a rule-based algorithm to determine whether a gaze direction vector reflects a fact of paying attention to the object 200. For example, during a given analysis window (e.g., 3 seconds), the controller 105 selects all valid gaze vectors. The angles between all pairs of valid vectors are then calculated by the controller 105. From the resulting set of angles, the controller 105 may calculate one or more statistical measures such as variance, and the controller 105 may compare the statistical measure to a statistical threshold. The controller 105 determines that the user is interested in the object 200 in response to the statistical measure indicating lower variability than the statistical threshold. For example, if the controller 105 determines that the variance does not exceed the threshold, theAtty. Doc. No. HARM0997PCTcontroller 105 then determines that the attention to the object 200 by the user is detected. The average gaze direction vector can then be calculated by averaging the coordinates of all valid vectors. This vector may become the vector of attention to the object 200.

[0064] For another example, the controller 105 may determine that the user is interested in the object 200 based on a voice command from the user. The controller 105 may listen for an activation phrase for a voice assistant, listen for a verbal command given while or shortly after the user’s eye gaze is directed to the virtual object 200, or listen for a response to output from the voice assistant. The controller 105 may execute the voice assistant, which generates responses based on the content of the user speech as detected by the microphone 120. An example of this may include delivering information about the weather forecast if the user says, “I’m wondering if it’s going to rain today.” For another example, the controller 105 can change settings in response to a user request extracted from the speech content. An example of this may include lowering temperature settings in the air conditioning in response to the user verbally indicating that they are feeling hot.

[0065] In one example, the voice assistant may need to be activated to start “listening” to the user. This can be done using an activation phrase (sometimes called a “hot word”). Another way to activate would be tapping on an element of the display, which is not always convenient when driving. An alternative way suggested in this proposal is glancing at an element of the display before talking, which is more human-like and natural, the same way the user would communicate with another person if they were sitting next to them in the vehicle 205. In another implementation, the system 100 is “listening” continuously, but glancing at the virtual object 200, such as a digital avatar with eyes, may be considered within the context input to indicate that the user is addressing the system 100, rather than talking to a passenger, talking on the phone, or thinking aloud. The controller 105 then analyses the content of the speech to change mode or respond accordingly. This is the most human-like way of communication because the user can glance to indicate they are addressing the system 100 at any time, before, during, or after their verbal interaction. In some instances, the voice assistant may start the interaction with the user after an involuntary action by the user such as yawning. In this example, the voice assistant may ask, “I see you may be tired, should I make the audio louder to help you stay alert?” The controller 105 may process any received verbal commands or dialog and transmit verbal outputs to the user.

[0066] To parse the voice command, the controller 105 may execute an artificial intelligence (Al) model or natural language processor to transform the speech content into text. The controllerAtty. Doc. No. HARM0997PCT105 can use any suitable algorithm for converting speech to text, e.g., hidden Markov models, dynamic time warping-based speech recognition, neural networks, end-to-end speech recognition, etc.

[0067] The above uses of the voice assistant may not apply in all situations. For example, in the event the vehicle 205 is moving (i.e., being driven or speed of the vehicle 205 is greater than zero), the voice assistant may simply respond to verbal input provided by the user / driver while recognizing that the user / driver cannot look at the virtual object 200 on the screen 305 to avoid distractions.

[0068] For another example, the controller 105 may execute an attention model based on the psychophysiological state of the user. For the purposes of this disclosure, “psychophysiological state” is defined as a current psychological, emotional, and / or physiological condition of someone. For example, the psychophysiological state may include at least one of alertness, arousal, cognitive load, stress, fatigue, drowsiness, emotion, eye gaze metrics, heart rate, heart rate variability, or electrodermal activity. Alertness is a state of active attention characterized by high sensory awareness, and it is a psychological and physiological state. Drowsiness is the opposite. Arousal is the physiological and psychological state of being awoken or of sense organs stimulated to a point of perception. It involves activation of the ascending reticular activating system in the brain, which mediates wakefulness, the autonomic nervous system, and the endocrine system, leading to increased heart rate and blood pressure and a condition of sensory alertness, desire, mobility, and reactivity. Cognitive load is the effort being used in the working memory when the user is processing information. Emotions are physical and mental states associated with thoughts, feelings, and a degree of pleasure or displeasure (e.g., happy, excited, scared, sad, angry, etc.). Eye gaze metrics are measures of the gaze direction of the user, such as where the user is looking, how frequently where the user is looking changes, etc. Heart rate is the frequency at which the user’s heart beats, generally measured in beats per minute. Heart rate variability is a measure of how the heart rate changes over time. Electrodermal activity is the property of the human body that causes continuous variation in the electrical characteristics of the skin, also been known as skin conductance, galvanic skin response, electrodermal response, psychogalvanic reflex, skin conductance response, sympathetic skin response, or skin conductance level. Electrodermal activity can indicate arousal.Atty. Doc. No. HARM0997PCT

[0069] The controller 105 may be programmed to determine the psychophysiological state of the user based on at least one biometric characteristic of the user, as indicated by sensor data depicting the user (e.g., the biometric sensor data) and possibly other data that directly affects the psychophysiological state such as time of day, ambient brightness, ambient noise level, etc. For example, the controller 105 may determine some aspects of the psychophysiological state by direct measurement of the sensor data, such as heart rate, heart rate variability, or electrodermal activity. For another example, the controller 105 may execute a machine-learning algorithm on image data from the user-facing camera 115, such as an object-recognition algorithm, a facial-detection algorithm, an eye-tracking algorithm, etc. The machine-learning algorithm may output an identification of a facial expression or an emotion or level or alertness or drowsiness associated with the identified facial expression, or a direction of the eye gaze. For another example, the controller 105 may combine multiple direct measurements indicative of a more generalized state such as stress or cognitive load. The direct measurements may include heart rate, electrodermal activity, and / or eye gaze movement. The controller 105 may combine the direct measurements by normalizing one or more of the direct measurements to make the units commensurable and then using a simple or weighted average. The output may be a metric indicating stress, cognitive load, etc. The psychophysiological state may include multiple results from the foregoing determinations.

[0070] The controller 105 may determine that the user is interested in the object 200 based on the psychophysiological state of the user. For example, with respect to the biometric characteristics forming the psychophysiological state, various types of attention may be estimated using a model that takes data about head and eye movements of the user from the camera as well as additional data from other biosensors, such as heart rate (HR) dynamics measured through the biometric sensors 125 such as photoplethysmography or radar sensors. The controller 105 may determine that the user is interested in the object 200 according to known correlations between aspects of psychophysiological state and attention. Examples of research that illustrate specific patterns of eye movements that include fixations and saccades are characteristic for certain tasks, including searching for an object 200 may be found, for example, in “Eye Tracking in Visual Search Experiments”, Hollingworth et al., Neuromethods (2 Oct. 2019) 151: 23 - 25 © Spring Science +Business Media, LLC 2019. In addition, tracking objects on videos may be found, for example, in Hyona J, Li J, Oksama L., “Eye Behavior During Multiple Object Tracking and Multiple Identity Tracking. Vision (Basel). 2019 lul 31;3(3):37. doi: 10.3390 / vision3030037. PMID:Atty. Doc. No. HARM0997PCT31735838; PMCID: PMC6802796. Similarly, in virtual reality may be found in, for example, Clay V, Konig P, Kbnig S. Eye Tracking in Virtual Reality. J Eye Mov Res. 2019 Apr 5;12(l):10.16910 / jemr.l2.1.3. doi: 10.16910 / jemr.l2.1.3. PMID: 33828721; PMCID: PMC7903250. In addition, information related to the natural environment when using smart glasses may be found in, for example, Fleming, W., Rizowy, B., & Shwartz, A. (2024). The nature gaze: Eye-tracking experiment reveals well-being benefits derived from directing visual attention towards elements of nature. People and Nature, 6, 1469-1485. In addition, for virtual reality, for example, may be found in Adhanom, IB., MacNeilage, P. & Folmer, E. “Eye Tracking in Virtual Reality: A Broad Review of Applications and Challenges”. Virtual Reality 27, 1481-1505 (2023). Additional research illustrating specific patterns of eye movements for intending to interact with an object may be found in, for example, “Human gaze-aware attentive object detection for ambient intelligence”, Engineering Applications of Artificial Intelligence, Volume 106, 2021, 104471, ISSN 0952-1976. Additional research illustrating specific patterns of eye movements for intending to interact with an object may be found in, for example, Chen, X.-L.; Hou, W.-J. “Gaze-Based Interaction Intention Recognition in Virtual Reality.” Electronics 2022, 11, 1647. https: / / doi.org / 10.3390 / electronicslll01647. Additional research illustrating specific patterns of eye movements for intending to collaborate may be found in, for example, Huang Chien-Ming, Andrist Sean, Sauppe Allison, Mutlu Bilge, “Using gaze patterns to predict task intent in collaboration”, Frontiers in Psychology, V0LUME=6, 2015. Other signals may also be used to improve the accuracy of attention / interest detection as set forth herein. For example, there are studies illustrating that average HR increases and the heart rate variability (HRV) decreases during tasks involving tracking of target stimuli as set forth in, for example, Porges, S. W., & Raskin, D. C. (1969). “Respiratory and heart rate components of attention. Journal of Experimental Psychology”, 81(3), 497-503. In addition, non-linear indexes of the HRV in certain contexts correlate with attention focus as discussed in, for example, Young, H., Benton, D. “We should be using nonlinear indices when relating heart-rate dynamics to cognition and mood.” Sci Rep 5, 16619 (2015). Cardiac activity may be a marker of voluntary attention, as set forth in, for example, Cobos MI, Guerra PM, Vila J, Chica AB. “Heart-rate modulations reveal attention and consciousness interactions.” Psychophysiology. 2019; 56:el3295. https: / / doi.org / 10.llll / psyp.13295. Research also shows that task-related attention is accompanied by specific patterns in respiratory variability and sigh rate as shown in, for example,Atty. Doc. No. HARM0997PCTVlemincx, E., Taelman, J., De Peuter, S., Van Diest, I. and Van Den Bergh, O. (2011), “Sigh rate and respiratory variability during mental load and sustained attention.” Psychophysiology, 48: 117-120. Sustained attention is also accompanied by decreased total breathing variability as shown in, for example, Elke Vlemincx, Use Van Diest, Omer Van den Bergh, “A sigh following sustained attention and mental stress: Effects on respiratory variability”, Physiology & Behavior, Volume 107, Issue 1, 2012. Thus, a variety of biometric characteristics can potentially be used in the development of a model that detects attention and its types.

[0071] With reference to Figures 4-6, the controller 105 is programmed to emit an output related to the object 200 from an output device, for example, from the screen 305, the speaker 135, or another component of the user interface 130. As a general overview, the controller 105 emits the output in response to (1) the eye gaze being directed to the object 200 and (2) the user being interested in the object 200, both as determined above. Emitting the output may include controlling the virtual object 200 (e.g., an avatar) as shown in the screen 305, outputting a statement by the voice assistant depicted as the virtual object 200, outputting a description of a physical object 200, etc.

[0072] For example, the object 200 at which the user’s eye gaze is directed may be a virtual object 200 generated by the user interface 130, e.g., on the screen 305. In particular, the virtual object 200 may be an anthropomorphic avatar with eyes. For the purposes of this disclosure, “anthropomorphic” is defined as, for a nonhuman entity, having human form or characteristics. Because the avatar has eyes, the avatar is capable of appearing to look at the user, for example, to return the user’s gaze and thereby indicate that the system 100 is ready to respond to the user. The illustrations below show examples of the user interaction with the avatar. Figure 4 shows when the system 100 detects the user’s eye gaze directed to the avatar and the avatar looks back at the driver. In Figure 5, the avatar is verbally interacting with the user and maintaining eye contact even though the user is now looking away. Figure 6 illustrates a default eye gaze for the avatar when the user is not looking at the digital display or interacting with it.

[0073] The output emitted by the output device related to the anthropomorphic avatar may be, for example, a specific gesture or motion by the avatar as shown by the screen 305 of the user interface 130 and / or speaking by the avatar as outputted by the speaker 135 of the user interface 130. As one example, the motion by the avatar may include changing a gaze direction of the anthropomorphic avatar. The gaze direction of the anthropomorphic avatar resulting from theAtty. Doc. No. HARM0997PCToutput is directed toward the eyes of the user, as shown in Figures 4 and 5. The default gaze direction of the avatar may be some direction that is not toward the user’s eyes, such as straight ahead as shown in Figure 6. Changing the gaze direction of the avatar toward the user indicates that the avatar is responsive to the user’ s initiation in a way that is easy for the user to immediately understand.

[0074] For another example, the output emitted by the output device related to the object 200 may be activating a voice assistant. As described above, the voice assistant is programmed to receive verbal commands from the user. As one example, the voice assistant may be integrated with the virtual object 200, e.g., the avatar. The verbal output from the voice assistant through the speaker 135 may be depicted on the screen 305 as the avatar speaking. For example, the avatar may say something indicating that the system 100 has responded to the initiation by the user, such as “I’m listening.”

[0075] As another example, the voice assistant may be unrelated to the user’s attention to the avatar. The output emitted by the output device may be related to the physical object 200, e.g., activating the voice assistant to provide information related to the physical object 200. The output may include conveying information about the physical object 200 to the user. The information conveyed about the object 200 may be based on the object recognition described above. As one example, the identification of the object 200 may be used as an input or prompt for another model such as a large language model (LLM), possibly along with other data such as a geographic location of the object 200, a current time or day, etc. The input or prompt to the LLM may further include a request to describe the physical object 200, or some other context-based request. The output from the LLM may be conveyed to the user via a text-to-speech algorithm and the speakers 135, or may be displayed on the screen 305.

[0076] The system 100 and techniques described herein can potentially be used for a variety of scenarios that would not only enrich the environment for the user, but also assist in solving various tasks. For example, information about behavior and human state of the user can be combined with additional contextual information (e.g., location determined based on vehicle global positioning system (GPS), navigation data, etc.) to aid with such activities such as parking. In this example, if the system 100 detects a searching eye gaze pattern and, based on contextual information, it can be assumed the user is looking for a place to park, it can provide information about available parking in the area before the user has to communicate their intention or need. ThisAtty. Doc. No. HARM0997PCTtype of seamless assistance based on attention detection would enrich the user perception of the immediate reality as well as facilitate achieving desired outcomes of various activities.

[0077] Figure 7 is a flowchart illustrating an example process 700 for operating the humanmachine interface. The controller 105 is programmed to perform the process 700; for example, the memory device of the controller 105 stores executable instructions for performing the steps of the process 700 and / or programming can be implemented in structures such as mentioned above. As a general overview of the process 700, the controller 105 receives data from the sensors, determines the direction of the user’s eye gaze, generates a model of the environment around the user, identifies the (virtual or physical) object 200 to which the eye gaze of the user is directed, performs object recognition on that object 200, converts speech from the user to text, and determines whether the user is interested in the object 200. In response to the user being interested in the object 200, the controller 105 emits an output related to the object 200 from the output device.

[0078] The process 700 begins in a block 705, in which the controller 105 receives data from the front-facing camera 110, the user-facing camera 115, the microphone 120, and the biometric sensors 125, as described above.

[0079] Next, in a block 710, the controller 105 determines the gaze direction of the user, as described above.

[0080] Next, in a block 715, the controller 105 generates a model of the environment from sensor data, as described above.

[0081] Next, in a block 720, the controller 105 determines that the eye gaze of the user is directed to an object 200 in the environment.

[0082] Next, in a block 725, the controller 105 performs object recognition on the object 200 at which the user is looking, as described above.

[0083] Next, in a block 730, the controller 105 converts a voice command or other speech from the user (if any) to text, as described above.

[0084] Next, in a decision block 735, the controller 105 determines whether the user is interested in the object 200 at which the user is gazing, as described above. Upon determining that the user is interested in the object 200, the process 700 proceeds to a block 740. In response to the user not indicating interest in the object 200, the process 700 ends.Atty. Doc. No. HARM0997PCT

[0085] In the block 740, the controller 105 emits an output related to the object 200 from an output device (e.g., the speaker 135, the screen 305). After the block 740, the process 700 ends.

[0086] As required, detailed embodiments of the present invention are disclosed herein; however, it is to be understood that the disclosed embodiments are merely exemplary of the invention that may be embodied in various and alternative forms. The figures are not necessarily to scale; some features may be exaggerated or minimized to show details of particular components. Therefore, specific structural and functional details disclosed herein are not to be interpreted as limiting, but merely as a representative basis for teaching one skilled in the art to variously employ the present invention.

[0087] The disclosure has been described in an illustrative manner, and it is to be understood that the terminology which has been used is intended to be in the nature of words of description rather than of limitation. Use of “in response to,” “upon determining,” etc. indicates a causal relationship, not merely a temporal relationship. Many modifications and variations of the present disclosure are possible in light of the above teachings, and the disclosure may be practiced otherwise than as specifically described. Operations, systems, and methods described herein should always be implemented and / or performed in accordance with an applicable owner’ s / user’s manual and / or safety guidelines.

Claims

Atty. Doc. No. HARM0997PCTCLAIMSWhat is claimed is:

1. A method comprising:determining that an eye gaze of a user is directed to an object;determining that the user is interested in the object; andin response to the eye gaze being directed to the object and the user being interested in the object, emitting an output related to the object from an output device.

2. The method of claim 1, wherein the object is a virtual object generated by a user interface.

3. The method of claim 2, wherein the virtual object is an anthropomorphic avatar with eyes.

4. The method of claim 3, wherein the output related to the object includes changing a gaze direction of the anthropomorphic avatar.

5. The method of claim 4, wherein the gaze direction of the anthropomorphic avatar resulting from the output is directed toward eyes of the user.

6. The method of claim 1, wherein determining that the user is interested in the object includes determining that the user is interested in the object based on a voice command from the user.

7. The method of claim 1, wherein determining that the user is interested in the object includes determining that the eye gaze of the user is directed to the object for a duration exceeding a time threshold.

8. The method of claim 1, wherein determining that the eye gaze of the user is directed to the object includes determining that the eye gaze of the user is directed to the object based on a position of eyes of the user, a gaze direction of the eyes of the user, and a position of the object, wherein the position of the object is represented in a same coordinate system as the position of the eyes of the user.Atty. Doc. No. HARM0997PCT9. The method of claim 8, wherein the object is a virtual object generated by a user interface, and the position of the object is prestored.

10. The method of claim 8, wherein the object is a physical object in an environment around the user, the method further comprising determining the position of the object based on data from sensors indicating the environment.

11. The method of claim 1, wherein determining that the user is interested in the object includes determining that the user is interested in the object based on a psychophysiological state of the user.

12. The method of claim 11, further comprising determining the psychophysiological state of the user based on at least one biometric characteristic of the user.

13. The method of claim 1, wherein emitting the output includes activating a voice assistant, the voice assistant being programmed to receive verbal commands from the user.

14. The method of claim 1, wherein the object is a physical object in an environment around the user.

15. The method of claim 14, wherein emitting the output includes conveying information about the physical object to the user.

16. The method of claim 15, further comprising performing object recognition on the object, wherein the information conveyed about the object is based on the object recognition.

17. A controller comprising:at least one memory device;wherein the controller is programmed to perform the method of one of claims 1-16.

18. A system comprising:the controller of claim 17.Atty. Doc. No. HARM0997PCT19. The system of claim 18, further comprising a camera having a field of view encompassing the user, wherein determining that the eye gaze of the user is directed to the object is based on data from the camera.

20. The system of claim 19, wherein the camera is located in a passenger compartment of a vehicle.

Images