580 results about "Gaze directions" patented technology

The invention discloses an eye movement tracking method, an eye movement tracking device, eye movement tracking equipment, computer equipment and a storage medium. The method comprises the following steps: calculating calibration gaze information and a calibration pupil position according to first event data collected by an event camera; determining a region of interest based on the calibrated pupil position; processing second event data collected by the event camera according to the region of interest to generate region of interest data, the collection time of the second event data being later than the collection time of the first event data; according to the region-of-interest data, calculating a current fixation position offset; according to the calibration gaze information and the current gaze position offset, the current gaze information is determined, the calibrated pupil position is updated based on the current gaze position offset, the gaze direction, the moving direction and the moving distance of the pupil are accurately captured in real time by utilizing the characteristics of high time resolution and the like of the event camera, and the eye movement tracking effect is effectively improved.

The invention provides a driving behavior intelligent identification method and system based on video analysis, and the method comprises the steps: obtaining a driver face video stream and a road environment video stream collected by a vehicle-mounted camera, and reading the driving information recorded by a whole vehicle communication network; recognizing an eyelid closing state, a sight line direction and a head posture in the driver face video stream based on a posture recognition model, and performing fatigue distraction analysis to obtain driver state information; performing motion trail analysis on the road environment video stream and the driving information, and performing driving risk assessment in combination with the driver state information to obtain driving assessment information; and performing early warning construction according to the driving evaluation information, generating early warning prompt information, and synchronously writing the early warning prompt information, the driving evaluation information and the driver state information into a safety data protection memory. The fatigue and distraction states of the driver can be recognized more accurately, and the accuracy of state judgment is improved.

The embodiment disclose a wearable augmented reality eyewear system including a frame supporting left and right transparent lenses. A plurality of outward-facing cameras mounted on the frame have overlapping fields of view for capturing real-time images of the environment. A plurality of inward-facing cameras integrated in the frame or lenses detect pupil size, pupil dilation, and gaze direction using infrared imaging. A digital processor analyzes gaze direction and fixation based on virtual grid coordinates, initiates an object or person recognition routine using images from the outward-facing cameras, and compares detected features with records in a database. At least one lens includes an AR projection system embedded therein and configured to superimpose data about the recognized object or person in the user's field of view. An audio speaker positioned near the user's ear outputs related audio. A wireless communication module enables real-time data exchange with a cloud-based platform.

Various implementations disclosed herein include devices, systems, and methods that initiate an action based on a detected gaze direction oriented towards a target area (e.g., a hot corner / zone). For example, an example process may include producing a reflection by directing light towards an eye using an illuminator, receiving sensor data from a sensor, wherein a direction of sensing by the sensor (e.g., the optical axis of the camera) and a direction from the eye to a target area are approximately aligned, determining a reflective property (e.g., a spectral property) of the reflection based on the sensor data, detecting that a gaze direction of the eye is approximately oriented towards the target area (e.g., a hot zone / spot) based on the reflective property, and initiating an action based on detecting that the gaze direction is approximately oriented towards the target area.

Systems and methods for eye tracking and content tagging in minimally invasive surgical videos are described herein. Minimally invasive surgical videos may be captured while performing robotic surgeries. Robotic surgical systems described herein include robotic arms with interchangeable surgical tools. An endoscope at the end of one of the robotic arms captures video of the surgical procedure. The video is displayed on a display of the surgical system and an eye tracking device captures data corresponding to a gaze direction of the user on the display. Content tags are automatically generated in the image data based on areas of focus of the user. Information within the content tag may be generated based on surgical procedure steps derived from the image data, a log of current surgical procedures, and image data of previous surgical procedures.

A head mounted display for use in a virtual reality, mixed reality or augmented reality system comprises a display device configured to display an image, a tracking system configured to determine a gaze direction of a user's eye and an optical system configured to project the image into the user's eye, the optical system comprising an adjustable optical device; and processing circuitry configured to control the adjustable optical device based on the gaze direction of the user's eye such that a position of the image projected into the user's eye is unchanged, when the gaze direction of the user's eye changes.

An electronic device may include a waveguide that directs light to an eye box. The waveguide may include a diffractive grating structure. A projector may generate image light coupled into the waveguide by a first input coupler. A light source may generate supplemental light coupled into the waveguide by a second input coupler. The diffractive grating structure may couple the image light out of the waveguide within a central region of the field of view (FOV) of an eye box and may couple the supplemental light out of the waveguide at a location within a peripheral region of the FOV. The location may be outside of the central region by at least 10 degrees. A gaze tracking sensor may estimate a user's gaze direction at the eye box. A processor may wake or power down the projector responsive to the gaze direction overlapping the location for a predetermined time period.

Techniques are provided for real-time calibration of an eye-tracking system in a wearable display device using user interactions with a graphical user interface. A background calibration process continuously or intermittently updates a calibration matrix based on stable gaze-target pairs inferred from natural user behavior, without requiring explicit calibration routines. The system detects user interactions such as selections of UI elements, and associates them with gaze direction data captured immediately beforehand. Fixation filtering is applied to identify stable gaze intervals based on dispersion thresholds and minimum duration criteria.

An electronic device that includes a display and an eye tracker configured to collect eye tracking data regarding a gaze of one or more of a user's eyes across the display. The electronic device also includes processing circuitry that is operatively coupled to the display and configured to generate pixel data for frames of content based on the eye tracking data such that the content is configured to be shown on the display in a dynamically foveated manner. The processing circuitry is configured to determine dither blocks, each of which corresponds to a subset of the plurality of pixels. The processing circuitry is configured to apply a dither pattern to the frames of the plurality of frames of content independent of the gaze of one or more of the user's eyes and based on the dither blocks.

An audiovisual system uses a spatial position detection module to process audio signals and determine the location and orientation of a speaking participant. Based on this data, the system dynamically controls sensors to optimize audio and video capture. Behavioral and contextual information may also be used to train intelligence models for improved system performance. Further, sensor arrays may be utilized to identify gaze vectors of participants to select a camera sensor of the sensor array. Meeting content collected with the sensor array may be organized into meeting metrics in accordance with an analytics strategy before training an intelligence module with the meeting metrics. Meeting content may also be configured into digital tiles in accordance with a tile strategy. At least one of the digital tiles may be altered in response to a meeting condition detected by the sensor array.

The invention discloses an automatic adjustment and feedback control method for an intelligent rearview mirror in an automatic driving environment, and relates to the technical field of vehicle accessories. Comprising the following steps: defining an intelligent rearview mirror focusing area based on a driver view position, so that the focusing area is matched with the gazing direction of a driver; the displacement track of the reference target in the focusing area is analyzed based on the time sequence analysis technology; and dynamically adjusting the angle of the intelligent rearview mirror according to the displacement track analysis result and the driving posture of the driver. By constructing the three-dimensional coordinates based on the intelligent rearview mirrors on the two sides of the A column and the intelligent rearview mirrors in the vehicle, the head datum point position and the sight direction vector of the driver are obtained, so that the real-time view position of the driver is determined, the focusing area on the rearview mirror imaging plane is dynamically defined, the gazing direction of the driver is effectively matched, and the driving safety is improved. Real-time and dynamic recognition of the visual field of the driver is achieved, and the consistency of the imaging area of the rearview mirror and the actual attention area of the driver is guaranteed.

The voice directional enhancement method comprises the steps that a current frame and a staring direction of mixed voice are acquired, and the mixed voice is acquired by a microphone array; constructing a direction feature based on the current frame of the mixed voice and the gazing direction, and constructing an audio feature based on the current frame of the mixed voice; determining a current frame mask according to the direction features and the audio features; and determining a target enhanced voice of the current frame according to the current frame mask, the current frame of the mixed voice and the plurality of historical frames of the mixed voice. In the embodiment of the invention, the mask of the current frame is generated under the guidance of the gazing direction, so that the target voice can be accurately separated from the mixed voice. Besides, based on the current frame mask and the current frame of the mixed voice, the target voice enhancement is performed frame by frame from the overall characteristics of the mixed voice by referring to a plurality of historical frames, so that the accuracy of the target enhanced voice of the current frame can be effectively improved, and thus the accuracy of directional pickup is improved.

A light-transmitting substrate is deployed with a first of two major surfaces in facing relation to an eye of a viewer and guides light by internal reflection between the two major surfaces. An optical coupling-out configuration couples image light, that corresponds to a collimated image and that is guided by internal reflection between the two major surfaces, out of the light-transmitting substrate. A first optical coupling configuration collimates light from the eye to produce collimated light, and couples the collimated light into the light-transmitting substrate for guiding by internal reflection. A second optical coupling configuration couples the collimated light out of the light-transmitting substrate toward an optical sensor that senses the coupled-out light. A processing system derives current gaze direction of the eye by processing signals from the optical sensor.

A head mounted display system for displaying image content to a user comprises at least one display configured to be worn by a user to present virtual content to first and second eyes of a user, one or more inwardly facing sensors or camera configured to monitor one or both of the users eye and processing electronics. This head mounted display system is configured such that virtual content activity can be initiated and / or driven from eye inputs such as gaze direction, eyelid motions (e.g., blinking), and / or other eye gestures.

Embodiments of the disclosure provides a method, an apparatus of line-of-sight detection, an electronic device and a storage medium. The method includes: obtaining a current image frame and a first reference image frame, where in the current image frame is an eye image collected in real time, and the first reference image frame is an eye image collected before the current image frame; obtaining an inter-frame variation amount of the current image frame relative to the first reference image frame; and deriving a target gaze direction based on the inter-frame variation amount, where the target gaze direction is a first gaze direction or a second gaze direction, the first gaze direction is a gaze direction pre-generated based on the first reference image frame, and the second gaze direction is a gaze direction generated in real time based on the current image frame.

The present invention relates to a blind spot display device for construction equipment including a detector provided in the construction equipment to detect an object around the construction equipment; a display provided in front of eyes of a driver of the construction equipment; a gaze tracker tracking a direction of a gaze of the driver; and a controller acquiring information from the detector and the gaze tracker and analyzing information on a direction consistent with the gaze of the driver, wherein the controller displays a marker for the object included in the direction of the gaze of the driver on the display.

The invention relates to the technical field of man-machine interaction, and discloses an eye movement tracking method, intelligent glasses, a mobile terminal and an eye movement tracking system.The method comprises the steps that current head movement data of a wearer is collected and sent to the mobile terminal; receiving the current gazing direction of the wearing user fed back by the mobile terminal, and tracking the eye movement of the wearing user according to the current gazing direction; wherein the current gazing direction is obtained through the mobile terminal according to the current head movement data and a current eyeball image of the wearing user, and the current eyeball image is obtained by shooting eyeballs of the wearing user through a camera. Since the mobile terminal with the camera arranged outside is utilized to complete eyeball collection and fixation direction determination of the wearing user, an independent camera and an independent computing power unit do not need to be arranged in the intelligent glasses, and the size of the intelligent glasses is reduced.

A retinal projection display system includes at least one visible light source for projecting a visible light image, an infrared light source for projecting infrared light, a scanning mirror having a field of view larger than the visible light image, a reflective surface on which the visible light image is projected and on which the infrared light is reflected at least partially towards an eye of a user, wherein the reflective surface is larger than the visible light image, at least one infrared photodetector for receiving reflected infrared light that reflects off of the eye of the user, and a hardware computation module comprising a processor and a memory, the hardware computation module configured to determine a gaze direction of the user based at least in part on the reflected infrared light.

Eye tracking is performed by determining an initial pupil position of a user in relation to a lens situated in front of the user, detecting a change in pupil position in relation to the lens to an updated pupil position in relation to the lens, and determining that the updated pupil position is outside a bounding box associated with the lens. The updated pupil position is a replacement pupil position with a replacement pupil position within the bounding box associated with the lens, and the updated pupil position is utilized for eye-tracking functionality. Eye tracking is also performed by determining that a first pixel associated with a gaze direction is outside a visibility region, identifying a replacement pixel within the visibility region, determining an updated gaze angle based on the replacement pixel, and performing eye tracking using the updated gaze angle.

This disclosure relates to a system and method for real-time monitoring of driver in a vehicle. The method includes capturing at least one of a head movement, steering direction, and a direction of gaze of the driver. The method further includes determining a Plane of Vision (POV) for the driver based on end point of each side view mirrors, a rear-view mirror and bottom of a windshield of the vehicle. The method further includes determining a Region of Interest (ROI) for the driver based on one of a bounding box regression model and a direction of movement of the vehicle. The method further includes determining a Line of Gaze (LOG) for the driver based on the direction of gaze. The method further includes identifying a distraction level of the driver while driving the vehicle based on determining at least one of the POV, the ROI, and the LOG.

A remote operator terminal is used by a remote operator for a remote operation of a moving body. A first image is captured at a first timing by a camera mounted on the moving body from a first point of view. A second point of view is one at a second timing later than the first timing. A homography process converts the first image viewed from the first point of view into a second image viewed from the second point of view. The remote operator terminal acquires a plurality of second images by applying the homography process to a plurality of first images, respectively. The remote operator system displays the plurality of second images side-by-side on a display such that a priority second image corresponding to a gaze direction of the remote operator appears to be in front of the other second images.

Systems and methods for determining the gaze direction of a subject and projecting this gaze direction onto specific regions of an arbitrary three-dimensional geometry. In an exemplary embodiment, gaze direction may be determined by a regression-based machine learning model. The determined gaze direction is then projected onto a three-dimensional map or set of surfaces that may represent any desired object or system. Maps may represent any three-dimensional layout or geometry, whether actual or virtual. Gaze vectors can thus be used to determine the object of gaze within any environment. Systems can also readily and efficiently adapt for use in different environments by retrieving a different set of surfaces or regions for each environment.

A system for controlling a vehicle operation based on a driver gaze direction includes a plurality of sensors and an electronic processor. The electronic processor is configured to determine a gaze direction of a driver using the plurality of sensors and selectively activating a vehicle function based on the gaze direction.

A method includes receiving scene image data, comprising left and right images for each eye. Through an eye tracking module, the method includes determining the user's gaze direction or eye vergence. Using this information, the method includes identifying an object in the scene that the user is focusing on, and determining, using a depth estimation module, a left depth from the left eye to the object and a right depth from the right eye to the object. Further, based on the computed left and right depths, the method includes generating, for the left and right eye, constant left and right depth meshes, generating a left and right output images by projecting the left and right images on the corresponding constant left and right depth meshes. Additionally, the method includes displaying the left output image for the left eye and displaying the right output image for the right eye.

The invention provides an iris collection method and device based on eye movement prediction, equipment and a storage medium, and the method comprises the steps: collecting image frames and event streams in a mixed manner through an event camera and an RGB camera, obtaining eye feature point coordinates through key point detection, and converting the event streams into an event frame sequence; performing spatial-temporal feature extraction and filtering processing on the eye feature point coordinates and the event frame sequence by using a time sequence prediction network, and predicting eyeball positions and gazing directions at future moments; calculating the optimal optical axis direction of iris imaging according to the predicted eyeball position and gazing direction, and converting the optimal optical axis direction into control parameters such as a pan-tilt angle and a lens focal length of an imaging system; and synchronously adjusting the imaging system and triggering acquisition when the prediction time arrives. According to the method, active iris tracking acquisition is realized by predicting the eyeball movement track, the problem of response lag of traditional passive tracking is solved, and the success rate of iris acquisition is improved.

Provided herein are system, apparatus, device, method and / or computer program product embodiments, and / or combinations and sub-combinations thereof, for improving object detection based on gaze direction. In the method, a first image of surroundings of a vehicle is received. A second image of a driver of the vehicle is also received. The second image captured contemporaneously with the first image. Based on the second image, a gaze direction of the driver is determined. Based on the gaze direction, a region of the first image being viewed by the driver is determined. An object detection algorithm is applied to recognize an object in the surroundings of the vehicle at a greater level of detail within the region.

In an exemplary process for interacting with affordances using eye gaze, a first affordance is displayed concurrently with a second affordance. In response to determining that a first gaze direction or a first gaze depth of a user corresponds to both the first affordance and the second affordance, the first affordance and the second affordance are enlarged.

An adaptive display buffering method for enhanced visual comfort includes: original display data is obtained from an off-screen display buffer; an encoded vector is generated based on the original display data, and content feature scoring is performed based on the encoded vector; content features of the original display data are modified based on the content feature scoring and a user gaze direction, and a content regulator applies different modification methods to a foveal region and a peripheral region; and modified display data is stored into an on-screen display buffer. An adaptive display buffering system for enhanced visual comfort is also provided.

A driving training system based on virtual reality comprises a VR head-mounted device used for projecting a road condition and a virtual vehicle in a virtual driving process to a driver; the eyeball tracking module is used for capturing gazing direction gazing data of the driver for different virtual components in virtual driving; a hand tracking module for capturing motion data associated with driver hand motion to access different virtual components of the virtual vehicle during virtual driving; the driving learning module is used for introducing a driving principle and improving a driving skill through system courses of different themes of a virtual vehicle based on the watching data and the motion data of the driver in virtual driving; the performance evaluation module is used for determining indexes related to driving skills and performance of the driver in virtual driving based on the captured gaze data, the motion data and the tracked navigation; and the danger simulation module is used for simulating dangerous events possibly occurring in the real world in virtual driving so as to improve the response and alertness of the driver.

To reliably determine the direction of the subject's gaze. [Solution] The gaze direction detection device 1 comprises a camera 10 that acquires a face image by capturing the face of subject A, and an image processing device 20 that processes the face image acquired by the camera 10 to determine the gaze direction of subject A. The image processing device 20 detects the three-dimensional positions of the right pupil, left pupil, and nose based on the face image, identifies the three-dimensional position of the intersection of a second straight line that passes through the three-dimensional position of the nose and intersects perpendicularly with a first straight line connecting the three-dimensional positions of the right pupil and the three-dimensional positions of the left pupil, and the first straight line, and determines the gaze direction of subject A using the first distance between the three-dimensional position of the intersection and the three-dimensional position of the right pupil, the second distance between the three-dimensional position of the intersection and the three-dimensional position of the left pupil, and the third distance between the three-dimensional position of the intersection and the three-dimensional position of the nose.