885 results about "Gaze directions" patented technology

Apparatuses, systems, and techniques are presented to predict gaze of an observer. In at least one embodiment, a network is trained to predict a gaze of one or more users based, at least in part, on one or more gazes corresponding to objects not always visible to the one or more users.

A vehicular driver monitoring system includes an interior rearview mirror assembly including a mirror head that accommodates a mirror reflective element, a video display screen, and a driver monitoring camera that views at least a driver's head region of the cabin of the vehicle. The vehicular driver monitoring system, based at least in part on processing of image data captured by the driver monitoring camera, determines a gaze direction of the driver. Based at least in part on determination that the gaze direction of the driver is directed is toward the mirror assembly, the video display screen is electrically operated to display video images at a first brightness, and responsive to determination that the gaze direction of the driver is not directed toward the mirror assembly, the video display screen is electrically operated to display video images at a second brightness that is less than the first brightness.

A video perspective method based on virtual reality relates to the technical field of virtual reality, and comprises: at least two cameras for capturing video information in real time, the cameras being provided with spatial positions and angles; performing geometric correction on the video information; the eye movement tracking module monitors the gazing direction of the eyes of the user in real time and adjusts the display position; the eye movement tracking module tracks eyeball movement; reconstructing a three-dimensional model of the surrounding environment of the user; based on the three-dimensional model, determining a relative position relationship between the virtual object and the real object; the head posture sensor obtains head movement information of a user; performing semantic segmentation on the video frame through a deep learning algorithm; and synthesizing the video information and the virtual content on the head-mounted display device, and outputting the synthesized video information and virtual content. The technical problems of visual discomfort and dizziness caused by position mismatching can be solved, and natural fusion of a real scene and a virtual scene is realized.

Eyewear dynamically adjusts viewing effects to match the wearer, the object or scene being viewed (luminance, color prominence, glare, visual blur / noise), other conditions: sensory parameters (gaze direction, focal length, eye gestures, other eye activity, other senses, wearer inputs), medical conditions, wearer location, environmental parameters, wearer activity, use by the wearer, the wearer's field of view. The eyewear can adjust visual features presented to the wearer, such as changes in refraction, polarization / shading, color, prismatic angles / functions, 3D displays. Eyewear can be tailored to form factor: glasses, contacts, RID, IOL, facemask / helmet, vehicles, windows, screens, scopes, AR / VR devices, nerve sensors, external devices. Eyewear can adjust refraction, polarization / shading, color filtering / injection, false coloring, color change; prismatic angles / functions. Eyewear can respond to wearer activity: police, military, firefighter, emergency responder, search and rescue, vehicle operation, sporting / theme-park events, viewing advertising / storefronts, conversation. Hybrid optimization of eyewear can be personalized to users.

The invention relates to an interactive projection optimization method for an exhibition hall, and the method comprises the following steps: before projection is started, sensing the change of ambient light in the exhibition hall through a sensor, and evaluating the illumination conditions of different regions; acquiring a spatial layout in the exhibition hall by using a spatial modeling technology, and identifying wall surface and object shielding information of a projection area; gestures, actions and watching directions of audiences are captured in real time through a camera or infrared induction equipment; dynamically adjusting the projection content according to the action and position of the audience; comprising the following steps: when an audience approaches a certain exhibit, amplifying related projection content or switching to a detailed display mode; collecting behavior data of each audience including stopping time and interaction times of the audience in front of an exhibit; interest points of audiences are predicted through a machine learning model, and projection content related to audience interests is recommended; and adjusting the brightness and color parameters of each projector according to the actual projection area and content of each projection device.

The invention relates to the technical field of pet behavior monitoring, in particular to a pet dangerous behavior monitoring system based on an optic neural network, which comprises the following steps: acquiring image data and spatial point cloud data, extracting image semantic features and spatial structure features by using a semantic segmentation module, and fusing to generate a three-dimensional semantic map; and dangerous areas such as high platforms, foreign matters easy to eat by mistake, fragile objects and the like are marked. A pet skeleton key point sequence is extracted through a posture estimation network, a gazing direction space vector is obtained through a convolutional neural network, a behavior state vector is constructed and then input into a time sequence model, a pet movement target area is predicted, and the pet movement target area is matched with a three-dimensional semantic map to judge whether a dangerous behavior intention exists or not, such as jumping, mistakenly eating or approaching a fragile object. According to the invention, early warning and active intervention of dangerous behaviors of the pet are realized, and the monitoring accuracy and the response capability are remarkably improved.

The invention discloses an eye movement tracking method, an eye movement tracking device, eye movement tracking equipment, computer equipment and a storage medium. The method comprises the following steps: calculating calibration gaze information and a calibration pupil position according to first event data collected by an event camera; determining a region of interest based on the calibrated pupil position; processing second event data collected by the event camera according to the region of interest to generate region of interest data, the collection time of the second event data being later than the collection time of the first event data; according to the region-of-interest data, calculating a current fixation position offset; according to the calibration gaze information and the current gaze position offset, the current gaze information is determined, the calibrated pupil position is updated based on the current gaze position offset, the gaze direction, the moving direction and the moving distance of the pupil are accurately captured in real time by utilizing the characteristics of high time resolution and the like of the event camera, and the eye movement tracking effect is effectively improved.

This disclosure relates to a system and method for real-time monitoring of driver in a vehicle. The method includes capturing at least one of a head movement, steering direction, and a direction of gaze of the driver. The method further includes determining a Plane of Vision (POV) for the driver based on end point of each side view mirrors, a rear-view mirror and bottom of a windshield of the vehicle. The method further includes determining a Region of Interest (ROI) for the driver based on one of a bounding box regression model and a direction of movement of the vehicle. The method further includes determining a Line of Gaze (LOG) for the driver based on the direction of gaze. The method further includes identifying a distraction level of the driver while driving the vehicle based on determining at least one of the POV, the ROI, and the LOG.

The invention discloses an AR glasses control system and method, and belongs to the field of artificial intelligence and AR glasses application, and the method comprises the steps: capturing a user face image, combining a voice assistant, collecting user voiceprint data, and carrying out the matching through a recurrent neural network; constructing a three-dimensional environment model, positioning and marking obstacles and interactive objects, and identifying objects in the environment; the method comprises the following steps of: tracking a gesture, acquiring a key point coordinate by adopting an OpenPose algorithm, identifying a predefined gesture by utilizing a support vector machine, and mapping the gesture into a specific operation instruction; analyzing the user intention based on natural language processing, and calling a corresponding function module according to an analysis result; acquiring a gazing direction of a user by using an eye movement tracking sensor, calculating a screen focus position through an algorithm, and dynamically highlighting options or displaying information according to a gazing point position; through multi-modal data fusion, gestures, voices and fixation point input are analyzed in a unified manner. By accurately understanding the intention of the user, the interaction precision of the AR glasses is improved, and the user experience is optimized.

The invention is a system and method of introducing an automated detection system for manual and semi-automated visual inspection processes. The system employs a facial tracking camera to monitor an operator's gaze direction and level of attention during the inspection process. One or more images of a subject are received, and facial landmarks / associated confidence values are output. The landmark points and any associated confidence values, and / or associated information, may then be input to another set of one or more models that may output any facial analysis quantity or quantities. The system can analyze the data and trigger an alert if it detects an attention lapse, allowing for immediate intervention.

A gaze estimation apparatus according to one or more embodiments may estimate the gaze direction of a target person using calibration information including feature information about the gaze of the eyes of the target person looking in a predetermined direction and true value information indicating the true value for the predetermined direction, in addition to a target image including the eye(s) of the target person. The gaze estimation apparatus may thus estimate the gaze direction of the target person reflecting individual differences, and may allow estimation of the gaze direction of the target person with higher accuracy.

Predictive mixed-scale encoding of a media stream is described. A composite gaze path is determined based on multiple gaze paths. Each of the gaze paths includes gaze direction(s) obtained when a user views the media stream with a user device. A foveation weight map associated with a scene of the media stream is generated. The generation includes: determining, based on the plurality of gaze paths, a gaze heatmap that includes for each direction within the scene a score based on a number of users that had that direction as a gaze direction when viewing the scene; determining, based on the gaze heatmap, a region of interest that includes region(s) of high gaze concentration; and determining, based on the region of interest, the foveation weight map that includes foveation weights within a foveation area surrounding the region of interest and centered at a composite gaze direction from the composite gaze path.

Disclosed herein are system, method, and device embodiments for determining the mental state of a driver using gaze tracking and object detection. Specifically, an Advanced Driver Assistance System may determine if the driver's gaze is associated with spatial locations of objects of interest in the surrounding environment. The surrounding environment may be represented with a 360-degree, three-dimensional model generated by the external-facing cameras and other sensors. The driver's gaze direction may be represented by a heatmap derived based on the driver's eye movements. Whether or not the driver's gaze is associated with a spatial location of an object may be used as evidence about the driver's mental state. The system may perform appropriate actions based on the mental state including generating alerts, engaging mechanical systems, and adjusting parameters in ancillary safety systems to increase or decrease tolerance. The mental state may be tracked throughout a driving session and stored in a driver profile.

The invention provides a driving behavior intelligent identification method and system based on video analysis, and the method comprises the steps: obtaining a driver face video stream and a road environment video stream collected by a vehicle-mounted camera, and reading the driving information recorded by a whole vehicle communication network; recognizing an eyelid closing state, a sight line direction and a head posture in the driver face video stream based on a posture recognition model, and performing fatigue distraction analysis to obtain driver state information; performing motion trail analysis on the road environment video stream and the driving information, and performing driving risk assessment in combination with the driver state information to obtain driving assessment information; and performing early warning construction according to the driving evaluation information, generating early warning prompt information, and synchronously writing the early warning prompt information, the driving evaluation information and the driver state information into a safety data protection memory. The fatigue and distraction states of the driver can be recognized more accurately, and the accuracy of state judgment is improved.

The embodiment disclose a wearable augmented reality eyewear system including a frame supporting left and right transparent lenses. A plurality of outward-facing cameras mounted on the frame have overlapping fields of view for capturing real-time images of the environment. A plurality of inward-facing cameras integrated in the frame or lenses detect pupil size, pupil dilation, and gaze direction using infrared imaging. A digital processor analyzes gaze direction and fixation based on virtual grid coordinates, initiates an object or person recognition routine using images from the outward-facing cameras, and compares detected features with records in a database. At least one lens includes an AR projection system embedded therein and configured to superimpose data about the recognized object or person in the user's field of view. An audio speaker positioned near the user's ear outputs related audio. A wireless communication module enables real-time data exchange with a cloud-based platform.

A method for calibrating eye tracking data for a head wearable optical device includes receiving eye tracking data from an imaging device oriented towards a combiner. The method includes determining a gaze direction of an eye of a user based on the eye tracking data. The method includes receiving calibration data from the imaging device that includes an indication of positions of one or more calibration indicators. The method includes determining the gaze direction based at least in part on the calibration data received from the imaging device (or factors adjusted by the calibration data).

A vehicle includes an eye tracker for tracking a gaze of an occupant of the vehicle. The vehicle has a processor to use gaze direction data to determine if the occupant is gazing at a nearby vehicle sensed by a vehicle-detecting sensor, e.g. a camera, LIDAR or RADAR. The processor determines an identification of the nearby vehicle by optically recognizing characters of a license plate of the nearby vehicle using the camera, or by detecting an identification beacon broadcast by the nearby vehicle using a short-range vehicle-to-vehicle transceiver. The vehicle includes a radiofrequency cellular transceiver to transmit a message to a server to look up the identification of the vehicle and to receive a communication-availability reply message indicating either a communication availability or communication unavailability of the nearby vehicle. The radiofrequency cellular transceiver or the short-range vehicle-to-vehicle transceiver initiates a voice communication with the nearby vehicle.

Disclosed is an imaging system of a display apparatus with gaze-tracking means and processor(s). The processor(s) is / are configured to: process gaze-tracking data, collected by the gaze-tracking means, to detect gaze directions of a user's eyes; determine a gaze convergence distance, based on a convergence of the gaze directions of the user's eyes; identify a region of interest in a given image frame, based on a gaze direction of a given eye of the user from a perspective of which the given image frame is rendered; and generate a reprojected image frame, by reprojecting the region of interest using six degrees-of-freedom reprojection, whilst considering the gaze convergence distance as an optical depth of pixels of the region of interest.

The driver monitoring device includes a processor configured to: detect an object having a possibility of affecting traveling of a vehicle based on an external sensor signal, detect, among a gaze direction of a driver of the vehicle and an orientation of a face of the driver, at least the gaze direction based on a driver image, determine whether an object direction to the detected object is included within a peripheral visual range corresponding to a peripheral visual field of the driver based on the gaze direction, set an inattentiveness warning condition when the object direction is included within the peripheral visual range based on a driving mode or the situation around the vehicle, and notify the driver of an inattentiveness warning via a notification device when at least detected one of the orientation of the face and the gaze direction satisfies the inattentiveness warning condition.

Disclosed is a system with at least one server that is communicably coupled to at least one display apparatus, wherein the at least one server is configured to detect a start of a Vestibulo-Ocular Reflex (VOR) movement (T1) based on a gaze tracking-data and a pose tracking-data received from the at least one display apparatus; control remote rendering of an extended reality video stream at the at least one display apparatus by modifying a head pose data indicated in the pose tracking-data and a gaze direction data indicated in the gaze tracking-data such that a total retinal angular velocity of a focus target approaches a zero value during a VOR movement (T2); detect an end of the VOR movement (T3); and stop the modification and revert video rendering parameters to pre-set default settings after the end of the VOR movement (T3).

A system and method for attention redirection is disclosed. The system may include a display, an environment imaging sensor configured to receive environment images comprising elements of an environment, a user state sensor configured to receive user images comprising a user gaze, and a controller. The controller may be configured to execute program instructions to receive environment images, identify elements and their identities and locations, receive user images comprising the user gaze, track the user gaze direction, determine salience values associated with the elements, and direct a focus adjustment of at least one region of the display based on the salience values.

The invention is applicable to the technical field of information interaction, and provides an asymptomatic patient interaction method based on an eye-controlled staring trigger communication interface, which comprises the following steps: sight tracking: acquiring facial image data of a patient through a camera, extracting facial and eye mark points, performing image processing on an eye area, and judging a staring direction; staring triggering: presetting a coordinate area of an interaction control, judging whether a fixation point of a patient falls into the control area in real time, and if the fixation point is continuously kept for preset time, triggering corresponding operation; executing a compensation mechanism: compensating head movement and illumination change in real time, and correcting a staring point coordinate; and interface design: designing an interactive interface with a daily communication function, and broadcasting the content of the control through voice. The invention provides an efficient, convenient and easy-to-use eye-controlled staring triggering communication interface for the patient with the asymptomatic disease, aims to reduce the learning cost and the operation complexity of the patient, and has great significance in improving the life quality of the patient with the asymptomatic disease.

A system is described for determining accuracy of an eye-tracking device using an oculomotor simulator assembly. The system receives instructions specifying movement for simulated eyeballs in the oculomotor simulator assembly. The system enables movement of the simulated eyeballs based on the received instructions. The system obtains ground-truth values for three-dimensional gaze directions of the simulated eyeballs in response to the enabled movement using projections of laser diodes from the center of the simulated eyeballs during the specified movement. The system additionally gathers three-dimensional gaze directions as determined by the eye-tracking device that is tracking the simulated eyeballs during the specified movement. The system determines accuracy of the eye-tracking device based on the obtained ground-truth values and the determined three-dimensional gaze directions from the eye-tracking device.

The invention discloses a multi-camera imaging method and system for head-mounted equipment, and relates to the technical field of multi-camera imaging, and the system comprises an image data acquisition module, a weight acquisition module, an image fusion module, a feedback control module, a visual angle movement determination module and a preloading module. The gaze weight of the image is obtained by obtaining the gaze direction of the user, an image weighted fusion mode with the current visual attention area of the user as the core is achieved, the generated main view angle image better conforms to the actual view field of the user, and immersive visual experience is enhanced; inter-frame time sequence consistency loss is introduced in the fusion process, the pixel change of the fusion image between continuous frames is restrained through time sequence smoothing, the image flicker and jump phenomena are remarkably reduced, and the visual stability in a dynamic scene is improved.

An apparatus for steering adjustment detects a gaze direction of a driver of a vehicle from a facial image in which a face of the driver is represented. During autonomous driving in which steering of the vehicle is autonomously controlled, when an operation by the driver on a steering wheel is detected, determines whether the gaze direction of the driver corresponds to the steering direction in the operation of the driver. The apparatus adjusts a reaction force with respect to the operation of the steering wheel smaller when the gaze direction corresponds to the steering direction than when the gaze direction does not correspond to the steering direction.

Various implementations disclosed herein include devices, systems, and methods that initiate an action based on a detected gaze direction oriented towards a target area (e.g., a hot corner / zone). For example, an example process may include producing a reflection by directing light towards an eye using an illuminator, receiving sensor data from a sensor, wherein a direction of sensing by the sensor (e.g., the optical axis of the camera) and a direction from the eye to a target area are approximately aligned, determining a reflective property (e.g., a spectral property) of the reflection based on the sensor data, detecting that a gaze direction of the eye is approximately oriented towards the target area (e.g., a hot zone / spot) based on the reflective property, and initiating an action based on detecting that the gaze direction is approximately oriented towards the target area.

A head mounted display (HMD) device is provided. The HMD includes a display, a gaze sensor configured to detect a gaze direction of a user of the HMD device, a camera, a position sensor configured to detect a position of the HMD device, communication circuitry, memory storing one or more computer programs, and one or more processors communicatively coupled to the display, the gaze sensor, the camera, the position sensor, the communication circuitry, and the memory, wherein the one or more computer programs include computer-executable instructions that, when executed by the one or more processors individually or collectively, cause the HMD device to, based on the position of the HMD device, acquire three-dimensional (3D) segmentation information for a space corresponding to the position of the HMD device, acquire gaze point-based interest information corresponding to a profile of the user, the gaze point-based interest information including interest information generated based on gaze points of a plurality of users, display, on the display, a first visual effect for a plurality of regions in the space based on the 3D segmentation information and the interest information, and display, on the display, a second visual effect for a plurality of objects in a first region among the plurality of regions based on the 3D segmentation information and the interest information when the position of the HMD device corresponds to the first region.

The disclosure relates to technology for monitoring driver attentiveness in a vehicle. A driver distraction system collects vehicle data and scene information from the vehicle while traveling on a route. The vehicle data and scene information are then processed to generate a reference heat map. At the same time, the driver distraction system may capture a gaze of a driver to track a gaze direction and duration of the driver while driving the vehicle on the route. The gaze direction and duration are processed to generate a driver gaze heat map. The driver gaze heat map and reference heat map are analyzed to determine a level of driver distraction of the driver in the vehicle, and a recommendation or warning is output to the driver of the vehicle according to the level of driver distraction.

Methods and systems for performing operations to control brightness in an AR device are disclosed. The operation comprises: displaying an image on an eye wearable device worn by a user; detecting a gazing direction of pupils of the user; identifying a first region of the image corresponding to a gaze direction of the pupil; and modifying a luminance level or luminance value of pixels in the image based on the gaze direction such that pixels in a first region of the image are set to a first luminance value and pixels in a second region of the image are set to a second luminance value lower than the first luminance value.

The invention relates to an intelligent cabin interaction method and system and a medium, and relates to the field of intelligent cabins, and the method comprises the steps: obtaining the facial expression and voice information of a user; performing sentiment classification on the facial expression and the voice information by using a deep learning model; based on the sentiment classification result, adjusting the environment atmosphere in the cabin; creating a three-dimensional holographic image on the vehicle-mounted head-up display through holographic projection equipment; receiving a voice instruction, a gesture action or a staring direction of a user; and controlling functions in the cabin according to the voice instruction, the gesture action or the staring direction. The user interaction experience, the emotion response ability and the intelligent level of the intelligent cabin can be improved, and safer, more comfortable and more personalized driving experience is provided for a driver.