726 results about "Gaze directions" patented technology

Apparatuses, systems, and techniques are presented to predict gaze of an observer. In at least one embodiment, a network is trained to predict a gaze of one or more users based, at least in part, on one or more gazes corresponding to objects not always visible to the one or more users.

The invention relates to the technical field of pet behavior monitoring, in particular to a pet dangerous behavior monitoring system based on an optic neural network, which comprises the following steps: acquiring image data and spatial point cloud data, extracting image semantic features and spatial structure features by using a semantic segmentation module, and fusing to generate a three-dimensional semantic map; and dangerous areas such as high platforms, foreign matters easy to eat by mistake, fragile objects and the like are marked. A pet skeleton key point sequence is extracted through a posture estimation network, a gazing direction space vector is obtained through a convolutional neural network, a behavior state vector is constructed and then input into a time sequence model, a pet movement target area is predicted, and the pet movement target area is matched with a three-dimensional semantic map to judge whether a dangerous behavior intention exists or not, such as jumping, mistakenly eating or approaching a fragile object. According to the invention, early warning and active intervention of dangerous behaviors of the pet are realized, and the monitoring accuracy and the response capability are remarkably improved.

The invention discloses an eye movement tracking method, an eye movement tracking device, eye movement tracking equipment, computer equipment and a storage medium. The method comprises the following steps: calculating calibration gaze information and a calibration pupil position according to first event data collected by an event camera; determining a region of interest based on the calibrated pupil position; processing second event data collected by the event camera according to the region of interest to generate region of interest data, the collection time of the second event data being later than the collection time of the first event data; according to the region-of-interest data, calculating a current fixation position offset; according to the calibration gaze information and the current gaze position offset, the current gaze information is determined, the calibrated pupil position is updated based on the current gaze position offset, the gaze direction, the moving direction and the moving distance of the pupil are accurately captured in real time by utilizing the characteristics of high time resolution and the like of the event camera, and the eye movement tracking effect is effectively improved.

The invention discloses an AR glasses control system and method, and belongs to the field of artificial intelligence and AR glasses application, and the method comprises the steps: capturing a user face image, combining a voice assistant, collecting user voiceprint data, and carrying out the matching through a recurrent neural network; constructing a three-dimensional environment model, positioning and marking obstacles and interactive objects, and identifying objects in the environment; the method comprises the following steps of: tracking a gesture, acquiring a key point coordinate by adopting an OpenPose algorithm, identifying a predefined gesture by utilizing a support vector machine, and mapping the gesture into a specific operation instruction; analyzing the user intention based on natural language processing, and calling a corresponding function module according to an analysis result; acquiring a gazing direction of a user by using an eye movement tracking sensor, calculating a screen focus position through an algorithm, and dynamically highlighting options or displaying information according to a gazing point position; through multi-modal data fusion, gestures, voices and fixation point input are analyzed in a unified manner. By accurately understanding the intention of the user, the interaction precision of the AR glasses is improved, and the user experience is optimized.

Disclosed herein are system, method, and device embodiments for determining the mental state of a driver using gaze tracking and object detection. Specifically, an Advanced Driver Assistance System may determine if the driver's gaze is associated with spatial locations of objects of interest in the surrounding environment. The surrounding environment may be represented with a 360-degree, three-dimensional model generated by the external-facing cameras and other sensors. The driver's gaze direction may be represented by a heatmap derived based on the driver's eye movements. Whether or not the driver's gaze is associated with a spatial location of an object may be used as evidence about the driver's mental state. The system may perform appropriate actions based on the mental state including generating alerts, engaging mechanical systems, and adjusting parameters in ancillary safety systems to increase or decrease tolerance. The mental state may be tracked throughout a driving session and stored in a driver profile.

The invention provides a driving behavior intelligent identification method and system based on video analysis, and the method comprises the steps: obtaining a driver face video stream and a road environment video stream collected by a vehicle-mounted camera, and reading the driving information recorded by a whole vehicle communication network; recognizing an eyelid closing state, a sight line direction and a head posture in the driver face video stream based on a posture recognition model, and performing fatigue distraction analysis to obtain driver state information; performing motion trail analysis on the road environment video stream and the driving information, and performing driving risk assessment in combination with the driver state information to obtain driving assessment information; and performing early warning construction according to the driving evaluation information, generating early warning prompt information, and synchronously writing the early warning prompt information, the driving evaluation information and the driver state information into a safety data protection memory. The fatigue and distraction states of the driver can be recognized more accurately, and the accuracy of state judgment is improved.

The embodiment disclose a wearable augmented reality eyewear system including a frame supporting left and right transparent lenses. A plurality of outward-facing cameras mounted on the frame have overlapping fields of view for capturing real-time images of the environment. A plurality of inward-facing cameras integrated in the frame or lenses detect pupil size, pupil dilation, and gaze direction using infrared imaging. A digital processor analyzes gaze direction and fixation based on virtual grid coordinates, initiates an object or person recognition routine using images from the outward-facing cameras, and compares detected features with records in a database. At least one lens includes an AR projection system embedded therein and configured to superimpose data about the recognized object or person in the user's field of view. An audio speaker positioned near the user's ear outputs related audio. A wireless communication module enables real-time data exchange with a cloud-based platform.

The invention is applicable to the technical field of information interaction, and provides an asymptomatic patient interaction method based on an eye-controlled staring trigger communication interface, which comprises the following steps: sight tracking: acquiring facial image data of a patient through a camera, extracting facial and eye mark points, performing image processing on an eye area, and judging a staring direction; staring triggering: presetting a coordinate area of an interaction control, judging whether a fixation point of a patient falls into the control area in real time, and if the fixation point is continuously kept for preset time, triggering corresponding operation; executing a compensation mechanism: compensating head movement and illumination change in real time, and correcting a staring point coordinate; and interface design: designing an interactive interface with a daily communication function, and broadcasting the content of the control through voice. The invention provides an efficient, convenient and easy-to-use eye-controlled staring triggering communication interface for the patient with the asymptomatic disease, aims to reduce the learning cost and the operation complexity of the patient, and has great significance in improving the life quality of the patient with the asymptomatic disease.

The invention discloses a multi-camera imaging method and system for head-mounted equipment, and relates to the technical field of multi-camera imaging, and the system comprises an image data acquisition module, a weight acquisition module, an image fusion module, a feedback control module, a visual angle movement determination module and a preloading module. The gaze weight of the image is obtained by obtaining the gaze direction of the user, an image weighted fusion mode with the current visual attention area of the user as the core is achieved, the generated main view angle image better conforms to the actual view field of the user, and immersive visual experience is enhanced; inter-frame time sequence consistency loss is introduced in the fusion process, the pixel change of the fusion image between continuous frames is restrained through time sequence smoothing, the image flicker and jump phenomena are remarkably reduced, and the visual stability in a dynamic scene is improved.

An apparatus for steering adjustment detects a gaze direction of a driver of a vehicle from a facial image in which a face of the driver is represented. During autonomous driving in which steering of the vehicle is autonomously controlled, when an operation by the driver on a steering wheel is detected, determines whether the gaze direction of the driver corresponds to the steering direction in the operation of the driver. The apparatus adjusts a reaction force with respect to the operation of the steering wheel smaller when the gaze direction corresponds to the steering direction than when the gaze direction does not correspond to the steering direction.

Various implementations disclosed herein include devices, systems, and methods that initiate an action based on a detected gaze direction oriented towards a target area (e.g., a hot corner / zone). For example, an example process may include producing a reflection by directing light towards an eye using an illuminator, receiving sensor data from a sensor, wherein a direction of sensing by the sensor (e.g., the optical axis of the camera) and a direction from the eye to a target area are approximately aligned, determining a reflective property (e.g., a spectral property) of the reflection based on the sensor data, detecting that a gaze direction of the eye is approximately oriented towards the target area (e.g., a hot zone / spot) based on the reflective property, and initiating an action based on detecting that the gaze direction is approximately oriented towards the target area.

A head mounted display (HMD) device is provided. The HMD includes a display, a gaze sensor configured to detect a gaze direction of a user of the HMD device, a camera, a position sensor configured to detect a position of the HMD device, communication circuitry, memory storing one or more computer programs, and one or more processors communicatively coupled to the display, the gaze sensor, the camera, the position sensor, the communication circuitry, and the memory, wherein the one or more computer programs include computer-executable instructions that, when executed by the one or more processors individually or collectively, cause the HMD device to, based on the position of the HMD device, acquire three-dimensional (3D) segmentation information for a space corresponding to the position of the HMD device, acquire gaze point-based interest information corresponding to a profile of the user, the gaze point-based interest information including interest information generated based on gaze points of a plurality of users, display, on the display, a first visual effect for a plurality of regions in the space based on the 3D segmentation information and the interest information, and display, on the display, a second visual effect for a plurality of objects in a first region among the plurality of regions based on the 3D segmentation information and the interest information when the position of the HMD device corresponds to the first region.

Methods and systems for performing operations to control brightness in an AR device are disclosed. The operation comprises: displaying an image on an eye wearable device worn by a user; detecting a gazing direction of pupils of the user; identifying a first region of the image corresponding to a gaze direction of the pupil; and modifying a luminance level or luminance value of pixels in the image based on the gaze direction such that pixels in a first region of the image are set to a first luminance value and pixels in a second region of the image are set to a second luminance value lower than the first luminance value.

The invention relates to an intelligent cabin interaction method and system and a medium, and relates to the field of intelligent cabins, and the method comprises the steps: obtaining the facial expression and voice information of a user; performing sentiment classification on the facial expression and the voice information by using a deep learning model; based on the sentiment classification result, adjusting the environment atmosphere in the cabin; creating a three-dimensional holographic image on the vehicle-mounted head-up display through holographic projection equipment; receiving a voice instruction, a gesture action or a staring direction of a user; and controlling functions in the cabin according to the voice instruction, the gesture action or the staring direction. The user interaction experience, the emotion response ability and the intelligent level of the intelligent cabin can be improved, and safer, more comfortable and more personalized driving experience is provided for a driver.

Systems and methods for eye tracking and content tagging in minimally invasive surgical videos are described herein. Minimally invasive surgical videos may be captured while performing robotic surgeries. Robotic surgical systems described herein include robotic arms with interchangeable surgical tools. An endoscope at the end of one of the robotic arms captures video of the surgical procedure. The video is displayed on a display of the surgical system and an eye tracking device captures data corresponding to a gaze direction of the user on the display. Content tags are automatically generated in the image data based on areas of focus of the user. Information within the content tag may be generated based on surgical procedure steps derived from the image data, a log of current surgical procedures, and image data of previous surgical procedures.

A head mounted display for use in a virtual reality, mixed reality or augmented reality system comprises a display device configured to display an image, a tracking system configured to determine a gaze direction of a user's eye and an optical system configured to project the image into the user's eye, the optical system comprising an adjustable optical device; and processing circuitry configured to control the adjustable optical device based on the gaze direction of the user's eye such that a position of the image projected into the user's eye is unchanged, when the gaze direction of the user's eye changes.

The invention discloses a method and device for recalibrating the gazing direction of a driver, and belongs to the technical field of intelligent driving. The method comprises the steps that a driver face image in a natural driving scene is collected for head posture estimation, and head posture data of a driver standardized image are obtained; on the basis of a gaze area prediction model, gaze data corresponding to the head posture data of the driver standardized image is obtained, the gaze data is processed, and a calibration image and a target image are further obtained; and respectively inputting the calibration image and the target image into a feature extraction network to extract image features, inputting the calibration image features and the target image features into a direction classification network to find an association relationship between the target image and the calibration image, and completing sight line calibration based on the association relationship. The method can be used for self-adaptive deployment of the pre-training sight estimation model, implicit gaze direction mapping under the condition of few samples and even zero samples is achieved, and extra hardware calibration or model training is not needed.

An electronic device may include a waveguide that directs light to an eye box. The waveguide may include a diffractive grating structure. A projector may generate image light coupled into the waveguide by a first input coupler. A light source may generate supplemental light coupled into the waveguide by a second input coupler. The diffractive grating structure may couple the image light out of the waveguide within a central region of the field of view (FOV) of an eye box and may couple the supplemental light out of the waveguide at a location within a peripheral region of the FOV. The location may be outside of the central region by at least 10 degrees. A gaze tracking sensor may estimate a user's gaze direction at the eye box. A processor may wake or power down the projector responsive to the gaze direction overlapping the location for a predetermined time period.

The invention relates to the technical field of image processing and mode recognition, in particular to a head posture correction gaze direction estimation method based on multiple modes, which comprises the following steps of: 1, preprocessing a gaze estimation data set and dividing the gaze estimation data set into a training set and a test set; 2, constructing a head posture correction staring direction estimation model based on multiple modes; 3, taking the binocular image and the face image in the training set as input, and training a head posture correction staring direction estimation model based on multiple modes; and 4, predicting test set data by using the trained gaze direction estimation model. According to the method, the performance of staring direction estimation is remarkably improved, the experience of the user in the fields of virtual reality and human-computer interaction is improved, and a more intelligent and more natural interaction mode is provided for the user.

An information processing method includes: by using a processor, acquiring at least one index selected from an index group that includes an amount of variation in a face position of a user, an amount of variation in a face orientation of the user, an amount of variation in a gaze direction of the user, and an eye closure percentage of the user, and determining, based on the at least one acquired index, whether or not the user is in a divergent thinking state.

A system includes at least one server configured to receive, from a client device, information indicative of gaze directions of a user's eyes; determine a gaze point and a gaze depth of the user's eyes, using said information; identify, in an image, a gaze-contingent region that includes and surrounds the gaze point; identify a region of interest in the gaze-contingent region, wherein the region of interest comprises pixels whose optical depth lies within a first predefined distance (Δd1) from the gaze depth; encode the image by applying a first encoding setting to the pixels belonging to the region of interest; and send the image that is encoded, to the client device.

Techniques are provided for real-time calibration of an eye-tracking system in a wearable display device using user interactions with a graphical user interface. A background calibration process continuously or intermittently updates a calibration matrix based on stable gaze-target pairs inferred from natural user behavior, without requiring explicit calibration routines. The system detects user interactions such as selections of UI elements, and associates them with gaze direction data captured immediately beforehand. Fixation filtering is applied to identify stable gaze intervals based on dispersion thresholds and minimum duration criteria.

Systems and methods for dynamic dimming of a set of displays sharing a single energy source are provided. Image data is received from an external camera of a vehicle. A light intensity level of an external light source in the image data is determined based on the image data. A light angle of the external light source is determined with respect to the vehicle based on the image data. A desired brightness level is generated based on the light intensity level and the light angle. If a first display of a plurality of displays of the vehicle is in a line of sight of a driver gaze direction, a command is issued to adjust a brightness level of the first display to the desired brightness level and dim the other displays to a dim brightness level. The desired brightness level is greater than the dim brightness level.

An electronic device that includes a display and an eye tracker configured to collect eye tracking data regarding a gaze of one or more of a user's eyes across the display. The electronic device also includes processing circuitry that is operatively coupled to the display and configured to generate pixel data for frames of content based on the eye tracking data such that the content is configured to be shown on the display in a dynamically foveated manner. The processing circuitry is configured to determine dither blocks, each of which corresponds to a subset of the plurality of pixels. The processing circuitry is configured to apply a dither pattern to the frames of the plurality of frames of content independent of the gaze of one or more of the user's eyes and based on the dither blocks.

An audiovisual system uses a spatial position detection module to process audio signals and determine the location and orientation of a speaking participant. Based on this data, the system dynamically controls sensors to optimize audio and video capture. Behavioral and contextual information may also be used to train intelligence models for improved system performance. Further, sensor arrays may be utilized to identify gaze vectors of participants to select a camera sensor of the sensor array. Meeting content collected with the sensor array may be organized into meeting metrics in accordance with an analytics strategy before training an intelligence module with the meeting metrics. Meeting content may also be configured into digital tiles in accordance with a tile strategy. At least one of the digital tiles may be altered in response to a meeting condition detected by the sensor array.

The invention provides a sound source localization method and system, an electronic device and a storage medium, and belongs to the technical field of sound source localization, the method is applied to a wearable device, and the method comprises the following steps: determining a gaze space vector corresponding to a gaze direction of a target user based on an eye action of the target user, the target user being a user wearing the wearable device; based on the position coordinates of a microphone array, determining a microphone space vector corresponding to each microphone, the microphone array being arranged in the wearable device; establishing a mapping relation based on the fixation space vector and the microphone space vector; and positioning the sound source in the gazing direction of the target user based on the mapping relation. According to the sound source positioning method and system, the electronic equipment and the storage medium provided by the invention, the user experience can be improved.

The invention discloses an automatic adjustment and feedback control method for an intelligent rearview mirror in an automatic driving environment, and relates to the technical field of vehicle accessories. Comprising the following steps: defining an intelligent rearview mirror focusing area based on a driver view position, so that the focusing area is matched with the gazing direction of a driver; the displacement track of the reference target in the focusing area is analyzed based on the time sequence analysis technology; and dynamically adjusting the angle of the intelligent rearview mirror according to the displacement track analysis result and the driving posture of the driver. By constructing the three-dimensional coordinates based on the intelligent rearview mirrors on the two sides of the A column and the intelligent rearview mirrors in the vehicle, the head datum point position and the sight direction vector of the driver are obtained, so that the real-time view position of the driver is determined, the focusing area on the rearview mirror imaging plane is dynamically defined, the gazing direction of the driver is effectively matched, and the driving safety is improved. Real-time and dynamic recognition of the visual field of the driver is achieved, and the consistency of the imaging area of the rearview mirror and the actual attention area of the driver is guaranteed.

The voice directional enhancement method comprises the steps that a current frame and a staring direction of mixed voice are acquired, and the mixed voice is acquired by a microphone array; constructing a direction feature based on the current frame of the mixed voice and the gazing direction, and constructing an audio feature based on the current frame of the mixed voice; determining a current frame mask according to the direction features and the audio features; and determining a target enhanced voice of the current frame according to the current frame mask, the current frame of the mixed voice and the plurality of historical frames of the mixed voice. In the embodiment of the invention, the mask of the current frame is generated under the guidance of the gazing direction, so that the target voice can be accurately separated from the mixed voice. Besides, based on the current frame mask and the current frame of the mixed voice, the target voice enhancement is performed frame by frame from the overall characteristics of the mixed voice by referring to a plurality of historical frames, so that the accuracy of the target enhanced voice of the current frame can be effectively improved, and thus the accuracy of directional pickup is improved.

An example vehicle control system includes a plurality of sensors, a driver monitoring system, a human-machine interface configured to provide at least one of a visual alert, a haptic alert and an audio alert to the driver, and a vehicle control module configured to identify at least one occluded region based on the plurality of sensors, determine a predicted trajectory of the vehicle, score the at least one occluded region based on importance of the at least one occluded region to the predicted trajectory of the vehicle, determine a driver attention direction based on detection of the at least one of a gaze direction and a head orientation of the driver of the vehicle, and generate an occluded region alert based on the score of the at least one occluded region and the driver attention direction.

Systems and methods for enabling higher uniform performance and higher voltage operation for micro-OLED displays, manufacturing a meniscus lens including thermo-forming a functional optical layer and printing a lens element over the formed functional optical layer, using a hybrid process used to form a functionalized lens having a controlled surface profile, modeling the polarization properties of a human eye using a polymer thin film, and improving optical sparce eye-tracking by collecting an optical output signal from each detector through a corresponding optical fiber, and determining a gaze direction of a user's eye based on the electrical signal may be disclosed.