1230 results about "Gaze" patented technology

In some embodiments, an electronic device facilitates cursor interactions in different regions in a three-dimensional environment. In some embodiments, an electronic device facilitates cursor interactions in content. In some embodiments, an electronic device facilitates cursor movement. In some embodiments, an electronic device facilitates interaction with multiple input devices. In some embodiments, a computer system facilitates cursor movement based on movement of a hand of a user of the computer system and a location of a gaze of the user in the three-dimensional environment. In some embodiments, a computer system facilitates cursor selection and scrolling of content in the three-dimensional environment.

In some embodiments, while displaying a content entry user interface element and while a first content entry tool is selected, an electronic device detects a first movement of a predefined portion of a user while the predefined portion has a first shape. In some embodiments, in response to detecting the first movement, in accordance with a determination that a gaze of the user was directed toward the content entry user interface element when the first movement was detected, the electronic device enters first content, based on the first content entry tool, corresponding to the first movement into the content entry user interface element. In some embodiments, in accordance with a determination that the gaze of the user was directed toward a menu user interface element when the first movement was detected, the electronic device selects a second content entry tool for entering content into the content entry user interface element.

The invention discloses an intelligent control method and system for AR equipment, and the method specifically comprises the steps: determining a pupil center position coordinate and a cornea reflection point position coordinate of a user according to an eye image of the user, and inputting the pupil center position coordinate and the cornea reflection point position coordinate into a geometric mapping model for matching, obtaining user sight direction information; acquiring image brightness value distribution of an area near a user fixation point, determining a plurality of candidate saliency areas according to the image brightness value distribution, and comparing the candidate saliency areas with a preset saliency area template to determine a user attention focus; and acquiring head posture information of the user in real time by adopting a head posture tracking algorithm, and dynamically controlling the shooting direction of a camera of the AR equipment according to the head posture information, the sight direction information of the user and the attention focus of the user. According to the method and the device, accurate alignment of the AR content and the visual focus of the user is realized, and the immersion and interaction experience of the AR application are greatly improved.

In some embodiments, an electronic device emphasizes and / or deemphasizes user interfaces based on the gaze of a user. In some embodiments, an electronic device defines levels of immersion for different user interfaces independently of one another. In some embodiments, an electronic device resumes display of a user interface at a previously-displayed level of immersion after (e.g., temporarily) reducing the level of immersion associated with the user interface. In some embodiments, an electronic device allows objects, people, and / or portions of an environment to be visible through a user interface displayed by the electronic device. In some embodiments, an electronic device reduces the level of immersion associated with a user interface based on characteristics of the electronic device and / or physical environment of the electronic device.

The invention relates to the field of multi-modal interaction analysis, in particular to a multi-modal interaction method and system of a digital human agent. The method comprises the following steps: acquiring a real-time face image and a voice signal input stream of an interactive user based on an intelligent agent; performing real-time micro-expression recognition and deep emotion analysis based on the real-time facial image to obtain real-time emotion features of the user; performing time sequence evolution analysis on the real-time emotion characteristics of the user, performing holographic user emotion deep mining, and constructing a user emotion holographic characteristic spectrum; carrying out adaptive acoustic gain processing on the voice signal input stream, and carrying out voice-emotion association analysis based on the user emotion holographic characteristic spectrum to generate a voice-emotion linkage mapping spectrum; and carrying out eyeball fixation point migration tracking based on the user emotion holographic feature map and the real-time face image, and generating a user interaction depth intention signal. Through the real-time deep semantic understanding and emotion perception ability, the intelligent agent interaction intelligence and response accuracy are improved.

The present disclosure generally relates to controlling electronic devices. In some examples, the electronic device uses gaze information to activate a digital assistant. In some examples, the electronic device uses gaze information to identify an external device on which to act. In some examples, the electronic device provides an indication that distinguishes between different speakers.

The invention relates to a method for evaluating the situation awareness ability of a pilot based on a Ti-SA model. The Ti-SA model comprises an eye movement trace multi-dimensional attention enhancement module, an AOI multi-scale comprehensive feature module and a fusion module. The evaluation method comprises the steps of collecting eye movement trace time sequence data and flight training videos of a pilot at continuous moments in real time, and obtaining corresponding AOI fixation time sequence data based on the eye movement trace time sequence data, the flight training videos and a preset AOI subdivision area; preprocessing the eye movement trace time sequence data and the AOI fixation time sequence data; inputting the preprocessed eye movement trace time sequence data into an eye movement trace multi-dimensional attention enhancement module to obtain a category probability value corresponding to the eye movement trace time sequence data; inputting the preprocessed AOI fixation time sequence data into an AOI multi-scale comprehensive feature module to obtain a category probability value corresponding to an AOI fusion feature; and inputting the two category probability values into a fusion module to obtain a predicted situation awareness level.

In some embodiments, while a keyboard is visible in a three-dimensional environment, the computer system detects a gaze of a user move from a position away from a first key of the keyboard to the first key. In some embodiments, in response to detecting the gaze of the user moving from the position away from the first key to the first key, in accordance with a determination the one or more criteria are satisfied, the computer system initiates a process to select a first character corresponding to the first key for entry. In some embodiments, in response to detecting the gaze of the user moving from the position away from the first key to the first key, in accordance with a determination that the one or more criteria are not satisfied, the computer system forgoes initiating the process.

Methods and systems are described for imaging and content generation. During a road trip, a driver or passenger observes an object or scene from a vehicle. From a perspective of the driver or passenger, the object or scene flows out of view in a relatively short amount of time. Images and video of the object or scene are captured with cameras. A field of view is extended beyond what the driver or passenger can easily observe. The capture is configurable to focus on particular types of objects or scenes. Recordings of billboards and road signs are easily displayed after passing the object. Object identification, gaze determination, and interest determination are provided. Curated content is generated. Applications to extended reality environments are provided. Artificial intelligence systems, including neural networks, and models are utilized to improve the imaging and content generation. Related apparatuses, devices, techniques, and articles are also described.

The invention relates to the technical field of image recognition, in particular to a lightning arrester fault thermal imaging picture recognition method and system combined with gaze prediction, and the method comprises the steps: carrying out the down-sampling of an original thermal imaging image to a fixed size, sequentially passing through a multi-layer convolution and a Spatial Softmax layer, and outputting a predicted gaze point track sequence, extracting a key area and a non-key area according to the intensity of the fixation point track sequence; reconstructing the key area to obtain a compressed and recombined target area; inputting the non-key region into a variational auto-encoder to obtain a low-dimensional potential feature vector; and inputting the image-structure joint feature representation into an EffiCroprViT model, and finally obtaining a lightning arrester fault classification result. Local and global features are efficiently fused, the calculation complexity is reduced, background noise interference is effectively suppressed, real-time and accurate identification and early warning of the fault state of power equipment are realized, and the fault classification method has the advantages of high efficiency, high reliability and high reliability. Therefore, the safety and stability of power grid operation are ensured.

Techniques for gaze-based video conference prompts are described that leverage eye-tracking algorithms within a video conference setting to perform a variety of functionality. For instance, a computing device displays a user interface of a video conference session that includes representations of video conference attendees. The computing device receives video content that depicts a user of the computing device and uses eye tracking techniques to determine a gaze location of the user based on the video content. The gaze location corresponds to one of the representations of an attendee. The computing device then communicates a prompt to one or more of the attendees that indicates the gaze location. In another example, the computing device detects a behavior of the user, e.g., a scanning behavior. Responsive to detection of the behavior, the computing device performs an action within the user interface, such as to display a roster view of the attendees.

Approaches presented herein provide for the automated determination of a level of impairment of a person, as may be relevant to the performance of a task. A light and camera-based system can be used to determine factors such as gaze nystagmus that are indicative of inebriation or impairment. A test system can simulate motion of a light using a determined pattern, and capture image data of at least the eye region of a person attempting to follow the motion. The captured image data can be analyzed using a neural network to infer at least one behavior of the user, and the behavior determination(s) can be used to determine a capacity or level of impairment of a user. An appropriate action can be taken, such as to allow a person with full capacity to operate a vehicle or perform a task, or to block access to such operation or performance if the person is determined to be impaired beyond an allowable amount.

The invention discloses a lightweight display interface rendering optimization system, which relates to the technical field of image processing, and comprises an acquisition module, an interface analysis module, a behavior analysis module and an attention analysis and rendering module, dynamically calculating an effective view radius by utilizing a visual tunneling effect; the interface analysis module performs character string fuzzy matching in combination with the context input by the user, identifies the search intention of the user and improves the weight of a corresponding component; the attention analysis module performs multi-modal weighted fusion on the physiological fixation data and the sparse content saliency thermodynamic diagram; according to the method, by recognizing the semantic intention and the physiological fixation point of the user, on the premise that core visual experience continuity is guaranteed, GPU load and video memory bandwidth occupation are remarkably reduced, and balance of high-performance display and low-power-consumption operation is achieved.

Methods and devices are described that enable user interactions using a combination of gaze input and touch input. During display of interactive elements on a display of an electronic device, an estimated gaze location of a gaze input is determined. A target interactive element is identified based on the estimated gaze location. Touch input is detected that traverses a first distance in a first direction. A preview of information related to the target interactive element is displayed, where an amount of the information displayed in the preview is based on the first distance traversed by the touch input.

The invention provides a physiological state risk detection method, device, equipment and program product, which can be applied to the technical field of computers. The method comprises the following steps: acquiring a facial expression image, an eyeball tracking image, a limb balance image and audio data; key point coordinate values of a plurality of facial key points in the facial expression image are processed based on a first random forest to obtain a facial result, and the facial result represents a facial expression attribute and a facial symmetry attribute; inputting an eyeball movement distance obtained from the eyeball tracking image into a second random forest, and outputting a pupil staring result; inputting the related coordinate values of the limb key points in the limb balance image into a third random forest, and outputting a limb balance state result; performing voice analysis on the audio data based on a convolutional neural network to obtain a voice definition result; and inputting the face result, the pupil staring result, the limb balance state result and the voice definition result into a fourth random forest, and outputting a stroke risk detection result.

The invention discloses an ideological and political large-scale course teaching attention assessment data acquisition method based on sight tracking. The method comprises the following steps: S1, high-definition large-scene eye movement tracking photographing equipment captures eye images of students in a course teaching process in real time through an infrared camera; s2, performing pupil area detection and pupil center fitting on the eye image to obtain sight tracking data; s3, performing gaze hotspot analysis and gaze duration statistics according to the sight tracking data to quantify attention distribution and change trend of the students and visually display the attention distribution and change trend; and S4, generating a teaching quality evaluation report and teaching optimization suggestions according to the gazing hotspot analysis and gazing duration statistical results. The classroom teaching effect can be objectively evaluated in real time, subjective evaluation errors are reduced, the accuracy and efficiency of teaching feedback are improved, the method is suitable for various scenes such as traditional classrooms, online education and experiment teaching, and the method has the industrial application prospect of education digital transformation.

Provided in the embodiments of the present disclosure are a wearing deviation measurement method and apparatus, and an electronic device and a storage medium. The method comprises: collecting an eye image; obtaining at least two eye feature points according to the eye image, wherein the eye feature points are distributed on the contour of an eye area in the eye image; and obtaining deviation information according to location features of the at least two eye feature points, wherein the deviation information represents a deviation type and / or a deviation amount of the current wearing pose of a terminal device relative to a standard wearing pose. An eye image is collected, and a deviation type and / or a deviation amount of the current wearing pose of a terminal device relative to a standard wearing pose are / is obtained on the basis of eye feature points on the contour of an eye area in the eye image, so as to realize early detection and prompt when a wearing pose is abnormal, thereby avoiding the problem of the accuracy of gaze point detection being reduced due to the fact that the wearing pose of the terminal device is abnormal, and improving the image display effect and the human-computer interaction efficiency.

The invention discloses a cheating detection method and system applied to online interview, relates to the technical field of computer vision, and solves the problems that in the prior art, multiple behavior characteristics in interview personnel are often adopted for monitoring in the whole process, multi-data analysis in the presence of non-cheating behaviors causes waste of computing resources, and the monitoring efficiency is high. And the cheating detection method is low in efficiency. A gaze anomaly coefficient is generated according to gaze data; when the fixation abnormal coefficient is greater than the abnormal threshold value, generating a multi-modal feature according to the monitoring video; generating a multi-modal decision weight according to the interview stage label; fusing the multi-modal decision-making weight and the multi-modal features to obtain multi-modal decision-making data; and inputting the multi-modal decision data into the cheating judgment model to obtain a monitoring result, carrying out tendency evaluation on cheating monitoring, and carrying out multi-modal dynamic fusion when the cheating tendency exists so as to accurately judge the cheating condition, so that the cheating monitoring efficiency and the cheating monitoring accuracy are improved.

An electronic device and method for video engagement determination based on statistical positional object tracking is provided. The electronic device receives media content including a set of video frames based on a content playability factor and detects a set of objects included in each video frame based on an application of a first NN model on each video frame. The electronic device receives a set of images of an audience watching the media content. The electronic device estimates gaze co-ordinates associated with the audience based on an application of a second NN model on each image. The electronic device splits each video frame into a set of segments and maps the gaze co-ordinates to the set of segments. The electronic device determines an engagement score for each object based on the mapping of the gaze co-ordinates and renders the engagement score on a display device.

The invention discloses a fixation spectrum estimation method based on a first-view-angle high-dynamic long video, and belongs to the field of image processing. According to the gaze spectrum prediction method, the key problems that in the prior art, time sequence information in a long-time video is lost, local and global information fusion is insufficient, and static masks are difficult to adapt to dynamic scene changes are effectively solved through three innovative modules. The enhanced long-time memory encoder effectively encodes a long-time video through multi-scale attention and a hierarchical memory mechanism, the problems of information loss and redundancy under long-time dependence are solved, and the integrity of time sequence information is ensured; the high-pass global-local information aggregation module enables a global background and local details to be effectively combined in a multi-layer network by designing a cross-layer dynamic information transmission channel, so that the gaze spectrum prediction capability in a complex dynamic scene is enhanced; and the dynamic mask fusion module adopts a self-adaptive mechanism, so that the mask weight can be adjusted in real time, the problem that static attention cannot cope with a rapid change background is solved, and the flexibility and accuracy of the model are improved.

Apparatuses, systems, and techniques are presented to estimate user gaze. In at least one embodiment, one or more neural networks are used to determine coarse and fine gaze estimates for one or more users.

Methods and systems are described for imaging and content generation. During a road trip, a driver or passenger observes an object or scene from a vehicle. From a perspective of the driver or passenger, the object or scene flows out of view in a relatively short amount of time. Images and video of the object or scene are captured with cameras. A field of view is extended beyond what the driver or passenger can easily observe. The capture is configurable to focus on particular types of objects or scenes. Recordings of billboards and road signs are easily displayed after passing the object. Object identification, gaze determination, and interest determination are provided. Curated content is generated. Applications to extended reality environments are provided. Artificial intelligence systems, including neural networks, and models are utilized to improve the imaging and content generation. Related apparatuses, devices, techniques, and articles are also described.

A method of determining a target of a subject's gaze from an image of the subject within a scene is disclosed. The method comprises analyzing the image and determining information associated with the image. A processor configured to analyze the image and determine information associated with the image is also disclosed. Finally, a non-transitory computer readable storage device comprising computer executable instructions for determining a target of a subject's gaze is disclosed.

According to an embodiment, a wearable device may display, based on contact on a second surface identified using a touch sensor, a visual object indicating a first position of the touch input on the second surface in a screen through a display. The wearable device may identify, in response to the touch input, a second position in the screen of a gaze identified based on an image obtained through a camera exposed outside at a portion of a first surface. The wearable device may provide, in response to the second position identified within a specified distance from the first position of the visual object, feedback with respect to the touch input. The wearable device may cease to provide the feedback in response to the second position identified outside from the first position by the specified distance.

Generally disclosed herein is a system for automatically controlling media being viewed by a user based on the user attention levels predicted from gaze and drowsiness estimation and the user's position in a pre-marked monitoring zone. The system may be configured to adjust the monitoring zones based on varying lighting conditions, varying poses of the subjects or users, and the varying distances between the media devices and the users. The system may be further configured to automatically pause or play back the media content based on the user's attention level or changes in positions of the user.

Systems, methods, and apparatuses are described for causing a camera of a head-mounted computing device to capture a first video, the head-mounted computing device comprising a camera direction control element for controlling a capture direction of the camera, and a camera zoom control element for controlling zoom of the camera. One or more objects in the captured first video may be identified based on a detected gaze angle of a user wearing the head-mounted computing device. A target location in an environment may be determined, and based on such target location, the capture direction and zoom of the camera may be adjusted using the camera direction control element and the camera zoom control element, respectively. The camera may capture, based on the adjusted capture direction and the adjusted zoom of the camera, a second video using the camera of the head-mounted computing device.

The invention relates to the technical field of mixed reality and artificial intelligence fusion application, in particular to a mixed reality intelligent inspection method and system based on multi-modal fusion. According to the technical scheme, the method comprises the following steps: synchronously acquiring a user view image and voice stream data through mixed reality equipment; performing target detection on the view image by adopting an improved YOLOv10 model, and outputting a candidate target area and a confidence score # imgabs0 # thereof; voice input is transferred through a Whisper voice recognition model, and a semantic vector # imgabs1 # is generated by using a BERT semantic embedding model. According to the invention, through a multi-mode cooperative sensing mechanism of fusing image, voice and space gazing information, in combination with context semantic optimization and lightweight terminal deployment, intelligent target identification and natural interaction with high precision, real-time performance and strong robustness are realized in a complex industrial inspection scene.

The invention discloses a high-altitude falling simulation system based on VR equipment, and relates to the technical field of virtual reality simulation, and the system comprises a data collection module which collects the pupil distance and initial sight line data of a user; the visual disturbance module is used for dynamically adjusting local visual disturbance of a corresponding area in the high-altitude virtual scene based on the gaze data to form a visual disturbance effect; the psychological index calculation module is used for calculating an anxiety index and a focus drift speed by adopting a time sequence convolutional neural network according to the visual disturbance effect and the eye movement behavior of the user to obtain an anxiety index and a focus drift speed index; and the disturbance adjustment module adaptively adjusts the frequency and amplitude of the visual disturbance effect by adopting a method of combining fuzzy logic control and adaptive gain adjustment to obtain disturbance rhythm parameters. According to the method, the personalized matching ability and training effectiveness in psychological behavior intervention and virtual high-altitude adaptation training are improved.

The invention provides a hybrid wireless eye movement tracking method and system based on multi-sensor fusion, electronic equipment and a storage medium, and relates to the technical field of eye movement tracking. The method comprises the following steps: firstly, acquiring various data by using various different types of sensors such as a Q-Var sensor, an inertial measurement unit and a high-speed panoramic camera; secondly, preprocessing the various sensor data by means of graph Fourier transform, a Kalman filter, a spectral clustering technology and the like; fusing the preprocessed sensor data based on a graph theory method; and finally, performing fixation track prediction based on a convolutional neural network and a gating circulation unit. According to the technical scheme, the performance of eye movement tracking under different environment conditions is improved, the eye movement tracking precision is higher, and the stability is better. In addition, the technology of the application also uses a 5G millimeter wave technology or an IEEE 802.11 ac protocol to transmit gaze tracking data, thereby achieving the purpose of bandwidth enhancement and low-delay data transmission.

Systems and methods are provided for generating adaptive content. The system may implement a machine learning model including training data pre-trained, or trained in real-time based on captured content or prestored content associated with gazes of users, pupil dilations, facial expressions, muscle movements, heart rates, or gaze dwell times of users determined previously or in real-time. The system may determine a gaze(s) of an eye of a user or facial features of a face associated with the user viewing, by a device, items of content in an environment. The system may include determining, based on the gaze(s) or facial features, a state(s) or interest(s) of the user. The system may determine, by implementing the machine learning model and based on the state(s) or interest(s) of the user, content to generate a modification of the items of content or to generate new content items associated with the items of content.