978 results about "Gaze" patented technology

In some embodiments, an electronic device facilitates cursor interactions in different regions in a three-dimensional environment. In some embodiments, an electronic device facilitates cursor interactions in content. In some embodiments, an electronic device facilitates cursor movement. In some embodiments, an electronic device facilitates interaction with multiple input devices. In some embodiments, a computer system facilitates cursor movement based on movement of a hand of a user of the computer system and a location of a gaze of the user in the three-dimensional environment. In some embodiments, a computer system facilitates cursor selection and scrolling of content in the three-dimensional environment.

In some embodiments, while displaying a content entry user interface element and while a first content entry tool is selected, an electronic device detects a first movement of a predefined portion of a user while the predefined portion has a first shape. In some embodiments, in response to detecting the first movement, in accordance with a determination that a gaze of the user was directed toward the content entry user interface element when the first movement was detected, the electronic device enters first content, based on the first content entry tool, corresponding to the first movement into the content entry user interface element. In some embodiments, in accordance with a determination that the gaze of the user was directed toward a menu user interface element when the first movement was detected, the electronic device selects a second content entry tool for entering content into the content entry user interface element.

The invention discloses an intelligent control method and system for AR equipment, and the method specifically comprises the steps: determining a pupil center position coordinate and a cornea reflection point position coordinate of a user according to an eye image of the user, and inputting the pupil center position coordinate and the cornea reflection point position coordinate into a geometric mapping model for matching, obtaining user sight direction information; acquiring image brightness value distribution of an area near a user fixation point, determining a plurality of candidate saliency areas according to the image brightness value distribution, and comparing the candidate saliency areas with a preset saliency area template to determine a user attention focus; and acquiring head posture information of the user in real time by adopting a head posture tracking algorithm, and dynamically controlling the shooting direction of a camera of the AR equipment according to the head posture information, the sight direction information of the user and the attention focus of the user. According to the method and the device, accurate alignment of the AR content and the visual focus of the user is realized, and the immersion and interaction experience of the AR application are greatly improved.

In some embodiments, an electronic device emphasizes and / or deemphasizes user interfaces based on the gaze of a user. In some embodiments, an electronic device defines levels of immersion for different user interfaces independently of one another. In some embodiments, an electronic device resumes display of a user interface at a previously-displayed level of immersion after (e.g., temporarily) reducing the level of immersion associated with the user interface. In some embodiments, an electronic device allows objects, people, and / or portions of an environment to be visible through a user interface displayed by the electronic device. In some embodiments, an electronic device reduces the level of immersion associated with a user interface based on characteristics of the electronic device and / or physical environment of the electronic device.

The invention relates to the field of multi-modal interaction analysis, in particular to a multi-modal interaction method and system of a digital human agent. The method comprises the following steps: acquiring a real-time face image and a voice signal input stream of an interactive user based on an intelligent agent; performing real-time micro-expression recognition and deep emotion analysis based on the real-time facial image to obtain real-time emotion features of the user; performing time sequence evolution analysis on the real-time emotion characteristics of the user, performing holographic user emotion deep mining, and constructing a user emotion holographic characteristic spectrum; carrying out adaptive acoustic gain processing on the voice signal input stream, and carrying out voice-emotion association analysis based on the user emotion holographic characteristic spectrum to generate a voice-emotion linkage mapping spectrum; and carrying out eyeball fixation point migration tracking based on the user emotion holographic feature map and the real-time face image, and generating a user interaction depth intention signal. Through the real-time deep semantic understanding and emotion perception ability, the intelligent agent interaction intelligence and response accuracy are improved.

In some embodiments, while a keyboard is visible in a three-dimensional environment, the computer system detects a gaze of a user move from a position away from a first key of the keyboard to the first key. In some embodiments, in response to detecting the gaze of the user moving from the position away from the first key to the first key, in accordance with a determination the one or more criteria are satisfied, the computer system initiates a process to select a first character corresponding to the first key for entry. In some embodiments, in response to detecting the gaze of the user moving from the position away from the first key to the first key, in accordance with a determination that the one or more criteria are not satisfied, the computer system forgoes initiating the process.

Methods and systems are described for imaging and content generation. During a road trip, a driver or passenger observes an object or scene from a vehicle. From a perspective of the driver or passenger, the object or scene flows out of view in a relatively short amount of time. Images and video of the object or scene are captured with cameras. A field of view is extended beyond what the driver or passenger can easily observe. The capture is configurable to focus on particular types of objects or scenes. Recordings of billboards and road signs are easily displayed after passing the object. Object identification, gaze determination, and interest determination are provided. Curated content is generated. Applications to extended reality environments are provided. Artificial intelligence systems, including neural networks, and models are utilized to improve the imaging and content generation. Related apparatuses, devices, techniques, and articles are also described.

The invention relates to the technical field of image recognition, in particular to a lightning arrester fault thermal imaging picture recognition method and system combined with gaze prediction, and the method comprises the steps: carrying out the down-sampling of an original thermal imaging image to a fixed size, sequentially passing through a multi-layer convolution and a Spatial Softmax layer, and outputting a predicted gaze point track sequence, extracting a key area and a non-key area according to the intensity of the fixation point track sequence; reconstructing the key area to obtain a compressed and recombined target area; inputting the non-key region into a variational auto-encoder to obtain a low-dimensional potential feature vector; and inputting the image-structure joint feature representation into an EffiCroprViT model, and finally obtaining a lightning arrester fault classification result. Local and global features are efficiently fused, the calculation complexity is reduced, background noise interference is effectively suppressed, real-time and accurate identification and early warning of the fault state of power equipment are realized, and the fault classification method has the advantages of high efficiency, high reliability and high reliability. Therefore, the safety and stability of power grid operation are ensured.

Approaches presented herein provide for the automated determination of a level of impairment of a person, as may be relevant to the performance of a task. A light and camera-based system can be used to determine factors such as gaze nystagmus that are indicative of inebriation or impairment. A test system can simulate motion of a light using a determined pattern, and capture image data of at least the eye region of a person attempting to follow the motion. The captured image data can be analyzed using a neural network to infer at least one behavior of the user, and the behavior determination(s) can be used to determine a capacity or level of impairment of a user. An appropriate action can be taken, such as to allow a person with full capacity to operate a vehicle or perform a task, or to block access to such operation or performance if the person is determined to be impaired beyond an allowable amount.

The invention discloses a lightweight display interface rendering optimization system, which relates to the technical field of image processing, and comprises an acquisition module, an interface analysis module, a behavior analysis module and an attention analysis and rendering module, dynamically calculating an effective view radius by utilizing a visual tunneling effect; the interface analysis module performs character string fuzzy matching in combination with the context input by the user, identifies the search intention of the user and improves the weight of a corresponding component; the attention analysis module performs multi-modal weighted fusion on the physiological fixation data and the sparse content saliency thermodynamic diagram; according to the method, by recognizing the semantic intention and the physiological fixation point of the user, on the premise that core visual experience continuity is guaranteed, GPU load and video memory bandwidth occupation are remarkably reduced, and balance of high-performance display and low-power-consumption operation is achieved.

Methods and devices are described that enable user interactions using a combination of gaze input and touch input. During display of interactive elements on a display of an electronic device, an estimated gaze location of a gaze input is determined. A target interactive element is identified based on the estimated gaze location. Touch input is detected that traverses a first distance in a first direction. A preview of information related to the target interactive element is displayed, where an amount of the information displayed in the preview is based on the first distance traversed by the touch input.

The invention discloses an ideological and political large-scale course teaching attention assessment data acquisition method based on sight tracking. The method comprises the following steps: S1, high-definition large-scene eye movement tracking photographing equipment captures eye images of students in a course teaching process in real time through an infrared camera; s2, performing pupil area detection and pupil center fitting on the eye image to obtain sight tracking data; s3, performing gaze hotspot analysis and gaze duration statistics according to the sight tracking data to quantify attention distribution and change trend of the students and visually display the attention distribution and change trend; and S4, generating a teaching quality evaluation report and teaching optimization suggestions according to the gazing hotspot analysis and gazing duration statistical results. The classroom teaching effect can be objectively evaluated in real time, subjective evaluation errors are reduced, the accuracy and efficiency of teaching feedback are improved, the method is suitable for various scenes such as traditional classrooms, online education and experiment teaching, and the method has the industrial application prospect of education digital transformation.

The invention discloses a cheating detection method and system applied to online interview, relates to the technical field of computer vision, and solves the problems that in the prior art, multiple behavior characteristics in interview personnel are often adopted for monitoring in the whole process, multi-data analysis in the presence of non-cheating behaviors causes waste of computing resources, and the monitoring efficiency is high. And the cheating detection method is low in efficiency. A gaze anomaly coefficient is generated according to gaze data; when the fixation abnormal coefficient is greater than the abnormal threshold value, generating a multi-modal feature according to the monitoring video; generating a multi-modal decision weight according to the interview stage label; fusing the multi-modal decision-making weight and the multi-modal features to obtain multi-modal decision-making data; and inputting the multi-modal decision data into the cheating judgment model to obtain a monitoring result, carrying out tendency evaluation on cheating monitoring, and carrying out multi-modal dynamic fusion when the cheating tendency exists so as to accurately judge the cheating condition, so that the cheating monitoring efficiency and the cheating monitoring accuracy are improved.

Apparatuses, systems, and techniques are presented to estimate user gaze. In at least one embodiment, one or more neural networks are used to determine coarse and fine gaze estimates for one or more users.

Methods and systems are described for imaging and content generation. During a road trip, a driver or passenger observes an object or scene from a vehicle. From a perspective of the driver or passenger, the object or scene flows out of view in a relatively short amount of time. Images and video of the object or scene are captured with cameras. A field of view is extended beyond what the driver or passenger can easily observe. The capture is configurable to focus on particular types of objects or scenes. Recordings of billboards and road signs are easily displayed after passing the object. Object identification, gaze determination, and interest determination are provided. Curated content is generated. Applications to extended reality environments are provided. Artificial intelligence systems, including neural networks, and models are utilized to improve the imaging and content generation. Related apparatuses, devices, techniques, and articles are also described.

According to an embodiment, a wearable device may display, based on contact on a second surface identified using a touch sensor, a visual object indicating a first position of the touch input on the second surface in a screen through a display. The wearable device may identify, in response to the touch input, a second position in the screen of a gaze identified based on an image obtained through a camera exposed outside at a portion of a first surface. The wearable device may provide, in response to the second position identified within a specified distance from the first position of the visual object, feedback with respect to the touch input. The wearable device may cease to provide the feedback in response to the second position identified outside from the first position by the specified distance.

Systems, methods, and apparatuses are described for causing a camera of a head-mounted computing device to capture a first video, the head-mounted computing device comprising a camera direction control element for controlling a capture direction of the camera, and a camera zoom control element for controlling zoom of the camera. One or more objects in the captured first video may be identified based on a detected gaze angle of a user wearing the head-mounted computing device. A target location in an environment may be determined, and based on such target location, the capture direction and zoom of the camera may be adjusted using the camera direction control element and the camera zoom control element, respectively. The camera may capture, based on the adjusted capture direction and the adjusted zoom of the camera, a second video using the camera of the head-mounted computing device.

The invention relates to the technical field of mixed reality and artificial intelligence fusion application, in particular to a mixed reality intelligent inspection method and system based on multi-modal fusion. According to the technical scheme, the method comprises the following steps: synchronously acquiring a user view image and voice stream data through mixed reality equipment; performing target detection on the view image by adopting an improved YOLOv10 model, and outputting a candidate target area and a confidence score # imgabs0 # thereof; voice input is transferred through a Whisper voice recognition model, and a semantic vector # imgabs1 # is generated by using a BERT semantic embedding model. According to the invention, through a multi-mode cooperative sensing mechanism of fusing image, voice and space gazing information, in combination with context semantic optimization and lightweight terminal deployment, intelligent target identification and natural interaction with high precision, real-time performance and strong robustness are realized in a complex industrial inspection scene.

The invention discloses a high-altitude falling simulation system based on VR equipment, and relates to the technical field of virtual reality simulation, and the system comprises a data collection module which collects the pupil distance and initial sight line data of a user; the visual disturbance module is used for dynamically adjusting local visual disturbance of a corresponding area in the high-altitude virtual scene based on the gaze data to form a visual disturbance effect; the psychological index calculation module is used for calculating an anxiety index and a focus drift speed by adopting a time sequence convolutional neural network according to the visual disturbance effect and the eye movement behavior of the user to obtain an anxiety index and a focus drift speed index; and the disturbance adjustment module adaptively adjusts the frequency and amplitude of the visual disturbance effect by adopting a method of combining fuzzy logic control and adaptive gain adjustment to obtain disturbance rhythm parameters. According to the method, the personalized matching ability and training effectiveness in psychological behavior intervention and virtual high-altitude adaptation training are improved.

The invention provides a hybrid wireless eye movement tracking method and system based on multi-sensor fusion, electronic equipment and a storage medium, and relates to the technical field of eye movement tracking. The method comprises the following steps: firstly, acquiring various data by using various different types of sensors such as a Q-Var sensor, an inertial measurement unit and a high-speed panoramic camera; secondly, preprocessing the various sensor data by means of graph Fourier transform, a Kalman filter, a spectral clustering technology and the like; fusing the preprocessed sensor data based on a graph theory method; and finally, performing fixation track prediction based on a convolutional neural network and a gating circulation unit. According to the technical scheme, the performance of eye movement tracking under different environment conditions is improved, the eye movement tracking precision is higher, and the stability is better. In addition, the technology of the application also uses a 5G millimeter wave technology or an IEEE 802.11 ac protocol to transmit gaze tracking data, thereby achieving the purpose of bandwidth enhancement and low-delay data transmission.

Systems and methods are provided for generating adaptive content. The system may implement a machine learning model including training data pre-trained, or trained in real-time based on captured content or prestored content associated with gazes of users, pupil dilations, facial expressions, muscle movements, heart rates, or gaze dwell times of users determined previously or in real-time. The system may determine a gaze(s) of an eye of a user or facial features of a face associated with the user viewing, by a device, items of content in an environment. The system may include determining, based on the gaze(s) or facial features, a state(s) or interest(s) of the user. The system may determine, by implementing the machine learning model and based on the state(s) or interest(s) of the user, content to generate a modification of the items of content or to generate new content items associated with the items of content.

The invention relates to the technical field of intelligent education, in particular to an interactive English teaching method and system based on AI vision, and the method comprises the steps: obtaining the voice and mouth dynamic image frame segments of a student, extracting an offset frame segment to generate a semantic motion disjunction interval, analyzing the time sequence consistency of a behavior signal and a motion cause word triggering frame segment, and calculating the motion matching degree. Time synchronism of the speech output and the visual action is evaluated. According to the invention, through real-time analysis of the voice and mouth dynamic image of the student, the alignment offset of the pronunciation and the mouth action is identified, and in combination with behavior signals such as eye fixation and head rotation of the student, the interactivity between the language action and the behavior of the learner is comprehensively evaluated, so that the efficient alignment of the language expression and the standard template is ensured; by analyzing the distribution frequency and the matching degree of the voice behaviors, a personalized teaching path is optimized. In addition, delay frames and contour offset are deeply analyzed, the synchronism of voice and visual actions is enhanced, and the real-time feedback and interaction effect in the learning process is improved.

Various implementations disclosed herein include devices, systems, and methods for using a gaze vector and head pose information to activate a display interface in an environment. In some implementations, a device includes a sensor for sensing a head pose of a user, a display, one or more processors, and a memory. In various implementations, a method includes displaying an environment comprising a field of view. Based on a gaze vector, it is determined that a gaze of the user is directed to a first location within the field of view. A head pose value corresponding to the head pose of the user is obtained. On a condition that the head pose value corresponds to a motion of the head of the user toward the first location, a user interface is displayed in the environment.

Various implementations disclosed herein include devices, systems, and methods for modifying display of virtual content. In some implementations, the method is performed by an electronic device including a non-transitory memory, one or more processors, a display, an eye tracker and a movement sensor. In some implementations, the method includes displaying, via the display, virtual content. In some implementations, the method includes detecting, via the eye tracker, a movement of a gaze in a first direction. In some implementations, the method includes detecting, via the movement sensor, a movement of a head in a second direction while detecting the movement of the gaze in the first direction. In some implementations, the method includes, in accordance with a determination that the movement of the gaze in the first direction and the movement of the head in the second direction are corresponding opposing movements, modifying a display characteristic of the virtual content.

The present application provides a human-computer interaction method and apparatus, a device, and a medium. The method includes: determining a gaze point of a line of sight of a user on a target object, the target object being located in a virtual space; adjusting, in response to an interaction gesture for the gaze point, a display form of the gaze point to generate a first interaction point; and interacting with the target object based on the first interaction point.

The invention discloses a new media promotion method and system based on the Internet, and belongs to the technical field of new media promotion, and the method specifically comprises the steps that the distribution density and the stop position of people in a waiting area are monitored in real time, load distribution data in a lift car are collected, and the vibration frequency and the start-stop state in the elevator running process are recorded; when it is detected that the density of people in the waiting area exceeds a set threshold value, switching to duration compressed advertisement content, and when the load distribution of the lift car displays that one side is crowded, displaying interactive advertisements in the free area; capturing a user eyeball movement track, calculating an advertisement interface fixation thermodynamic diagram, identifying a user gesture operation by using a millimeter wave radar, mapping a specific gesture into an advertisement content switching instruction, and generating a personalized advertisement recommendation scheme; and the advertisement interaction data is converted into a pushing strategy adjustment basis of adjacent elevators, a propagation path is established according to the flow rule of people in the elevator room, and when the advertisement click rate of a certain elevator exceeds a set value, a content recommendation signal is sent to the associated elevator.

The invention belongs to the field of virtual reality teaching application, and provides a teaching atmosphere automatic adaptation method and system in an immersive teaching environment. The method comprises the following steps: acquiring an eye movement image; recognizing a fixation point; carrying out ROI tracking; carrying out ROI boundary fusion; adaptively optimizing the object; adjusting the brightness of the ROI; and watching object interaction. According to the method, the immersion and interactivity of a future classroom can be improved, the use experience of an immersive virtual environment is facilitated, and deep fusion of an intelligent teaching environment and self cognition of a user is promoted.

Various implementations assess user interactions to trigger user interface responses. Some implementations associate user gaze with user interface elements using one or more criteria that enable interpreting the user's gaze in ways that are consistent with user intentions and / or expectations. In some implementations, a user's gaze is associated with a user interface element when the user gazes at the user interface element. When the gaze leaves the user interface element (e.g., intentionally or unintentionally), the association may be maintained or discontinued using one or more criteria to be consistent with user intentions and / or expectations. In some implementations, this involves determining whether to continue associating a user's gaze with a user interface element so that the gaze-to-element association is maintained in certain circumstances (e.g., blinks and saccades) but discontinued in other circumstances (e.g., when the user is fixating outside of a zone around the element for more than a threshold time / number of frames).

The invention relates to the technical field of deep faking attribution and detection methods, in particular to a deep faking attribution and detection method based on a gaze guide CLIP model, and the method specifically comprises the following steps: collecting face images, constructing a deep faking attribution and detection data set, enabling each face image in the data set to have a corresponding faking attribution label and a faking detection label, preprocessing images in the data set, and then dividing into a training set and a test set; constructing a zero-sample deep counterfeit attribution and detection model, inputting the preprocessed data set into the model, and performing model processing to obtain predicted image counterfeit detection category features and predicted image counterfeit attribution category features; and performing optimization training on the deep counterfeit attribution and detection model through the training set to obtain an optimized and trained model, and testing the optimized and trained model by using data in the test set. According to the invention, the source of the deep pseudo image is traced through the deep learning method, and the faked attribution and detection can be carried out more accurately.

The invention discloses a nursing scene data enhancement and generation method for intention recognition model training, and belongs to the technical field of artificial intelligence and smart old-age care. The method aims at solving the problems that in the prior art, high-quality nursing scene training data is deficient, and the obtaining cost is high. According to the core technical scheme, the method comprises the steps that firstly, a staring sequence and other context information (such as time, place and physiological signals) of a user are processed through a multi-modal information fusion model, and a structured initial situation vector is generated; secondly, inputting the vector into a scene generation model combined with a nursing knowledge base, and automatically generating a batch of basic nursing scene data with intention labels; key points are that a data enhancement module is introduced, and a plurality of innovative strategies such as situation element disturbance, virtual physiological data synthesis, gaze path variation and virtual scene deduction are adopted to deeply process basic data, so that the diversity and complexity of a data set are greatly enriched; and finally, combining the basic data with the enhanced data to construct a final comprehensive training data set. According to the method, large-scale and high-fidelity training data can be generated in a low-cost and high-efficiency manner, and the accuracy and robustness of the intention recognition model in a real nursing environment are remarkably improved.