Extended reality (XR) systems and methods for predicting eye state

Abstract

A XR device configured to be worn by a user, where the XR device includes a display for displaying images to the user, a first eye sensor for producing data for use in predicting the duration of an eye-closed state, and processing circuitry. The XR device is configured to perform a method that includes, based on data from the first eye sensor, predicting the duration of an eye-closed state. The method also includes determining whether the predicted duration satisfies a duration condition (e.g., exceeds a time threshold). The method further includes in response to determining that the predicted duration satisfies the duration condition, adjusting an operational mode of the XR device.

Description

EXTENDED REALITY (XR) SYSTEMS AND METHODS FOR PREDICTING EYE STATETECHNICAL FIELD

[0001] Disclosed are embodiments related to extended reality (XR) systems and method for predicting an eye state of a user.BACKGROUND

[0002] Extended reality (XR) is an umbrella term that encompasses virtual reality (VR) and augmented reality (AR).

[0003] Virtual reality (VR) technology is technology that involves creating a virtual environment that a user can interact with. Today, a user’s interaction with a virtual environment is often performed via head-mounted displays (HMDs), as they allow a user to view a 3D representation of the virtual environment with real-life motions being tracked and mirrored inside the virtual environment. A virtual environment includes not only a virtual visual environment, but also can be represented in other senses, most commonly being spatial audio and haptic feedback within hand controllers.

[0004] Augmented Reality (AR) technology is technology that involves combining the real world with virtual environments, enhancing a user’s perception of the real world, and / or changing the user’s perception of the real world. For example, in an AR application, virtual objects can be represented by using displays, speakers, haptics, or other mediums, intersecting with the physical real-world environment. An example is a pair of glasses that has an integrated heads-up display that can show, for example, a virtual model of a building in the place of where another building is already standing. Depending on the application, objects in the real world can influence virtual objects and vice versa.

[0005] Gaze tracking refers to determining the direction in which a user is looking. Gaze tracking systems often use a camera to monitor the user’s eyes and a trained algorithm using data from the camera to calculate where the user is looking, showing the result either as a vector from the user’s eyes or as a point in space / projected onto a 2D screen. In HMDs this is most commonly achieved via an inwards-facing camera located on the inside rim of the headset.

[0006] Closed-eye gaze gestures can be recognized with e.g., gaze tracking camera or EOG (electrooculogram), of which there are existing solutions integrated in smart glasses frames (see, e.g., references [1, 2, 4]).SUMMARY

[0007] Certain challenges presently exist. For instance, current implementations of “sleep mode” for XR devices are simplistic, reactive, and binary. As explained in reference [7], sleep mode activates after a set time of inactivity. That is, for example, after the period of inactivity is detected, the device powers down or otherwise reduces its functionality.

[0008] In some XR systems, the XR system is configured to detect inactivity by determining whether the user is utilizing an XR headset. The determination as to whether a user is utilizing the headset may be based on a distance detector that can provide an indication as to whether the user is wearing the headset (a common way to bypass this is to use a piece of tape over this detector) or based on images from an inward facing camera for detecting the eyes. Another way to detect whether a user is utilizing the headset is to employ an eye pressure sensor.

[0009] Eye pressure sensors are based on two physical electrodes which sense the pressure of muscles of eye that active eye-lid movement. In order to detect the movement of eyelid, the XR headset (e.g., AR / VR glasses) needs to have electrodes connected, which are supposed to touch close to the eyes. This method is limited by the presence of the electrodes and their contact to the user’s nose pads. Any XR headset would need a fixture with nose support. This will not be relevant for all headset use cases, such as helmet with visor.

[0010] As noted above, an alternative method to detect inactivity relies on camera-based eye movement monitoring, which is described in reference [9], However, any use of camera and corresponding detection is energy hungry and typically needs a frame by frame analysis and / or a complex machine learning algorithm. Hence, any power saved by using this method to detect inactivity may be eclipsed by the amount of power needed to perform the inactivity detection.

[0011] Yet another method to detect inactivity could utilize RADAR-based eye tracking. Implementing RADAR-based eye tracking in an HMD, however, is not straight-forward and costly.

[0012] The other problem of existing solutions is the reactive nature of sensing and the corresponding power saving events. The process of sensing an eyelid down state requires a set of detection steps which may cause a delay and / or have variable accuracy. This can result in either the headset powering down more than necessary (e.g., when mis-detected), or still running on full power despite the user having their eyes closed (e.g., in case of a delay).

[0013] Accordingly, in one aspect there is provided an XR device configured to be worn by a user, where the XR device includes a display for displaying images to the user, a first eye sensor for producing data for use in predicting the duration of an eye-closed state, and processing circuitry. The XR device is configured to perform a method that includes, based on data from the first eye sensor, predicting the duration of an eye-closed state. The method also includes determining whether the predicted duration satisfies a duration condition (e.g., exceeds a time threshold). The method further includes in response to determining that the predicted duration satisfies the duration condition, adjusting an operational mode of the XR device.

[0014] In another aspect, there is provide a method performed by an XR device configured to be worn by a user. The method includes a first eye sensor producing data for use in predicting the duration of an eye-closed state. The method also includes, based on the data from the first eye sensor, predicting the duration of an eye-closed state. The method also includes determining whether the predicted duration satisfies a duration condition (e.g., exceeds a time threshold). The method further includes in response to determining that the predicted duration satisfies the duration condition, adjusting an operational mode of the XR device and / or notifying an application to prepare for a prolonged eye-closed state.

[0015] In another aspect there is provided a computer program comprising instructions which when executed by processing circuitry of an apparatus causes the apparatus to perform any of the methods disclosed herein. In one embodiment, there is provided a carrier containing the computer program wherein the carrier is one of an electronic signal, an optical signal, a radio signal, and a computer readable storage medium.

[0016] An advantage of the embodiments disclosed herein is that they increase the energy efficiency of the XR device, thereby experiencing less battery-drain in the XR device.BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate various embodiments.

[0018] FIG. 1A illustrates a system according to an embodiment.

[0019] FIG. 1B illustrates components of an XR device according to embodiment.

[0020] FIG. 2 illustrates an embodiment.

[0021] FIG. 3 illustrates an example configuration.

[0022] FIG. 4 shows multiple sensors attached to the display.

[0023] FIG. 5 is a data flow diagram illustrating a process according to an embodiment.

[0024] FIG. 6 is a flowchart illustrating a process according to an embodiment.

[0025] FIG. 7 illustrates components of an XR device according to an embodiment.DETAILED DESCRIPTION

[0026] FIG. 1 A illustrates an XR system 100 in which the embodiments disclosed herein may be applied. XR system 100 includes speakers 104 and 105 (which may be speakers of headphones worn by the user) and an XR device 110 that may include a display for displaying images to the user and that, in some embodiments, is configured to be worn by the user. In the illustrated XR system 100, XR device 110 has a display and is designed to be worn on the user‘s head and is commonly referred to as a head-mounted display (HMD).

[0027] As shown in FIG. 1B, XR device 110 may comprise an orientation sensing unit (OSU) 101, a position sensing unit (PSU) 102, a gaze tracking unit (GTU) 169, and a processing unit 103 coupled (directly or indirectly) to an audio renderer 151 for producing output audio signals (e.g., a left audio signal 181 for a left speaker and a right audio signal 182 for a right speaker as shown) and a video renderer 171 for producing an output video signal 193 for display.

[0028] Orientation sensing unit 101 is configured to detect a change in the orientation of the user and provides information regarding the detected change to processing unit 103. In some embodiments, processing unit 103 determines the absolute orientation (in relation to some coordinate system) given the change in orientation detected by orientation sensing unit 101. There could also be different systems for determination of orientation and position, e.g. asystem using lighthouse trackers (LIDAR). In one embodiment, orientation sensing unit 101 may determine the absolute orientation (in relation to some coordinate system) given the detected change in orientation. In this case the processing unit 103 may simply multiplex the absolute orientation data from orientation sensing unit 101 and positional data from position sensing unit 102. In some embodiments, orientation sensing unit 101 may comprise one or more accelerometers and / or one or more gyroscopes.

[0029] GTU 169 is configured to track the user’s gaze. In one embodiment, GTU 169 employes an “eyes-open” algorithm to track the user’s gaze when the user’s eyes are open and employes an “eyes-closed” algorithm to track the user’s gaze when the user’s eyes are closed. GTU 169 may track the user’s gaze using an inwards-facing camera or the capacitive sensor described herein.

[0030] Audio renderer 151 produces the audio output signals based on input audio signals 161, metadata 162 regarding the XR scene the user is experiencing, and information 163 about the location and orientation of the user. The metadata 162 for the XR scene may include metadata for each object and sound source included in the XR scene, as well as metadata for the XR space (“acoustic environment) in which the user is virtually located. The metadata for an object may include information about the dimensions of the object and occlusion factors for the object (e.g., the metadata may specify a set of occlusion factors where each occlusion factor is applicable for a different frequency or frequency range). The metadata 162 may also include control parameters, such as a reverberation time value, a reverberation level value, and / or absorption parameter(s). Audio renderer 151 may be a component of XR device 110 or it may be remote from the XR device 110 (e.g., audio renderer 151, or components thereof, may be implemented in the cloud).

[0031] Video renderer 171 produces the video output signal 193 based on input video signal 191, metadata 162 regarding the XR scene the user is experiencing, and information 163 about the location and orientation of the user. Video Tenderer 171 may be a component of XR device 110 or it may be remote from the XR device 110 (e.g., video renderer 171, or components thereof, may be implemented in the cloud).

[0032] As noted above, certain challenges presently exist because, for instance, current implementations of sleep mode for XR devices are simplistic, reactive, and binary.

[0033] This disclosure provides a method in which XR device 110 uses data from a sensor (e.g., a moisture sensor) to predict when, and for how long, a user will close their eyes. Using this prediction, the functionalities of XR device 110 are limited, thereby allowing for a more gradual and efficient use of the XR device’s systems and power. For instance, when it is predicted that the user’s eyes will close within a certain period of time for at least some predetermined amount of time, the components of XR device 110 that are critical for accurately detecting eye gaze and / or the components for displaying high-definition visuals can be deprioritized and eventually turned off, with resources instead being devoted to a less precise eye-open detection algorithm.

[0034] When a user is wearing XR device 110, there may be instances where the user desires to close their eyes for an extended period of time (e.g., >2 seconds). This could, as examples, be due to resting, meditation, or to visually avoid content being displayed (e.g. a frightening game). The two former examples are more likely as headsets become lighter and more comfortable to wear - resting in bed while wearing XR glasses could become more of a possibility.

[0035] During these instances, the solution proposes that the XR device utilizes its sensor to not only detect that the user’s eyes have closed, but also predict how long the eyes will remain closed. In response to predicting that the eyes will remain closed for at least a predetermined threshold of time, XR device 110 may begin to gradually lower its resource usage (e.g., reduce the brightness of its display, cease gaze tracking, etc.).

[0036] The amount of resource reduction can be a function of time, with more parts of the systems’ functionalities being disabled the longer the predicted duration of the eye-closed state. The application(s) currently active could also be a factor in determining how fast the device reduces the resource usage, by the device providing an API that allows applications to set an appropriate reduction level based on their contents (e.g. a meditation app may shut down systems faster than a horror experience).

[0037] Contactless Low Power Detection of Closed Eyes

[0038] At any given time, eyes have a particular level of moisture. The moisture of the user’s eyes affects the frequency of eyelid dropping, corresponding to an eye-closed event. The lower the moisture, the more frequently the eye will be closed. Also, the duration and the amountof closed eye is dependent on the variation in moisture of the eye. Publications have summarized the dependencies of eye-closed events, or eye closures, and moisture of the eye, such as reference

[0010] , Table 1 below is a copy of a table from reference

[0010] showing a determined relationship between eye moisture and eye-closed events.Table I Proportions of blink types in dry-eye (n=IO) and normal (n= 11] subjectsT ype of blink based Blink types: proportions and odds ratioson lid closure Proportions, Proportions, Ratio of Ratio of odds,confidence P-value fordry eye normals proportions, dry eydiwmals interwls fa* equal odds(n 3,576) (n 1,414) dry eyefnormals odds ratioBWHs O.t 0.032 0.05 Ofil ip.22, 3 J5) 0. M501 blinks OdW 0236 0. M3 0, W3 (0.33, 1.93) 0.62575 blinks 0J« 0J3I 1® LOffl i058, 204i 0. WAll lid closures OJffl 0368 1.041 IM (043, 1621 0.8B(>0 second in du|raLGn)Extended lid closures 0.015 0032 0.463 0.455 (0.14, 1.44) 0.179i G.l se’cncindumo"|SUMMON lid ■: osurcs 0,023 0.002 10.02? 10.242 (1.38.76.3) 0.023i 0.5 second in durationNotes' D n o ' ir u ctjie'c J-t cm onent kniri. r uJ 'It * Arre J bi / J tU / iH D,-c»i m J hOfiwI ww CMpttftf xh jucs ntiL’i Cuds-ritie.1. tutusl’5’:.tnfiJ.r.i titeml -r u b ‘ 1 l.vj-s oic test tr ap.l it. pt s.r tu Grjip loikurs.r c ivi i.t x - 1 - i.pi cst ntc d feiwu If s 51 ik« jt 0051 lAr tr its fujx it-.- nflh if.hty. in Lid

[0039] The presence of moisture influences the capacitance of a capacitive structure; thus, moisture sensors can be based on capacitance as described in reference

[0011] , Any variation in moisture, in the space in which the electric field extends between the electrodes of the capacitor, results in variation in the capacitance of the capacitor. In this disclosure it is proposed to have the display of XR device have electrodes to create a planar capacitor. The capacitor may be mounted in a way that its fringe field touches the eyes. Thus, the capacitance of the capacitor will vary with changes in moisture of the eye. Additionally, the capacitance also will change when the eye-lids move. A detector circuit can be used to monitor continuous variation of the capacitance of the capacitor, to detect the variation in moisture and, based on that, to predict the eye-lids closing — i.e., to predict an eye-closed event. Additionally, from the continuous value of capacitance reading, it will detect prolonged closing of the eye-lids.

[0040] FIG. 2 illustrates an embodiment where the sensor comprises a capacitor having a first electrode 201 and a second electrode 202, both of which are attached to an inside surface ofa component 204 of display 206 of XR device 110. Display component 204 may be a thin glass component for displaying images to the user. Display component 204 stays at a certain distance from the iris of the eye. The distance and intensity of the fringe field 211 is a function of the wavelength. With a very low frequency signal in kHz, the electrodes 201, 202 of the capacitor with certain separation between them will have far enough fringe field 211 to touch the eyes to detect moisture of the eye and eye-lid movement, as shown in FIG. 2.

[0041] FIG. 3 illustrates an example configuration of electrodes 201 and 202. That is, in the embodiment shown in FIG. 3 electrodes 201 and 202 form a transparent capacitor 301. The orientation of the capacitor is not limited. However, the separation between the electrodes and thus the size of the capacitor will be dependent on the frequency of signal for humidity detection.

[0042] FIG. 4 shows an example of display component 204 having multiple transparent capacitors 301. The presence of multiple capacitors can be utilized to track the eye more accurately.

[0043] Capacitor 301 is designed so as not to obstruct visibility. For example, in one embodiment, a transparent conducting paint with metallic ink, which has been used for RIS antenna design (see, e.g., reference [6]), is used for the capacitor formation on glass. In another embodiment, graphene is used for creating a transparent capacitor 301. Graphene is a good conductor and therefore is suitable for capacitor design. Graphene can be formed on glass using different methods, including with ink-based painting, and offers sufficient transparency as shown in reference [5],

[0044] FIG. 5 is a data flow diagram illustrating a process, according to one embodiment, for making predictions about eye-closed events — that is, for example, predicting how long a user will keep their eye(s) closed and determining a probability that a user will transition from having eyes open to eyes closed within a predetermined amount of time (e.g., 100 milliseconds). In the example shown (but not limited to) this figure indicates capacitance variation can be extracted in the form of Voltage Standing Wave Ratio (VSWR). Whenever there is a variation in the capacitance, the capacitive load will change relative to the port impedance and thus VSWR. The VSWR is to be measured continuously with a certain sampling frequency. Considering that the event for eye blinking is in the range of seconds, the required sampling frequency may not be more than a KHz. Noise cancellation and artifact removal can then be applied to the capturedsignal. For example, unwanted physical movement-related artefacts may be cancelled out to have a low noise signal representing eyelid movement.

[0045] The low noise signal is then used in an eye-closed event detector to determine whether the user’s eyes are closed. This detector also keeps track of the actual interval between two eye-closed events. The low noise signal is then processed to remove the portion of the signal corresponding to when the user’s eye was closed, thereby producing a signal that only corresponds to when the user’s eye is open. This signal is then filtered, and the filtered signal is then used to determine an initial moisture level value. For example, the initial moisture level value can be determined using the filtered signal and look-up table such as the one shown in Table 1 of reference

[0010] , As another example, the filtered signal is input into a trained Machine Learning (ML) model that then maps the input signal to an initial moisture level value and outputs this value.

[0046] In the example shown, a correction value is added to the initial moisture value to produce a calibrated moisture level value. As shown in FIG. 5, in one embodiment, the correction value is determined based on a comparison of the estimated interval between two eye-closed events and the measured actual interval between the two eye-closed events.

[0047] The calibrated moisture level value is then used to predict the duration of an eye-closed state. That is, the calibrated moisture level value is used to produce an estimate of how long it is expected that the user’s eye will remain closed the next time the user closes the eye. The calibrated moisture level value is also used to estimate an interval between two eye-closed events and may further be used to determine, while the user’s eye is open, a probability that the user’s eye will close within a predetermined amount of time.

[0048] FIG. 6 is a flowchart illustrating a process 600 according to an embodiment. Process 600 may begin in step s602.

[0049] Step s602 comprises a first eye sensor (e.g. sensor 301) producing data for use in predicting the duration of an eye-closed state.

[0050] Step s604 comprises determining whether the user’s eyes are closed. If the eyes are not closed, process 600 proceeds to step s606, otherwise it proceeds to step s612.

[0051] Step s606 comprises the XR device 110 operating in its eyes open mode.

[0052] Step s608 comprises using the data obtained in step s602 to determine a probability (p) that the user’s eye will close within a predetermined amount of time (e.g., within 100 milliseconds). That is, the data obtained in step s602 is used to determine a probability within the predetermined amount of time the user will experience an eye-closed event, and thereby transition from an eye-open state to an eye-closed state.

[0053] Step s610 comprises determining whether p is less than a probability threshold (P). If p is less than P, process 600 returns to step s602, otherwise process 600 proceeds to step 612.

[0054] Step s612 comprises predicting the duration (d) of the next eye-closed state using the data produced in step s602.

[0055] Step s614 comprises determining whether d is less than a duration threshold (D). If it is not less than the duration threshold, process 600 proceeds to step 616, otherwise process 600 returns to step s602.

[0056] Step s616 comprises adjusting an operational mode of the XR device and / or notifying an application of the XR device to prepare for a prolonged eye-closed state. The notification may include information specifying the predicted duration. The notification may further include information specifying the value of p. In one embodiment, the application is notified that XR device 110 is switching to an eyes-closed mode (e.g. via an API call). In response to the notification, the application may change its content to better suit the user, for example changing a football game viewing experience to radio narration which typically has more spoken information.

[0057] After step s616, process 600 may go back to step s602.

[0058] In some embodiments, the first eye sensor comprises a transparent capacitive sensor for use in determining a moisture level in the vicinity of the transparent capacitive sensor.

[0059] In some embodiments, the transparent capacitive sensor is attached to an inner surface of the display of XR device 110.

[0060] In some embodiments, predicting the duration of an eye-closed state based on data from the first eye sensor comprises: determining a moisture level using the data from thefirst eye sensor; and predicting the duration of an eye-closed state based on the determined moisture level.

[0061] In some embodiments, determining the moisture level comprises determining the moisture level based on a measured average moisture level and a correction value.

[0062] In some embodiments, the method further comprises determining the correction value based on a measured interval between two eye-closed events and a predicted interval between the two eye-closed events.

[0063] In some embodiments, the method further comprises determining the predicted interval based on the determined moisture level.

[0064] In some embodiments, the magnitude of the adjustment of the operational mode of the XR device is proportional to the predicted duration.

[0065] In some embodiments, adjusting an operational mode of the XR device comprises: modifying a functionality of the XR device, reducing the brightness of the display of the XR device, reducing the amount of data processed by the XR device, reducing the XR device’s power consumption, and / or reducing the quality of video and / or audio output by the XR device.

[0066] In some embodiments, the method further comprises, while the user’s eye is open, determining a probability that the user’s eye will close within a predetermined amount of time.

[0067] Gaze Tracking While Eyes Closed

[0068] In one embodiment, in addition to the optimizations listed above, once the device detects that the user’s eyes are closed, gaze tracking unit (GTU) 169 of XR device 110 shifts over to the eyes-closed gaze tracking algorithm. This eyes-closed gaze tracking algorithm can be implemented for the existing inwards-facing camera, or utilize a sensor made specifically for it. While active, the eyes-closed algorithm would allow tracking the user’s eyes behind their eyelids based on movements seen on the outside of the eyelid. This movement could then be utilized as input for applications, should it be appropriate to do so. As this method is less precise than proper gaze tracking it could possibly be used for more binary operations, like raising and lowering the volume based on which direction the eyes have shifted (and if the eyes e.g. are being kept in said direction for a time duration X ms).

[0069] In this mode, an application of XR device 110 is also able to utilize a rudimentary user interface (UI) system adapted to the user’s eyes being closed. The XR device can, for example, shine light in various colors, patterns (including e.g. lights pulsating in various sequences), and strength to indicate UI elements. An example of a UI pattern would be a light shining from left to right across the XR device’s screen, with the location of the light indicating a sliding value like e.g., the volume of the device audio. This can be recognized by the user with eyes closed. In another embodiment the inwards-facing gaze tracking cameras are replaced with, or complemented with, EOG sensors for detecting eye gestures.

[0070] In another embodiment, the user could deliberately have one eye open and one eye closed, and perform eye gestures that, with the additional left or right eye being closed while performing said gestures, have a special meaning to the system, e.g., trigger a specific function, etc. For instance, closing just the left eye while keeping the right eye open can be interpreted by XR device 110 as a command to lower a parameter (e.g., lower volume of audio, lower brightness of display screen, etc.), whereas closing just the right eye while keeping the left eye open can be interpreted by XR device 110 as a command to increase a parameter (e.g., increase volume of audio, increase brightness of display screen, etc.).

[0071] FIG. 7 is a block diagram of a XR device 110, according to some embodiments. As shown in FIG. 7, XR device 110 may comprise: PC 102, which comprises one or more processors (P) 755 (e.g., one or more general purpose microprocessors and / or one or more other processors, such as an application specific integrated circuit (ASIC), field-programmable gate arrays (FPGAs), and the like), which processors may be co-located in a single housing or in a single data center or may be geographically distributed (e.g., XR device 110 may be a distributed, cloud computing system comprising two or more computers or a monolithic computing system consisting of a single computer); at least one network interface 748 (e.g., a physical interface or air interface) comprising a transmitter (Tx) 745 and a receiver (Rx) 747 for enabling XR device 110 to transmit data to and receive data from other nodes connected to network 110 (e.g., an Internet Protocol (IP) network) to which network interface 748 is connected (physically or wirelessly) (e.g., network interface 748 may be coupled to an antenna arrangement comprising one or more antennas for enabling XR device 110 to wirelessly transmit / receive data); and a storage unit (a.k.a., “data storage system”) 708, which may include one or more non-volatile storage devices and / or one or more volatile storage devices. Inembodiments where PC 103 includes a programmable processor, a computer readable storage medium (CRSM) 742 may be provided. CRSM 742 may store a computer program (CP) 743 comprising computer readable instructions (CRI) 744. CRSM 742 may be a non-transitory computer readable medium, such as, magnetic media (e.g., a hard disk), optical media, memory devices (e.g., random access memory, flash memory), and the like. In some embodiments, the CRI 744 of computer program 743 is configured such that when executed by PC 103, the CRI causes XR device 110 to perform steps described herein (e.g., steps described herein with reference to the flow charts). In other embodiments, XR device 110 may be configured to perform steps described herein without the need for code. That is, for example, PC 103 may consist merely of one or more ASICs. Hence, the features of the embodiments described herein may be implemented in hardware and / or software.

[0072] While various embodiments are described herein, it should be understood that they have been presented by way of example only, and not limitation. Thus, the breadth and scope of this disclosure should not be limited by any of the above-described exemplary embodiments. Moreover, any combination of the above-described elements in all possible variations thereof is encompassed by the disclosure unless otherwise indicated herein or otherwise clearly contradicted by context.

[0073] As used herein transmitting a message “to” or “toward” an intended recipient encompasses transmitting the message directly to the intended recipient or transmitting the message indirectly to the intended recipient (i.e., one or more other nodes are used to relay the message from the source node to the intended recipient). Likewise, as used herein receiving a message “from” a sender encompasses receiving the message directly from the sender or indirectly from the sender (i.e., one or more nodes are used to relay the message from the sender to the receiving node). Further, as used herein “a” means “at least one” or “one or more.”

[0074] Additionally, while the processes described above and illustrated in the drawings are shown as a sequence of steps, this was done solely for the sake of illustration. Accordingly, it is contemplated that some steps may be added, some steps may be omitted, the order of the steps may be re-arranged, and some steps may be performed in parallel.

[0075] References

[0076] [1] Findling, R. D., et. al., “Hide my Gaze with EOG! Towards Closed-Eye Gaze Gesture Passwords that Resist Observation-Attacks with Electrooculography in Smart Glasses,” In 17th International Conference on Advances in Mobile Computing and Multimedia, 2019.

[0077] [2] Friström, E., et. al., “Free-Form Gaze Passwords from Cameras Embedded in Smart Glasses,” In 17th International Conference on Advances in Mobile Computing and Multimedia, 2019.

[0078] [3] www.tobii.com / learn-and-support / get-started / what-is-eye-tracking

[0079] [4] Alam, M., et. al., “High Precision Eye Tracking Based on Electrooculography (EOG) Signal Using Artificial Neural Network (ANN) for Smart Technology Application”, 2021 24th International Conference on Computer and Information Technology (ICCIT) (available at ieeexplore.ieee.org / document / 9689821).

[0080] [5] Ivanov, I., et. al., “Layer-number determination in graphene on SiC by reflectance mapping” Elsevier 2014 (available at liu.diva-portal.org / smash / get / diva2:749471 / FULLTEXT01)

[0081] [6] Yang, Y., “Polarization Insensitive and Transparent Frequency Selective Surface for Dual Band GSM Shielding,” IEEE Transactions on Antennas and Propagation (Volume: 69, Issue: 5, May 2021) (available ati eeexpl ore. i eee. org / stamp / stamp. j sp?tp=& arnumb er=9241398

[0082] [7] EP3109689A1

[0083] [8] Chen, R., et. al., “Blink-sensing glasses: A flexible iontronic sensing wearable for continuous blink monitoring” iScience, May 2021 (available at www.sciencedirect.com / science / article / pii / S2589004221003679)

[0084] [9] Smart Eye, “Interior Sensing” (available at www.smarteye.se / solutions / automotive / interior-sensing / )

[0085]

[0010] Ousler, G., etl. al., “Blink patterns and lid-contact times in dry-eye and normal subjects,” Clinical Ophthalmology, May 2014 (available at pmc.ncbi.nlm.nih.gov / articles / PMC4015796 / pdf / opth-8-869. pdf)

[0086]

[0011] Feng, J., et. al., “Fabrication and Evaluation of a Graphene Oxide-Based Capacitive Humidity Sensor,” Sensors 2016 (available at www.mdpi.com / 1424-8220 / 16 / 3 / 314)

[0087]

[0012] Elman Retina Group, “How Much Do You Know About the Anatomy of Your Eye,” March 2022 (available at www.elmanretina.com / how-much-do-you-know-about-the-anatomy-of-your-eye / )

Claims

CLAIMS1. An extended reality, XR, device (110) configured to be worn by a user, the XR device (100) comprising:a display (206) for displaying images to the user;a first eye sensor (301) for producing data for use in predicting the duration of an eye-closed state; andprocessing circuitry, wherein the XR device is configured to perform a method comprising:based on data from the first eye sensor, predicting (s612) the duration of an eye- closed state;determining (s614) whether the predicted duration satisfies a duration condition (e.g., exceeds a time threshold); andin response to determining that the predicted duration satisfies the duration condition, adjusting an operational mode of the XR device and / or notifying (s616) an application to prepare for a prolonged eye-closed state.

2. The XR device of claim 1, whereinthe first eye sensor comprises a transparent capacitive sensor for use in determining a moisture level in the vicinity of the transparent capacitive sensor.

3. The XR device of claim 2, whereinthe transparent capacitive sensor is attached to an inner surface of the display.

4. The XR device of claim 2 or 3, whereinpredicting the duration of an eye-closed state based on data from the first eye sensor comprises:determining a moisture level using the data from the first eye sensor; and predicting the duration of an eye-closed state based on the determined moisture level.

5. The XR device of claim 4, whereindetermining the moisture level comprises determining the moisture level based on a measured average moisture level and a correction value.

6. The XR device of claim 5, whereinthe method further comprises determining the correction value based on a measured interval between two eye-closed events and a predicted interval between the two eye-closed events.

7. The XR device of claim 6, whereinthe method further comprises determining the predicted interval between the two eye-closed events based on the determined moisture level.

8. The XR device of any one of claims 1-7, whereinthe method comprises adjusting an operational mode of the XR device, andthe magnitude of the adjustment of the operational mode of the XR device is proportional to the predicted duration of the eye-closed state.

9. The XR device of any one of claims 1-8, whereinthe method comprises adjusting an operational mode of the XR device, and adjusting an operational mode of the XR device comprises:modifying a functionality of the XR device,reducing the brightness of the display of the XR device,reducing the amount of data processed by the XR device,reducing the XR device’s power consumption, and / orreducing the quality of video and / or audio output by the XR device.

10. The XR device of any one of claims 1-8, whereinthe method further comprises, while the user’s eye is open, determining a probability that the user’s eye will close within a predetermined amount of time.

11. A method (600) performed by an extended reality, XR, device (110) configured to be worn by a user, the method comprising:using a first eye sensor (301), producing (s602) data for use in predicting the duration of an eye-closed state;based on the data from the first eye sensor, predicting (s612) the duration of an eye-closed state;determining (s614) whether the predicted duration satisfies a duration condition (e.g., exceeds a time threshold); andin response to determining that the predicted duration satisfies the duration condition, adjusting an operational mode of the XR device and / or notifying (s616) an application to prepare for a prolonged eye-closed state.

12. The method of claim 11, whereinthe first eye sensor comprises a transparent capacitive sensor for use in determining a moisture level in the vicinity of the transparent capacitive sensor.

13. The method of claim 12, whereinthe transparent capacitive sensor is attached to an inner surface of a display (206) of the XR device.

14. The method of claim 12 or 13, whereinpredicting the duration of an eye-closed state based on data from the first eye sensor comprises:determining a moisture level using the data from the first eye sensor; and predicting the duration of an eye-closed state based on the determined moisture level.

15. The method of claim 14, whereindetermining the moisture level comprises determining the moisture level based on a measured average moisture level and a correction value.

16. The method of claim 15, whereinthe method further comprises determining the correction value based on a measured interval between two eye-closed events and a predicted interval between the two eye-closed events.

17. The method of claim 16, whereinthe method further comprises determining the predicted interval between the two eye-closed events based on the determined moisture level.

18. The method of any one of claims 11-17, whereinthe method comprises adjusting an operational mode of the XR device, andthe magnitude of the adjustment of the operational mode of the XR device is proportional to the predicted duration of the eye-closed state.

19. The method of any one of claims 11-18, whereinthe method comprises adjusting an operational mode of the XR device, and adjusting an operational mode of the XR device comprises:modifying a functionality of the XR device,reducing the brightness of a display (206) of the XR device,reducing the amount of data processed by the XR device,reducing the XR device’s power consumption, and / orreducing the quality of video and / or audio output by the XR device.

20. The method of any one of claims 11-18, whereinthe method further comprises, while the user’s eye is open, determining a probability that the user’s eye will close within a predetermined amount of time.

21. A computer program comprising instructions for configuring the XR device to perform the method of claim 11.

Images