Unsupervised event-based pupil tracking

Abstract

A method for tracking eye movements comprising: receiving, from an event camera, event data corresponding to an eye; encoding the event data into event frames and extracting encoded spatial features; providing the encoded features to an unsupervised neural network configured to temporally model the encoded features and to estimate a predicted 3D surface field of the eye, wherein the unsupervised neural network is trained using a loss function that includes a physics-based loss; determining a 3D rigid motion of the eye by solving a linear system derived from an event-based motion constraint using the predicted 3D surface field; computing the physics-based loss from a variational energy functional that enforces consistency between the predicted 3D surface field and the determined 3D rigid motion; and determining a pupil position based on the predicted 3D surface field and the determined 3D rigid motion.

Description

UNSUPERVISED EVENT-BASED PUPIL TRACKINGFIELD OF TECHNOLOGY

[0001] The present disclosure relates to tracking eye movements. Particular embodiments provide methods and systems for determining a pupil position.BACKGROUND

[0002] Eye tracking is increasingly crucial in a wide range of domains, including human-computer interaction, assistive technologies, behavioral neuroscience, and immersive systems such as augmented reality (AR) and virtual reality (VR). It provides a natural and intuitive interface by capturing subtle eye movements that reflect attention, intent, and cognitive states. In medical diagnostics, eye tracking has been used to assess neurological disorders, monitor fatigue, and aid in rehabilitation. As the demand for high precision, low-latency gaze tracking grows, so does the need for more robust and scalable technological solutions.

[0003] Traditional eye-tracking systems typically rely on frame-based cameras that operate at fixed frame rates. While effective in controlled environments, these systems pose challenges for real-time applications, often requiring high computational power to process dense image data and depending on large labeled datasets for supervised learning-based tracking models. In addition, frame-based systems struggle to capture rapid and minute eye movements due to motion blur and temporal aliasing, especially in low-light or high-speed scenarios.

[0004] Recently, event cameras, also known as neuromorphic cameras, have emerged as a promising alternative to conventional imaging sensors. Unlike traditional cameras, event cameras asynchronously record changes in pixel intensity with microsecond-level resolution. This allows them to capture motion with minimal latency and no redundant data, enabling faster and more energyefficient processing. Moreover, they perform exceptionally well in low-lightenvironments and during rapid motion transitions, making them particularly well suited for tracking fast and subtle eye movements such as microsaccades and blinks.

[0005] Deep learning approaches that use event-based inputs take advantage of the sparsity and high temporal resolution of event streams. These approaches are generally more robust than traditional methods, especially under challenging conditions. In particular, supervised learning models have dominated recent eyetracking research using event cameras. However, these models suffer from key limitations: they require large-scale annotated datasets with precise labels for pupil position, gaze direction, or eye motion patterns, labels that are both difficult and expensive to collect. For event data, labeling is further complicated by the asynchronous nature of the signal, making it hard to temporally align eye movement events with corresponding annotations.

[0006] Improvements in methods and systems for eye tracking are desirable.SUMMARY

[0007] According to one aspect of an embodiment, the present disclosure provides a method for tracking eye movements, the method comprising: receiving, from an event camera, event data corresponding to an eye; encoding the event data into event frames and extracting encoded spatial features; providing the encoded spatial features to an unsupervised neural network configured to temporally model the encoded features and to jointly estimate a predicted three-dimensional (3D) surface field of the eye; determining a 3D rigid motion of the eye by solving a linear system derived from an event-based motion constraint using the predicted 3D surface field; computing a physics-based loss from a variational energy functional derived from partial differential equations that enforce physical consistency between the predicted 3D surface field and the determined 3D rigid motion; training the unsupervised neural network using a loss function that includes the physics-based loss together with a variational loss comprising an event-based data term and a smoothness term; and determining apupil position based on the predicted 3D surface field and the determined 3D rigid motion.

[0008] According to another aspect of an embodiment, the present disclosure provides a system for tracking eye movements, the system comprising: an event camera for capturing event data corresponding to an eye; one or more processors operatively coupled to one or more memory devices, the one or more processors configured to: receive the event data from the event camera; encode the event data into event frames and extracting encoded spatial features; provide the encoded features to an unsupervised neural network configured to temporally model the encoded features and to estimate a predicted 3D surface field of the eye, wherein the unsupervised neural network is trained using a loss function that includes a physics-based loss; determine a 3D rigid motion of the eye by solving a linear system derived from an event-based motion constraint using the predicted 3D surface field; compute the physics-based loss from a variational energy functional that enforces consistency between the predicted 3D surface field and the determined 3D rigid motion; and determine a pupil position based on the predicted 3D surface field and the determined 3D rigid motion; and an output device for outputting the determined pupil position.

[0009] Further aspects of the present disclosure and details of example embodiments are set forth below.BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Embodiments of the present disclosure will now be described, by way of example only, with reference to the attached figures.

[0011] FIG. 1 schematically illustrates an example network architecture for an unsupervised neural network for eye-tracking in accordance with an aspect of an embodiment.

[0012] FIG. 2 is a flowchart illustrating an example method for tracking eyemovements in accordance with an aspect of an embodiment.

[0013] FIG. 3 schematically illustrates an example system for tracking eye movements in accordance with an aspect of an embodiment.

[0014] FIG. 4 is a graph of training curves showing the evolution of various terms of the loss function.DETAILED DESCRIPTION

[0015] For simplicity and clarity of illustration, reference numerals may be repeated among the figures to indicate corresponding or analogous elements. Numerous details are set forth to provide an understanding of the examples described herein. The examples may be practiced without these details. In other instances, well-known methods, procedures, and components are not described in detail to avoid obscuring the examples described. The description is not to be considered as limited to the scope of the examples described herein.Introduction

[0016] The present disclosure provides a novel unsupervised framework and related methods and systems (which is sometimes referred to below as the " PhysEye™ framework") for event-based eye tracking that integrates physics-informed constraints into a deep learning architecture. The inventors have determined that application of physics-based unsupervised learning methods is particularly valuable in event-based eye tracking, where labeled data is expensive or impractical to obtain, and physical dynamics such as 3D motion and shape evolution of the human eye are well understood. Unsupervised learning offers a compelling alternative to typical supervised neural networks used for eye tracking by learning patterns and dynamics directly from raw event streams without requiring ground truth annotations.

[0017] Recent work has shown that incorporating physics-based priors can enhance the interpretability, stability, and performance of learned representations, and physics-based unsupervised learning methods haveemerged as a powerful paradigm for solving partial differential equations (PDEs) without relying on explicit supervision These methods do not require labeled ground truth outputs; instead, they utilize the physical laws embedded in PDEs as implicit supervision. By incorporating the structure of governing equations directly into the learning process, physics-informed neural networks (PINNs) and their variants enable models to learn accurate, physically plausible solutions while generalizing better across unseen conditions. Classical PINNs typically optimize PDE residuals directly in the continuous domain, and have been applied to problems such as optical flow, motion estimation, or depth cues in neuromorphic vision. However, most existing PINN formulations rely on brightness-constancy assumptions or analytic priors that are insufficient for the complex, nonlinear, and rigid-body dynamics of ocular motion. The physics-informed components of the framework and related systems and methods disclosed herein differs from these existing methods in a number of ways, as described in detail below, including by deriving a discrete Euler-Lagrange update specific to ocular kinematics, which constrains the predicted motion and 3D surface to remain consistent with the biomechanical energy of the eye. This targeted form of regularization provides additional stability in low-event or highspeed regimes, where purely unsupervised constraints become underconstrained.

[0018] PINNs complement or even replace traditional numerical solvers (e.g., finite difference, finite element, or spectral methods), which can be computationally expensive and sensitive to noise or mesh discretization.ConvLSTM (Convolutional Long Short-Term Memory) networks

[0011] are especially well-suited for modeling nonlinear spatiotemporal dynamics governed by PDEs. Unlike traditional LSTMs that rely on fully connected operations and disregard spatial structure, ConvLSTMs employ convolutional filters within the LSTM cell gates to maintain spatial locality. This allows them to efficiently capture both spatial correlations (such as gradients of shape fields) and temporal dependencies (such as motion trajectories), making them ideal for PDE-based tasks. Furthermore, ConvLSTMs naturally support sequential prediction, allowingthem to model dynamic systems that evolve over time. They can implicitly learn adaptive time-stepping mechanisms, which is advantageous when solving PDEs over varying temporal resolutions. This provides a distinct advantage over traditional solvers, which often rely on fixed time steps and are susceptible to instability or discretization errors, particularly in highly nonlinear or multi-scale systems like eye motion dynamics.

[0019] The PhysEye framework leverages a Convolutional Long Short-Term Memory (ConvLSTM) network to model the spatiotemporal dynamics of eye motion and shape. Crucially, it embeds physical principles, specifically in preferred embodiments a variational energy functional derived from Euler-Lagrange partial differential equations (PDEs), as part of the learning objective. This enables the model to predict both 3D motion and 3D shape of the eye while ensuring that its outputs remain physically plausible. By minimizing the energy functional as a loss, the network is guided to learn dynamics that are consistent with real-world eye behavior. The PhysEye framework does not attempt to reconstruct a metrically accurate 3D surface from monocular event data.Instead, the PhysEye framework predicts a relative depth-like surface field, defined up to an arbitrary global scale and offset. This predicted 3D field represents geometric variations that are sufficient for enforcing the Euler-Lagrange physics prior and deriving consistent 2D motion constraints, thus avoiding classical scale ambiguity while retaining the benefits of a physics-informed geometric representation. The PhysEye framework addresses the limitations of supervised methods by removing the need for ground truth labels during training while achieving competitive and often superior performance, as discussed below, and demonstrates strong generalization across users, lighting conditions, and eye movement scenarios. Experimental data from a prototype embodiment of a system based on the PhysEye framework shows robust tracking in challenging cases such as fast saccades, involuntary blinks, and resting phases, where other models often falter due to sparse or ambiguous input signals. The design of the PhysEye framework also opens up possibilities for real-time, low-power, and label-free eye-tracking solutions suited for embedded platforms.

[0020] The following sections describe data representation and processing strategies utilized in the PhysEye framework, detail the formulation of the variational energy functional, explain how the ConvLSTM network is trained to approximate PDE solutions, and example methods and systems implementing the PhysEye framework. Experimental results on a public dataset and comparing performance of a prototype embodiment of the PhysEye framework to two supervised baselines, are also discussed below, demonstrating the effectiveness, robustness, and real-world applicability of the PhysEye framework.Data Representation

[0021] Event cameras, unlike traditional cameras, do not capture frames at fixed intervals. Instead, they capture asynchronous brightness changes at the pixel level. An event occurs whenever there is a significant change in the brightness at a given pixel, with high temporal resolution (on the order of microseconds), which makes them ideal for eye-tracking. An event stream is a collection of these discrete events over time. Each event is represented by eᵢ = (xᵢ, yᵢ, tᵢ, pᵢ), where xᵢ, yᵢ are image coordinates of the pixel where the event occurred, t, is the timestamp (in microseconds) when the event was triggered and Pi is the polarity (+1 or -1) indicating whether the brightness increased or decreased. The data is stored as a continuous event stream in memory or in a data file format, typically as a list of events or a sparse matrix. Event cameras output asynchronous events; therefore spatial derivatives cannot be directly estimated using the finite-difference method but instead rely on event data to accumulate events over a small temporal window [t, t+Δt] into an event image I(x, y), where:I(x, y) = ∑ pᵢ δ(x − xᵢ, y − yᵢ)

[0022] Then, finite differences can be applied to compute:

[0023] On the other hand, the temporal derivative from events is computed by taking the time difference between consecutive events at the same pixel:where tᵢ and tᵢ₋₁ are timestamps of the current and previous events at pixel (xᵢ, yᵢ).Event-Based Motion Contraint (EBMC)

[0024] The Event-Based Motion Constraint (EBMC) is a key concept in estimating optical flow from event cameras. Unlike traditional cameras that capture images at fixed intervals, event cameras detect changes in the scene asynchronously, producing a stream of events based on brightness changes. This constraint helps relate the spatial and temporal characteristics of these events to the motion of objects in the scene. The EBMC states that the change in brightness observed by an event camera is related to the motion of objects in the scene in terms of the temporal and spatial gradients of the brightness function I(x, y, t). Given a stream of events, the motion constraint can be expressed as∂I / ∂t + ∇I · (u, v) = 0where ∇I = (∂I / ∂x, ∂I / ∂y) are the spatial gradients of the brightness function, indicating how brightness changes in space, and (u, v) are the optical flow components representing the motion in the x and y directions.Determination of 3D Motion from Optical Flow

[0025] Deriving 3D motion from optical flow involves estimating the motion in depth (z-direction) along with the 2D motion on the image plane. Since optical flow provides only the 2D projection of 3D motion, additional assumptions or techniques are required to recover the complete 3D motion. With the pinhole camera model, the projection relationship between a 3D point P - (X, Y, Z) in the world and its projection in the image plane p - (x, y) is given by x - X / Z, y - Y / Z, where (X, Y, Z) are the 3D coordinates of a point relative to the camera and (x, y) are the normalized 2D coordinates on the image plane (assuming the camera focal length is 1. Let (u, v) be the 2D optical flow at the image coordinates (x, y). Using the pinhole camera projection model and differentiating the 2D image coordinates (x, y), the optical flow is related to the 3D motion as:> Ax _ XZ — XZ, > dy > YZ - YZ“ ~ ~dt ~ Z 'V~ ~dt ~ 2where Ẋ, Ẏ, Ż are the time derivatives of the 3D coordinates and describe 3D motion of the 3D point P.

[0026] One can describe the 3D motion of the pupil and eyelid with a rigid motion. Pupil movement primarily involves rotational motion of the eye, while the eyelid undergoes translational and rotational motion during blinking or gaze adjustments. For a rigid body, every point moves with respect to the camera frame, hence 3D rigid motion can be described by a translation T = (T₁, T₂, T₃) and a rotation ω = (ω₁, ω₂, ω₃). These motion components describe how a point in 3D space translates and rotates with respect to the camera frame. The instantaneous velocity of the 3D point due to both translation and rotation is:(Ẋ, Ẏ, Ż) = T + ω × Pdtwhere x denotes the cross-product. By substituting the velocities into the 2D flow equations above, the following flow equations are obtained:Z‘2(T2+ w3X - uYZ)Z - Y(T3+ UiF - w2X)(Equation (1))Variational Energy Functional

[0027] To infer the 3D shape of the eye and its motion from event-based data, the PhysEye framework is configured for the minimization of a variational energy functional. This energy functional incorporates both data fidelity terms, derived from the event-based motion constraint (EBMC), and regularization terms that ensure spatial smoothness of the reconstructed shape. The resulting optimization problem is governed by a set of partial differential equations (PDEs), derived from the Euler-Lagrange equations, which express the necessary conditions for minimizing the energy.

[0028] Let ξ ∈ {X, Y, Z} represent the 3D coordinates of points on the eye's surface in camera space. The total energy functional E is defined over the image domain and can be written as:E(X, Y, Z, T, ω) = ∫_Ω L(X, Y, Z, ∇X, ∇Y, ∇Z) dΩ

[0029] Here, L is the Lagrangian density that combines: a data term enforcingthe event-based brightness constancy constraint in 3D; and a smoothness term that penalizes sharp variations in the estimated 3D shape. The data term enforces that the predicted motion field aligns with observed brightness changes, and may be formulated as follows:£data — ( I.r fl + ly l' + If )"The smoothness term encourages continuity in the 3D shape surface, and may be formulated as follows:^smooth = λ( ||∇X||2+ ||∇Y||2+ ||∇Z||2)where ∇X, ∇Y, ∇Z are spatial derivatives of the 3D coordinates, and A is a regularization coefficient.Partial Differential Equations (PDEs) for 3D Shape

[0030] To find the shape fields ξ(x, y) that minimize this energy, the following Euler-Lagrange equations are derived for each variable:

[0031] This results in a set of coupled, nonlinear second-order PDEs that describe how the 3D shape fields evolve to satisfy the energy minimization condition. Solving these PDEs explicitly using numerical methods (e.g., finite differences or finite elements) can be computationally expensive and sensitive to noise, especially on sparse event data. Moreover, such solvers are difficult to integrate into end-to- end learning pipelines. Instead, the PhysEye framework utilizes a learning-based approach using a ConvLSTM neural network to approximate the temporal evolution of the 3D shape fields. Rather than solving the PDEs directly, the ConvLSTM learns to predict an updated state ^(t + At) from a sequence of past states {^(ti), ^(t2),..., ^(tn)}, effectively learning a mappingthat mimics the solution trajectory of the governing equations. To guide the learning process, the PhysEye framework utilizes a physics-informed update rule derived from the gradient descent form of the Euler-Lagrange equations:dt (Equation (2))where N(-) is a nonlinear operator representing the residual force. The ConvLSTM is trained to minimize the difference between its prediction and the expected physics-driven update, ensuring that learned dynamics are physically consistent. By using neural networks to approximate the PDE solutions the PhysEye framework combines the flexibility of learning-based models with the rigor of physics-based constraints, enabling accurate and interpretable 3D reconstructions of eye motion in an unsupervised manner.Determination of 3D Rigid Motion by Linear System

[0032] The brightness constraint can be rewritten as a linear system:Aη = b(Equation (3))where A ∈ Rn×6is the coefficient matrix that depends on the 3D shape coordinates, b =· · · ,−It(wn)) ∈ Rnis the known result vector, and η = (T,w) ∈ R6is the unknown vector to be determined. A is large, sparse, and noisy matrix, that depends on the 3D shape ξ. Due to the ill-conditioned nature of matrix A, solving the linear system of Equation (3) directly using traditional numerical methods such as Gaussian elimination or LU decomposition can be difficult, and computationally expensive. Instead, in currently preferred embodiments the PhysEye framework utilizes iterative least squares solvers, such as LSQR, which can handle sparse matrices efficiently without the need to store a large matrix in memory.Example Network Architecture

[0033] FIG. 1 schematically illustrates an example network architecture 100 for an unsupervised neural network for eye-tracking in accordance with an aspect of an embodiment. The network architecture 100 is designed to reflect the structure of spatiotemporal PDEs while being compatible with deep learning optimization, and integrates spatial feature encoding, temporal modeling, and physics-informed constraints. The network architecture 100 comprises input layers 110, an encoder 120, an unsupervised neural network 130 (in the form of a ConvLSTM module), a decoder 140 and a loss layer 160 / 170 / 190.

[0034] The input layers 110 receive the initial conditions (e.g. from an event accumulation block as described below) in the form of spatial grids for the 3D shape fields X, Y, Z and a 6-dimensional rigid motion vector η = (T,ω), where T denotes translation and ω rotation. These fields define the current physical state of the eye and are provided over a sequence of time steps.

[0035] The encoder 120, which may for example comprise a convolutional neural network (CNN) encoder, encodes the spatial inputs into a compact latent representation. The encoder 120 consists of several layers of convolution, batch normalization, and rectified linear unit (ReLU) activations, gradually reducing spatial resolution while increasing channel depth. This stage extracts multi-scale spatial features that capture local geometry and shape gradients to generate encoded spatial features.

[0036] The unsupervised neural network / ConvLSTM module 130 comprises a stack of ConvLSTM2D layers which receive the encoded spatial features. Each ConvLSTM2D layer is equipped with convolutional gates for input, forget, and output mechanisms. These ConvLSTM2D layers maintain both temporal memory and spatial continuity, enabling the network 130 to learn the dynamics of eye shape and motion over time. Stacking multiple ConvLSTM layers deepens the capacity of network 130 to model complex, nonlinear spatiotemporaldependencies.

[0037] After temporal modeling by the ConvLSTM module 130, the decoder 140, which may for example comprise a convolutional neural network (CNN) decoder, upsamples latent features to reconstruct predicted future states " X, " Y, " Z (e.g. a predicted 3D surface field 150) and a motion vector "q (e.g. a determined 3D rigid motion) at the next time step. The decoder 140 typically mirrors the encoder 120 with transposed convolutions or bilinear upsampling followed by convolutional refinements.

[0038] The output of the decoder 150 is fed into a physics-based loss module, which incorporates multiple components including a combiner 160 that receives the output of the decoder 120 and the event frames from the input layers 110, a linear solver block 170 that determines a rigid 3D motion, and a physics-based loss determination block 190, to ensure that the network's predictions conform to the governing PDEs and physical constraints of eye motion. This architecture 100 facilitates an end-to-end trainable pipeline that learns to approximate the temporal evolution of physical fields without any labeled data, guided solely by the embedded PDEs.

[0039] To ensure that the predictions from architecture 100 remain faithful to the underlying physical laws, the physics-based loss determination block 190 integrates a physics-informed loss function based on the update rule derived from the Euler-Lagrange PDEs. Rather than explicitly solving the PDE defined in Equation (2), the network 130 is trained to approximate its solution over time through iterative refinement. At each time step, the network 130 computes a residual term for each spatial fielde {X, Y, Z} as follows:Residualξ= ξ(t+1)− (ξ(t)− α · 풩(ξ(t), ∇ξ(t))) (4)(Equation (4))

[0040] In Equation (4), N (■) is a nonlinear operator derived from the variational energy functional described above, representing the physical evolution of the system, and a is a scalar step size. This formulation mimics the structure of gradient descent in PDE solvers, where the field is updated based on the energy functional's gradient. The residual represents the deviation of the network's predicted next statefrom the expected physics-consistent update. The network architecture 100 penalizes this discrepancy with the following residual loss:ℒphys= ∑ ||Residualξ||2(5)ξ∈{X, Y, Z}(Equation (5))

[0041] This term encourages predictions from the network 130 to remain close to the trajectory defined by the underlying physics. Additionally, the network 130 utilizes a variational physics loss directly from the energy functional E( ■ ), which enforces brightness constancy and smoothness constraints as discussed above. The total training objective of network 130 is a weighted sum of the physics loss and the residual consistency loss:^ℒtotal= ℒdata+ ℒsmooth+ β · ℒphys(6)where / ? is a hyperparameter that balances the influence of residual minimization relative to the overall energy minimization.

[0042] By jointly optimizing both terms, the network 130 is guided not only to fit the observed dynamics but to do so in a physically plausible manner. This physics-informed training approach provides a robust inductive bias that improves generalization, enables stable long-term predictions, and reducesoverfitting to data artifacts. It is particularly advantageous in sparse or noisy settings typical of event camera data, where purely data-driven models often struggle.

[0043] Because the total loss combines heterogeneous terms, experiments conducted with a prototype embodiment explicitly monitored each loss component during training to ensure that no single term dominates the gradients. Experimental results indicate that no gradient collapse was observed. The PhysEye framework utilizes two mechanisms to prevent gradient collapse: (i) each loss is normalized by the number of active pixels in the event frame, which prevents scale explosion in low-activity segments; and (ii) the physics residual term is weighted by a bounded coefficient p e [0.1, 1], which empirically prevents the variational term from overshadowing the data-driven component. FIG. 4. shows the evolution of the three loss components LPhys410, Ldata 420 and Lsmooth 430) during training of an example prototype embodiment. Ldata 420 corresponds to an event-based data fidelity term and naturally exhibits higher variance during training, particularly in early epochs, and Lsmooth 430 exhibit smooth, monotonic decay, reflecting the stability of the photometric and regularization terms. In contrast, the physics-based loss LPhysdecreases more irregularly due to its sensitivity to temporal derivatives and higher-order residuals. Despite these heterogeneous behaviors, the combined optimization remains stable, confirming that the PhysEye framework can effectively balance data fidelity, spatial smoothness, and physics consistency throughout training. Training Algorithm Overview

[0044] In preferred embodiments, the training process for the PhysEye framework combines a spatiotemporal deep ConvLSTM network with physics-informed supervision via PDE-constrained losses. The network predicts the future state of the 3D eye shape and motion using event-based inputs, and updates its parameters by minimizing a combination of variational and residual losses. Anexample of an overall training procedure is described by the following algorithm:Algorithm 1 Physics-Informed Training of PhysEyeInput: Event frames {Et}, initial 3D shape fields {Xt, Yt, Zt}, motion vector ηt= (Tt, ωt), learning rate α, residual weight βOutput: Trained network parameters 01 Initialize ConvLSTM network parameters θ for each training sequence do2 for each time step t do3 Encode Etusing CNN encoder to obtain latent featuresPass features through stacked ConvLSTM layersDecode predicted next-step lields: A'(_i. l’ / +i. Z(_iCompute the 3D motion vector by solving the linear system A = η̂t+1bCompute variational loss ℒdata+ℒsmoothusing equations (2) and (3)Compute residual loss using PDE-deriscd energy= ||ξ̂t+1− (ξt− α · 풩(ξt, ∇ξt))||2«e(.\ )Compute total loss:Update parameters: θ ← θ −4 return θ

[0045] FIG. 2 is a flowchart illustrating an example method 200 for tracking eye movements in accordance with an aspect of an embodiment. Method 200 may be carried out by one or more processors operatively coupled with one or more memory devices configured to receive event data from an event camera, process the event data in a neural network configured according to the PhysEye framework described above, and determine a pupil position.

[0046] At step 202, event data corresponding to an eye is received from an event camera. At step 204, the event data is encoded into event frames and spatial features are extracted. At step 206, the encoded spatial features are provided to an unsupervised neural network as described above to estimate a predicted 3D surface field of the eye. At step 208, a 3D rigid motion of the eye is determined by solving a linear system derived from an Event-Based Motion Constraint. At step 210, a physics-based loss is determined based on a variational energy functional that enforces consistency between the predicted 3D surface field and the determined 3D rigid motion. At step 212, a pupil position isdetermined based on the predicted 3D surface field and the determined 3D rigid motion. The determined pupil position may be utilized in various contexts including human-computer interaction, assistive technologies, behavioral neuroscience, and immersive systems such as augmented reality (AR) and virtual reality (VR).

[0047] FIG. 3 schematically illustrates an example system 300 for tracking eye movements in accordance with an aspect of an embodiment. System 300 comprises an event camera 310 which is oriented toward a subject's eye and configured to asynchronously detect brightness changes at the pixel level. The event camera 310 outputs a stream of events, each event being represented as a tuple (x, y, t, p), where (x, y) denote pixel coordinates, t denotes a timestamp, and p denotes a polarity indicating an increase or decrease in brightness.

[0048] System 300 may also optionally comprise an illumination source 312, such for example as a near-infrared (NIR) light-emitting diode or similar light source. The illumination source 312 may be used to illuminate the eye, to improve contrast between pupil and iris regions while avoiding the use of structured light or active projection.System 300 comprises one or more processing units, such as a graphics processing unit (GPU), an embedded processor, or a system-on-chip (SoC), and is configured to execute event-based eye-tracking operations, including: accumulation of asynchronous events into event frames by an event accumulation block 320; spatial feature extraction by a convolutional neural network (CNN) encoder 330; temporal modeling using a Convolutional Long Short-Term Memory (ConvLSTM) network 340; 3D surface field estimation by a CNN decoder 350; solving a linear system to estimate rigid motion parameters of the eye by a linear solver block 360 computation of a physics-based loss derived from a variational energy functional by physics-based loss block 370. System 300 also comprises one or more memory components configured to store event streams, intermediate feature representations, and predicted three-dimensionalsurface fields generated during processing. System 300 also comprises a pupil / gaze output interface 380 configured to provide eye-tracking results, including pupil position, gaze direction, or other eye-motion parameters, for use in applications such as augmented and virtual reality (AR / VR), human-computer interaction (HCI), or medical diagnostics.Experimental Results

[0049] Experiments were conducted using a prototype embodiment of a system based on the PhysEye architecture, in conjunction with the EET+ dataset as described by Wang et al in Event-based eye tracking, ais 2024 challenge survey, in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 5810-5825, 2024 (" Wang et al."). The dataset includes 13 participants, each participating in 2 to 6 recording sessions. During these sessions, subjects performed five types of activities: random movements, saccades, reading text, smooth pursuit, and blinking. Ground truth labels are recorded at 100 Hz and include three components for each frame (x, y, close). The EET+ dataset was divided into training, validation, and test subsets, with the training and validation splits containing annotated data, while the test set was provided without annotations. Evaluation of the prototype embodiment used the labeled portion of the EET+ dataset, as no other public event-based eye-tracking dataset provides compatible annotations. Specifically, all available annotated sessions were used for training and validation, holding out 12 unannotated sessions exclusively for testing. Since the test data lacks ground truth labels, it is intended solely for qualitative evaluation or leaderboard-based benchmarking, and cannot be used for quantitative error metrics. As a result, all quantitative results discussed herein are reported on the validation set using the provided annotations. The x and y coordinates represent the pupil center, while the binary close value indicates whether the eye is blinking (1 for closed, 0 for open).Unless otherwise stated, p = 0.5, which offered the most stable balance between the data term and the physics-informed term. To validate this choice, a small sensitivity analysis was performed over p e {0.1, 0.5, 1.0}.

[0050] As shown in Table 2, performance varies smoothly with / ?, confirming that the PhysEye architecture is not overly sensitive to this hyperparameter. The value / ? = 0.5 provides the best tradeoff between Euclidean error and threshold accuracy. For the training split, labels are initially available at 100 Hz but are downsampled to 20 Hz, to be consistent with the requirements in Wang et al.. The close label is not used by default. A total of 12 recordings were reserved for testing, while the remaining recordings were used for training and validation. The evaluation was assessed by the Euclidean distance between the predicted pupil center and the ground truth. The metrics pn(where n = 5, 10, 15) represent the percentage of predicted pupil positions that lie within n pixels of the ground truth position. They are formally defined as:z=lwhere ^piis the predicted pupil center at frame i, piis the ground truth pupil center at frame i, N is the total number of frames, 핀(·) is the indicator function, which equals 1 if the condition inside is true and 0 otherwise, and || · ||2denotes the Euclidean distance.

[0051] This section, and the tables below, provide an assessment of the training performance of a prototype embodiment of an unsupervised network based on the PhysEye architecture and compares it with a leading state-of-the-art supervised method. The comparison includes a standard baseline using CNN and a bidirectional Gated Recurrent Unit (GRU) and the Mambapupil approach (as described by Wang et al. in MambaPupil: Bidirectional Selective Recurrent model for Event-based Eye tracking in 2024 IEEE / CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 5762-5770, Los Alamitos,CA, USA, 2024. IEEE Computer Society) using both GRU and a Linear Time-Varying State Space Module (LTV-SSM). Table 3 shows that the PhysEye framework performs competitively with existing supervised techniques, achieving the highest improvements in ps and showing strong competitive improvements for p10and p15, while reducing the error distance to 2.5 pixels.

[0052] Qualitative comparisons of the PhysEye framework to state-of-the-art supervised models have shown robust performance in scenarios where eyetracking is difficult for many existing techniques. Four challenging scenarios were selected for evaluation: a) saccades, which are eye movement oscillations with subtle intensity changes caused by light reflections, shadows, or textures on the eye surface, creating a significant disadvantage for supervised networks that rely on temporal correlations; b) blinking and c) rapid eye movement (" Fastmove"), both of which generate numerous interfering events, leading to shifts and jitters in the network's predictions and d) eye rest, where event cameras struggle to capture static objects, offering limited information to infer the pupil position when the eyes are still. These scenarios often result in suboptimal predictions by most existing supervised methods. The visual results show that the PhysEye framework achieves the best performance on the strict ps metric and obtains the lowest Euclidean prediction error, while remaining competitive on p10andp15. The results highlight that the physics-informed constraint improves accuracy in the high-precision regime. To further analyze the contribution of each component, Table 1 reports an ablation study in which the variational term and ConvLSTM temporal module are individually removed. Both components provide measurable improvements, confirming their complementary roles.Table 1. Ablation of core components.Model Variant Euclidean ↓ p5 ↑ Full model (ours) 2.18 74.6 w / o physics term 2.41 69.3w / o ConvLSTM 2.52 66.7Table 2. Sensitivity of the method to the hyperparameter β.β Euclidean ↓ p5 ↑ p10 ↑ 0.1 2.31 71.2 89.4 0.5 2.18 74.6 90.11.0 2.26 72.9 89.7Method p5 p10 p15 peCNN-GRU

[0012] 0.615 0.844 0.931 5.90 0.0737 MambaPupil

[0013] 0.937 0.984 0.990 2.03 0.02870.0258PhysEye (Ours) 0.987 0.977 0.978 1.87Table 3. Training performance and comparison with state-of-the-arts: pnrepresents the prediction accuracy within n pixels, with higher values indicating better performance. Meanwhile. de¬ notes the average Euclidean distance of predictions, where lower values are desirable.

[0053] All experiments were executed on a single NVIDIA GeForce RTX 2080 Ti GPU. For the complete PhysEye pipeline, including the event encoder, the CNN backbone, the ConvLSTM temporal module, the decoder, and the evaluation of the physics residuals, the prototype system achieved an end-to-end inference speed of 126 FPS (7.9 ms per frame). This number reflects the real operational cost of the physics-informed formulation. For completeness, the forward-only latency of the backbone (i.e., CNN + ConvLSTM + decoder without physics residual computation) runs at 325 FPS (3.1 ms per frame), which illustrates that the physics constraints introduce only a moderate overhead while significantly improving the stability and physical plausibility of the predictions.

[0054] In contrast to supervised approaches that require large annotated datasets, the PhysEye framework embeds a variational energy derived from Euler-Lagrange PDEs directly into the loss function, enabling the joint estimation of 2D pupil motion, 3D shape, and 3D rigid motion without ground-truth labels. This physics-informed formulation improves data efficiency and enforces predictions consistent with ocular biomechanics. Experiments on the EET+ dataset show that systems and methods based on the PhysEye framework perform competitively with state-of-the-art supervised methods across challenging conditions such as saccades, blinks, and stillness, while exhibiting strong generalization across subjects and environments. Overall, the PhysEye framework advances the development of real-time, label-free, and physically grounded eye-tracking systems for AR / VR, medical diagnostics, and neuromorphic vision applications.

[0055] The scope of the claims should not be limited by the preferred embodiments set forth in the examples, but should be given the broadest interpretation consistent with the description as a whole.

Claims

What is claimed is:Claims1. A method for tracking eye movements, the method comprising:receiving, from an event camera, event data corresponding to an eye; encoding the event data into event frames and extracting encoded spatial features;providing the encoded spatial features to an unsupervised neural network configured to temporally model the encoded features and to jointly estimate a predicted three dimensional (3D) surface field of the eye, wherein the unsupervised neural network is trained using a loss function that includes a physics-based loss;determining a 3D rigid motion of the eye by solving a linear system derived from an event-based motion constraint using the predicted 3D surface field;computing the physics-based loss from a variational energy functional that enforces consistency between the predicted 3D surface field and the determined 3D rigid motion; anddetermining a pupil position based on the predicted 3D surface field and the determined 3D rigid motion.

2. The method of claim 1 wherein the unsupervised neural network comprises a ConvLSTM neural network.

3. The method of claim 2 wherein the ConvLSTM neural network comprises a plurality of ConvLSTM2D layers arranged in a stack, each ConvLSTM2D layer having an input gate, a forget gate, and an output gate.

4. The method of any one of claims 1 to 3 wherein encoding the event data comprises encoding the event data with a convolutional neural network (CNN) encoder comprising a plurality of layers configured to gradually reduce spatial resolution while increasing channel depth.

5. The method of any one of claims 1 to 4 comprising decoding latent features using a convolutional neural network (CNN) decoder to generate the predicted 3D surface field of the eye.

6. The method of any one of claims 1 to 5 wherein the loss function comprises a weighted sum of the physics-based loss and a variational loss.

7. The method of claim 6 wherein the variational loss comprises a data term derived from an event-based motion constraint, and a smoothness term.

8. The method of any one of claims 1 to 7 wherein the variational energy functional is derived from a set of Euler-Lagrange partial differential equations.

9. The method of claim 8 wherein the unsupervised neural network is configured to approximate a temporal evolution of 3D surface fields governed by the set of Euler-Lagrange partial differential equations.

10. The method of any one of claims 1 to 9 comprising determining a center position of the pupil directly from the predicted 3D surface field and the determined 3D rigid motion.

11. A system for tracking eye movements, the system comprising:an event camera for capturing event data corresponding to an eye; one or more processors operatively coupled to one or more memory devices, the one or more processors configured to:receive the event data from the event camera;encode the event data into event frames and extracting encoded spatial features;provide the encoded features to an unsupervised neural network configured to temporally model the encoded features and to estimate a predicted 3D surface field of the eye, wherein the unsupervised neural network is trained using a loss function that includes a physics-based loss;determine a 3D rigid motion of the eye by solving a linear system derived from an event-based motion constraint using the predicted 3D surface field;compute the physics-based loss from a variational energy functional that enforces consistency between the predicted 3D surface field and the determined 3D rigid motion; anddetermine a pupil position based on the predicted 3D surface field and the determined 3D rigid motion; andan output device for outputting the determined pupil position.

Images