Retinal image generation based on eye motion modelling-based capture parameters

Abstract

Introduced here are approaches to generating an image with a fundus camera based on parameters determined through modeling of eye motion. The method captures a series of images to generate pupil position data over time. Using this data, the method determines a model for estimating pupil positions and predicts a time associated with a minimum pupil velocity. The method can estimate the pupil position at this time and generate a fundus camera configuration based on the position. The method can transmit a command to capture an image of the eye using the fundus camera configuration. By implementing these approaches, an imaging system can improve the quality of retinal imaging for patients with eye motion.

Description

PATENTAtorney Docket No. 124824.8143.WO01RETINAL IMAGE GENERATION BASED ON EYE MOTION MODELLING-BASED CAPTURE PARAMETERSCROSS-REFERENCE TO RELATED APPLICATION(S)

[0001] This application claims priority to U.S. Provisional Application No. 63 / 737,057, titled "RETINAL IMAGE GENERATION BASED ON EYE MOTION MODELLING-BASED CAPTURE PARAMETERS," and filed on December 20, 2024, which is incorporated by reference herein in its entirety.TECHNICAL FIELD

[0002] Various embodiments concern computer programs and associated computer-implemented techniques for generating retinal images of an eye by an imaging system.BACKGROUND

[0003] The process by which visual representations of a human body are captured is referred to as "medical imaging" or "biological imaging." Generally, medical imaging seeks to reveal internal structures that are hidden by the skin, bones, or organs in order to detect the presence of a disease. For example, a series of digital images corresponding to different aspects of the anatomy of the human body may make it possible to more readily identify abnormalities that are indicative of a disease.

[0004] A variety of different technologies may be used to capture these digital images. Examples of such technologies include x-ray, magnetic resonance imaging (MRI), ultrasonography or ultrasound, endoscopy, microscopy, elastography, tactile imaging, thermography, computed tomography (CT), fluoroscopy, angiography, mammography, positron emission tomography (PET), single photon emission computed tomography (SPECT), and the like. The quality of medical images, particularly in fields like ophthalmology, is dependent on the relationship between subject movement and image capture. Even slight movements can introduce blur or distortion, potentially obscuring important diagnostic details.

[0005] One area in medical imaging is retinal imaging. Retinal imaging devices, such as fundus cameras (also called "retinal cameras"), capture a digital image of thePATENT Atorney Docket No. 124824.8143.WO01 fundus to document the retina, which is the neurosensory tissue in the eye that translates optical images into the electrical impulses that can be understood by the brain. The fundus can include the retina, optic disc, macula, fovea, and posterior pole. Fundus cameras are designed to provide an upright, magnified view of the fundus. Figure 1 illustrates an example of a retinal camera in operation. Generally, an individual (also called a "subject" or "patient") will sit at the retinal camera with their chin set within a chin rest and their forehead pressed against a bar. An operator may be responsible for visually aligning the retinal camera and then pressing a shutter release that causes a digital image of the retina to be generated.

[0006] As shown in Figure 1 , light may be focused via a series of lenses through a masked aperture to form an annulus that passes through an objective lens onto the retina. The illuminating light rays are generated by one or more light sources, each of which is electrically coupled to a power source. When the objective lens is aligned with the retina, light reflected by the retina will pass through the un-illuminated hole in the annulus formed by the masked aperture. Normally, alignment is facilitated by having the subject place their eye proximate to a first eyepiece (also called the "subject eyepiece" or "patient eyepiece"). Those skilled in the art will recognize that the optics of the retinal camera are generally similar to those of an indirect ophthalmoscope in that the illuminating light rays entering the eye and the imaging light rays exiting the eye follow dissimilar paths.

[0007] The imaging light rays exiting the eye may initially be guided toward a second eyepiece (also called the "operator eyepiece") that is used by the operator to assist in aligning and / or focusing the illuminating light rays. When the operator presses the shutter release, a first mirror can interrupt the path of the illuminating light rays and a second mirror can fall in front of the operator eyepiece, which causes the imaging light rays to be redirected onto a capturing medium. Examples of capturing mediums include film, digital charge-coupled devices ("CCDs"), and complementary metal-oxide- sem iconductors ("CMOSs").

[0008] Healthcare professionals, such as optometrists, ophthalmologists, and orthoptists, may use the digital images generated by a retinal camera to determine whether diseases are present and / or monitor progression of diseases. For instance, these digital images may be used to document indicators of diabetes, age-macularPATENTAtorney Docket No. 124824.8143.WO01 degeneration ("AMD"), glaucoma, and the like. Accordingly, it is important that the digital images be high quality, so that these healthcare professionals can readily identify pathological features, as well as distinguish pathological features from non-pathological features (also called "artifacts").

[0009] However, achieving perfect stillness in biological subjects, particularly in organs that move rapidly like the eye, is generally impractical, if not impossible. The eye, for instance, tends to be in constant motion due to involuntary movements such as microsaccades, drift, and tremor. The movements, which may be required for normal visual function, are typically imperceptible to the individual but can blur and / or distort images generated by fundus cameras, thus significantly degrading the quality of the images. Degraded images may potentially lead to missed diagnoses or the need for repeated imaging sessions.BRIEF DESCRIPTION OF THE DRAWINGS

[0010] This patent or application contains at least one drawing executed in color. Copies of this patent or application publication with color drawings will be provided by the Office upon request and payment of the necessary fee.

[0011] Figure 1 illustrates an example of a retinal camera in operation.

[0012] Figure 2 illustrates an example environment of an imaging system (here, a fundus camera) for generating a medical image (here, a retinal image).

[0013] Figure 3 illustrates a network environment that includes the imaging system of Figure 2.

[0014] Figure 4 depicts an example of a communication environment that includes the imaging system of Figure 2 configured to acquire data from one or more sources.

[0015] Figure 5A illustrates how, in some embodiments, a pair of secondary cameras are circumferentially arranged along opposing sides of the lens through which light is collected by a fundus camera.

[0016] Figures 5B-C include examples of digital images that were generated by secondary cameras circumferentially arranged along opposing sides of the lens.

[0017] Figure 6 illustrates an example environment for determining an angle of the captured images from the optic axis of the imaging system of Figure 2.PATENTAtorney Docket No. 124824.8143.WO01

[0018] Figure 7 illustrates an example graphical chart for determining a model for estimating the positions of the pupil over time.

[0019] Figure 8 illustrates an example graphical chart for establishing values for parameters of a modelled eye motion.

[0020] Figure 9 depicts a flow diagram of a process for generating, with the imaging system of Figure 2, an image based on parameters determined through the modelled eye motion.

[0021] Figure 10 illustrates an example graphical chart for establishing a trigger time for the imaging system of Figure 2 based on the modelled eye motion.

[0022] Figure 11 illustrates an example graphical chart for a delay between a capture time and the trigger time for the imaging system of Figure 2 based on the modelled eye motion.

[0023] Figure 12 depicts a flow diagram of a process for generating, with the imaging system of Figure 2, an image based on a latency value associated with a delay between triggering the imaging system and image capture by the imaging system.

[0024] Figure 13 illustrates an example environment for determining an illumination pattern of the imaging system of Figure 2 based on a predicted location of the eye.

[0025] Figure 14 depicts a flow diagram of a process for generating, with the imaging system of Figure 2, an image using a predicted illumination pattern based on the modelled eye motion.

[0026] Figure 15 is a block diagram illustrating an example of a processing system in which at least some operations described herein can be implemented.

[0027] Various features of the technologies described herein will become more apparent to those skilled in the art from a study of the Detailed Description in conjunction with the drawings. Embodiments are illustrated by way of example and not limitation in the drawings. While the drawings depict various embodiments for the purpose of illustration, those skilled in the art will recognize that alternative embodiments may be employed without departing from the principles of the technologies. Accordingly, whilePATENTAtorney Docket No. 124824.8143.WO01 specific embodiments are shown in the drawings, the technology is amenable to various modifications.DETAILED DESCRIPTION

[0028] Retinal imaging has historically been an effective means of detecting a variety of eye conditions. For example, retinal imaging has been shown to be useful for early detection of diabetic retinopathy and age-related macular degeneration, among other conditions. While advancements in imaging technology have significantly improved the quality of retinal images, a higher resolution often means a greater susceptibility to minute movements, such as involuntary eye movements. Even microsaccades, which are rapid, small-scale eye movements, can significantly degrade image quality. These movements are typically imperceptible to the patient but can result in blurred images that lack the clarity and detail needed for accurate diagnosis. Degradation in image quality can be problematic for humans (doctors) who grade the blurred images and / or use the blurred images for diagnosis.

[0029] Further, the degradation in image quality can be particularly problematic when applying a model, such as a classification model, to predict a presence or stage of an eye-related disease. Models are typically trained to examine the underlying pixel data on a per-pixel basis to identify abnormalities indicative of disease presence or state. If a movement causes a decrease in quality, the model may output a false positive because an aberration from the movement appears to be an indicator of disease. Conversely, the model may output a false negative because an aberration causes an actual indicator of disease to be missed or misunderstood.

[0030] The degradation in image quality further affects the model during the training stage. During training, the model typically learns to recognize patterns and features associated with different disease states of a large dataset of images. If the training images are affected by inconsistent and unpredictable eye movements, the model may learn to associate motion artifacts with disease indicators, leading to incorrect conclusions. For instance, if some training images contain motion blur that coincidentally aligns with disease features, the model may incorrectly learn that such blur is a sign of disease. This can result in a model that is overly sensitive to motion artifacts, increasing the likelihood of false positives during inference.PATENTAtorney Docket No. 124824.8143.WO01

[0031] Moreover, certain pathological conditions exacerbate the difficulty of obtaining high-quality retinal images. For instance, nystagmus, a condition characterized by uncontrolled and repetitive movements of the eyes, can substantially affect the stability of retinal images. Patients with nystagmus experience continuous eye movements that can blur the images and make it challenging to capture a stable and clear view of the retina.

[0032] Another problem faced by retinal imaging systems (e.g., fundus cameras, retinal cameras) is the variability in eye anatomy and pathology among patients. Different eye conditions, such as cataracts or corneal abnormalities, may affect the quality of the images obtained. Cataracts, for instance, cause clouding of the lens, which can obstruct the passage of light and result in blurred or obscured images. Corneal abnormalities, such as keratoconus or scarring, can distort the light entering the eye, leading to image artifacts and reduced image quality.

[0033] Additionally, pupil size and reactivity can vary greatly between individuals, which may impact the amount of light entering the eye and, consequently, the quality of the retinal image. Factors such as age, ambient lighting, medications, and underlying health conditions can influence pupil dilation and constriction. A smaller pupil size may limit the amount of light entering the eye, resulting in darker and less detailed images. Conversely, a larger pupil may allow more light but can also introduce glare and reflections that degrade image quality. This variability makes it challenging to develop a one-size-fits-all approach to configuring the fundus camera. The variability often requires multiple attempts to capture a usable image, thereby increasing the time and cost associated with retinal examinations.

[0034] To address the challenges of eye movement during retinal imaging, a system - also called an "imaging system" - may generate a model that can be used for predicting eye motion to determine a timing and configuration of image capture. The imaging system can include one or more computer systems, camera systems (e.g., retinal cameras, fundus cameras, secondary cameras, pupil cameras), and / or illumination elements. The imaging system can capture a series of images and, using the images, generate pupil position data over time. Using the pupil position data, the imaging system can determine a model for estimating pupil positions and predict a time associated with a minimum pupil velocity. For example, the model can be determinedPATENTAtorney Docket No. 124824.8143.WO01 based on a periodicity of the periodic motion, an amplitude of the periodic motion, a temporal offset, a pattern of the periodic motion, and so forth. The imaging system can estimate the pupil position at this time and generate a fundus camera configuration (e.g., of a fundus camera within the imaging system) based on the position.

[0035] In some embodiments, the imaging system determines a latency value associated with a delay between triggering the fundus camera and image capture by the fundus camera. The imaging system determines a second time, subsequent to the first time, in accordance with the latency value and the first time. The imaging system can transmit a command to capture an image of the eye using the fundus camera configuration. By predicting periods of minimal pupil velocity (e.g., minimum velocity, zero velocity, near zero velocity), the system can trigger the capture of retinal images at particular times, thereby reducing motion artifacts and improving image clarity.

[0036] The imaging system can further improve the quality of retinal images by adjusting the illumination / brightness (e.g., using a lookup table) based on the predicted position of the pupil at the time of image capture, pre-existing conditions, and so forth. For example, the imaging system can determine a target angular region for illumination and select a subset of illuminants that can best light this region. By controlling the brightness and distribution of light, the imaging system ensures that the retina is evenly illuminated, reducing shadows and improving the quality of the captured image.

[0037] The imaging system that is disclosed herein is able to mitigate the impact of involuntary eye movements that may cause blurring and distortion in conventional imaging methods. By timing the image capture to coincide with periods of minimal eye motion, the system can reduce the effects of blurring and distortion, resulting in higher quality images. Further, the adaptive illumination compensates for variations in eye anatomy and pathology among patients. By adjusting the illumination based on the predicted pupil position, the system can individually tailor lighting conditions for each individual patient, which may reduce the need for multiple image capture attempts. In addition, the imaging system reduces the reliance on operator skill and judgment, which can lead to more standardized imaging results across different operators and clinical settings, potentially improving diagnostic accuracy and patient care.

[0038] Embodiments may be described with reference to particular medical conditions, imaging devices, computer programs, etc. However, those skilled in the artPATENTAtorney Docket No. 124824.8143.WO01 will recognize that the features are similarly applicable to other medical conditions, imaging devices, computer programs, etc. For example, although embodiments may be described in the context of fundus cameras that generate digital images of retinas over the course of diagnostic sessions, the relevant features may be similarly applicable to imaging devices designed to generate digital images of other anatomical regions of the human body.

[0039] Moreover, while embodiments may be described in the context of computer-executable instructions for the purpose of illustration, aspects of the technology can be implemented via hardware, firmware, or software. As an example, embodiments may include a machine-readable medium having instructions that may be used to program a computing device to perform a process for generating, with a fundus camera having a lens through which light is reflected by a retina of an eye and / or a secondary camera that is located proximate to the lens, an image based on parameters determined through modeling of eye motion.Terminology

[0040] References in this description to "an embodiment" or "one embodiment" means that the particular feature, function, structure, or characteristic being described is included in at least one embodiment. Occurrences of such phrases do not necessarily refer to the same embodiment, nor are they necessarily referring to alternative embodiments that are mutually exclusive of one another.

[0041] Unless the context clearly requires otherwise, the terms "comprise," "comprising," and "comprised of" are to be construed in an inclusive sense rather than an exclusive or exhaustive sense (i.e., in the sense of "including but not limited to"). The term "based on" is also to be construed in an inclusive sense rather than an exclusive or exhaustive sense. Thus, unless otherwise noted, the term "based on" is intended to mean "based at least in part on."

[0042] The terms "connected," "coupled," or any variant thereof is intended to include any connection or coupling between two or more elements, either direct or indirect. The connection / coupling can be physical, logical, or a combination thereof. For example, objects may be electrically or communicatively coupled to one another despite not sharing a physical connection.PATENTAtorney Docket No. 124824.8143.WO01

[0043] The term "module" refers broadly to software components, firmware components, and / or hardware components. Modules are typically functional components that generate output(s) based on specified input(s). A computer program may include one or more modules. Thus, a computer program may include multiple modules responsible for completing different tasks or a single module responsible for completing all tasks.

[0044] When used in reference to a list of multiple items, the word "or" is intended to cover all of the following interpretations: any of the items in the list, all of the items in the list, and any combination of items in the list.

[0045] The sequences of steps performed in any of the processes described here are exemplary. However, unless contrary to physical possibility, the steps may be performed in various sequences and combinations. For example, steps could be added to, or removed from, the processes described here. Similarly, steps could be replaced or reordered. Thus, descriptions of any processes are intended to be open-ended.Overview of the Imaging System

[0046] Figure 2 illustrates an example environment 200 of an imaging system 204 (here, a fundus camera) for generating a medical image (here, a retinal image). Environment 200 includes eye 202, imaging system 204 (which can include pupil camera 206, fundus camera 208, and so forth), imaging system's 204 optic axis 210, modeled eye motion 212, slow motion image 214, and fast motion image 216. Embodiments of the environment 200 can include different and / or additional components or can be connected in different ways.

[0047] The eye 202 can refer to the human eye being observed and imaged by the imaging system 204. The eye 202 includes anatomical features such as the cornea, lens, retina, optic nerve, and so forth. The pupil camera 206 and the fundus camera 208 of the imaging system 204 capture images of the eye 202. The pupil camera 206 can operate by directing light through the eye's cornea and lens to illuminate the pupil. In some embodiments, the pupil camera 206 can use one or more illumination sources (e.g., infrared illuminants, white light illuminants, and so forth) to capture images of the pupil of the eye by capturing the reflection of the pupil from the light. The pupil camera 206 can track the position and movement of the pupil using methods discussed withPATENT Atorney Docket No. 124824.8143.WO01 reference to Figure 9. On the other hand, the fundus camera 208 can capture images of the retina (i.e., fundus) of the eye. The fundus camera 208 operates by directing light through the eye's cornea and lens to illuminate the retina. The fundus camera 208 can use visible light or specific wavelengths of light, such as blue or green, to capture images of the retina. The light passes through the cornea and lens, reaching the retina at the back of the eye. The retina reflects the light, and the reflections are captured by the fundus camera's 208 sensor.

[0048] The imaging system 204 continuously monitors the eye's movements and uses the data collected by the pupil camera 206 to create the modeled eye motion 212. The modeled eye motion 212 is a set of predicted eye movements to enable the imaging system 204 to anticipate the eye 202 position when capturing images. The imaging system 204 can capture both slow motion images 214 and fast motion images 216. Slow motion images 214 are captured when the eye 202 is moving slowly (e.g., at a velocity less than the mean velocity, near a minimum velocity) relative to the speed throughout a particular time period. Fast motion images 216 are captured when the eye 202 is moving quickly (e.g., at a velocity greater than the mean velocity) relative to the speed throughout a particular time period. However, motion artifacts in fast motion images 216 can occur because the eye's rapid movement can cause misalignment between the different parts of the image captured during the short exposure time. The artifacts can manifest as blurring, ghosting, or distortion in the image, making it difficult to interpret the retinal structures.

[0049] In some embodiments, an imaging system 204 models the eye's 202 position, angle, and / or motion (e.g., modeled eye motion 212) to guide the patient in moving the eye 202 to a designated position and angle. In some embodiments, the imaging system 204 can direct the gaze of the eye 202 to a particular orientation. For example, the gaze of the eye 202 can be directed so that one of the edges of the eye 202 motion aligns with the optic axis 210 (or an orientation in accordance with a particular gaze angle relative to the optic axis 210, such as angle 602) of the imaging system 204. The particular gaze angle can be predetermined and based on the imaging system's 204 view from the particular gaze angle of a particular structure within the eye 202, (e.g., retina, optic nerve).PATENTAtorney Docket No. 124824.8143.WO01

[0050] Guidance can be provided by moving a gaze target visualized on a display (e.g., the same gaze target used for focusing, guiding an eye 202 to an eye box of the imaging system 204). The displayed target can be displaced to adjust the gaze angle and ensure that the edge of the motion aligns with the particular orientation (e.g., the gaze of the eye can be guided such that the relationship (position, gaze) between the eye position and the imaging system 204 is adjusted to the edge of the eye motion, where speed is near minimal). For example, the gaze target can be displaced from standard position so that the gaze angle moves, and the edge of the motion matches the optic axis. Various means can be used to guide patients, including visual displays, lights, sounds, vibrations, temperature cues, and so forth.

[0051] In some embodiments, instead of asking the patient to move their eyes 202 or head, the optics (e.g., the pupil camera 206, the fundus camera 208, and so forth) of the imaging system 204 can be moved to align the optic axis 210 with the edge of the eye motion. The optics can be designed to be manually movable, with guidance provided to the operator to indicate the appropriate adjustments. Alternatively, the imaging system can be equipped with motorized stages, allowing the optics to be moved automatically to achieve the desired alignment.

[0052] Figure 3 illustrates a network environment 300 that includes the imaging system 302 (e.g., imaging system 204 in Figure 2). Individuals can interact with the imaging system 302 via an interface 304. For example, an operator of an imaging device may access the interface 304 to initiate an imaging session in which a patient is to be imaged. As another example, a healthcare professional may access the interface 304 to review the digital images generated by an imaging device in order to diagnose the human bodies captured in those digital images. As another example, a user may access the interface 304 to view different visualizations (or visualization components) of imaging data. The interface 304 may display various types of visualizations, such as three-dimensional (3D) renderings, cross-sectional views, heat maps, and so forth, highlighting regions of interest. In some cases, the interface 304 may allow users to toggle between different visualization modes or adjust parameters like contrast and color mapping. The interface 304 may present quantitative metrics or measurements extracted from the imaging data. Additionally, in some aspects, the interface 304 may enable comparison of multiple images side-by-side or overlay of images from differentPATENTAtorney Docket No. 124824.8143.WO01 time points. The specific visualizations and capabilities of the interface 304 may depend on the type of imaging being performed and the needs of the users accessing the system.

[0053] As shown in Figure 3, the imaging system 302 may reside in a network environment 300. Thus, the imaging system 302 may be connected to one or more networks 306a-b. The network(s) 306a-b can include personal area networks (PANs), local area networks (LANs), wide area networks (WANs), metropolitan area networks (MANs), cellular networks, the Internet, etc. Additionally or alternatively, the imaging system 302 can be communicatively coupled to computing device(s) over a short-range wireless connectivity technology, such as Bluetooth® or Near Field Communication (NFC).

[0054] The interface 304 is preferably accessible via a web browser, desktop application, mobile application, or over-the-top (OTT) application. Accordingly, the interface 304 may be viewed on an imaging device, personal computer, tablet computer, mobile workstation, mobile phone, game console, wearable electronic device (e.g., a watch or fitness accessory), network-connected ("smart") electronic device, (e.g., a television or home assistant device), or virtual / augmented reality system (e.g., a headmounted display).

[0055] Some embodiments of the imaging system 302 are hosted locally. That is, the imaging system 302 may reside on the computing device used to access the interface 304. For example, the imaging system 302 may be embodied as a mobile application executing on a mobile phone. As another example, the imaging system 302 may be embodied as a desktop application executing on an imaging device.

[0056] Other embodiments of the imaging system 302 are executed by a cloud computing service operated by, for example, Amazon Web Services®, Google Cloud Platform™, or Microsoft Azure®. In such embodiments, the imaging system 302 may reside on a network-accessible server system 308 comprised of one or more computer servers. These computer servers can include digital images generated by imaging devices, patient information (e.g., age, sex, health diagnoses, etc.), imaging device information (e.g., resolution, expected file size, etc.), imaging models, and other assets. For example, profiles that include digital images associated with different patients, imaging sessions, imaging devices, healthcare facilities (e.g., hospitals, clinics,PATENTAtorney Docket No. 124824.8143.WO01 optometry offices), healthcare networks, etc. Those skilled in the art will recognize that this information could also be distributed amongst a network-accessible server system and one or more computing devices.

[0057] Figure 4 depicts an example of a communication environment 400 that includes an imaging system 402 configured to acquire data from one or more sources. Here, the imaging system 402 may receive data from a fundus camera 406, laptop computer 408, or network-accessible server system 410 (collectively referred to as the "networked devices"). For example, the imaging system 402 may obtain pixel data from the fundus camera 406 and other data (e.g., patient information, imaging models, processing operations) from the laptop computer 408 or network-accessible server system 410.

[0058] The networked devices can be connected to the imaging system 402 via one or more networks 404a-c. The network(s) 404a-c can include PANs, LANs, WANs, MANs, cellular networks, the Internet, etc. Additionally or alternatively, the networked devices may communicate with one another over a short-range wireless connectivity technology, such as Bluetooth or NFC. For example, if the imaging system 402 resides on the network-accessible server system 410, data received from the network- accessible server system 410 need not traverse any networks. However, the network- accessible server system 410 may be connected to the fundus camera 406 and laptop computer 408 via separate Wi-Fi communication channels.

[0059] Embodiments of the communication environment 400 may include a subset of the networked devices. For example, some embodiments of the communication environment 400 include an imaging system 402 that receives pixel data from the fundus camera 406 (e.g., in the form of DICOM data objects) and additional data from the network-accessible server system 410 on which it resides. As another example, some embodiments of the communication environment 400 include an imaging system 402 that receives pixel data from a series of fundus cameras located in different environments (e.g., different clinics).Methodologies for Generating Images Using the Imaging System

[0060] Introduced here are several approaches to generating images using the imaging system 204. Historically, retinal imaging has been performed using fundusPATENT Atorney Docket No. 124824.8143.WO01 cameras that capture single images of the retina during brief moments when a patient's eye is relatively still. The process often requires multiple attempts to obtain clear, usable images due to involuntary eye movements, which could blur or distort the captured images. For patients with certain medical conditions causing more pronounced eye movements, such as nystagmus, obtaining high-quality retinal images was particularly challenging and time-consuming.

[0061] The approaches described herein improve the quality of the retinal image captured by modeling the eye motion of the patient and dynamically determining a trigger time and / or illumination pattern based on the eye motion. As such, these approaches may enable the capture of higher quality retinal images, even for patients with involuntary eye movements. Moreover, these approaches may serve to reduce the time and number of attempts required to obtain diagnostically useful images. Simply put, these approaches may permit more reliable and consistent retinal imaging across a broader range of patients, potentially improving the detection and monitoring of various eye conditions.

[0062] Note that these approaches can be employed independent of one another. Accordingly, while the approaches are described separately for the purpose of simplification, those skilled in the art will recognize that these approaches (or aspects of each approach) could be performed in combination. Moreover, while these approaches are described in the context of a fundus camera, those skilled in the art will recognize that the approaches are similarly applicable to other imaging devices.

[0063] Figure 5A illustrates how, in some embodiments, a pair of secondary cameras are circumferentially arranged along opposing sides of the lens through which light is collected by a fundus camera. Generally, the secondary cameras are arranged such that the area surrounding the eye is observable when the patient aligns her retina with the lens. Accordingly, these secondary cameras may be referred to as "pupil cameras" or "ocular cameras."

[0064] However, other facial features may also be observable by the secondary cameras as can be seen in Figures 5B-C. Figures 5B-C include examples of digital images that were generated secondary cameras circumferentially arranged along opposing sides of the lens. As discussed above with respect to Figure 6, these digital images may be analyzed in order to determine whether the left eye or the right eye isPATENT Atorney Docket No. 124824.8143.WO01 presently located in front of the lens. The secondary cameras may be selected so that digital images generated by those cameras provide different information than digital images generated based on light reflected through the lens. For example, the secondary camera may include image sensors that are sensitive to infrared light or visible light.

[0065] Figure 6 illustrates an example environment 600 for determining an angle 602 of the captured images from the optic axis 210 of the imaging system 204 of Figure 2. Environment 600 includes eye 202, imaging system's 204 optic axis 210, angle 602, slow motion images 604 (e.g., first slow motion image 604a, second slow motion image 604b, and so forth), and fast motion image 606. Embodiments of the environment 600 can include different and / or additional components or can be connected in different ways.

[0066] The angle 602 refers to the gaze angle between the eye 202 and the optic axis 210 of the imaging system 204. The angle 602 can be used to determine the alignment and orientation of the captured images. When capturing images, the angle 602 can vary depending on the position and movement of the eye. For instance, the angle 602 between the imaging system's 204 optic axis 210 and the eye 202 can be equal when capturing at locations with the first slow motion image 604a and the second slow motion image 604b (e.g., at the edge locations 802 discussed with reference to Figure 8). Conversely, the angle 602 can be zero or near zero when capturing a fast motion image 606. The slow motion images 604 are the same as or similar to slow motion image 214 in Figure 2. In some embodiments, slow motion images 604 is captured from a shifted field of view (e.g., different angle and / or position) as that of the slow motion image 214 in accordance with the angle 602. The fast motion image 606 are the same as or similar to fast motion image 216 in Figure 2. Similar to the slow motion images 604 and 214, the fast motion images 606 can be captured from different angles and / or positions.

[0067] In Figure 6, the center of the eye's 202 motion is aligned with the optic axis, which is the central axis of the optical system. The alignment improves image quality (e.g., reduces motion artifacts) at both extremities (e.g., edge locations 802 in Figure 8) of the eye's 202 motion range. In some embodiments, guidance can be provided either to patients to move their eyes 202 or to adjust the optical systems (as shown in FigurePATENT Atorney Docket No. 124824.8143.WO012) to align the eye 202 with the angle 602 of slow motion images 604. In some embodiments, images of the pupil are segmented and triangulated to ascertain the coordinates of the pupil center, and changes in the coordinates of the pupil center are used to determine the angle 602. In some embodiments, the angle 602 can be measured using the imaging system 204 (e.g., via a pupil camera 206) that projects a reference beam onto the eye 202. The reflections from the eye can be used to determine the angle 602 between the eye 202 and the optic axis 210. Other modalities like Optical coherence tomography (OCT), visible camera, fundus imaging, and so forth can be used to measure the angle 602. Further eye movement characterization methods can be performed using various methodologies discussed with reference to Figure 9.

[0068] Figure 7 illustrates an example graphical chart 700 for determining a model for estimating the positions of the pupil over time. Chart 700 includes eye position 702, triangular wave model 704, and sinusoidal wave model 706. Embodiments of the chart 700 can include different and / or additional components or can be connected in different ways.

[0069] The eye position 702 refers to the spatial coordinates of the pupil as the pupil moves over time. Eye position 702 can be tracked by the imaging system 204 to capture the pupil's location at different time intervals. The data collected can be used to create a time-series representation of the pupil's movement. The triangular wave model 704 is a mathematical representation used to estimate the periodic motion of the pupil. The triangular wave model 704 can assume that the pupil's movement follows a triangular waveform, characterized by linear increases and decreases in position over time. The triangular wave model 704 can be used, for example, for simulating saccadic eye movements, which involve rapid shifts in gaze direction. The sinusoidal wave model 706 is a further mathematical representation that can be used to estimate the periodic motion of the pupil. The sinusoidal wave model 706 assumes that the pupil's movement follows a sinusoidal waveform, characterized by smooth, continuous oscillations. The sinusoidal wave model 706 can be used, for example, for simulating pendular eye movements. Examples of periodic functions used to model the eye position 702 include sinusoidal, triangular wave, square wave, sawtooth wave, and so forth.PATENTAtorney Docket No. 124824.8143.WO01

[0070] Figure 8 illustrates an example graphical chart 800 for establishing values for parameters of a modelled eye motion. Chart 800 includes eye position 702 and edge locations 802 (e.g., first edge location 802a, second edge location 802b, and so forth). Embodiments of the chart 800 can include different and / or additional components or can be connected in different ways.

[0071] The edge locations 802 refer to the extreme positions of the eye's 202 motion, such as the first edge location 802a and the second edge location 802b. Edge locations 802 can be estimated from the peak values of the eye position measurements, which represent the maximum and minimum positions reached by the eye 202 during its motion. In some embodiments, edge locations 802 can be determined by computing the maximum and minimum values in the position or angle data. For example, the edge locations 802 can be positions where the eye 202 has a zero or near zero velocity. By examining the peak values and the periodicity of the eye's motion, the parameters of the motion model can be determined using methods discussed with reference to Figure 9.

[0072] Figure 9 depicts a flow diagram of a process 900 for generating, with the imaging system 204 of Figure 2, an image based on parameters determined through the modelled eye motion. In some embodiments, the process 900 is performed by components of example computer systems 1500 illustrated and described in more detail with reference to Figure 15. Particular entities, for example, the model, are illustrated and described in more detail with reference to chart 700 and chart 800 in Figure 7 and Figure 8, respectively. Embodiments of the process 900 can include different and / or additional operations or can perform the operations in different orders.

[0073] In operation 902, the imaging system 204 can receive an indication of an interaction by a human with a control of the fundus camera having a lens through which light is reflected by a retina of an eye. For example, a user can be positioned at the fundus camera and a button can be depressed (e.g., the imaging system 204 detects that the forehead is against the fundus camera apparatus). In some embodiments, the interaction can be a physical action such as pressing a button, turning a dial, or touching a screen on the fundus camera. In other embodiments, the interaction can be a voice command or a gesture recognized by the camera's integrated sensors.PATENTAtorney Docket No. 124824.8143.WO01

[0074] In response to said receiving, in operation 904, the imaging system 204 can transmit a first command to a set of secondary cameras that is located proximate to the lens to cause the secondary camera to generate a series of images. In some embodiments, the set of secondary cameras is a stereoscopic camera system including a pair of digital cameras. For example, two or more cameras can be placed at fixed distances from each other, capturing images simultaneously.

[0075] In operation 906, the imaging system 204 can generate (by the fundus camera and / or the secondary camera), based on the series of images, pupil position data that includes positions of a pupil of the human, in temporal order, over a time period of pre-determined length. The imaging system 204 can detect the pupil in each image by identifying the dark circular region within the eye (e.g., via edge detection, thresholding, and so forth). Once the pupil is detected, pupil position can estimated using the location and / or shape of the pupil (e.g., ellipses) within each image. In some embodiments, one or more cameras (e.g., pupil cameras) of the imaging system 204 can be calibrated to map image coordinates (e.g., pixel coordinates) to 3D coordinates (i.e., x-, y-, z-coordinates), and estimate pupil position data based on the mapped 3D coordinates. The shape of the pupil can be used as input to the imaging system 204 to estimate pupil position data (e.g., a gaze angle, an angle of the pupil from the optic axis). For example, if the pupil appears as a circle, it can indicate that the eye is looking directly at the camera. If the pupil appears as an elongated ellipse, it can indicate that the pupil is positioned at an angle from the imaging system 204. In some embodiments, the imaging system 204 can use one or more machine learning models trained to recognize the pupil in various lighting conditions and eye orientations. Convolutional neural networks (CNNs) can be used to identify the pupil's features from a dataset of labeled images. For example, a CNN can be trained on a dataset of labeled images, where the pupil's position is annotated. The network can learn to identify the features of the pupil, such as its shape and texture, and can generalize this knowledge to new images. The trained model can be applied to the series of images to automatically detect and track the pupil's position over time.

[0076] In operation 908, the imaging system 204 can determine a model for estimating the positions of the pupil overtime using the pupil position data. The imaging system 204 can establish values for parameters of the model. In some embodiments,PATENTAtorney Docket No. 124824.8143.WO01 the imaging system 204 can use the pupil position data to fit a line, polynomial equation, piecewise polynomials, and so forth, to represent the pupil's movement. The pupil's movement can be repetitive (e.g., patterned motion) or non-patterned. In some embodiments, the imaging system 204 determines that the pupil position data is associated with periodic motion. The imaging system 204 can select the model from a set of models. The parameters associated with the model include the periodicity (or frequency) of the motion, the amplitude (the maximum deviation from the mean position), the temporal offset (the phase shift), and / or the specific pattern of the motion. For example, by examining the frequency spectrum, the imaging system 204 can detect dominant periodic components, indicating that the pupil's motion follows a regular, repeating pattern.

[0077] In some embodiments, recurrent neural networks (RNNs) or long shortterm memory (LSTM) networks can be used to learn temporal dependencies in the pupil position data. RNNs maintain a hidden state that updates with each time step based on the current input (pupil position) and the previous hidden state, enabling the model to capture patterns in the sequence. LSTMs, a type of RNN, use memory cells and gating mechanisms (input, forget, and output gates) to control the flow of information, enabling LSTMs to learn long-term dependencies. The parameters of the model (e.g., slope, intercept, weights, biases), can be determined by minimizing the difference between the observed positions and the estimated positions.

[0078] In operation 910, the imaging system 204 can predict, based on the model and the parameters, a first time at which the positions of the pupil are associated with a minimum velocity (e.g., a near zero velocity). In some embodiments, the imaging system 204 can determine a first position (e.g., 2D coordinate, 3D coordinate, angular position, and so forth) associated with the pupil at a first time step associated with the time period, and determine a second position associated with the pupil at a second time step associated with the time period, where the second time step immediately succeeds the first time step. The imaging system 204 can determine an angular difference between the first position and the second position. In some embodiments, the imaging system 204 can determine that the angular difference is less than or equal to a threshold difference. Based on determining that the angular difference is less than or equal toPATENT Atorney Docket No. 124824.8143.WO01 the threshold difference, the imaging system 204 can determine the first time associated with the first time step.

[0079] In operation 912, the imaging system 204 (e.g., via the fundus camera) can estimate a position of the pupil at the first time. In some embodiments, the imaging system 204 estimates the position of the pupil at the first time by determining a horizontal angular coordinate and / or a vertical angular coordinate associated with the pupil at which the pupil is at a maximum angle from a central axis of pupil motion during the time period. In some embodiments, the imaging system 204 can use the model and parameters established in previous steps to estimate the position of the pupil at the first time of minimum velocity. Given the periodic nature of the pupil's motion, the imaging system 204 can calculate the position by analyzing the motion pattern and identifying the specific time when the velocity is at its minimum. The horizontal angular coordinate refers to the angle between the pupil's position and the central vertical axis, while the vertical angular coordinate refers to the angle between the pupil's position and the central horizontal axis.

[0080] In some embodiments, the position of the pupil is represented by a coordinate on a three-dimensional coordinate system by defining the pupil's position in terms of x, y, and z coordinates, where the x-coordinate can represent the horizontal position, the y-coordinate can represent the vertical position, and the z-coordinate can represent the depth or distance from the camera. For example, the imaging system 204 can capture images from different angles and triangulate the pupil's position and calculate the x, y, and z coordinates.

[0081] In some embodiments, the position of the pupil is represented as a pitch angle and / or yaw angle. The pitch angle can correspond to a vertical angle of a gaze vector of the pupil relative to a reference plane, such as the horizontal plane passing through the center of the eye. The yaw angle can correspond to a horizontal angle of the gaze vector of the pupil relative to a reference axis, such as the vertical axis passing through the center of the eye. To calculate the pitch and yaw angles, the imaging system 204 can determine the gaze vector, which is a line extending from the center of the pupil in the direction of the gaze. The imaging system 204 can project the gaze vector onto the reference plane and reference axis to calculate the angles. The pitch angle can be calculated by measuring the angle between the gaze vector and thePATENT Atorney Docket No. 124824.8143.WO01 reference plane, while the yaw angle can be calculated by measuring the angle between the gaze vector and the reference axis.

[0082] In operation 914, the imaging system 204 can generate, based on the position, a fundus camera configuration that includes values of measurement parameters associated with the fundus camera (e.g., focus, exposure time, gain, illumination intensity, and so forth). For example, the imaging system 204 can use a lookup table to quickly determine the fundus camera configuration settings based on the pupil's position. The lookup table can be precomputed and stored in memory, allowing the imaging system 204 to retrieve the appropriate settings without performing calculations in real-time.

[0083] In operation 916, the imaging system 204 can transmit, at a second time that is subsequent to the first time (e.g., accounting for a latency delay discussed with reference to Figures 10-12), a command to the fundus camera to cause capture of a second image of the eye based on the light being reflected through the lens according to the fundus camera configuration. In some embodiments, the imaging system 204 can transmit a command to the fundus camera to capture a second image of the eye at a specific time after the initial image capture. The imaging system 204 can predict, based on the model and the parameters, the second time at which the positions of the pupil are associated with the minimum velocity. The second time can be different from the first time. In some embodiments, the imaging system 204 can transmit a command to the fundus camera to capture a second image of the eye at a specific time after the initial image capture.

[0084] In some embodiments, the imaging system 204 can generate, for display on a user interface, the second image of the eye. The user interface can include interactive elements that enable the user to manipulate the image. For example, the user can be enabled to zoom in and out, pan across the image, or adjust the brightness and contrast. The interactive features can be implemented using event handlers that respond to user input, such as mouse clicks or touch gestures. In some embodiments, the imaging system 204 can display information alongside the image of metadata such as date and time of capture, camera settings used, annotations, measurements and so forth. The metadata can be displayed in a separate panel or overlayed on the image itself.PATENTAtorney Docket No. 124824.8143.WO01

[0085] Figure 10 illustrates an example graphical chart 1000 for establishing a trigger time for the imaging system 204 of Figure 2 based on the modelled eye motion. Chart 1000 includes edge locations 802 (e.g., first edge location 802a, second edge location 802b, and so forth), modeled position 1002, and trigger time 1004. Embodiments of the chart 1000 can include different and / or additional components or can be connected in different ways.

[0086] The modeled position 1002 represents the predicted position of the eye 202 based on a mathematical model (e.g., triangular wave model 704, sinusoidal wave model 706, saccadic model, sawtooth model, square model, and so forth) of the eye's 202 motion. The model can be derived from periodic patterns observed in the eye's movement, such as sinusoidal, triangular, or other waveforms. The modeled position 1002 can be used to anticipate the eye's location at a particular time. In some embodiments, the modeled position 1002 can be generated using machine learning models that use the historical eye movement data to predict future positions. Methods of using machine learning models to generate future positions are discussed further with reference to Figure 12.

[0087] The trigger time 1004 is a particular time when the imaging system 204 is instructed to capture an image. The trigger time can be determined based on the modeled position 1002 to ensure that the image is captured when the eye is in a favorable position, such as at the edge of its motion where velocity is lowest and motion artifacts are minimized. In some embodiments, the trigger time 1004 can be synchronized with the natural periodicity of the eye's motion to capture images at consistent intervals (e.g., capturing images every 5 periods).

[0088] Figure 11 illustrates an example graphical chart 1100 for a delay between a capture time and the trigger time for the imaging system 204 of Figure 2 based on the modelled eye motion. Chart 1100 includes measured trajectory 1102, trigger time 1004, predicted trajectory 1104, and capture time 1106. Embodiments of the chart 1100 can include different and / or additional components or can be connected in different ways.

[0089] The measured trajectory 1102 refers to the observed path taken by the eye 202 as the eye 202 moves over time. The trigger time 1004 is a particular time when the imaging system 204 is activated to capture an image. The trigger time 1004 can be determined based on the modeled position of the eye to ensure that the image isPATENTAtorney Docket No. 124824.8143.WO01 captured when the eye is in a favorable position, such as at the edge of the eye's 202 motion where velocity is lowest and motion artifacts are reduced.

[0090] The predicted trajectory 1104 represents the anticipated path of the eye based on a mathematical model of the eye's motion. This model can be derived from the periodic patterns and / or non-periodic patterns. The capture time 1106 is the moment when the imaging system 204 actually captures the image. Due to the delay between the trigger time 1004 and the capture time 1106, the eye's 202 position at the capture time 1106 can differ from the position at the trigger time 1004.

[0091] Figure 12 depicts a flow diagram of a process 1200 for generating, with the imaging system 204 of Figure 2, an image based on a latency value associated with a delay between triggering the imaging system and image capture by the imaging system. In some embodiments, the process 1200 is performed by components of example computer systems 1500 illustrated and described in more detail with reference to Figure 15. Particular entities, for example, the model, are illustrated and described in more detail with reference to chart 1000 and chart 1 100 in Figure 10 and Figure 11 , respectively. Embodiments of the process 1200 can include different and / or additional operations or can perform the operations in different orders.

[0092] In operation 1202, the imaging system 204 can determine a latency value associated with a delay between triggering the fundus camera and image capture by the fundus camera. The imaging system 204 can use a timestamp function provided by the system's operating system or hardware timer to record the time at which the trigger command is sent to the fundus camera. The imaging system 204 can determine the time at which the image is captured by the fundus camera by monitoring the camera's status signals that indicate when an image has been captured. Once both timestamps are recorded, the imaging system 204 can calculate the latency value by subtracting the trigger timestamp from the capture timestamp. The resulting value can represent the delay between triggering the camera and the actual image capture. Once the latency value is determined, the imaging system 204 can store the latency value in a log file or database.

[0093] In operation 1204, the imaging system 204 can determine a second time, subsequent to the first time, according to the latency value and the first time. The imaging system 204 can record the first time (e.g., time when the initial trigger commandPATENT Atorney Docket No. 124824.8143.WO01 is sent to the fundus camera). The imaging system 204 can retrieve the previously determined latency value, and add the latency value to the first time to determine the expected moment when the image capture will occur.

[0094] In operation 1206, the imaging system 204 can transmit a command, at the second time, to the fundus camera to cause capture, by the fundus camera and using the fundus camera configuration, of a second image of the eye. The command signal can include information about the timing of the capture and any specific parameters that are applied, such as exposure settings, focus adjustments, or illumination levels.

[0095] Figure 13 illustrates an example environment 1300 for determining an illumination pattern of the imaging system 204 of Figure 2 based on a predicted location of the eye. Environment 1300 includes eye 202, capture locations 1302 (e.g., first location 1302a, second location 1302b, third location 1302c, and so forth), and predicted illumination patterns 1304 (e.g., first illumination pattern 1304a, second illumination pattern 1304b, third illumination pattern 1304c, and so forth). Embodiments of the environment 1300 can include different and / or additional components or can be connected in different ways.

[0096] The capture locations 1302 refer to the specific positions where the imaging system captures images of the eye. In some embodiments, light source illumination sequences can be selected based on the predicted capture location 1302 to increase the illumination at the retina.

[0097] The predicted illumination patterns 1304 refer to the sequences of light sources used to illuminate the eye during image capture. Predicted illumination patterns 1304 can be created using an illumination lookup table (LUT) that specifies which light sources to activate based on the expected pupil locations. The LUT can map locations to illumination patterns with the most light throughput to and from the eye based on certain pupil size and location within the predicted eye motion range.

[0098] In some embodiments, predicted illumination patterns 1304 can be based on pupil sizes and / or eye conditions (e.g., cataracts). For example, the LUT can be adjusted to account for changes in pupil size that occur when a flash is triggered, ensuring that the retina receives illumination regardless of the pupil's state. In some embodiments, the imaging system can guide the eye to an optimal imaging start position that will ensure illumination for the entire range of predicted motion.PATENT Atorney Docket No. 124824.8143.WO01

[0099] Figure 14 depicts a flow diagram of a process 1400 for generating, with the imaging system 204 of Figure 2, an image using a predicted illumination pattern based on the modelled eye motion. In some embodiments, the process 1400 is performed by components of example computer systems 1500 illustrated and described in more detail with reference to Figure 15. Particular entities, for example, the model, are illustrated and described in more detail with reference to the environment 1300 in Figure 13. Embodiments of the process 1400 can include different and / or additional operations or can perform the operations in different orders.

[0100] In operation 1402, the imaging system 204 can determine, based on the position at the first time, a target angular region for illumination of the pupil. For example, the imaging system can calculate angles at which the illumination source are directed to ensure that the pupil is adequately illuminated. In some embodiments, the target angular region can be determined by specific eye conditions. For example, for patients with glaucoma, the imaging system 204 can need to focus on specific regions of the optic nerve head. The illumination angles can be adjusted to improve the visibility of these regions.

[0101] In operation 1404, the imaging system 204 can determine a subset of illuminants of the light source, where activation of the subset of illuminants enables illumination of the target angular region. Each illuminant in the array can have a specific position and angle of emission, which can be used to direct light towards different regions of the eye. The imaging system 204 can map the target angular region to the positions and angles of the illuminants in the array.

[0102] In operation 1406, the imaging system 204 can determine illuminant identifiers for the subset of illuminants. Each illuminant in the array can be assigned a unique identifier, which can be a numerical or alphanumeric code. For each selected illuminant, the imaging system 204 can retrieve its unique identifier from the database or map.

[0103] In operation 1408, the imaging system 204 can generate a command including an indication of the illuminant identifiers. The imaging system 204 can include additional parameters in the command, such as the intensity levels for each illuminant, the duration of illumination, and any timing information required for synchronization. For example, the command can include global control parameters that apply to the entirePATENT Atorney Docket No. 124824.8143.WO01 subset of illuminants (e.g., overall duration of illumination, synchronization information, and so forth). In some embodiments, the imaging system 204 can determine, based on the target angular region of illumination of the pupil, a brightness value associated with the target angular region of illumination, and generate the fundus camera configuration including the brightness value.

[0104] In operation 1410, the imaging system 204 can transmit the command to the capturing medium to cause the capturing medium to capture a second image of the eye during illumination of the pupil by the subset of illuminants. Upon receiving the command, a control unit of the imaging system 204 can activate the specified illuminants according to the provided parameters. For example, the imaging system 204 can turn on the illuminants, adjust their intensity levels, and so forth.Processing System

[0105] Figure 15 is a block diagram illustrating an example of a processing system 1500 in which at least some operations described herein can be implemented. For example, some components of the processing system 1500 may be hosted on a computing device that includes an imaging system (e.g., imaging system 204 of Figure 2 or imaging system 302 of Figure 3).

[0106] The processing system 1500 may include a central processing unit (also referred to as a "processor") 1502, main memory 1506, non-volatile memory 1510, network adapter 1512, video display 1518, input / output devices 1520, control device 1522 (e.g., keyboard and pointing devices), drive unit 1524 including a storage medium 1526, and signal generation device 1530 that are communicatively connected to a bus 1516. The bus 1516 is illustrated as an abstraction that represents one or more physical buses and / or point-to-point connections that are connected by appropriate bridges, adapters, or controllers. The bus 1516, therefore, can include a system bus, a Peripheral Component Interconnect (PCI) bus or PCI-Express bus, a HyperTransport or industry standard architecture (ISA) bus, a small computer system interface (SCSI) bus, a universal serial bus (USB), an Inter-Integrated Circuit (l2C) bus, or an Institute of Electrical and Electronics Engineers (IEEE) standard 1494 bus (also referred to as "Firewire").PATENTAtorney Docket No. 124824.8143.WO01

[0107] The processing system 1500 may share a similar computer processor architecture as that of a desktop computer, tablet computer, personal digital assistant (PDA), mobile phone, game console, music player, wearable electronic device (e.g., a watch or fitness tracker), network-connected ("smart") device (e.g., a television or home assistant device), virtual / augmented reality systems (e.g., a head-mounted display), or another electronic device capable of executing a set of instructions (sequential or otherwise) that specify action(s) to be taken by the processing system 1500.

[0108] While the main memory 1506, non-volatile memory 1510, and storage medium 1526 (also called a "machine-readable medium") are shown to be a single medium, the term "machine-readable medium" and "storage medium" should be taken to include a single medium or multiple media (e.g., a centralized / distributed database and / or associated caches and servers) that store one or more sets of instructions 1528. The term "machine-readable medium" and "storage medium" shall also be taken to include any medium that is capable of storing, encoding, or carrying a set of instructions for execution by the processing system 1500.

[0109] In general, the routines executed to implement the embodiments of the disclosure may be implemented as part of an operating system or a specific application, component, program, object, module, or sequence of instructions (collectively referred to as "computer programs"). The computer programs typically comprise one or more instructions (e.g. , instructions 1504, 1508, 1528) set at various times in various memory and storage devices in a computing device. When read and executed by the processor 1502, the instruction(s) cause the processing system 1500 to perform operations to execute elements involving the various aspects of the disclosure.

[0110] Moreover, while embodiments have been described in the context of fully functioning computing devices, those skilled in the art will appreciate that the various embodiments are capable of being distributed as a program product in a variety of forms. The disclosure applies regardless of the particular type of machine or computer- readable media used to actually effect the distribution.

[0111] Further examples of machine-readable storage media, machine-readable media, or computer-readable media include recordable-type media such as volatile memory, non-volatile memory 1510, floppy and other removable disks, hard disk drives, optical disks (e.g., compact disc read-only memory (CD-ROMs) and Digital VersatilePATENT Atorney Docket No. 124824.8143.WO01Discs (DVDs)), and transmission-type media such as digital and analog communication links.

[0112] The network adapter 1512 enables the processing system 1500 to mediate data in a network 1514 with an entity that is external to the processing system 1500 through any communication protocol supported by the processing system 1500 and the external entity. The network adapter 1512 can include a network adaptor card, a wireless network interface card, a router, an access point, a wireless router, a switch, a multilayer switch, a protocol converter, a gateway, a bridge, a bridge router, a hub, a digital media receiver, or a repeater.

[0113] The network adapter 1512 may include a firewall that governs and / or manages permission to access / proxy data in a computer network and tracks varying levels of trust between different machines and / or applications. The firewall can be any number of modules having any combination of hardware and / or software components able to enforce a predetermined set of access rights between a particular set of machines and applications, machines and machines, and / or applications and applications (e.g., to regulate the flow of traffic and resource sharing between these entities). The firewall may additionally manage and / or have access to an access control list that details permissions including the access and operation rights of an object by an individual, a machine, and / or an application, and the circumstances under which the permission rights stand.

[0114] The techniques introduced here can be implemented by programmable circuitry (e.g., one or more microprocessors), software and / or firmware, special-purpose hardwired (i.e. , non-programmable) circuitry, or a combination of such forms. Specialpurpose circuitry can be in the form of one or more application-specific integrated circuits (ASICs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), etc.Remarks

[0115] The foregoing description of various embodiments of the claimed subject matter has been provided for the purposes of illustration and description. It is not intended to be exhaustive or to limit the claimed subject matter to the precise forms disclosed. Many modifications and variations will be apparent to one skilled in the art.PATENTAtorney Docket No. 124824.8143.WO01Embodiments were chosen and described in order to best describe the principles of the invention and its practical applications, thereby enabling those skilled in the relevant art to understand the claimed subject matter, the various embodiments, and the various modifications that are suited to the particular uses contemplated.

[0116] Although the Detailed Description describes certain embodiments and the best mode contemplated, the technology can be practiced in many ways no matter how detailed the Detailed Description appears. Embodiments may vary considerably in their embodiment details, while still being encompassed by the specification. Particular terminology used when describing certain features or aspects of various embodiments should not be taken to imply that the terminology is being redefined herein to be restricted to any specific characteristics, features, or aspects of the technology with which that terminology is associated. In general, the terms used in the following claims should not be construed to limit the technology to the specific embodiments disclosed in the specification, unless those terms are explicitly defined herein. Accordingly, the actual scope of the technology encompasses not only the disclosed embodiments, but also all equivalent ways of practicing or implementing the embodiments.

[0117] The language used in the specification has been principally selected for readability and instructional purposes. It may not have been selected to delineate or circumscribe the subject matter. It is therefore intended that the scope of the technology be limited not by this Detailed Description, but rather by any claims that issue on an application based hereon. Accordingly, the disclosure of various embodiments is intended to be illustrative, but not limiting, of the scope of the technology as set forth in the following claims.

Claims

PATENTAtorney Docket No. 124824.8143.WO01CLAIMSWe claim:

1. A method for generating, with a fundus camera having a lens through which light is reflected by a retina of an eye and a secondary camera that is located proximate to the lens, an image based on parameters determined through modeling of eye motion, the method comprising: receiving an indication of an interaction by a human with a control of the fundus camera; in response to said receiving, transmitting a first command to the secondary camera to cause the secondary camera to generate a series of images; generating, based on the series of images, pupil position data that includes positions of a pupil of the human, in temporal order, over a time period of pre-determined length; using the pupil position data: determining a model for estimating the positions of the pupil over time, and establishing values for parameters of the model; predicting, based on the model and the parameters, a first time at which the positions of the pupil are associated with a minimum velocity; estimating a position of the pupil at the first time; generating, based on the position, a fundus camera configuration that includes values of measurement parameters associated with the fundus camera; and transmitting, at a second time that is subsequent to the first time, a command to the fundus camera to cause capture of a second image of the eye based on the light being reflected through the lens according to the fundus camera configuration.

2. The method of claim 1 , wherein determining the model comprises: determining that the pupil position data is associated with periodic motion; and selecting the model from a set of models,PATENTAtorney Docket No. 124824.8143.WO01 wherein the parameters associated with the model include one or more of: (1) a periodicity of the periodic motion, (2) an amplitude of the periodic motion, (3) a temporal offset, or (4) a pattern of the periodic motion.

3. The method of claim 1 , wherein predicting the first time at which the positions of the pupil are associated with the minimum velocity comprises: determining a first position associated with the pupil at a first time step associated with the time period; determining a second position associated with the pupil at a second time step associated with the time period, wherein the second time step immediately succeeds the first time step; determining an angular difference between the first position and the second position; determining that the angular difference is less than or equal to a threshold difference; and based on determining that the angular difference is less than or equal to the threshold difference, determining the first time associated with the first time step.

4. The method of claim 1 , further comprising: predicting, based on the model and the parameters, the second time at which the positions of the pupil are associated with the minimum velocity, wherein the second time is different from the first time.

5. The method of claim 1 , wherein estimating the position of the pupil at the first time comprises determining one or more of: a horizontal angular coordinate or a vertical angular coordinate associated with the pupil at which the pupil is at a maximum angle from a central axis of pupil motion during the time period.PATENTAtorney Docket No. 124824.8143.WO016. The method of claim 1 , wherein generating, based on the position, the fundus camera configuration based on the position comprises: determining, based on the position at the first time, a target angular region of illumination of the pupil; determining a subset of illuminants of a light source associated with the fundus camera, wherein activation of the subset of illuminants enables illumination of the target angular region of illumination; and generating the fundus camera configuration including identifiers of the subset of illuminants.

7. The method of claim 6, wherein generating, based on the position, the fundus camera configuration comprises: determining, based on the target angular region of illumination of the pupil, a brightness value associated with the target angular region of illumination; and generating the fundus camera configuration including the brightness value.

8. The method of claim 7, wherein causing the capture of the second image of the eye according to the fundus camera configuration comprises: generating the command including an indication of the identifiers of the subset of illuminants and an indication of the brightness value; and transmitting the command to the fundus camera to cause the fundus camera to capture a retinal image of the retina of the eye during illumination of the pupil by the subset of illuminants at the brightness value.

9. The method of claim 1 , further comprising generating, for display on a user interface, the second image of the eye.

10. The method of claim 1 , wherein estimating the position of the pupil at the first time is performed by the fundus camera.11 . The method of claim 1 , wherein the position of the pupil is represented as a coordinate on a three-dimensional coordinate system.PATENTAtorney Docket No. 124824.8143.WO0112. The method of claim 1 , wherein the position of the pupil is represented as one or more of a pitch angle or yaw angle, wherein the pitch angle corresponds to a vertical angle of a gaze vector of the pupil relative to a reference plane, and wherein the yaw angle corresponds to a horizontal angle of the gaze vector of the pupil relative to a reference axis.

13. An imaging system including: a lens through which light reflected by a retina of an eye is collected; a capturing medium that is configured to create an image based on the light reflected by the retina and collected through the lens; a set of secondary cameras that are located proximate the lens and are configured to generate a series of images from which position of the retina can be established based on a determination of positions of a pupil; a light source that includes multiple illuminants, each of which is separately controllable to illuminate the eye and surrounding area; a processor; and a non-transitory medium storing instructions that, when executed by the processor, cause the imaging system to perform operations comprising: transmitting a first command to the set of secondary cameras to cause the set of secondary cameras to generate a series of images; in response to transmitting the first command to the set of secondary cameras, generating, based on the series of images, pupil position data that includes horizontal and vertical coordinates of a pupil, in temporal order, over a time period of pre-determined length; using the pupil position data, predicting a first time at which the positions of the pupil are associated with a minimum velocity; estimating a position of the pupil at the first time, wherein the position is represented as a horizontal angular coordinate and a vertical angular coordinate; determining, based on the position at the first time, a target angular region for illumination of the pupil;PATENTAtorney Docket No. 124824.8143.WO01 determining a subset of illuminants of the light source, wherein activation of the subset of illuminants enables illumination of the target angular region; determining illuminant identifiers for the subset of illuminants; generating a command including an indication of the illuminant identifiers; and transmitting the command to the capturing medium to cause the capturing medium to capture a second image of the eye during illumination of the pupil by the subset of illuminants.

14. The imaging system of claim 13, wherein the instructions for predicting the first time at which the positions of the pupil are associated with the minimum velocity cause the imaging system to perform operations comprising: based on the pupil position data, determining a model enabling estimation of positions of the pupil over time, wherein the model is associated with model parameters; using the pupil position data, calculating parameter values associated with of the model parameters; and using the parameter values, predicting the first time at which the positions of the pupil are associated with the minimum velocity.

15. The imaging system of claim 14, wherein the instructions for determining the model cause the imaging system to perform operations comprising: determining that the pupil position data is associated with periodic motion; and selecting the model from a set of models, wherein the model parameters include (1 ) a periodicity of the periodic motion, (2) an amplitude of the periodic motion, and (3) a temporal offset.PATENTAtorney Docket No. 124824.8143.WO0116. The imaging system of claim 14, wherein the instructions for predicting the first time at which the positions of the pupil are associated with the minimum velocity cause the imaging system to perform operations comprising: determining a first angular position associated with the pupil at a first time step associated with the time period; determining a second angular position associated with the pupil at a second time step associated with the time period, wherein the second time step immediately succeeds the first time step; determining an angular difference between the first angular position and the second angular position; determining that the angular difference is less than or equal to a threshold difference; and based on determining that the angular difference is less than or equal to the threshold difference, determining the first time associated with the first time step.

17. The imaging system of claim 13, wherein the instructions for estimating the position at the first time comprise determining the horizontal angular coordinate and the vertical angular coordinate at which the pupil is, during the time period, at a maximum angle from a central axis of pupil motion.

18. The imaging system of claim 13, wherein the instructions for generating the command cause the imaging system to cause operations comprising: determining, based on the target angular region of illumination of the pupil, a brightness value; and generating the command including the brightness value.

19. The imaging system of claim 13, wherein the instructions for transmitting the command to the capturing medium to cause the capturing medium to capture the second image cause the imaging system to cause operations comprising: determining a second time associated with a trigger for capturing the second image at the first time; and transmitting the command to the capturing medium at the second time.PATENTAtorney Docket No. 124824.8143.WO0120. The imaging system of claim 13, wherein the set of secondary cameras is a stereoscopic camera system including a pair of digital cameras.21 . A method comprising: receiving an indication of an interaction by a human with a control of a retinal camera; in response to said receiving, transmitting a first command to a second camera associated with the retinal camera to cause the second camera to generate a series of images of an eye of the human; generating, based on the series of images, pupil position data that includes horizontal and vertical coordinates of a pupil of the human, in temporal order, over a time period of pre-determined length; using the pupil position data, determining a model for estimating positions of the pupil over time, and establishing values for parameters of the model; predicting, based on the model and the parameters, a first time at which the positions of the pupil are associated with a minimum velocity; estimating a position at the first time, wherein the position is represented as a horizontal angular coordinate and a vertical angular coordinate of the pupil; generating, based on the position, a retinal camera configuration that includes values of measurement parameters associated with the retinal camera; determining a latency value associated with a delay between triggering the retinal camera and image capture by the retinal camera; determining a second time, subsequent to the first time, according to the latency value and the first time; and transmitting a command, at the second time, to the retinal camera to cause capture, by the retinal camera and using the retinal camera configuration, of a second image of the eye.

22. The method of claim 21 , wherein determining the model comprises: determining that the pupil position data is associated with periodic motion; and selecting the model from a set of models,PATENTAtorney Docket No. 124824.8143.WO01 wherein the parameters associated with the model include (1 ) a periodicity of the periodic motion, (2) an amplitude of the periodic motion, and (3) a temporal offset.

23. The method of claim 21 , wherein predicting the first time at which the positions of the pupil are associated with the minimum velocity comprises: determining a first angular position associated with the pupil at a first time step within the time period; determining a second angular position associated with the pupil at a second time step within the time period, wherein the second time step immediately succeeds the first time step; determining an angular difference between the first angular position and the second angular position; determining that the angular difference is less than or equal to a threshold difference; and based on determining that the angular difference is less than or equal to the threshold difference, determining the first time.

24. The method of claim 21 , wherein estimating the position at the first time comprises determining the horizontal angular coordinate and the vertical angular coordinate of the pupil at which the pupil is at a maximum angle from a central axis of pupil motion during the time period.

25. The method of claim 21 , wherein generating, based on the position, the retinal camera configuration comprises: determining, based on the position at the first time, a target angular region of illumination of the pupil; determining a subset of illuminants of a light source associated with the retinal camera, wherein activation of the subset of illuminants enables illumination of the target angular region; and generating the retinal camera configuration including identifiers of the subset of illuminants.PATENTAtorney Docket No. 124824.8143.WO0126. The method of claim 25, wherein generating, based on the position, the retinal camera configuration comprises: determining, based on the target angular region, a brightness value associated with the target angular region; and generating the retinal camera configuration including the brightness value.

27. The method of claim 26, wherein causing the capture of the second image of the eye comprises: generating the command including an indication of the identifiers of the subset of illuminants and an indication of the brightness value; and transmitting the command to the retinal camera to cause the retinal camera to capture the second image of the eye during illumination of the pupil by the subset of illuminants at the brightness value.

Images