Real-time refraction-correction for intraoperative oct imaging

Abstract

Described herein are systems and methods for refraction correction of OCT images configured for intraoperative use, including: a deep-learning model configured to receive a plurality of OCT images; and an imaging device configured to generate the plurality OCT images during an intraoperative procedure; wherein the deep-learning model is configured to map, in real-time, the plurality of OCT images to a two-dimensional vector field over a spatial domain of the plurality of OCT images; wherein the mapping is configured for real-time correction of refraction errors on the plurality of OCT images by employing Snell's law to generate a plurality of refraction-corrected OCT images via the deep-learning model. Other examples are described including features of the deep-learning model uniquely configured for interoperative usage such as an OCT image processing speed of 20Hz-50Hz.

Description

REAL-TIME REFRACTION-CORRECTION FOR INTRAOPERATIVE OCTIMAGINGPRIORITY CLAIM

[0001] This application claims priority to United States Provisional Patent Application Serial Nr. 63 / 725,503 filed November 26, 2024 entitled, “REAL-TIME REFRACTIONCORRECTION FOR INTRAOPERATIVE OCT IMAGING,” the contents of which are incorporated herein by reference.INCORPORATION BY REFERENCE

[0002] All publications and patent applications mentioned in this specification are herein incorporated by reference in their entirety to the same extent as if each individual publication or patent application was specifically and individually indicated to be incorporated by reference.FIELD

[0003] The present invention relates to the utility of OCT images obtained in intraoperative settings, and more particularly, to systems and method for real-time refraction correction of OCT images in intraoperative settings via a deep-learning model having fast processing speeds.BACKGROUND

[0004] Optical Coherence Tomography (OCT) scanners are widely used in ophthalmology to generate high-resolution volumetric images of the eye, assisting in the diagnosis of various pathologies. Intra-operatively, this technology has the potential to be utilized for determining the depth of anatomical landmarks and surgical instruments, enabling enhanced safety monitoring and automation. OCT systems can provide quantitative depth information, as the OCT datasets are depth-resolved and three-dimensional. However, OCT datasets may suffer from errors due to distortion inherent to the imaging optics, and refraction in the subject eye. Both of these effects can be numerically corrected, but this requires additional processing. While intra-surgical OCT offers surgeons a new dimension of information, a fully three- dimensional dataset of the entire surgical field takes much longer to acquire than the two- dimensional datasets of a traditional microscope. Thus, even for the fastest OCT systems,- 1 of 28 -SG Docket No.: 14843-710.600OCT acquisition necessarily has slower refresh rates and longer latencies than surgical microscope video.

[0005] Because OCT volume acquisition rates are relatively slow, they cannot selfcontextualize on time scales relevant to ophthalmic surgery. In other words, although a full volumetric OCT dataset contains enough information to identify key surgical landmarks (such as surgical tools and major anatomical features), the rate at which full volumetric OCT datasets can be acquired, at least with current OCT technologies, is not fast enough to provide a surgeon with feedback on a reasonable timescale to facilitate performing ophthalmic surgery using OCT data alone. As a result, intraoperative OCT systems may operate simultaneously with traditional surgical microscopes, relying on some form of digital recording of surgical microscope video and then registering the OCT volume to the surgical microscope images.

[0006] OCT operates based on low-coherence interferometry, measuring the optical path length difference between the reference and sample arms, typically assuming a constant refractive index and straight-line light propagation.

[0007] However, these assumptions do not hold true when imaging the eye. The refractive indices of the cornea and aqueous humor are approximately 30% higher than that of air, and the curved surface of the cornea causes light paths to deviate due to refraction. This discrepancy introduces significant localization errors, which can pose safety risks, particularly for automated surgical procedures. While computational techniques exist to correct for this refraction error, none have been adapted for intra-operative application.

[0008] Existing solutions that address refraction in OCT images typically use Snell’s law to model light paths through media with differing refractive indices. The correction involves two components: a geometric transformation and a speed adjustment. The geometric transformation models the light path as piecewise-linear, with direction changes at points where light enters and exits the cornea. The speed adjustment accounts for the variation in light propagation speed across different media. Together, these methods enable tracing of a corrected light path for each pixel in the image, yielding a more accurate representation of anatomical features.

[0009] Problematically, existing techniques for refraction correction of OCT images are not practical for intraoperative use given their relatively slow processing and image scan cycle times and reliance on external algorithms that do not take into consideration intraoperative changes in the eye’s geometry.- 2 of 28 -SG Docket No.: 14843-710.600

[0010] As such, there remains a need for systems and methods that can both estimate anatomical geometry and apply OCT refraction correction intraoperatively in real-time using suitable learning models.SUMMARY OF THE DISCLOSURE

[0011] Described herein is a system for refraction correction of OCT images configured for intraoperative use, having: a refraction deep-learning model configured to receive a plurality of OCT images; a feature deep-learning model configured to receive a plurality of OCT images; and an imaging device configured to generate the plurality OCT images during an intraoperative procedure; in which the feature deep-learning model is configured to map, in real-time, the plurality of OCT images to spatial features; in which the refraction deeplearning model is configured to map, in real-time, the plurality of OCT images to a two- dimensional vector field over a spatial domain of the plurality of OCT images; in which the mapping is configured for real-time correction of refraction errors on the plurality of OCT images to generate a plurality of refraction-corrected features produced by the feature deeplearning model via the refraction deep-leaming model.

[0012] According to certain examples of the system, the vector field may encode a displacement required to correct for distortions caused by light refraction for each pixel in each of the plurality of OCT images.

[0013] According to certain examples of the system, the vector field may be configured to serve as a lookup table by repositioning a detected feature at a pixel location p=(x,y) in each of the plurality of OCT images to pixel location p+v; in which v is a vector associated with pixel location p.

[0014] According to certain examples of the system, the deep-learning model may be configured to be trained on a plurality of OCT training images; in which the plurality of OCT training images are one or more of annotated manually and annotated via automation.

[0015] According to certain examples of the system, the deep learning model may employ Snell based techniques to generate training targets. In one exemplary optional implementation, the correction of refraction errors may employ Snell’s law including modeling a light path of a plurality of light rays traversing two or more media; in which the media may have differing refractive indices; further in which a deflection of the plurality of light rays at interfaces between the two of the two or more media may be computed and adjusted for changes in propagation speed of the plurality of light rays to obtain a- 3 of 28 -SG Docket No.: 14843-710.600displacement field correlating to a true optical path of the plurality of light rays. Advantageously, this method finds particular utility is the various tissues of the eye.

[0016] According to certain examples of the system, the deep-learning model may be a neural network configured to output displacement fields, optionally implemented using encoderdecoder structures interconnected via skip connections. In certain examples, the neural network may utilize a first output map to encode a horizontal displacement for correcting refraction and a second map encoding a vertical displacement for correcting refraction to generate the vector field.

[0017] According to certain examples of the system, the imaging device may be an OCT scanner configured to continuously acquire the plurality of OCT images at a frame rate of 20Hz- 200Hz; in which the plurality of OCT images are B-scans.

[0018] According to certain examples of the system, the plurality of OCT images may be configured to be processed in parallel via: (i) a feature extraction model configured to process the plurality of OCT images to identify anatomical structures and surgical tools, and (ii) a refraction correction model configured for real-time estimating of refraction errors on the plurality of OCT images; in which the refraction correction model is the deep-learning model; further in which the feature extraction model and deep-learning model operate at a processing speed of 20Hz-50Hz configured for intraoperative usage.

[0019] According to certain examples of the system, the feature extraction model may be configured to output a list of points of interest representing the anatomical structure and surgical tools; in which the refraction correction model may be configured to output the vector field, in which the vector field may be configured to provide a displacement vector indicating a refraction correction for each pixel in each of the plurality of OCT images.

[0020] According to certain examples of the system, correction features may be configured to be generated by applying the displacement vector for each pixel in each of the plurality of OCT images to an associated point of interest from the list of points of interest; further in which the correction features may be configured for one or more of: path planning tasks, visualization and use by virtual fixtures.

[0021] According to certain examples of the system, generating and refraction correction of the plurality of OCT images may be applied to one or more of: an eye, an esophagus, and other anatomical regions suitable for OCT scanning.

[0022] According to certain examples of the system, the imaging device may be a- 4 of 28 -SG Docket No.: 14843-710.600multimodal imaging device including OCT and one or more of: digital microscope (DM) and ultrasound (U / S).

[0023] According to certain examples of the system, the generating and refraction correction of the plurality of OCT images may be applied to diagnosis and assessment of: Barrett’s esophagus, tumor size measurement, scar reduction and formation measurement, and radiation treatment.

[0024] In other aspects, there is a method of real-time refraction correction of OCT images in an intraoperative setting, including: training a deep-learning model based on a dataset of OCT images; intraoperatively generating a plurality of OCT images using an imaging device; mapping, via the deep-learning model, the plurality of OCT images to a two-dimensional vector field over a spatial domain of the plurality of OCT images; training the deep-learning model using refraction corrections computed by Snell’s law; correcting the plurality of OCT images in real-time based on an output of the deep-learning model; and generating a plurality of real-time refraction-corrected OCT images from the plurality of OCT images.

[0025] According to certain examples, the method may further include repositioning a detected feature at a pixel location p=(x,y) in each of the plurality of OCT images to pixel location p+v; in which v is a vector associated with each pixel location p.

[0026] According to certain examples, the method further includes processing the plurality of OCT images through an encoder-decoder structure of a neural network interconnected via skip connections and encoding, via a first and second output map of the neural network, a horizontal and vertical displacement, respectively, for refraction correction of the plurality of OCT images.

[0027] According to certain examples, the method may further include acquiring the plurality of OCT images at a frame rate of 20Hz-200Hz.

[0028] According to certain examples, the method may further include processing the plurality of OCT images in parallel via: processing the plurality of OCT images via a feature extraction model to identify anatomical structures and surgical tools; estimating refraction errors on the plurality of OCT images in real-time via a refraction correction model; in which the refraction correction model is the deep-leaming model; and operating the feature extraction model and deep-learning model for intraoperative usage at a processing speed of 20Hz-50Hz.

[0029] According to certain examples, the method may further include outputting, via the feature extraction model, a list of points of interest representing the anatomical structure and surgical tools; outputting, via the refraction correction model, the vector field; and generating- 5 of 28 -SG Docket No.: 14843-710.600correction features configured for path planning tasks by applying the vector field to the points of interest representing the anatomical structure and surgical tools.

[0030] All and each of the methods and apparatuses described herein, in any combination, are herein contemplated and can be used to achieve the benefits as described herein.

[0031] In one embodiment there is provided a system for refraction correction of OCT images configured for intraoperative use including a deep-learning model configured to receive a plurality of OCT images: and an imaging device configured to generate the plurality OCT images during an intraoperative procedure. In one aspect, the deep-learning model is configured to map, in real-time, the plurality of OCT images to a two-dimensional vector field over a spatial domain of the plurality of OCT images and the mapping is configured for real-time correction of refraction errors on the plurality of OCT images to generate a plurality of refraction-corrected OCT images via the deep-learning model. In one variation of the system, the vector field encodes a displacement required to correct for distortions caused by light refraction for each pixel in each of the plurality of OCT images. In yet another variation of the system, the vector field is configured to serve as a lookup table by repositioning a detected feature at a pixel location p=(x,y) in each of the plurality of OCT images to pixel location p+v; wherein v is a vector associated with pixel location p. In one aspect, the deeplearning model is configured to be trained on a plurality of OCT training images; wherein the plurality of OCT training images are one or more of annotated manually and annotated via automation or the deep-learning model is configured to utilize Snell-based techniques to generate training targets. In still other variations, the process for correcting refraction errors includes determining surface normals at a corneal interface from OCT data; modeling incident rays traversing multiple ocular media of differing refractive index; applying Snell's law at each interface to compute refracted ray directions; adjusting each refracted ray for differences in propagation speed between the media; and generating a displacement field correlating to a time optical path of each ray.

[0032] In still other variations of the system, the deep-learning model is a neural network having an encoder-decoder structure interconnected via skip connections; further wherein the neural network utilizes a first output map to encode a horizontal displacement for correcting refraction and a second map encoding a vertical displacement for correcting refraction to generate the vector field. In yet other alterative implementations, the imaging device is an OCT scanner configured to continuously acquire the plurality of OCT images at a frame rate of 20Hz-200Hz; wherein the plurality of OCT images are B-scans. In another aspect of the system, the plurality of OCT images are configured to be processed in parallel via: (i) a- 6 of 28 -SG Docket No.: 14843-710.600feature extraction model configured to process the plurality of OCT images to identify anatomical structures and surgical tools, and (ii) a refraction correction model configured for real-time estimating of refraction errors on the plurality of OCT images; wherein the refraction correction model is the deep-learning model; further wherein the feature extraction model and deep-learning model operate at a processing speed of 20Hz-50Hz configured for intraoperative usage. Additionally or optionally, the system may include a version where the feature extraction model is configured to output a list of points of interest representing the anatomical structure and surgical tools; wherein the refraction correction model is configured to output the vector field, wherein the vector field is configured to provide a displacement vector indicating a refraction correction for each pixel in each of the plurality of OCT images. In another alternative there may be correction features configured to be generated by applying the displacement vector for each pixel in each of the plurality of OCT images to an associated point of interest from the list of points of interest; further wherein the correction features are configured for one or more of: path planning tasks, visualization, and use by virtual fixtures. In still other alternatives of the above, the steps of generating and refraction correction of the plurality of OCT images is applied to one or more of: an eye, an esophagus, and other anatomical regions suitable for OCT scanning. In another aspect of the system, the generating and refraction correction of the plurality of OCT images is applied to the movement of a tool relative to a portion of a posterior capsule of a lens. In another version, the imaging device is a multimodal imaging device including OCT and one or more of: digital microscope (DM) and ultrasound (U / S). In still another variation of the system, the generating and refraction correction of the plurality of OCT images is applied to diagnosis and assessment of: Barrett's esophagus, tumor size measurement, scar reduction and formation measurement, or a radiation treatment.

[0033] In another alterative embodiment, there is a method of real-time refraction correction of OCT images in an intraoperative setting including training a refraction deep-learning model based on a dataset of OCT images; training a feature deep-learning model based on a dataset of OCT images; intraoperatively generating a plurality of OCT images using an imaging device. There are also steps of mapping, via the refraction deep-learning model, the plurality of OCT images to a two-dimensional vector field over a spatial domain of the plurality of OCT images; and mapping, via the feature deep-learning model, the plurality of OCT images to points of interest representing anatomical features and surgical tools. Additionally, there are steps for training the deep-learning model using refraction corrections computed by Snell’s law; correcting the points of interest in real-time based on an output of- 7 of 28 -SG Docket No.: 14843-710.600the deep-learning model; and generating a plurality of real-time refraction-corrected features from the plurality of OCT images. In one aspect of the method, there is also a step of repositioning a detected feature at a pixel location p=(x,y) in each of the plurality of OCT images to pixel location p+v; wherein v is a vector associated with each pixel location p. In another variation, there is a step of processing the plurality of OCT images through an encoder-decoder structure of a neural network interconnected via skip connections; and encoding, via a first and second output map of the neural network, a horizontal and vertical displacement, respectively, for refraction correction of the plurality of OCT images. In some embodiments, the step of acquiring the plurality of OCT images is performed at a frame rate of 20-200Hz. In still other aspects of the method, there are steps of processing the plurality of OCT images in parallel via: processing the plurality of OCT images via a feature extraction model to identify anatomical structures and surgical tools; estimating refraction errors on the plurality of OCT images in real-time via a refraction correction model; wherein the refraction correction model is the deep-learning model; or operating the feature extraction model and deep-learning model for intraoperative usage at a processing speed of 20Hz-50Hz.

[0034] In still further variations of the method, there are steps of outputting, via the feature extraction model, a list of points of interest representing the anatomical structure and surgical tools; outputting, via the refraction correction model, the vector field; and generating correction features configured for path planning tasks by applying the vector field to the points of interest representing the anatomical structure and surgical tools. In still further variations of the above, the OCT images are of a portion of an eye.BRIEF DESCRIPTION OF THE DRAWINGS

[0035] A better understanding of the features and advantages of the methods and apparatuses described herein will be obtained by reference to the following detailed description that sets forth illustrative embodiments, and the accompanying drawings of which:

[0036] FIG. 1 depicts registration between imaging modalities.

[0037] FIGS. 2A-2B depict a ground-truth OCT image and a manually annotated image for correction, respectively.

[0038] FIG. 3 is a schematic for an exemplary light ray traveling between two media with differing refractive indices.

[0039] FIG. 4 is an exemplary architecture for image estimation and correction having an encoder-decoder structure.- 8 of 28 -SG Docket No.: 14843-710.600

[0040] FIG. 5 is an exemplary workflow of OCT feature extraction and refraction correction models configured for intra-operative use.

[0041] FIG. 6 is a flowchart depicting a method for intra-operative OCT refraction correction.DETAILED DESCRIPTION

[0042] For examples of robotic systems that may be used with the systems and methods described herein, see U.S. Patent 12,396,810 entitled SYSTEM AND METHOD FOR AUTOMATED IMAGE-GUIDED ROBOTIC INTRAOCULAR SURGERY filed on November 3, 2020, incorporated herein by reference in its entirety and with specific reference to FIGs. IB, 1C, 2A-3B and Col. 3 line 7 to Col. 6 line 52 and Col. 6 line 54 to Col. 7 line 67 and see as well International Application No. PCT / US2024 / 038892 entitled, “ROBOTIC ASSISTED OPHTHALMIC SURGERY SYSTEM” filed on July 19, 2024, the entire contents of which are incorporated by reference herein and with specific reference to FIGs. 1- 6 and paragraphs [0045-0106] and [0137-0146].

[0043] For examples of multi-modal imaging systems that may be used with the systems and methods described herein, see International Patent Application No. PCT / US2025 / 042484 entitled SYSTEMS AND METHODS FOR REGISTRATION OF MULTIMODAL IMAGING SYSTEMS INCORPORATED IN SURGICAL MICROSCOPE filed on August 18, 2025, the entire contents of which are incorporated by reference herein with specific reference to paragraphs [0089-0091], [0095-0123] and FIGs 2C, 2D, 3, 4, 5, 6, 7A and 7B. FIG. 1 depicts registration between imaging modalities.

[0044] Shown here is a field of view for OCT 124 and telecentric digital microscope 110, as well as registration 199 between the imaging system 102 and a robotic system 103. In certain examples, there may be generalized setup for a robotic system having a surgical robot based on the systems described in U.S. Patent 12,396,810 entitled SYSTEM AND METHOD FOR AUTOMATED IMAGE-GUIDED ROBOTIC INTRAOCULAR SURGERY filed on November 3, 2020 and International Application No. PCT / US2024 / 038892 entitled ROBOTIC ASSISTED OPHTHALMIC SURGERY SYSTEM filed on July 19, 2024, as indicated above.OCT Targeting via DM / Al

[0045] In some embodiments, OCT scan patterns may be generated from the data originating in the telecentric microscope channel. The data of the microscope channel is first processed to detect anatomical features, through traditional image-processing, artificial-intelligence, or- 9 of 28 -SG Docket No.: 14843-710.600other such means. The coordinates of these detected features in the microscope channel data can be translated into OCT scan pattern coordinates of any arbitrary pattern. The generated scan pattern can be used to acquire OCT data at the coordinates specified in the scan pattern.

[0046] As described above in International Patent Application No. PCT / US2025 / 042484 entitled, “SYSTEMS AND METHODS FOR REGISTRATION OF MULTIMODAL IMAGING SYSTEMS INCORPORATED IN SURGICAL MICROSCOPE” filed on August 18, 2025, refraction introduces errors in the depth measurements made by the OCT, which registration cannot account for since these errors are induced by the anatomy of the eye which is not known a priori. This application relates to the benefits of combining OCT and digital microscope (DM), such as OCT 124 and a telecentric digital microscope 110 described in FIG. 1. However, the uncorrected OCT such as OCT 124 induces refraction errors into this multimodal imaging process and registration 199 between the OCT 124 and digital microscope 110. The techniques in registration 199 may be improved by modifying the process to include one or more of the OCT correction techniques described herein.

[0047] Shown here is a field of view for OCT 124 and telecentric digital microscope 110, as well as registration 199 between the imaging system 102 and a robotic system 103.

[0048] In one aspect, the system applies an OCT refraction-correction method that accounts for changes in light direction and propagation speed as the OCT beam traverses ocular tissues. By modeling these refraction effects through Snell’s law, the system produces corrected representations of anatomical structures suitable for intra-operative robotic guidance.

[0049] The application of Snell’s law requires prior knowledge of the eye’s geometry, which determines the interfaces where refractions occur. Methods for estimating comeal geometry in OCT B -scans exist, including mathematical morphology combined with polynomial regression and deep learning-based segmentation. However, these techniques are typically treated as separate from the refraction correction process. This dependency on external algorithms introduces a vulnerability, as any inaccuracies in geometry estimation can propagate through subsequent corrections. Furthermore, current solutions assume that the eye’s geometry remains constant throughout the time required for the correction algorithm to execute, which can range from seconds to minutes. This assumption is not feasible for intraoperative use, where the geometry of the eye is continuously altered by physical interactions with surgical instruments.

[0050] The present disclosure addresses these three limitations by introducing a learningbased method that simultaneously estimates the eye's geometry and refraction errors in real- 10 of 28 -SG Docket No.: 14843-710.600time, implicitly utilizing Snell's law. This approach enables accurate, adaptive correction suited for intra-operative environments, even as the geometry of the eye changes dynamically due to surgical manipulation.

[0051] To achieve this, the system utilizes two complementary deep-learning models: a refraction model, trained using displacement fields derived from Snell’s law, and a feature model, trained to identify anatomical structures and surgical tools within OCT images. During intraoperative use, the imaging device acquires continuous OCT B-scans, which are processed concurrently by these models. The refraction model maps each OCT B-scan to a two-dimensional vector field representing the pixel-wise displacement required to correct refractive distortion. In parallel, the feature model localizes points of interest within the same spatial domain. By applying the refraction model’s output to the detected features in real time, the system generates refraction-corrected anatomical and tool locations, which are usable for downstream functions such as robotic path planning, intraoperative visualization, and the enforcement of virtual-fixture safety constraints.

[0052] Described herein is a learning-based method that simultaneously estimates the geometry of the eye and corrects for refraction errors in real time, leveraging the principles of Snell’s law implicitly. The method employs a deep-learning approach to map an OCT B- scan — represented as a 2-dimensional array or grayscale image — to a 2-dimensional vector field over the same spatial domain. This vector field encodes, for each pixel in the image, the displacement required to correct for the distortions caused by light refraction. Essentially, the vector field serves as a lookup table: a detected feature at a pixel location p=(x,y) in the original image is repositioned to p+v, where v is the vector associated with p. This direct mapping enables efficient real-time correction of refraction errors, crucial for intra-operative use.

[0053] The following sections outline the dataset generation process, model architecture, training protocol, and considerations for intra-operative deployment.

[0054] Dataset creation

[0055] To train and validate an exemplary model, tens of thousands of OCT B-scans may be collected using the Polaris OCT system, capturing data before, during, and after cataract surgery. A significant subset of these B-scans may be meticulously annotated by experienced annotators, who manually segment the comeal geometry and identify other key anatomical features of the eye. The annotations may then undergo several rounds of quality control, ensuring precise and accurate segmentations. The resulting dataset captures the variability of- 11 of 28 -SG Docket No.: 14843-710.600the eye’s anatomy, particularly under intra-operative conditions where typical assumptions of corneal smoothness may be violated due to the presence of surgical instruments.

[0056] FIGS. 2A-2B depict a ground-troth OCT image and a manually annotated image for correction, respectively.

[0057] FIG. 2A depicts an exemplary OCT B-scan acquired on the Polaris system. For examples of robotic systems that may be used with the Polaris system, see International Application No. PCT / US2024 / 038892 entitled ROBOTIC ASSISTED OPHTHALMIC SURGERY SYSTEM filed on July 19, 2024 discussed above and with specific reference here to the benefit of OCT refraction correction during operations in surgeon supervised mode or automated mode in paragraphs [0144-0146] along with use of OCT data by the surgical path planner. Distinctive parts of the eye anatomy can be seen, including: docking 202, which may be vertical bars flanking the eye that may be part of an eye fixation system, cornea 204, sclera 206, anterior lens capsule 208, iris 210, anterior surface of the lens 212, and posterior lens capsule 214.

[0058] FIG. 2B depicts a manually annotated corneal region for correction of the OCT B- scan of FIG. 2A. The top 216 and bottom 218 of the region may define boundaries of the cornea 204 extracted from FIG. 2A, and may be located at the interfaces where Snell’s Law applies and light is refracted.

[0059] The annotated corneal geometry provides ground troth data necessary for refraction correction using methods described in existing literature. In this offline preparation phase, the absence of time constraints allows us to evaluate and verify the accuracy of the corneal geometry estimation thoroughly. This step enables us to generate corrected mappings for each pixel without the real-time limitations present during surgery.

[0060] For each pixel in the B-scan image, we compute a corrected position based on the true light path, which is adjusted using Snell’s law. The difference between the original and corrected positions forms a 2-dimensional displacement vector anchored at the original pixel location. Applying this vector to the pixel effectively corrects for the refraction error, allowing us to create a corrected version of the OCT image.

[0061] The output of this preprocessing step is a comprehensive dataset of B-scans with associated displacement fields, which quantify the displacement required at each pixel to account for the effects of refraction. This dataset is used to train the model to learn the underlying mappings required for real-time refraction correction.

[0062] To support the generation of the displacement fields used for refraction correction, the preprocessing step reconstructs a detailed geometric model of the cornea from segmented- 12 of 28 -SG Docket No.: 14843-710.600OCT volumes. Raw and segmented B-scan images are stacked into a 3D representation, from which comeal boundaries are extracted using gradient-based edge-detection methods that identify the anterior and posterior interfaces with high spatial precision. These boundaries are separated to isolate the posterior corneal surface from the anterior comeal surface. From this reconstructed geometry, surface normals are computed across the anterior and posterior corneal interfaces using numerical gradient estimation methods that produce stable, unitlength normals. These normals, together with the known refractive indices of air, comeal tissue, and aqueous humor, are used to calculate refracted ray directions at each surface point through a vectorized formulation of Snell’s law. By applying this computation systematically across the comeal volume, the system produces accurate estimates of the true light paths within the eye. The difference between these refracted paths and the paths assumed in uncorrected OCT imaging yields the displacement fields that form the basis of the refractionestimation model.

[0063] Refraction correction using Snell’s law

[0064] FIG. 3 is a schematic for an exemplary light ray traveling between two media 302 / 304 with differing refractive indices.

[0065] For each vertical column in the B-scan, the traditional assumptions of straight-line light propagation and constant speed are corrected by applying Snell’s law. This involves modeling the light path of an incident ray 310 as it traverses interfaces between media with differing refractive indices 306, such as from air 302 to the cornea 304, or in other examples from the cornea 302 to the aqueous humor 304. By computing the angle of incidence 311 from normal 308 of the incident ray 310 and deflection of ray 310 by an angle of refraction 313 from normal 308 (resulting in a refracted ray 312) at these interfaces and adjusting for changes in propagation speed, an accurate displacement field that reflects the true optical paths may be obtained. This displacement field represents, for each pixel in the B-scan, the offset between the apparent location of a structure in the uncorrected OCT image and its true location under a physically accurate refraction model. These Snell-based calculations serve both as the basis for generating training targets for the deep-learning refraction model and as a reference for validating real-time corrections. In some exemplary implementations, the method generates the displacement fields at an OCT sensor acquisition rate that may range from 20 - 200 Hz, or from 30 - 50 Hz, or 30 - 75 Hz, or 50 - 100 Hz.

[0066] Architecture Choice

[0067] FIG. 4 is an exemplary architecture for image segmentation and correction having an encoder-decoder structure.- 13 of 28 -SG Docket No.: 14843-710.600

[0068] In certain examples of the methods described herein, such a U-Net architecture, a widely adopted model for pixel-level tasks in medical imaging, may be selected. It should be appreciated that a variety of neural network architectures may be selected for image estimation / segmentation / correction.

[0069] The U-Net architecture is a neural network designed to effectively capture both detailed local features and broader contextual information within an image, making it particularly effective for tasks requiring precise spatial predictions.

[0070] The model operates on an OCT B-scan image as input 402 and produces two output maps of identical spatial dimensions 404 (displayed as a combined visualization). Its structure is composed of two primary components — an encoder 406 and a decoder 408 — which are interconnected via "skip connections" and concatenation represented as central horizontal arrows 410.

[0071] Encoder: The encoder 406, illustrated by the blocks on the left of FIG. 4, processes the input image 402 through successive layers of convolutional operations. As the image 402 progresses through the encoder 406, its spatial resolution is systematically reduced. This approach enables the model to generate a hierarchical representation of the image, capturing increasingly abstract features without an excessive increase in the model's parameter count. At the shallowest layers, the encoder 406 identifies fundamental features such as edges and gradients; at deeper layers, it extracts more complex patterns such as shapes and object-level representations.

[0072] Decoder: The decoder 408, represented by the blocks on the right of FIG. 4, reconstructs the processed information into the final output maps. It achieves this by utilizing both the high-level encoded representation from the deepest layer of the encoder 406 and the intermediate features transmitted through the skip connections 410. These skip connections 410 ensure that finer details from earlier encoding stages are retained and incorporated into the final output, thereby enhancing spatial accuracy.

[0073] Through this encoder 406-decoder 408 structure, augmented by skip connections 410, the U-Net effectively combines local feature precision with global contextual awareness to generate high-quality spatial predictions, such as the displacement field referenced above.

[0074] In certain examples, a first output map encodes the horizontal displacement (3x3 convolution (ReLU) 412 / 1x1 convolution (sigmoid) 414 required to correct for refraction, while a second map encodes the vertical displacement 2x2 max pooling 416 / 2x2 transposed convolution 418. These displacement maps represent the adjustments needed for each pixel, providing a correction vector that accounts for the light path deviations induced by refraction.- 14 of 28 -SG Docket No.: 14843-710.600Importantly, the values in these maps are scaled to be invariant to the input image size, ensuring the model’s generalizability across different resolutions and imaging setups.

[0075] Training Methodology

[0076] The model may be trained using a mean squared error (MSE) loss function, which Measures the difference between the predicted displacement maps and the ground truth displacement fields derived from our annotated dataset. The MSE loss penalizes large deviations, encouraging the model to learn accurate pixel-wise corrections. During training, the dataset of OCT B-scans and corresponding displacement fields may be randomly shuffled at the eye level and split into training and validation sets. This guarantees that the performance of the model is evaluated on previously unseen eyes, allowing for assessment of the model to both estimate the eye geometry along with the errors induced by refraction. Both photometric and geometric data augmentation techniques may be employed to enhance the robustness of the model and prevent overfitting. Specifically, actions include:

[0077] Modifying the image contrast, brightness and gamma; and

[0078] Cropping and resizing the image (such as input image 402);

[0079] The U-Net may be optimized using an Adam optimizer, with a learning rate schedule That adjusts based on the validation performance. Convergence may be monitored using the validation loss and employ early stopping to avoid overfitting, ensuring the model retains generalizability for intra-operative use.

[0080] Intra-operative Use

[0081] FIG. 5 is an exemplary workflow of OCT feature extraction and refraction correction models configured for intra-operative use.

[0082] During surgery, the OCT scanner may continuously acquire B-scans at a frame rate of 20Hz-200Hz. In certain examples, processing speed of acquired OCT B-scans may be 20Hz- 50Hz for OCT B-scans acquired at a rate larger than ~50Hz. These B-scans may be processed in parallel by two models:

[0083] Feature Extraction Models: These may include feature model 502 that identifies anatomical structures and surgical tools in the OCT images. Shown here is the segmented image 503, where pixels belonging to the cornea, anterior lens surface, iris and sclera may be assigned different values.

[0084] Refraction Estimation Model: In certain examples, refraction estimation or refraction model 504 may be the deep-learning model described in FIG. 4, which may compensate for distortions caused by light refraction. The output 505 associates each pixel below the anterior surface of the cornea to a nonzero vector (represented by arrows 506) which represents the- 15 of 28 -SG Docket No.: 14843-710.600distance between the location of the object images by that pixel with the location that the imaged object in the scene would have been if refraction had been taken into account. The normal 508 and refracted 509 vectors to the anterior and posterior surfaces of the cornea are also displayed.

[0085] Both segmented model 502 and correction model 504 may operate at a processing speed of 20Hz-50Hz, meeting the real-time latency requirements for intra-operative applications. The parallel execution may ensure that each B-scan is analyzed efficiently, providing timely information to assist in surgical decision-making.

[0086] In certain examples, the output of this parallel processing may consist of two complementary components:

[0087] A list of points of interest, representing the locations of key anatomical landmarks and surgical instruments as identified by the feature extraction models 502 / 504. These points are critical for planning the robot's motion, but initially do not account for the effects of refraction; and

[0088] A vector field, generated by the refraction correction model 504, which provides a displacement vector for each pixel in the image. This vector may indicate the adjustment needed to account for the altered light paths due to refraction.

[0089] To obtain accurate feature locations, the vector field may be used as a lookup table. For each point of interest, the associated displacement vector may be retrieved and applied in real time. This correction step 507, performed in milliseconds, can ensure precise localization of anatomical and surgical features even in the presence of refraction-induced distortions.The corrected feature 507 locations may then be used for path planning tasks, enabling safe and accurate robotic motion during the procedure.

[0090] FIG. 6 is a flowchart depicting a method for real-time refraction correction of OCT images in an intraoperative setting 600.

[0091] Method 600 begins at block 605 with training a deep-learning model based on a dataset of OCT images.

[0092] At block 610, method 600 continues with intraoperatively generating a plurality of OCT images using an imaging device.

[0093] At block 615, method 600 includes mapping, via a refraction model, the plurality of OCT images to a two-dimensional vector field over the spatial domain of the plurality of OCT images.

[0094] Next, at block 620, method 600 continues with correcting the refraction errors of the- 16 of 28 -SG Docket No.: 14843-710.600features obtained from the plurality of OCT images by applying the two-dimensional vector field in real-time.

[0095] Method 600 concludes at block 625 with generating a plurality of refraction-corrected features from the plurality of OCT images.

[0096] According to certain examples, method 600 further includes repositioning a detected feature at a pixel location p=(x,y) in each of the plurality of OCT images to pixel location p+v; in which v is a vector associated with each pixel location p.

[0097] According to certain examples, method 600 further includes processing the plurality of OCT images through an encoder-decoder structure of a U-Net architecture neural network interconnected via skip connections; and encoding, via a first and second output map of the U-Net architecture, a horizontal and vertical displacement, respectively, for refraction correction of the plurality of OCT images.

[0098] According to certain examples, method 600 further includes acquiring the plurality of OCT images at a frame rate of 20Hz-200Hz.

[0099] According to certain examples, method 600 further includes processing the plurality of OCT images in parallel via: segmenting the plurality of OCT images via a feature extraction model to identify anatomical structures and surgical tools; correcting refraction errors on the plurality of OCT images in real-time via a refraction correction model; in which the refraction correction model is the deep-leaming model; and operating the feature extraction model and deep-learning model for intraoperative usage at a processing speed of 20Hz-50Hz.

[0100] According to certain examples, method 600 further includes outputting, via the feature extraction model, a list of points of interest representing the anatomical structure and surgical tools; outputting, via the refraction correction model, the vector field; generating, via the vector field, a displacement vector indicating a refraction correction for each pixel in each of the plurality of OCT images; and generating correction features configured for path planning tasks by applying the displacement vector for each pixel in each of the plurality of OCT images to an associated point of interest from the list of points of interest.

[0101] In certain embodiments, any of the foregoing steps may be combined, reordered, or executed concurrently, and may be used in conjunction with additional features described elsewhere in this Specification.

[0102] Advantages of Real-Time Refraction-Corrected OCT

[0103] Some conventional refraction correction and extraction techniques such as those described in Mingtao Zhao, Anthony N Kuo, Joseph A Izatt, "3D refraction correction- 17 of 28 -SG Docket No.: 14843-710.600and extraction of clinical parameters from spectral domain optical coherence tomography of the cornea," Opt. Express 18, 8923-8936 (2010); A. N. Kuo, M. Zhao, J. A. Izatt: "Corneal Aberration Measurement With Three-dimensional Refraction Correction for High-speed Spectral Domain Optical Coherence Tomography," Invest. Ophthalmol. Vis. Sci. 2009;50(13):3671; Yuan Tian, Mark Draelos, Ryan P. McNabb, Kris Hauser, Anthony N. Kuo, and Joseph A. Izatt, "Optical coherence tomography refraction and optical path length correction for image-guided corneal surgery," Biomed. Opt. Express 13, 5035-5049 (2022); and Brenton Keller, Mark Draelos, Gao Tang, Sina Farsiu, Anthony N. Kuo, Kris Hauser, and Joseph A. Izatt, "Real-time corneal segmentation and 3D needle tracking in intrasurgical OCT," Biomed. Opt. Express 9, 2716-2732 (2018) fall short because of their low processing rates. Such techniques as a result are only well suited for pre-operative and / or ex -vivo uses and are not suited to intra-operative use as they have imaging / B-scan cycles on the order of several seconds to a minute as compared to the much shorter imaging update cycle times / frequencies on the order of 20Hz-50Hz needed for real-time OCT imaging of anatomy for intraoperative and diagnostic uses. Furthermore, the refraction correction described in some of these conventional techniques is limited to a single point such as a tool tip.

[0104] In contrast, the systems and methods described herein are much better suited for intra-operative use by allowing for imaging cycles including B-scan cycles of 20Hz-50Hz (up to 50 times faster than the cited conventional techniques). Furthermore, the systems and methods contemplated herein may also allow for faster real-time imaging by decreasing processing load during certain portions of a procedure by limiting the imaging field to a safety margin around a surgical tool tip, thus allowing for refraction correction beyond a single point.

[0105] In one specific implementation, an embodiment of the inventive method is adapted and configured to address the challenge and safety concern of contact between tools and the posterior capsule of the eye (PC). Tools touching the PC can rupture it which leads to the collapse of the eye. Without refraction correction, in conventional approaches, the distance between the tool and the PC may be overestimated by as much as 30%. As such, without the advantageous imaging corrections described herein tool to PC distance may be overestimated where the system indicates a spacing further than the actual distance.

[0106] While described in various implementations for use in imaging of the eye, the inventive methods and techniques are not so limited. In various alternative embodiments the OCT correction and image combining methods may be applied to other portions of the eye as well as other anatomical targets of medical interest. Examples include regions and pathology- 18 of 28 -SG Docket No.: 14843-710.600with suitable tissue and transparent properties such as Barrett' s esophagus, in which OCT may provide images of the surface of the esophagus and its lining as well as subsurface structures, or intravascular imaging in which blood and arterial walls have different refractive indices.

[0107] Additionally or optionally, the techniques described herein may be used to improve OCT results for diagnosis and assessment errors in measurements such as for tumor shrinkage, scar formation, and scar reduction during healing in addition to real time correction of OCT errors.

[0108] In yet other examples, other dual or multimodal imaging modalities may be used with OCT such as digital microscope (DM), ultrasound, X-rays, or other surgical imaging modalities especially those for use in the eye.

[0109] In still other examples, OCT correction may be used in the context of other therapies such as guidance for radiation treatment.

[0110] It should be appreciated that all combinations of the foregoing concepts and additional concepts discussed in greater detail below (provided such concepts are not mutually inconsistent) are contemplated as being part of the inventive subject matter disclosed herein and may be used to achieve the benefits described herein.

[0111] The process parameters and sequence of steps described and / or illustrated herein are given by way of example only and can be varied as desired. For example, while the steps illustrated and / or described herein may be shown or discussed in a particular order, these steps do not necessarily need to be performed in the order illustrated or discussed. The various example methods described and / or illustrated herein may also omit one or more of the steps described or illustrated herein or include additional steps in addition to those disclosed.

[0112] Any of the methods (including user interfaces) described herein may be implemented as software, hardware or firmware, and may be described as a non-transitory computer-readable storage medium storing a set of instructions capable of being executed by a processor (e.g., computer, tablet, smartphone, etc.), that when executed by the processor causes the processor to control perform any of the steps, including but not limited to: displaying, communicating with the user, analyzing, modifying parameters (including timing, frequency, intensity, etc.), determining, alerting, or the like. For example, any of the methods described herein may be performed, at least in part, by an apparatus including one or more processors having a memory storing a non-transitory computer-readable storage medium storing a set of instructions for the processes(s) of the method.- 19 of 28 -SG Docket No.: 14843-710.600

[0113] While various embodiments have been described and / or illustrated herein in the context of fully functional computing systems, one or more of these example embodiments may be distributed as a program product in a variety of forms, regardless of the particular type of computer-readable media used to actually carry out the distribution. The embodiments disclosed herein may also be implemented using software modules that perform certain tasks. These software modules may include script, batch, or other executable files that may be stored on a computer-readable storage medium or in a computing system. In some embodiments, these software modules may configure a computing system to perform one or more of the example embodiments disclosed herein.

[0114] The various exemplary methods described and / or illustrated herein may also omit one or more of the steps described or illustrated herein or comprise additional steps in addition to those disclosed. Further, a step of any method as disclosed herein can be combined with any one or more steps of any other method as disclosed herein.

[0115] When a feature or element is herein referred to as being "on" another feature or element, it can be directly on the other feature or element or intervening features and / or elements may also be present. In contrast, when a feature or element is referred to as being "directly on" another feature or element, there are no intervening features or elements present. It will also be understood that, when a feature or element is referred to as being "connected", "attached" or "coupled" to another feature or element, it can be directly connected, attached or coupled to the other feature or element or intervening features or elements may be present. In contrast, when a feature or element is referred to as being "directly connected", "directly attached" or "directly coupled" to another feature or element, there are no intervening features or elements present. Although described or shown with respect to one embodiment, the features and elements so described or shown can apply to other embodiments. It will also be appreciated by those of skill in the art that references to a structure or feature that is disposed "adjacent" another feature may have portions that overlap or underlie the adjacent feature.

[0116] Terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. For example, as used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components,- 20 of 28 -SG Docket No.: 14843-710.600and / or groups thereof. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items and may be abbreviated as " / ".

[0117] Spatially relative terms, such as "under", "below", "lower", "over", "upper" and the like, may be used herein for ease of description to describe one element or feature's relationship to another element(s) or feature(s) as illustrated in the figures. It will be understood that the spatially relative terms are intended to encompass different orientations of the device in use or operation in addition to the orientation depicted in the figures. For example, if a device in the figures is inverted, elements described as "under" or "beneath" other elements or features would then be oriented "over" the other elements or features. Thus, the exemplary term "under" can encompass both an orientation of over and under. The device may be otherwise oriented (rotated 90 degrees or at other orientations) and the spatially relative descriptors used herein interpreted accordingly. Similarly, the terms "upwardly", "downwardly", "vertical", "horizontal" and the like are used herein for the purpose of explanation only unless specifically indicated otherwise.

[0118] Although the terms “first” and “second” may be used herein to describe various features / elements (including steps), these features / elements should not be limited by these terms, unless the context indicates otherwise. These terms may be used to distinguish one feature / element from another feature / element. Thus, a first feature / element discussed below could be termed a second feature / element, and similarly, a second feature / element discussed below could be termed a first feature / element without departing from the teachings of the present invention.

[0119] Throughout this specification and the claims which follow, unless the context requires otherwise, the word “comprise”, and variations such as “comprises” and “comprising” means various components can be co-jointly employed in the methods and articles (e.g., compositions and apparatuses including device and methods). For example, the term “comprising” will be understood to imply the inclusion of any stated elements or steps but not the exclusion of any other elements or steps.

[0120] In general, any of the apparatuses and methods described herein should be understood to be inclusive, but all or a sub-set of the components and / or steps may alternatively be exclusive, and may be expressed as “consisting of” or alternatively “consisting essentially of” the various components, steps, sub-components or sub-steps.

[0121] As used herein in the specification and claims, including as used in the examples and unless otherwise expressly specified, all numbers may be read as if prefaced by the word "about" or “approximately,” even if the term does not expressly appear. The phrase- 21 of 28 -SG Docket No.: 14843-710.600“about" or “approximately” may be used when describing magnitude and / or position to indicate that the value and / or position described is within a reasonable expected range of values and / or positions. For example, a numeric value may have a value that is + / - 0.1% of the stated value (or range of values), + / - 1% of the stated value (or range of values), + / - 2% of the stated value (or range of values), + / - 5% of the stated value (or range of values), + / - 10% of the stated value (or range of values), etc. Any numerical values given herein should also be understood to include about or approximately that value, unless the context indicates otherwise. For example, if the value " 10" is disclosed, then "about 10" is also disclosed. Any numerical range recited herein is intended to include all sub-ranges subsumed therein. It is also understood that when a value is disclosed that "less than or equal to" the value, "greater than or equal to the value" and possible ranges between values are also disclosed, as appropriately understood by the skilled artisan. For example, if the value "X" is disclosed the "less than or equal to X" as well as "greater than or equal to X" (e.g., where X is a numerical value) is also disclosed. It is also understood that the throughout the application, data is provided in a number of different formats, and that this data, represents endpoints and starting points, and ranges for any combination of the data points. For example, if a particular data point “10” and a particular data point “15” are disclosed, it is understood that greater than, greater than or equal to, less than, less than or equal to, and equal to 10 and 15 are considered disclosed as well as between 10 and 15. It is also understood that each unit between two particular units are also disclosed. For example, if 10 and 15 are disclosed, then 11, 12, 13, and 14 are also disclosed.

[0122] Although various illustrative embodiments are described above, any of a number of changes may be made to various embodiments without departing from the scope of the invention as described by the claims. For example, the order in which various described method steps are performed may often be changed in alternative embodiments, and in other alternative embodiments one or more method steps may be skipped altogether. Optional features of various device and system embodiments may be included in some embodiments and not in others. Therefore, the foregoing description is provided primarily for exemplary purposes and should not be interpreted to limit the scope of the invention as it is set forth in the claims.

[0123] The examples and illustrations included herein show, by way of illustration and not of limitation, specific embodiments in which the subject matter may be practiced. As mentioned, other embodiments may be utilized and derived there from, such that structural and logical substitutions and changes may be made without departing from the scope of this- 22 of 28 -SG Docket No.: 14843-710.600disclosure. Such embodiments of the inventive subject matter may be referred to herein individually or collectively by the term “invention” merely for convenience and without intending to voluntarily limit the scope of this application to any single invention or inventive concept, if more than one is, in fact, disclosed. Thus, although specific embodiments have been illustrated and described herein, any arrangement calculated to achieve the same purpose may be substituted for the specific embodiments shown. This disclosure is intended to cover any and all adaptations or variations of various embodiments. Combinations of the above embodiments, and other embodiments not specifically described herein, will be apparent to those of skill in the art upon reviewing the above description.- 23 of 28 -SG Docket No.: 14843-710.600

Claims

1. CLAIMSWhat is claimed is:

1. A system for refraction correction of OCT images configured for intraoperative use, comprising: a deep-learning model configured to receive a plurality of OCT images; and an imaging device configured to generate the plurality OCT images during an intraoperative procedure; wherein the deep-learning model is configured to map, in real-time, the plurality of OCT images to a two-dimensional vector field over a spatial domain of the plurality of OCT images; wherein the mapping is configured for real-time correction of refraction errors on the plurality of OCT images to generate a plurality of refraction-corrected OCT images via the deep-learning model.

2. The system of claim 1, wherein the vector field encodes a displacement required to correct for distortions caused by light refraction for each pixel in each of the plurality of OCT images.

3. The system of any of the above claims, wherein the vector field is configured to serve as a lookup table by repositioning a detected feature at a pixel location p=(x,y) in each of the plurality of OCT images to pixel location p+v; wherein v is a vector associated with pixel location p.

4. The system of any of the above claims, wherein the deep-learning model is configured to be trained on a plurality of OCT training images; wherein the plurality of OCT training images are one or more of annotated manually and annotated via automation.

5. The system of any of the above claims, wherein the deep-learning model is configured to utilize Snell-based techniques to generate training targets.

6. The system of any of the above claims, wherein correcting refraction errors comprises: determining surface normals at a comeal interface from OCT data; modeling incident rays traversing multiple ocular media of differing refractive index; applying Snell's law at each interface to compute refracted ray directions; adjusting each refracted ray for- 24 of 28 -SG Docket No.: 14843-710.600differences in propagation speed between the media; and generating a displacement field correlating to a true optical path of each ray.

7. The system of any of the above claims, wherein the deep-learning model is a neural network having an encoder-decoder structure interconnected via skip connections; further wherein the neural network utilizes a first output map to encode a horizontal displacement for correcting refraction and a second map encoding a vertical displacement for correcting refraction to generate the vector field.

8. The system of any of the above claims, wherein the imaging device is an OCT scanner configured to continuously acquire the plurality of OCT images at a frame rate of 20Hz-200Hz; wherein the plurality of OCT images are B-scans.

9. The system of claim 8, wherein the plurality of OCT images are configured to be processed in parallel via: (i) a feature extraction model configured to process the plurality of OCT images to identify anatomical structures and surgical tools, and (ii) a refraction correction model configured for real-time estimating of refraction errors on the plurality of OCT images; wherein the refraction correction model is the deep-learning model; further wherein the feature extraction model and deep-learning model operate at a processing speed of 20Hz-50Hz configured for intraoperative usage.

10. The system of claim 9, wherein the feature extraction model is configured to output a list of points of interest representing the anatomical structure and surgical tools; wherein the refraction correction model is configured to output the vector field, wherein the vector field is configured to provide a displacement vector indicating a refraction correction for each pixel in each of the plurality of OCT images.

11. The system of claim 10, further comprising correction features configured to be generated by applying the displacement vector for each pixel in each of the plurality of OCT images to an associated point of interest from the list of points of interest; further wherein the correction features are configured for one or more of: path planning tasks, visualization, and use by virtual fixtures.- 25 of 28 -SG Docket No.: 14843-710.60012. The system of one of claim 1 to claim 8 or claim 9 or claim 10 or claim 11, wherein the generating and refraction correction of the plurality of OCT images is applied to one or more of: an eye, an esophagus, and other anatomical regions suitable for OCT scanning.

13. The system of any of claim 1 to claim 8 wherein the generating and refraction correction of the plurality of OCT images is applied to the movement of a tool relative to a portion of a posterior capsule of a lens.

14. The system of one of claim 1 to claim 8 or claim 9 or claim 10 or claim 11, wherein the imaging device is a multimodal imaging device including OCT and one or more of: digital microscope (DM) and ultrasound (U / S).

15. The system of any one of claim 1 to claim 8 or claim 9 or claim 10 or claim 11, wherein the generating and refraction correction of the plurality of OCT images is applied to diagnosis and assessment of: Barrett’s esophagus, tumor size measurement, scar reduction and formation measurement, or a radiation treatment.

16. A method of real-time refraction correction of OCT images in an intraoperative setting, comprising: training a refraction deep-leaming model based on a dataset of OCT images; training a feature deep-learning model based on a dataset of OCT images; intraoperatively generating a plurality of OCT images using an imaging device; mapping, via the refraction deep-learning model, the plurality of OCT images to a two-dimensional vector field over a spatial domain of the plurality of OCT images; mapping, via the feature deep-learning model, the plurality of OCT images to points of interest representing anatomical features and surgical tools; training the deep-leaming model using refraction corrections computed by Snell’s law; correcting the points of interest in real-time based on an output of the deep-learning model; and generating a plurality of real-time refraction-corrected features from the plurality of OCT images.- 26 of 28 -SG Docket No.: 14843-710.60017. The method of claim 16, further comprising repositioning a detected feature at a pixel location p=(x,y) in each of the plurality of OCT images to pixel location p+v; wherein v is a vector associated with each pixel location p.

18. The method of claim 16, further comprising: processing the plurality of OCT images through an encoder-decoder structure of a neural network interconnected via skip connections; and encoding, via a first and second output map of the neural network, a horizontal and vertical displacement, respectively, for refraction correction of the plurality of OCT images.

19. The method of claim 16, further comprising: acquiring the plurality of OCT images at a frame rate of 20-200Hz.

20. The method of claim 16, further comprising processing the plurality of OCT images in parallel via: processing the plurality of OCT images via a feature extraction model to identify anatomical structures and surgical tools; estimating refraction errors on the plurality of OCT images in real-time via a refraction correction model; wherein the refraction correction model is the deep-learning model; and operating the feature extraction model and deep-learning model for intraoperative usage at a processing speed of 20Hz-50Hz.

21. The method of claim 20, further comprising : outputting, via the feature extraction model, a list of points of interest representing the anatomical structure and surgical tools; outputting, via the refraction correction model, the vector field; and generating correction features configured for path planning tasks by applying the vector field to the points of interest representing the anatomical structure and surgical tools.

22. The method of any of claims 16-21 wherein the OCT images are of a portion of an eye.- 27 of 28 -SG Docket No.: 14843-710.600

Images