System, method and computer program for a surgical or scientific imaging system and surgical or scientific imaging system

Abstract

Examples relate to a system (110), method and computer program for a surgical or scientific imaging system (100), and to a surgical or scientific imaging system (100) comprising such a system (110). The system (110) is configured to obtain imaging data of at least one optical imaging sensor (120) of the surgical or scientific imaging system, the imaging data comprising a first image representing a first angle of view and a second image representing a second angle of view on a sample (10), determine, for at least one of the first image or the second image, an area being affected by an obstruction obscuring a view on the sample, combine the first image and the second image using image projection based on the area being affected by an obstruction obscuring a view on the sample to obtain a combined view on the sample, and provide the combined view on the sample for a display device (130) of the surgical or scientific imaging system.

Description

[0001] System, Method and Computer Program for a Surgical or Scientific Imaging System and Surgical or Scientific Imaging System

[0002] Technical field

[0003] Examples relate to a system, method and computer program for a surgical or scientific imaging system, and to a surgical or scientific imaging system comprising such a system.

[0004] Background

[0005] In surgical microscopy, surgeons often encounter obstructions in their field of view caused by tissues, surgical tools, or other objects. Traditionally, overcoming these obstructions requires repositioning the tissue or tools or changing the microscope's observation angle. In physical repositioning, tissue or instruments are moved to clear the line of sight, which interrupts the surgical workflow. Additionally, moving the tissue or instruments may not be possible if the obstructing object cannot be moved. Moreover, when tissue is moved, there is a risk of causing unintended tissue damage. Adjusting the observation angle generally involves changing the microscope's position to view the patient from a different angle. This can be time-consuming, may disrupt the procedure, may be limited by physical constraints of the operating environment, and may not fully eliminate obstructions. These approaches may thus interrupt the surgical workflow and may not always be feasible, for example, when a detached vessel obstructs the view and cannot be moved.

[0006] There may be a desire for a concept for mitigating such visual obstructions without disrupting the procedure.

[0007] Summary

[0008] This desire is addressed by the subject matter of the independent claims.

[0009] The proposed concept is based on the finding that many surgical or scientific imaging devices, such as surgical or scientific microscopes or exoscopes, use multiple imaging channels that have different angles of view on the sample being viewed. This is, for example, the case in stereoscopic imaging devices, but also in imaging devices having separate imaging channels for white light imaging and fluorescence imaging, or imaging devices having a primary, magnifying channel and a secondary, widefield channel used forgenerating an overview of the sample. In such cases, the images having different angles of view are used to determine one or more areas of the sample that are only visible in one of the images (and thus obstructed in the other image(s)), and image projection is used to generate a combined view of the sample in which such obstructed areas are made visible using the images from the channel having a different angle of view on the sample. This way, the obstructions can be compensated for without interrupting the operator’s workflow and without altering the microscope's position.

[0010] Some aspects of the present disclosure relate to a system for a surgical or scientific imaging system. The system comprises one or more processors and one or more storage devices. The system is configured to obtain imaging data of at least one optical imaging sensor of the surgical or scientific imaging system. The imaging data comprises a first image representing a first angle of view and a second image representing a second angle of view on a sample. The system is configured to determine, for at least one of the first image or the second image, an area being affected by an obstruction obscuring a view on the sample. The system is configured to combine the first image and the second image using image projection based on the area being affected by an obstruction obscuring a view on the sample to obtain a combined view on the sample. The system is configured to provide the combined view on the sample for a display device of the surgical or scientific imaging system. This way, the obstructions can be compensated for without interrupting the operator’s workflow and without altering the microscope's position.

[0011] In general, the obstruction being referred to in the present context is a visual obstruction, i.e. , anything that blocks or hinders a clear view. It can be a physical object, like a surgical tool, a sponge, tissue or a part of a vessel, or even other conditions, such as specular reflections, that impede visibility. For example, the area being affected by an obstruction obscuring a view on the sample may be an area that may be hidden (e.g., due to an object blocking a view on the area from the respective angle or view, or due to reflections saturating the respective optical imaging sensor) in one of the first or second image and visible in the respective other of the first or second image.

[0012] One type of imaging system that is based on dual channel imaging is stereoscopic imaging devices. Accordingly, the system may be configured to obtain stereoscopic imaging data from the at least one optical imaging sensor of the surgical or scientific imaging system. For example, the stereoscopic imaging data may comprise the first image representing a first channel and the second image representing a second channel of the stereoscopic imagingdata. This way, the first and second image data may be obtained in a large number of use cases.

[0013] To generate the combined view, image information from both images is combined using image projection. In general, image projection is the process of mapping points from one space into another space. In many cases, image projection is used to project a three-dimensional object onto a two-dimensional image plane, simulating how a two-dimensional camera captures a three-dimensional scene. However, image projection can also be used between two two-dimensional spaces. In the proposed concept, image projection can be used to project pixels between the first and second image, or between a third image and the first or second image, to generate the combined view. For example, keypoint detection algorithms such as SIFT (Scale-Invariant Feature Transform) or SURF (Speeded-Up Robust Features) can be used to identify corresponding features between the respective images. For example, the system may be configured to use the corresponding features between the respective images to determine a mapping between pixels of the respective images. This mapping between the two two-dimensional spaces is called a homography, i.e. , a projective transformation between two planes in three-dimensional space. If an area of one of the images is obscured by an obstruction, the system may use the pixels of one of the other images to replace the pixels of the area being obstructed. This way, the combined view comprises versions of the first and second image that have been extended (e.g., improved) using pixels of another one of the images. Using a homography for image projection is useful if the sample is largely flat, with the homography accounting for perspective distortion.

[0014] More commonly, image projection is used between a three-dimensional representation of an object, such as the sample, and one or more two-dimensional image planes. To generate a combined view using image projection, mathematical operations may be used to transform the image information that is contained in the three-dimensional representation of the object (i.e., the sample) into corresponding two-dimensional image information. These mathematical operations are based on the perspective of the respective two-dimensional image relative to the three-dimensional object. This can be done with the help of a depth map, which can be used as a three-dimensional representation of the sample. A depth map of a sample is a three-dimensional representation of distances of points of the sample from a reference point, such as the surgical or scientific imaging system. For example, the system may be configured to determine the depth map of the sample, and to combine the first image and the second image using image projection based on the area being affected by an obstruction obscuring a view on the sample using the depth map to obtain the combined view on the sample. In this case, the pixels of the first and second (and optionally a third) imagesmay first be projected onto the depth map, and then re-projected, from the depth map, onto the image planes of the first and second images to generate the combined view. In particular, using both the first image and the second image, the color and / or intensity at each (three-dimensional) point of the depth map (e.g., each point being obstructed in one of the images) may be determined, and used to fill in the area(s) being obstructed in the combined view. This way, the combined view can show the area of the image being obstructed.

[0015] There are various approaches for determining suitable depth maps, such as optical or ultrasound time-of-flight sensors. As an alternative, the first and second image themselves may be used to generate the depth map. In other words, the system may be configured to determine the depth map of the sample using stereo disparity analysis based on the first and second image. Stereo disparity analysis, also known as stereopsis, is a technique used to determine depth information from two or more images taken from slightly different viewing angles. This process involves identifying corresponding points between the images and calculating the difference in their positions, which is referred to as disparity. The disparity values are then used to estimate the depth map of the sample. This way, the depth may can be generated without requiring an additional depth sensor, which saves cost and lowers implementation complexity.

[0016] The depth map is used to fill in the areas that are obstructed in one of the images. In particular, the system may be configured to map, using image projection, one or more image segments of the respective other image onto the area being affected by the obstruction using the depth map of the sample. This way, the combined view can show the area of the image being obstructed.

[0017] While replacing the obstructed areas with image information from the respective other angle of view, doing so may be disconcerting for the operator of the surgical or scientific imaging device, e.g., for the surgeon. Therefore, a visual overlay may be added to inform the operator that the respective area is being replaced using the proposed concept. Thus, the system may be configured to represent the area onto which the one or more image segments of the respective other image are mapped with an indicator representing the mapping operation. This way, the operator is made aware that image processing is being used to visualize this area of the sample.

[0018] To differentiate obstructions from permanent features of the sample, multiple depth maps may be determined and used to identify discrepancies that would indicate an obstruction from one angle of view. For example, the system may be configured to determine the areabeing affected by an obstruction obscuring a view on the sample based on a depth disparity between a first depth map being generated starting from the first image and a second depth map being generated starting from the second image. For example, if the depths for a point on the sample differ by at least a threshold value, e.g., at least 1 cm, or at least 10%, it may be assumed that the disparity is caused by an obstruction that obstructs the view on the sample for one of the angles of view (but not the other), which can be compensated for using the proposed concept.

[0019] In general, various aspects of the proposed concept can be further improved by adding one or more additional imaging channels having a different angle of view (than the first or second angle of view). Accordingly, the system may be configured to obtain the imaging data further with at least one third image representing at least one third angle of view on the sample. The system may be configured to use the first, second and third image to determine the area being affected by the obstruction obscuring a view on the sample, obtain the combined view on the sample and / or to determine a depth map. By using at least one third image representing at least one third angle of view, the quality and capability of the aforementioned operations may be improved.

[0020] Building surgical or scientific imaging devices with additional imaging channels (having different angles of view) is costly and adds complexity to the respective surgical or scientific imaging device. To avoid such costs or complexities, additional imaging channel(s) may be used that are already used for other purposes, e.g., to provide additional imaging modalities. For example, the system may be configured to obtain the at least one third image from an optical imaging sensor being associated with a different imaging modality than at least one optical imaging sensor providing the first and second image. For example, while the first and second image are obtained from an optical imaging sensor being used for white light imaging, the third optical imaging sensor might be used only for fluorescence imaging.

[0021] In various examples, the system may be configured to combine the first image and the second image using image projection to obtain a two-dimensional or three-dimensional combined view on the sample, and to provide the combined view for a 2D or 3D monitor of the surgical or scientific imaging system. This way, the combined view can be shown to multiple people, such as (surgical) staff and the operator (e.g., surgeon).

[0022] Additionally, or alternatively, the system may be configured to combine the first image and the second image using image projection to obtain a stereoscopic combined view on the sample, and to provide the stereoscopic combined view to a stereoscopic display device ofthe surgical or scientific imaging system. This way, the combined view can be shown via oculars or via a stereoscopic headset of the surgical or scientific imaging system.

[0023] Another aspect of the present disclosure relates to a surgical or scientific imaging system comprising at least one optical imaging sensor, a display device, and the above system.

[0024] In some case, an angle of view of one of the imaging channels can be changed, e.g., by laterally moving the imaging channel or by tilting the imaging channel. For example, the surgical or scientific imaging system may comprise a first optical imaging sensor for providing the first image and a second optical imaging sensor for providing the second image. The system may be configured to control the surgical or scientific imaging system to adjust the second angle of view based on the area being affected by an obstruction obscuring a view on the sample. This way, an angle of view can be chosen that is suitable for avoiding the obstruction obscuring the view on the sample.

[0025] In some cases, a similar approach can be chosen in a surgical or scientific imaging system having three imaging channels, of which one is adjustable with respect to the angle of view. For example, the surgical or scientific imaging system may comprise a first optical imaging sensor for providing the first image, a second optical imaging sensor for providing the second image and a third optical imaging sensor for providing a third image. The system may be configured to control the surgical or scientific imaging system to adjust a third angle of view on the sample of a the third image based on the area being affected by an obstruction obscuring a view on the sample. The system may be configured to use the first, second and third image to determine the area being affected by the obstruction obscuring a view on the sample, obtain the combined view on the sample and / or to determine a depth map. This way, a third angle of view can be chosen that is suitable for avoiding the obstruction obscuring the view on the sample.

[0026] Some aspects of the present disclosure relate to a method for a surgical or scientific imaging system. The method comprises obtaining imaging data of at least one optical imaging sensor of the surgical or scientific imaging system. The imaging data comprises a first image representing a first angle of view and a second image representing a second angle of view on a sample. The method comprises determining, for at least one of the first image or the second image, an area being affected by an obstruction obscuring a view on the sample. The method comprises combining the first image and the second image using image projection based on the area being affected by an obstruction obscuring a view on the sampleto obtain a combined view on the sample. The method comprises providing the combined view on the sample for a display device of the surgical or scientific imaging system.

[0027] Another aspect of the present disclosure relates to a computer program having a program code for performing the above method when the program is executed on processor.

[0028] Short description of the Figures

[0029] Some examples of apparatuses and / or methods will be described in the following by way of example only, and with reference to the accompanying figures, in which:

[0030] Fig. 1a shows a block diagram of an example of a system for a surgical or scientific imaging system;

[0031] Fig. 1b shows a schematic diagram of a surgical or scientific imaging system;

[0032] Fig. 2 shows an example of multi-angle image integration for obstruction mitigation;

[0033] Figs. 3a and 3b illustrates the imaging of a sample using multiple angles of view;

[0034] Fig. 4 shows an example of a method for a system for a surgical or scientific imaging system; and

[0035] Fig. 5 shows an example of a system comprising an imaging device and a computer system.

[0036] Detailed Description

[0037] Various examples will now be described more fully with reference to the accompanying drawings in which some examples are illustrated.

[0038] Fig. 1a shows a schematic diagram of an example of a system 110 for a surgical or scientific imaging system 100 (shown in Fig. 1b). The system 110 is a component of the surgical or scientific imaging system 100 and may be used to control various aspects of the surgical orscientific imaging system 100. In particular, it may be used for acquisition and / or processing of imaging data by at least one optical imaging sensor of an imaging device 120 of the surgical or scientific imaging system, through various means that will be introduced in more detail in the following. In addition, the system 110 may be configured to control additional aspects of the surgical or scientific imaging system 100, e.g., to provide a display signal for one or more display devices 130a, 130b of the surgical or scientific imaging system.

[0039] In general, the system 110 may be a computer system. The system 110 comprises one or more processors 114 and one or more storage devices 116. Optionally, the system 110 further comprises one or more interfaces 112. The one or more processors 114 are coupled to the one or more storage devices 116 and to the one or more interfaces 112. In general, the functionality of the system 110 may be provided by the one or more processors 114, in conjunction with the one or more interfaces 112 (for exchanging data / information with one or more other components of the surgical or scientific imaging system 100 and outside the surgical or scientific imaging system 100, such as an optical imaging sensor of the imaging device 120), and with the one or more storage devices 116 (for storing information, such as machine-readable instructions of a computer program being executed by the one or more processors). In general, the functionality of the one or more processors 114 may be implemented by one or more processors 114 executing machine-readable instructions. Accordingly, any feature ascribed to the one or more processors 114 may be defined by one or more instructions of a plurality of machine-readable instructions. The system 110 may comprise the machine-readable instructions, e.g., within the one or more storage devices 116.

[0040] As outlined above, the system 110 is part of the surgical or scientific imaging system 100, which comprises various components in addition to the system 110. For example, the surgical or scientific imaging system 100 comprises the imaging device 120, and may comprise one or more additional components, such as one or more display devices 130a-130b. Fig. 1b shows a schematic diagram of an example of such a surgical or scientific imaging system 100, and in particular of a surgical microscope system 100. In the following, the surgical or scientific imaging system 100 may also be referred to as surgical microscope system 100. A surgical microscope system is a surgical imaging system 100 that comprises a (surgical) microscope as imaging device 120. However, the proposed concept is not limited to such embodiments. The surgical or scientific imaging system 100 may be based on various (single or multiple) imaging devices, such as one or more microscopes, one or more endoscopes, and / or one or more exoscopes (also sometimes called an extracorporeal telescope). Exoscopes are camera-based imaging systems, and in particular camera-based3D imaging systems, which are suitable for providing images of surgical sites with high magnification and a large depth of field. Compared to microscopes, which may be used via oculars, exoscopes are only used via display modalities, such as a monitor or a headmounted display. Accordingly, the surgical or scientific imaging system 100 may alternatively be a surgical endoscope system, or a surgical exoscope system. Yet alternatively, the surgical or scientific imaging system may be a scientific imaging system comprising a microscope, e.g., a laboratory microscope. However, the following illustrations assume that the surgical imaging device 120 is a surgical microscope, and that the surgical or scientific imaging system 100 is a surgical microscope system 100.

[0041] Accordingly, the surgical or scientific imaging system or surgical microscope system 100 may comprise an imaging device, such as a microscope 120. In general, a microscope, such as the microscope 120, is an optical instrument that is suitable for examining objects that are too small to be examined by the human eye (alone). For example, a microscope may provide an optical magnification of a sample. In the present concept, the optical magnification is (also) provided for at least one optical imaging sensor. The microscope 120 thus comprises an optical imaging sensor, which is coupled with the system 110. The microscope 120 may further comprise one or more optical magnification components that are used to magnify a view on the sample, such as an objective (i.e., lens). For example, the surgical imaging device or microscope 120 is often referred to as the 'optics carrier' of the imaging system.

[0042] There are a variety of different types of surgical or scientific imaging devices. If the imaging device is used in the medical or biological fields, the object being viewed through the imaging device may be a sample of organic tissue, e.g., arranged within a petri dish or present in a part of a body of a patient. In various examples presented here, the imaging device 120 may be a microscope of a surgical microscope system, i.e., a microscope that is to be used during a surgical procedure, such as a neurosurgical procedure (i.e., brain surgery). Accordingly, the sample being viewed through the surgical imaging device may be a sample of organic tissue of a patient and may in particular be the surgical site that the surgeon operates on during the surgical procedure, e.g., the brain. However, the proposed concept is also suitable for other types of surgery, such as eye surgery or cardiac surgery.

[0043] Fig. 1b shows a schematic diagram of an example of a surgical or scientific imaging system 100, and in particular of a surgical microscope system 100, comprising the system 110 and a microscope 120. The surgical microscope system 100 shown in Fig. 1b comprises a number of optional components, such as a base unit 105 (comprising the system 110) with a (rolling) stand, ocular displays 130a that are arranged at the microscope 120, an auxiliarydisplay 130b that is arranged at the base unit, and the arm that holds the microscope 120 in place, and which is coupled to the base unit and to the microscope 120. In general, these optional and non-optional components may be coupled to the system 110 which may be configured to control and / or interact with the respective components.

[0044] In the proposed concept, the system 110 is used to perform multi-angle image integration. The imaging device 120 of the surgical or scientific imaging system captures images from multiple observation angles, e.g., using the imaging device’s stereo imaging capabilities (e.g., left and right channels). Obstructions in one image are identified where pixels are missing or obscured. Corresponding image segments from another channel, where the obstruction is not present, are mapped onto the obstructed areas. The result is a combined (i.e., composite) image with reduced obstructions.

[0045] Thus, the system 110 is configured to obtain imaging data of at least one optical imaging sensor 120 of the surgical or scientific imaging system 100. The imaging data comprises a first image representing a first angle of view and a second image representing a second angle of view on a sample 10. The system 110 is configured to determine, for at least one of the first image or the second image, an area being affected by an obstruction obscuring a view on the sample. For example, this area may be an area that may be hidden in one of the first or second image and visible in the respective other of the first or second image. The system 110 is configured to combine the first image and the second image using image projection based on the area being affected by an obstruction obscuring a view on the sample to obtain a combined view on the sample. The system 110 is configured to provide the combined view on the sample for a display device 130 of the surgical or scientific imaging system.

[0046] Fig. 2 shows an example of the proposed multi-angle image integration for obstruction mitigation. Column 210 includes the images of a left imaging channel and column 220 includes images of a right imaging channel of a stereoscopic microscope. Row 230 shows raw images, and row 240 shows combined (i.e., composite images). Fig. 2 shows the process of filling obstructed areas in one image with data from another channel where the obstruction is not present. For example, the “C” visible in the raw image of the left imaging channel is used for the combined image of the right imaging channel, and the A (and a portion of the B) visible in the raw image of the right imaging channel is used for the combined image of the left imaging channel. For example, the combined or composite image may be created using depth mapping to accurately align and integrate the image segments, resulting in a clearer view for the surgeon.The process starts with the imaging data of the optical imaging sensor(s) of the surgical or scientific imaging system 100. The system 110 is configured to obtain (i.e., receive or read out) the imaging data from the at least one optical imaging sensor. The imaging data may be obtained by receiving the imaging data from the optical imaging sensor(s) (e.g., via the interface 112), by reading the imaging data out from a memory of the optical imaging sensor (e.g., via the interface 112), or by reading the imaging data from a storage device 116 of the system 110, e.g., after the imaging data has been written to the storage device 116 by the optical imaging sensor or by another system or processor. In many cases, as the imaging data comprises at least two images representing at least two angles of view on the sample 10, the imaging data may be obtained of (e.g., from) at least two different optical imaging sensors. For example, as shown in Fig. 3a and 3b, the surgical or scientific imaging system 110 may comprise a first optical imaging sensor 120a for providing the first image and a second optical imaging sensor 120b for providing the second image. In some cases, different portions of a single optical imaging sensor may be used to generate the first and second image. In this case, the imaging data comprising the first and second image may be obtained or (e.g., from) the same optical imaging sensor.

[0047] In many cases, the imaging device 120 of the surgical or scientific imaging system 100 may be a stereoscopic imaging device. In this case, the system may be configured to obtain stereoscopic imaging data from the at least one optical imaging sensor of the surgical or scientific imaging system, with the stereoscopic imaging data comprising the first image representing a first channel and the second image representing a second channel of the stereoscopic imaging data. Such a configuration is shown in Figs. 3a and 3b, for example.

[0048] In some cases, more than two images (representing more than two imaging channels and more than two angles of view) may be used. The system 110 may incorporate more than two imaging channels if available, increasing the likelihood of obtaining unobstructed views. Such a configuration is shown in Fig. 3b, for example. For example, the system 110 may be configured to obtain the imaging data further with at least one third image representing at least one third angle of view on the sample. In this case, the surgical or scientific imaging system 110 may comprise a first optical imaging sensor 120a for providing the first image, a second optical imaging sensor 120b for providing the second image and a third optical imaging sensor 120c for providing a third image. For example, the third optical imaging sensor may be associated with a different imaging modality than at least one optical imaging sensor providing the first and second image. For example, the third optical imaging sensor may be used for fluorescence imaging while the optical imaging sensor(s) providing the firstand second image may be used for white light imaging. Alternatively, the third optical imaging sensor may be configured to provide a widefield view of the sample (e.g., of the patient) while optical imaging sensor(s) providing the first and second image may be used to provide a magnified view of the sample. The third image may be used to extend the image information being used in the subsequent image processing tasks. For example, the system 110 may be configured to use the first, second and third image to determine the area being affected by the obstruction obscuring a view on the sample and / or to obtain the combined view on the sample.

[0049] In some cases, the imaging device 120 comprises at least one imaging channel having an adjustable geometry. For example, the imaging device 120 (e.g., the microscope) may include an adjustable imaging channel that changes its observation angle to improve filling blind spots (i.e. , the area being affected by an obstruction obscuring a view on the sample) in other channels. For example, the second imaging channel may have an adjustable geometry. In this case, the system may be configured to control the surgical or scientific imaging system 110 (e.g., a motor 340 of the surgical or scientific imaging system 110 shown in Fig. 3b) to adjust the second angle of view based on the area being affected by an obstruction obscuring a view on the sample. Alternatively, if three (or more) imaging channels are used, the system 110 may be configured to control the surgical or scientific imaging system (e.g., a motor 340 of the surgical or scientific imaging system 110 shown in Fig. 3b) to adjust a third angle of view on the sample of the third image based on the area being affected by an obstruction obscuring a view on the sample.

[0050] In Figs. 3a and 3b, some examples are given illustrating the different angles of view on the sample 10. Figs. 3a and 3b illustrates the imaging of a sample using multiple angles of view. In Fig. 3a, a stereoscopic scenario is shown, with a first optical imaging sensor 120a and a second optical imaging sensor 120b. The first optical imaging sensor 120a and a second optical imaging sensor 120b observe the sample 10 via a first imaging channel 310a and a second imaging channel 310b, resulting in the first angle of view 320a and the second angle of view 320b. For example, the imaging channels 320a, 320b may, at least partially, use the same optical components 330, such as a lens or a zoom system of the imaging device.

[0051] In Fig. 3b, an imaging device 120 with a first optical imaging sensor 120a, a second optical imaging sensor 120b, and a third optical imaging sensor 120c observing the sample 10 via a first imaging channel 310a, a second imaging channel 310b, and a third imaging channel 310c, resulting in the first angle of view 320a, the second angle of view 320b and the third angle of view 320c is shown. In the case of Fig. 3b, the angle of view of the third opticalimaging sensor 120c is adjustable using motor 340. In particular, the system 110 may be configured to control the surgical or scientific imaging system to adjust the at least one third angle of view by changing at least one of a position or an orientation of the channel 310c between the third optical imaging sensor and the sample, e.g., using the motor 340. In particular, the third channel 310c may be moved laterally (to adjust its position). In other words, the system 110 may be configured to control the surgical or scientific imaging system 100 (e.g., the motor 340) to adjust a lateral offset of the channel 310c between the third optical imaging sensor and the sample from channels 310a, 310b between the first and second optical imaging sensors 120a, 120b and the sample 10 based on the area being affected by the obstruction. Additionally, the third channel 310c may be tilted (to change the orientation of the channel). In other words, the system 110 may be configured to control the surgical or scientific imaging system 100 (e.g., the motor 340) to adjust a tilt of the channel 310c between the third optical imaging sensor 120c and the sample 10 based on the area being affected by the obstruction.

[0052] The use of imaging channels having an adjustable geometry can improve the coverage of the sample, as the respective angle of view can be changed to avoid the obstruction. To determine an angle of view that avoids the obstruction, i.e., the desired angle of view, various approaches may betaken. In a first approach, machine learning may be used. In other words, the system 110 may be configured to adjust the second or third angle of view based on an output of a machine-learning model being trained to output a desired angle of view based on the area of the sample being affected by an obstruction. In general, the determination of a desired angle of view, based on a known angle of view for the firstand second optical imaging sensor, and optionally for the third optical imaging sensor, based on the known area being obstructed in one (or more) of the images is a finite challenge that can easily be solved using machine learning. For example, a regression machine learning model may be trained to output the desired angle of view, e.g., based on the second or third angle of view (being adjusted) and based on the area being affected by an obstruction obscuring a view on the sample. For example, the regression machine learning model may be trained to output, based on the second or third angle of view (being adjusted) and based on the area being affected by an obstruction obscuring a view on the sample (e.g., based on the area being affected by an obstruction obscuring a view on the sample for the first, second and / or third optical imaging sensor), the desired angle of view for the second or third optical imaging sensor. For example, the regression machine learning model may be trained using supervised learning based on an input comprising the second or third angle of view (being adjusted) and the area being affected by an obstruction obscuring a view on the sample and a corresponding desired angle of view as a desired result. Alternatively, the regressionmachine learning model may be trained using reinforcement learning based on a three-dimensional simulation of the area being affected by an obstruction obscuring a view on the sample, with the simulation being parametrized with the angle of view of the second or third image. A reward function being used for the reinforcement learning may be based on how much the area being affected by the obstruction in the first image is visible in the second image or how much the area being affected in the first and / or second image is visible in the third image, for example.

[0053] Alternatively, a three-dimensional simulation may be used to determine the desired angle of view (being used for generating the second or third image). For example, the system may be configured to determine a three-dimensional position of an object causing the obstruction relative to the area of the sample being obstructed (e.g., based on at least one depth map of the sample). For example, the system may be configured to adjust the second or third angle of view based on the three-dimensional position of an object causing the obstruction. For example, the system 110 may be configured to use ray tracing to determine the desired angle of view based on the area of the sample being obstructed and based on the three-dimensional position of the object causing the obstruction.

[0054] As a further option, the operator of the surgical or scientific imaging system may be requested to adjust the angle of view (e.g., the second or third angle of view) until visibility of the area being obstructed improves for the imaging channel being adjusted. In other words, the system 110 may be configured to adjust the second or third angle of view based on an input of a user being prompted to adjust the second or third angle of view until the obstruction is removed from at least one of the first image, the second image or the third image.

[0055] In the proposed concept, two operations are performed that benefit from knowing a topology of the sample being viewed using the surgical or scientific imaging system 110 - the determination of the area being affected by an obstruction obscuring a view on the sample, and the combination of the first and second image for the purpose of generating the combined view on the sample. Therefore, the topology of the sample may be determined. In other words, the system may be configured to determine a depth map of the sample. For example, the depth map may be calculated using stereo disparity analysis or other 3D scanning methods. In other words, the system 110 may be configured to determine the depth map of the sample using stereo disparity analysis based on the first and second image (and optionally based on the third image). Stereo disparity analysis is a technique used to determine the depth information of a scene by measuring the differences in the position of corresponding features between two images taken from slightly different viewing angles(stereo images). To perform stereo disparity analysis with a first image from the first stereo channel and a second image from the second stereo channel, the process typically involves the following tasks. For example, determining the depth map of the sample may comprise identifying and matching corresponding points or features between the two images. This can be done using algorithms like SIFT (Scale-Invariant Feature Transform), SURF (Speeded-Up Robust Features), or other feature detectors and descriptors. Determining the depth map of the sample may further comprise calculating the disparity for each matched pair of points. Disparity is the horizontal difference in position of the corresponding points in the two images. Determining the depth map of the sample may further comprise estimating the depth map of the sample using the calculated disparities, e.g., using triangulation based on the position geometry of the imaging channels.

[0056] In general, i.e. regardless of whether this is done based on the depth map, the area being affected by an obstruction obscuring a view on the sample may be determined for one of the images, or separately for the first and second image. For example, blind spots, i.e., the area being affected by an obstruction obscuring a view on the sample, may be detected when two or more pixels from one channel’s 2D image (e.g., from the first image or the second image) map to the same pixel in another channel's 2D image (e.g., in the respective other image). In other words, the system may be configured to determine the area being affected by an obstruction obscuring a view on the sample by identifying pixels of one of the images that map to the same pixel of another of the images. In this case, depth mapping ensures accurate correspondence between images.

[0057] When using the depth map to determine the area being affected by an obstruction obscuring a view on the sample, the following approach may also be used. In stereo disparity analysis, the resulting depth map changes depending on which of the images is used as a starting point. Swapping the images being used as an input for the stereo disparity analysis can thus lead to different depth maps, which represent the depths as perceived from the respective starting image. In case there is an obstruction that obstructs the view in only one of the images, there is a difference (i.e., a depth disparity) in the depth maps. This difference can be used to detect whether an area is obstructed in one of the images, and also to determine the three-dimensional position of the obstruction (for the image being obstructed by the obstruction). Accordingly, the system 110 may be configured to determine the area being affected by an obstruction obscuring a view on the sample (for one of the images) based on a depth disparity between a first depth map being generated starting from the first image and a second depth map being generated starting from the second image. This process can berepeated with the depth maps being switched to determine the area being affected by an obstruction obscuring a view on the sample for the respective other image.

[0058] The depth map can also be used to combine the images. For example, corresponding image segments from another image (and imaging channel), where the obstruction is not present, may be mapped onto the obstructed areas using the depth map. For example, the system 110 may be configured to combine the first image and the second image (and optionally the third image) using image projection based on the area being affected by an obstruction obscuring a view on the sample using the depth map to obtain the combined view on the sample. In particular, the system 110 may be configured to map, using image projection, one or more image segments of the respective other image onto the area being affected by the obstruction using the depth map of the sample.

[0059] In general, the purpose of the proposed concept is to perform obstruction transparency, i.e., obstruction mitigation via multi-angle image integration. Doing so may be disconcerting for the operator of the surgical or scientific imaging device, e.g., for the surgeon. Therefore, a visual overlay may be added to inform the operator that the respective area is being replaced using the proposed concept. For example, obstructed areas filled with data from another channel may be displayed in a way that informs the surgeon of the composite nature. For example, the system may be configured to represent the area onto which the one or more image segments of the respective other image are mapped with an indicator representing the mapping operation. For example, the area onto which the one or more image segments of the respective other image are mapped may be rendered as semi-transparent or with a distinct visual cue, such as a shading or color overlay.

[0060] The proposed concept may be used in different contexts (e.g., applications). For example, the proposed concept can be used in a monoscopic application. For example, the proposed methodology can be used to enhance a single 2D image by filling obstructions from one or more other channels (i.e., images). For example, the system 110 may be configured to combine the first image and the second image using image projection to obtain a two-dimensional combined view on the sample, and to provide the combined view for a 2D monitor 130b of the surgical or scientific imaging system.

[0061] Additionally, or alternatively, the proposed concept may be used in a stereoscopic application: It can be applied to both right and left images to create improved stereoscopic pairs for 3D visualization. In other words, the system 110 may be configured to combine the first image and the second image using image projection to obtain a stereoscopic combinedview on the sample, and to provide the stereoscopic combined view to a stereoscopic display device 130a, such as stereo oculars or a stereo headset, of the surgical or scientific imaging system 100. The same or similar techniques can also be used to supply a 3D monitor of the surgical or scientific imaging system. In other words, the system may be configured to combine the first image and the second image using image projection to obtain a three-dimensional combined view on the sample, and to provide the combined view for a 3D monitor 130b of the surgical or scientific imaging system.

[0062] In either case, the combined view is provided as part of a digital view on the sample. This digital view (e.g., of the surgical site) is created and provided to a display device 130a, 130b of the surgical imaging system as part of a display signal. In other words, the system may be configured to output a display signal to the display device 130a, 130b of the surgical or scientific imaging system, with the display signal comprising the combined view. The combined view may be viewed by the user, e.g., the surgeon, of the surgical or scientific imaging system. For this purpose, the display signal may be provided to the display device, e.g., an auxiliary display arranged at the base unit of the surgical microscope system, ocular displays integrated within the oculars of the surgical imaging device, or one or two displays of a head-mounted display. For example, the display signal may be a signal for driving (e.g., controlling) the respective display device. For example, the display signal may comprise video data and / or control instructions for driving the display. For example, the display signal may be provided via one of the one or more interfaces 112 of the system. Accordingly, the system 110 may comprise a video interface 112 that is suitable for providing the display signal to the display device 130a, 130b of the microscope system 100.

[0063] The proposed invention mitigates visual obstructions in surgical microscopy by utilizing multiangle image integration from stereo imaging. By combining images from two or more observation angles, the system fills in blind spots caused by obstructions in one view with corresponding image data from another view where the obstruction is not present. This creates a composite image that reduces the impact of obstructions without the need for physical adjustments. The method leverages depth mapping and image projection techniques to accurately integrate image segments from different angles, enhancing the surgeon's view and improving workflow efficiency.

[0064] The proposed concept provides an improved visualization, which reduces visual obstructions without physically altering the surgical field and enhances the surgeon's ability to see critical areas, aiding precision. It improves workflow efficiency by eliminating the need to pause theprocedure for adjustments, maintaining surgical flow. It is a non-invasive approach, as it does not require physical interaction with tissues or tools, reducing risk.

[0065] In the proposed concept, a surgical or scientific imaging system, such as a surgical microscope system, uses multi-angle image integration to digitally reduce obstructions. It is based on composite imaging from multiple observation angles. The proposed concept performs obstruction mitigation using image projection between channels. Enhanced views are provided by filling in obstructed areas with data from other channels. Visual indicators may be used to indicate that composite imaging is being used. The proposed concept provides digital obstruction reduction and enhanced visualization through multi-angle integration.

[0066] In the proposed scientific or surgical imaging system, the optical imaging sensor(s) 120a, 120b, 120c is / are used to provide the aforementioned imaging data. Accordingly, the optical imaging sensor(s), which may be part of the imaging device 120 (e.g., of the microscope) may be configured to generate the imaging data. For example, the optical imaging sensor(s) of the imaging device 120 may comprise or be an APS (Active Pixel Sensor) - or a CCD (Charge-Coupled-Device)-based imaging sensor. For example, in APS-based imaging sensors, light is recorded at each pixel using a photodetector and an active amplifier of the pixel. APS-based imaging sensors are often based on CMOS (Complementary Metal-Oxide-Semiconductor) or S-CMOS (Scientific CMOS) technology. In CCD-based imaging sensors, incoming photons are converted into electron charges at a semiconductor-oxide interface, which are subsequently moved between capacitive bins in the imaging sensors by a circuitry of the imaging sensors to perform the imaging.

[0067] The one or more interfaces 112 of the system 110 may correspond to one or more inputs and / or outputs for receiving and / or transmitting information, which may be in digital (bit) values according to a specified code, within a module, between modules or between modules of different entities. For example, the one or more interfaces 112 may comprise interface circuitry configured to receive and / or transmit information. The one or more processors 114 of the system 110 may be implemented using one or more processing units, one or more processing devices, any means for processing, such as a processor, a computer or a programmable hardware component being operable with accordingly adapted software. In other words, the described function of the one or more processors 114 may as well be implemented in software, which is then executed on one or more programmable hardware components. Such hardware components may comprise a general-purpose processor, a Digital Signal Processor (DSP), a micro-controller, etc. The one or more storage devices 116of the system 110 may comprise at least one element of the group of a computer readable storage medium, such as a magnetic or optical storage medium, e.g., a hard disk drive, a flash memory, Floppy-Disk, Random Access Memory (RAM), Programmable Read Only Memory (PROM), Erasable Programmable Read Only Memory (EPROM), an Electronically Erasable Programmable Read Only Memory (EEPROM), or a network storage.

[0068] More details and aspects of the system 110 and of the surgical or scientific imaging system 100 are mentioned in connection with the proposed concept or one or more examples described above or below (e.g. Fig. 4 to 5). The system 110 and the surgical or scientific imaging system 100 may comprise one or more additional optional features corresponding to one or more aspects of the proposed concept or one or more examples described above or below.

[0069] Fig. 4 shows an example of a corresponding method for a system for a surgical or scientific imaging system. For example, the method may be performed by the surgical or scientific imaging system 100 shown in Fig. 1b, and in particular by the system 110 shown in Figs. 1a and 1b. The method comprises obtaining 410 imaging data of at least one optical imaging sensor of the surgical or scientific imaging system, The imaging data comprising a first image representing a first angle of view and a second image representing a second angle of view on a sample. The method comprises determining 420, for at least one of the first image or the second image, an area being affected by an obstruction obscuring a view on the sample. The method comprises combining 430 the first image and the second image using image projection based on the area being affected by an obstruction obscuring a view on the sample to obtain a combined view on the sample. The method comprises providing 440 the combined view on the sample for a display device of the surgical or scientific imaging system. Features discussed in connection with the surgical or scientific imaging system of Figs. 1a to 1b and 2 to 3b may likewise be included in the method of Fig. 4.

[0070] More details and aspects of the method are mentioned in connection with the proposed concept, or one or more examples described above or below (e.g., Fig. 1a to 3b, 5). The method may comprise one or more additional optional features corresponding to one or more aspects of the proposed concept, or one or more examples described above or below.

[0071] As used herein the term “and / or” includes any and all combinations of one or more of the associated listed items and may be abbreviated as 7”.Although some aspects have been described in the context of an apparatus, it is clear that these aspects also represent a description of the corresponding method, where a block or device corresponds to a method step or a feature of a method step. Analogously, aspects described in the context of a method step also represent a description of a corresponding block or item or feature of a corresponding apparatus.

[0072] Some embodiments relate to an imaging device or imaging system, such as the surgical or scientific imaging device 120 or the surgical or scientific imaging system 100 of Fig. 1a and / or 1b comprising a system 110 as described in connection with one or more of the Figs. 1a to 4. Alternatively, an imaging device, such as a microscope or an exoscope, may be part of or connected to a system as described in connection with one or more of the Figs. 1a to 4. Fig.

[0073] 5 shows a schematic illustration of a system 500 configured to perform a method described herein. The system 500 comprises an imaging device 510, such as a microscope or exoscope, and a computer system 520. The imaging device 510 is configured to take images and is connected to the computer system 520. The computer system 520 is configured to execute at least a part of a method described herein. The computer system 520 may be configured to execute a machine learning algorithm. The computer system 520 and microscope 510 may be separate entities but can also be integrated together in one common housing. The computer system 520 may be part of a central processing system of the imaging device 510 and / or the computer system 520 may be part of a subcomponent of the imaging device 510, such as a sensor, an actor, a camera or an illumination unit, etc. of the imaging device 510.

[0074] The computer system 520 may be a local computer device (e.g. personal computer, laptop, tablet computer or mobile phone) with one or more processors and one or more storage devices or may be a distributed computer system (e.g. a cloud computing system with one or more processors and one or more storage devices distributed at various locations, for example, at a local client and / or one or more remote server farms and / or data centers). The computer system 520 may comprise any circuit or combination of circuits. In one embodiment, the computer system 520 may include one or more processors which can be of any type. As used herein, processor may mean any type of computational circuit, such as but not limited to a microprocessor, a microcontroller, a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a graphics processor, a digital signal processor (DSP), multiple core processor, a field programmable gate array (FPGA), for example, of a microscope or a microscope component (e.g. camera) or any other type of processor or processing circuit. Other types of circuits that may be included in the computer system 520may be a custom circuit, an application-specific integrated circuit (ASIC), or the like, such as, for example, one or more circuits (such as a communication circuit) for use in wireless devices like mobile telephones, tablet computers, laptop computers, two-way radios, and similar electronic systems. The computer system 520 may include one or more storage devices, which may include one or more memory elements suitable to the particular application, such as a main memory in the form of random access memory (RAM), one or more hard drives, and / or one or more drives that handle removable media such as compact disks (CD), flash memory cards, digital video disk (DVD), and the like. The computer system 520 may also include a display device, one or more speakers, and a keyboard and / or controller, which can include a mouse, trackball, touch screen, voice-recognition device, or any other device that permits a system user to input information into and receive information from the computer system 520.

[0075] Some or all of the method steps may be executed by (or using) a hardware apparatus, like for example, a processor, a microprocessor, a programmable computer or an electronic circuit. In some embodiments, some one or more of the most important method steps may be executed by such an apparatus.

[0076] Depending on certain implementation requirements, embodiments of the invention can be implemented in hardware or in software. The implementation can be performed using a non-transitory storage medium such as a digital storage medium, for example a floppy disc, a DVD, a Blu-Ray, a CD, a ROM, a PROM, and EPROM, an EEPROM or a FLASH memory, having electronically readable control signals stored thereon, which cooperate (or are capable of cooperating) with a programmable computer system such that the respective method is performed. Therefore, the digital storage medium may be computer readable.

[0077] Some embodiments according to the invention comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that one of the methods described herein is performed.

[0078] Generally, embodiments of the present invention can be implemented as a computer program product with a program code, the program code being operative for performing one of the methods when the computer program product runs on a computer. The program code may, for example, be stored on a machine readable carrier.

[0079] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.In other words, an embodiment of the present invention is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0080] A further embodiment of the present invention is, therefore, a storage medium (or a data carrier, or a computer-readable medium) comprising, stored thereon, the computer program for performing one of the methods described herein when it is performed by a processor. The data carrier, the digital storage medium or the recorded medium are typically tangible and / or non-transitionary. A further embodiment of the present invention is an apparatus as described herein comprising a processor and the storage medium.

[0081] A further embodiment of the invention is, therefore, a data stream or a sequence of signals representing the computer program for performing one of the methods described herein. The data stream or the sequence of signals may, for example, be configured to be transferred via a data communication connection, for example, via the internet.

[0082] A further embodiment comprises a processing means, for example, a computer or a programmable logic device, configured to, or adapted to, perform one of the methods described herein.

[0083] A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0084] A further embodiment according to the invention comprises an apparatus or a system configured to transfer (for example, electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may, for example, be a computer, a mobile device, a memory device or the like. The apparatus or system may, for example, comprise a file server for transferring the computer program to the receiver.

[0085] In some embodiments, a programmable logic device (for example, a field programmable gate array) may be used to perform some or all the functionalities of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. Generally, the methods are preferably performed by any hardware apparatus.Embodiments may be based on using a machine-learning model or machine-learning algorithm. Machine learning may refer to algorithms and statistical models that computer systems may use to perform a specific task without using explicit instructions, instead relying on models and inference. For example, in machine-learning, instead of a rule-based transformation of data, a transformation of data may be used, that is inferred from an analysis of historical and / or training data. For example, the content of images may be analyzed using a machine-learning model or using a machine-learning algorithm. In order for the machinelearning model to analyze the content of an image, the machine-learning model may be trained using training images as input and training content information as output. By training the machine-learning model with a large number of training images and / or training sequences (e.g. words or sentences) and associated training content information (e.g. labels or annotations), the machine-learning model "learns" to recognize the content of the images, so the content of images that are not included in the training data can be recognized using the machine-learning model. The same principle may be used for other kinds of sensor data as well: By training a machine-learning model using training sensor data and a desired output, the machine-learning model "learns" a transformation between the sensor data and the output, which can be used to provide an output based on non-training sensor data provided to the machine-learning model. The provided data (e.g. sensor data, meta data and / or image data) may be preprocessed to obtain a feature vector, which is used as input to the machine-learning model.

[0086] Machine-learning models may be trained using training input data. The examples specified above use a training method called "supervised learning". In supervised learning, the machine-learning model is trained using a plurality of training samples, wherein each sample may comprise a plurality of input data values, and a plurality of desired output values, i.e. each training sample is associated with a desired output value. By specifying both training samples and desired output values, the machine-learning model "learns" which output value to provide based on an input sample that is similar to the samples provided during the training. Apart from supervised learning, semi-supervised learning may be used. In semisupervised learning, some of the training samples lack a corresponding desired output value. Supervised learning may be based on a supervised learning algorithm (e.g. a classification algorithm, a regression algorithm or a similarity learning algorithm. Classification algorithms may be used when the outputs are restricted to a limited set of values (categorical variables), i.e. the input is classified to one of the limited set of values. Regression algorithms may be used when the outputs may have any numerical value (within a range). Similarity learning algorithms may be similar to both classification and regression algorithms but are based on learning from examples using a similarity function that measures how similar or related twoobjects are. Apart from supervised or semi-supervised learning, unsupervised learning may be used to train the machine-learning model. In unsupervised learning, (only) input data might be supplied and an unsupervised learning algorithm may be used to find structure in the input data (e.g. by grouping or clustering the input data, finding commonalities in the data). Clustering is the assignment of input data comprising a plurality of input values into subsets (clusters) so that input values within the same cluster are similar according to one or more (pre-defined) similarity criteria, while being dissimilar to input values that are included in other clusters.

[0087] Reinforcement learning is a third group of machine-learning algorithms. In other words, reinforcement learning may be used to train the machine-learning model. In reinforcement learning, one or more software actors (called "software agents") are trained to take actions in an environment. Based on the taken actions, a reward is calculated. Reinforcement learning is based on training the one or more software agents to choose the actions such, that the cumulative reward is increased, leading to software agents that become better at the task they are given (as evidenced by increasing rewards).

[0088] Furthermore, some techniques may be applied to some of the machine-learning algorithms. For example, feature learning may be used. In other words, the machine-learning model may at least partially be trained using feature learning, and / or the machine-learning algorithm may comprise a feature learning component. Feature learning algorithms, which may be called representation learning algorithms, may preserve the information in their input but also transform it in a way that makes it useful, often as a pre-processing step before performing classification or predictions. Feature learning may be based on principal components analysis or cluster analysis, for example.

[0089] In some examples, anomaly detection (i.e. outlier detection) may be used, which is aimed at providing an identification of input values that raise suspicions by differing significantly from the majority of input or training data. In other words, the machine-learning model may at least partially be trained using anomaly detection, and / or the machine-learning algorithm may comprise an anomaly detection component.

[0090] In some examples, the machine-learning algorithm may use a decision tree as a predictive model. In other words, the machine-learning model may be based on a decision tree. In a decision tree, observations about an item (e.g. a set of input values) may be represented by the branches of the decision tree, and an output value corresponding to the item may be represented by the leaves of the decision tree. Decision trees may support both discretevalues and continuous values as output values. If discrete values are used, the decision tree may be denoted a classification tree, if continuous values are used, the decision tree may be denoted a regression tree.

[0091] Association rules are a further technique that may be used in machine-learning algorithms. In other words, the machine-learning model may be based on one or more association rules. Association rules are created by identifying relationships between variables in large amounts of data. The machine-learning algorithm may identify and / or utilize one or more relational rules that represent the knowledge that is derived from the data. The rules may e.g. be used to store, manipulate or apply the knowledge.

[0092] Machine-learning algorithms are usually based on a machine-learning model. In otherwords, the term "machine-learning algorithm" may denote a set of instructions that may be used to create, train or use a machine-learning model. The term "machine-learning model" may denote a data structure and / or set of rules that represents the learned knowledge (e.g. based on the training performed by the machine-learning algorithm). In embodiments, the usage of a machine-learning algorithm may imply the usage of an underlying machine-learning model (or of a plurality of underlying machine-learning models). The usage of a machine-learning model may imply that the machine-learning model and / or the data structure / set of rules that is the machine-learning model is trained by a machine-learning algorithm.

[0093] For example, the machine-learning model may be an artificial neural network (ANN). ANNs are systems that are inspired by biological neural networks, such as can be found in a retina or a brain. ANNs comprise a plurality of interconnected nodes and a plurality of connections, so-called edges, between the nodes. There are usually three types of nodes, input nodes that receiving input values, hidden nodes that are (only) connected to other nodes, and output nodes that provide output values. Each node may represent an artificial neuron. Each edge may transmit information, from one node to another. The output of a node may be defined as a (non-linear) function of its inputs (e.g. of the sum of its inputs). The inputs of a node may be used in the function based on a "weight" of the edge or of the node that provides the input. The weight of nodes and / or of edges may be adjusted in the learning process. In other words, the training of an artificial neural network may comprise adjusting the weights of the nodes and / or edges of the artificial neural network, i.e. to achieve a desired output for a given input.

[0094] Alternatively, the machine-learning model may be a support vector machine, a random forest model or a gradient boosting model. Support vector machines (i.e. support vector networks) are supervised learning models with associated learning algorithms that may be used toanalyze data (e.g. in classification or regression analysis). Support vector machines may be trained by providing an input with a plurality of training input values that belong to one of two categories. The support vector machine may be trained to assign a new input value to one of the two categories. Alternatively, the machine-learning model may be a Bayesian network, which is a probabilistic directed acyclic graphical model. A Bayesian network may represent a set of random variables and their conditional dependencies using a directed acyclic graph. Alternatively, the machine-learning model may be based on a genetic algorithm, which is a search algorithm and heuristic technique that mimics the process of natural selection.List of Reference Numerals

[0095] 10: Sample

[0096] 100: Surgical or scientific imaging system

[0097] 105: Base unit

[0098] 110: System

[0099] 112: One or more interfaces

[0100] 114: One or more processors

[0101] 116: One or more storage devices

[0102] 120: Imaging device

[0103] 130a: Ocular displays

[0104] 130b: Auxiliary display

[0105] 210: Column including images of a left imaging channel

[0106] 220: Column including images of a right imaging channel

[0107] 230: Row showing raw images

[0108] 240: Row showing combined (i.e., composite) images

[0109] 310a: First imaging channel

[0110] 310b: Second imaging channel

[0111] 310c: Third imaging channel

[0112] 320a: First angle of view

[0113] 320b: Second angle of view

[0114] 320c: Third angle of view

[0115] 330: Optical component

[0116] 340: Motor for adjusting the angle of view

[0117] 410: Obtaining imaging data

[0118] 420: Determining an area being affected by an obstruction obscuring a view on a sample 430: Combining a first and second image to obtain a combined view on the sample 440: Providing the combined view

[0119] 500: System

[0120] 510: Imaging device

[0121] 520: Computer system

Claims

Claims1. A system (110) for a surgical or scientific imaging system (100), the system comprising one or more processors and one or more storage devices, wherein the system is configured to:obtain imaging data of at least one optical imaging sensor (120) of the surgical or scientific imaging system, the imaging data comprising a first image representing a first angle of view and a second image representing a second angle of view on a sample (10);determine, for at least one of the first image or the second image, an area being affected by an obstruction obscuring a view on the sample;combine the first image and the second image using image projection based on the area being affected by an obstruction obscuring a view on the sample to obtain a combined view on the sample; andprovide the combined view on the sample for a display device (130) of the surgical or scientific imaging system.

2. The system according to claim 1, wherein the area being affected by an obstruction obscuring a view on the sample is an area that is hidden in one of the first or second image and visible in the respective other of the first or second image.

3. The system according to one of the claims 1 or 2, wherein the system is configured to obtain stereoscopic imaging data from the at least one optical imaging sensor of the surgical or scientific imaging system, the stereoscopic imaging data comprising the first image representing a first channel and the second image representing a second channel of the stereoscopic imaging data.

4. The system according to one of the claims 1 to 3, wherein the system is configured to determine a depth map of the sample, and to combine the first image and the second image using image projection based on the area being affected by an obstruction obscuring a view on the sample using the depth map to obtain the combined view on the sample5. The system according to claim 4, wherein the system is configured to determine the depth map of the sample using stereo disparity analysis based on the first and second image.

6. The system according to one of the claims 4 or 5, wherein the system is configured to map, using image projection, one or more image segments of the respective other image onto the area being affected by the obstruction using the depth map of the sample.

7. The system according to claim 6, wherein the system is configured to represent the area onto which the one or more image segments of the respective other image are mapped with an indicator representing the mapping operation.

8. The system according to one of the claims 4 to 7, wherein the system is configured to determine the area being affected by an obstruction obscuring a view on the sample based on a depth disparity between a first depth map being generated starting from the first image and a second depth map being generated starting from the second image.

9. The system according to one of the claims 1 to 8, wherein the system is configured to obtain the imaging data further with at least one third image representing at least one third angle of view on the sample, and to use the first, second and third image to determine the area being affected by the obstruction obscuring a view on the sample, obtain the combined view on the sample and / or to determine a depth map.

10. The system according to claim 9, wherein the system is configured to obtain the at least one third image from an optical imaging sensor being associated with a different imaging modality than at least one optical imaging sensor providing the first and second image.

11. The system according to one of the claims 1 to 10, wherein the system is configured to combine the first image and the second image using image projection to obtain a two-dimensional or three-dimensional combined view on the sample, and to provide the combined view for a 2D or 3D monitor (130b) of the surgical or scientific imaging system,and / or wherein the system is configured to combine the first image and the second image using image projection to obtain a stereoscopic combined view on the sample, and to provide the stereoscopic combined view to a stereoscopic display device (130a) of the surgical or scientific imaging system.

12. A surgical or scientific imaging system comprising at least one optical imaging sensor, a display device, and the system according to one of the claims 1 to 11.

13. The surgical or scientific imaging system according to claim 12, wherein the surgical or scientific imaging system comprises a first optical imaging sensor for providing the first image and a second optical imaging sensor for providing the second image, with the system being configured to control the surgical or scientific imaging system to adjust the second angle of view based on the area being affected by an obstruction obscuring a view on the sample,or wherein the surgical or scientific imaging system comprises a first optical imaging sensor for providing the first image, a second optical imaging sensor for providing the second image and a third optical imaging sensor for providing a third image, with the system being configured to control the surgical or scientific imaging system to adjust a third angle of view on the sample of a the third image based on the area being affected by an obstruction obscuring a view on the sample, and to use the first, second and third image to determine the area being affected by the obstruction obscuring a view on the sample, obtain the combined view on the sample and / or to determine a depth map.

14. A method for a surgical or scientific imaging system, the method comprising:obtaining (410) imaging data of at least one optical imaging sensor of the surgical or scientific imaging system, the imaging data comprising a first image representing a first angle of view and a second image representing a second angle of view on a sample;determining (420), for at least one of the first image or the second image, an area being affected by an obstruction obscuring a view on the sample;combining (430) the first image and the second image using image projection based on the area being affected by an obstruction obscuring a view on the sample to obtain a combined view on the sample, andproviding (440) the combined view on the sample for a display device of the surgical or scientific imaging system.

15. A computer program having a program code for performing a method according to claim 14 when the program is executed on processor.

Images