Medical image processing device, medical image processing system, and medical image processing method

Abstract

A medical image processing device according to an embodiment comprises: an observation image generation unit that generates a medical observation image from a medical captured image; a depth estimation processing unit that generates depth information related to a depth in the medical captured image so that the depth resolution of a site of interest in the medical captured image is higher than that of a region other than the site of interest; and a display image generation unit that generates a display image including the medical observation image and the depth information.

Description

Medical image processing apparatus, medical image processing system, and medical image processing method

[0001] The present disclosure relates to a medical image processing apparatus, a medical image processing system, and a medical image processing method.

[0002] In an endoscope system, for example, the inside of a patient is imaged by a monocular camera attached to the distal end of an endoscope. In order to obtain depth information (depth-of-field information) from the endoscope image captured by this monocular camera, a technique for estimating depth information from the endoscope image using a learning model has been proposed (for example, see Patent Document 1).

[0003] Japanese Unexamined Patent Application Publication No. 2022-172654

[0004] However, even if the endoscope image and depth information are displayed, it is difficult for users such as operators and assistants to grasp the sense of depth (sense of depth of field) of a region of interest (such as a tumor or a polyp) in the endoscope image, and the visibility of the region of interest in the depth direction (depth-of-field direction) is low.

[0005] Accordingly, the present disclosure provides a technique capable of improving the visibility of a region of interest in the depth direction.

[0006] A medical image processing apparatus according to an embodiment includes an observation image generation unit that generates a medical observation image from a medical captured image, a depth estimation processing unit that generates depth information related to the depth of the medical captured image such that the depth resolution of a region of interest in the medical captured image is higher than that of other regions other than the region of interest, and a display image generation unit that generates a display image including the medical observation image and the depth information.

[0007] A medical image processing system according to an embodiment includes an imaging apparatus that acquires a medical captured image, and a medical image processing apparatus that processes the medical captured image, wherein the medical image processing apparatus has an observation image generation unit that generates a medical observation image from the medical captured image, a depth estimation processing unit that generates depth information related to the depth of the medical captured image such that the depth resolution of a region of interest in the medical captured image is higher than that of other regions other than the region of interest, and a display image generation unit that generates a display image including the medical observation image and the depth information.

[0008] The medical image processing method according to the embodiment includes a medical image processing device generating a medical observation image from a medical image captured image, generating depth information relating to the depth of the medical image captured image such that the depth resolution of the area of ​​interest in the medical image captured image is higher than that of other areas other than the area of ​​interest, and generating a display image including the medical observation image and the depth information.

[0009] This is a diagram illustrating an example configuration of an endoscope system according to an embodiment. This is a diagram illustrating an example configuration of a control device according to an embodiment. This is a diagram illustrating a first example of image generation according to an embodiment. This is a diagram illustrating a second example of image generation according to an embodiment. This is a diagram illustrating a third example of image generation according to an embodiment. This is a diagram illustrating a fourth example of image generation according to an embodiment. This is a diagram illustrating a fifth example of image generation according to an embodiment. This is a flowchart illustrating a first example of image generation according to an embodiment. This is a flowchart illustrating a second example of image generation according to an embodiment. This is a diagram illustrating an example of a depth estimation method according to an embodiment. This is a diagram illustrating an example of a depth estimation method according to an embodiment. This is a diagram illustrating an example configuration of an endoscope system according to a first application example. This is a block diagram illustrating an example configuration of an endoscope system according to a first application example. This is a block diagram illustrating an example configuration of a CCU (Camera Control Unit) according to a first application example. This is a diagram illustrating an example configuration of an endoscope system according to a second application example. This is a diagram illustrating an example configuration of an endoscope and CCU according to a second application example. This is a diagram illustrating an example configuration of hardware.

[0010] Embodiments of this disclosure will be described in detail below with reference to the drawings. Embodiments include examples and modifications. However, the technology relating to this disclosure is not limited by the embodiments. In addition, in the following embodiments, the same reference numerals are used for essentially the same parts to omit redundant explanations.

[0011] This disclosure will be described in the following order of items: 1. Embodiments 1-1. Example of Endoscope System Configuration 1-2. Example of Control Device Configuration 1-3. Example of Image Generation 1-4. Example of Image Generation Processing 1-5. Example of Depth Estimation Method 1-6. Overview 2. Other Embodiments 3. Application Examples 3-1. First Application Example 3-2. Second Application Example 4. Hardware Configuration Example 5. Notes

[0012] <1. Embodiments> <1-1. Example of Endoscope System Configuration> An example of the configuration of the endoscope system 1 according to the embodiment will be described with reference to Figure 1. Figure 1 is a diagram showing an example of the configuration of the endoscope system 1 according to the embodiment. The endoscope system 1 in Figure 1 is configured assuming a flexible endoscope, but it may also be configured assuming a rigid endoscope.

[0013] As shown in Figure 1, the endoscope system 1 according to this embodiment comprises a light source device 10, a light guide unit 20, an imaging device 30, a signal line 40, and a control device 50. This endoscope system 1 is a device for observing the inside of a living body, and generates an image of the observation target 2 while irradiating the observation target 2 with light 3. The endoscope system 1 is an example of a medical image processing system, and the control device 50 is an example of a medical image processing device.

[0014] The light source device 10 generates light 3 to irradiate the object to be observed 2. This light source device 10 generates, for example, white light for observation. The light source device 10 transmits the generated white light to the object to be observed 2 via the light guide unit 20.

[0015] The light guide unit 20 guides the light 3 emitted from the light source device 10. This light guide unit 20 is made of, for example, an optical fiber. The light guide unit 20 has a light emitting unit 21 that emits the light 3. The light 3 guided by the light guide unit 20 is irradiated onto the object to be observed 2 from the light emitting unit 21. The light guide unit 20 is also called a light guide.

[0016] The imaging device 30 generates an image of the object to be observed 2. Based on the control of the control device 50, the imaging device 30 performs imaging and generates an image (medical image). The imaging device 30 transmits the generated image to the control device 50 via the signal line 40.

[0017] The signal line 40 electrically connects the imaging device 30 and the control device 50. In other words, the imaging device 30 and the control device 50 are communicated with each other via the signal line 40.

[0018] The control device 50 controls the imaging device 30. This control device 50 includes, for example, a camera control unit (CCU). The control device 50 also processes, for example, the captured images transmitted from the imaging device 30.

[0019] In this type of endoscope system 1, when observing the object to be observed 2, the light source device 10 emits white light, the imaging device 30 takes images of the object to be observed 2 based on the white light and acquires the captured images, and the control device 50 generates observation images (medical observation images) from the captured images. This is done based on the control of the control device 50.

[0020] <1-2. Example of Control Device Configuration> An example of the configuration of the control device 50 according to the embodiment will be described with reference to Figure 2. Figure 2 is a diagram showing an example of the configuration of the control device 50 according to the embodiment.

[0021] As shown in Figure 2, the control device 50 according to this embodiment includes an observation image generation unit 51, an object detection processing unit 52, a depth estimation processing unit 53, and a display image generation unit 54. An imaging device 30 is connected to this control device 50 for communication, and a monitor 60 and a touch sensor 61 are also connected for communication. The monitor 60 is an example of a display device, and the touch sensor 61 is an example of an input device.

[0022] The imaging device 30 captures images of the subject. For example, the imaging device 30 captures images of a patient's body part (e.g., the large intestine) as the subject and transmits the captured images to the control device 50. Various types of cameras can be used as the imaging device 30.

[0023] The observation image generation unit 51 generates an observation image from the captured image and outputs the generated observation image to the display image generation unit 54. For example, the observation image generation unit 51 performs various processes on the captured image to generate an observation image. These various processes include, for example, development processing. Development processing may include, for example, exposure adjustment, contrast adjustment, white balance adjustment, saturation adjustment, cropping, and perspective adjustment.

[0024] The object detection processing unit 52 detects the user's area of ​​interest (e.g., a tumor or polyp) from the captured image and outputs object position information of the detected area of ​​interest (e.g., object region information of the area of ​​interest) to the depth estimation processing unit 53. For example, the object detection processing unit 52 detects the area of ​​interest from the captured image using an AI (Artificial Intelligence) object detection method such as deep learning. The AI ​​functions, for example, as a recognizer composed of a trained model.

[0025] Object detection methods include, for example, YOLO (You Only Look Once) and SSD (Single Shot MultiBox Detector). YOLO, for example, divides an image into a grid and determines whether or not an object exists in each grid cell. SSD, for example, processes an image using a convolutional neural network (CNN) and detects objects using grids of different sizes.

[0026] In addition to object detection methods such as YOLO and SSD, other AI-based object detection methods may also be used.

[0027] The depth estimation processing unit 53 generates depth information related to the depth of the captured image and outputs the generated depth information to the display image generation unit 54. Depth is the position in the depth direction (depth direction), and depth information is also called depth information. For example, the depth estimation processing unit 53 generates depth information based on the object position information of the area of ​​interest so that the depth resolution (position resolution in the depth direction) of the area of ​​interest in the captured image is higher than that of other areas other than the area of ​​interest. For example, the depth estimation processing unit 53 generates depth information from the captured image using an AI-based depth estimation method such as deep learning.

[0028] As a depth estimation method, for example, a trained model (monocular depth estimation model) that has learned the correlation between depth (depth) and brightness can be used. Alternatively, methods using structured light and triangulation techniques, or methods using distance sensors such as Lidar sensors may also be used. A more detailed explanation of depth estimation methods using monocular depth estimation models will be provided later.

[0029] The display image generation unit 54 generates a display image that includes the observation image and depth information. For example, the display image generation unit 54 generates a depth image that indicates the depth of the observation image based on the depth information, and generates a display image that includes the observation image and the depth image. In image generation, for example, the display image is generated by superimposing the depth image on the observation image, or by placing the observation image and the depth image side by side.

[0030] Depth images can be used in various ways, such as heatmaps or meshmaps. Heatmaps visually represent numerical data (depth) using color gradients. Meshmaps divide a surface into many meshes and show numerical data (depth) for each mesh.

[0031] The monitor 60 displays various images. These various images include, for example, display images that include observation images and depth images. As mentioned above, display images include, for example, display images in which depth images are superimposed on observation images, and display images in which observation images and depth images are displayed side by side.

[0032] The touch sensor 61 is located on the monitor 60. The touch sensor 61 accepts input operations from a user, such as a surgeon or assistant. For example, the touch sensor 61 detects the user's touch position on the monitor 60 screen and transmits touch position information indicating the detected touch position to the depth estimation processing unit 53. The user performs an input operation, for example, by touching and specifying a planar position on the monitor 60 screen (display image).

[0033] Although the touch sensor 61 is used as an input device, other input devices such as a mouse, stick, or buttons, or an audio input unit may also be used.

[0034] <1-3. Examples of Image Generation> Examples of image generation according to the embodiment (first to fifth generation examples) will be explained with reference to Figures 3 to 7.

[0035] (First Generation Example) Figure 3 is a diagram illustrating a first generation example of image generation according to the embodiment.

[0036] As shown in Figure 3, in the first generation example, the observation image G1 and depth image G2 are superimposed by the display image generation unit 54 to generate the display image G3. This display image G3 is displayed on the monitor 60. In the example in Figure 3, the depth image G2 is a heatmap, and the display image G3 also includes a color bar A1. The color bar A1 is a tool that shows the correspondence between colors and numerical values ​​(depth). For example, the depth resolution of the depth image G2 is set by dividing the depth range (e.g., reference depth range) from the foreground (near distance) to the background (far distance) of the entire captured image into a predetermined number (e.g., 256).

[0037] If the object of interest is not present in the captured image, the object of interest is not detected by the object detection processing unit 52. The depth estimation processing unit 53 acquires depth information up to the distant view of the observation image G1 with the predetermined depth resolution mentioned above, and the depth image G2 is generated by the display image generation unit 54 based on that depth information. Subsequently, the depth image G2 is superimposed on the observation image G1 to generate the display image G3.

[0038] For example, the depth image G2 and the observation image G1 are overlaid so that the user can view the observation image G1 from the display image G3. For example, the depth image G2 may be transparent, and the shape of a part (e.g., the large intestine) within the display image G3 may be emphasized. In these cases, the user can understand the depth information while viewing the observation image G1 together with the depth image G2. Furthermore, if the depth image G2 is transparent, the degree of transparency may be changeable by the user. For example, the user may perform an input operation such as touching the monitor 60, and the control device 50 may change the transparency in response to that input operation.

[0039] (Second Generation Example) Figure 4 is a diagram illustrating a second generation example of image generation according to the embodiment.

[0040] As shown in Figure 4, in the second generation example, the observation image G1 and depth image G2 are superimposed by the display image generation unit 54 to generate the display image G3. The display image G3 is displayed on the monitor 60. As in the example in Figure 3, the depth image G2 is a heat map, and the display image G3 also includes a color bar A1.

[0041] In the example in Figure 4, the depth resolution of area A2 is higher than that of other areas. The depth resolution of area A2 is set by dividing the depth range from the area in front of (near) to the area behind (far) of area A2 into a predetermined number of divisions (for example, 256). For example, when area A2 is detected, the minimum and maximum depths of area A2 are calculated, and the depth range from the minimum to the maximum depth is divided into 256 divisions, thereby increasing the depth resolution of area A2.

[0042] When a region of interest A2 is present in a captured image, the region of interest A2 is detected by an object detection processing unit 52. When the region of interest A2 is detected, depth information for an observation image G1 including the region of interest A2 is acquired by a depth estimation processing unit 53. At this time, the position of the region of interest A2 is specified based on object position information of the region of interest A2, and depth information in which the depth resolution of the region of interest A2 is higher than that of other portions is generated. Thereafter, a depth image G2 is generated by a display image generation unit 54 based on the depth information, and the depth image G2 is superimposed on the observation image G1 to generate a display image G3.

[0043] According to such image generation, depth information with an increased depth resolution for the region of interest A2 is generated, a depth image G2 based on the depth information is generated, and a display image G3 including the depth image G2 is generated. Thereby, the visibility of the region of interest A2 in the depth direction can be improved. Normally, it is difficult to perceive the depth sensation (depth perception) of the shape of the region of interest A2 only with a two-dimensional observation image G1 or a simple display of depth information, and the visibility of the region of interest A2 in the depth direction is low.

[0044] (Third Generation Example) FIG. 5 is a diagram for describing a third generation example of image generation according to the embodiment.

[0045] As illustrated in FIG. 5, in the third generation example, unlike the first generation example and the second generation example, the observation image G1 and the depth image G2 are arranged side by side by the display image generation unit 54, and the display image G3 is generated. The display image G3 is displayed on a monitor 60. In the example of FIG. 5, the depth image G2 is reduced. Note that, similarly to the example of FIG. 3, the depth image G2 is a heat map, and the display image G3 also includes a color bar A1. According to the third generation example, the same effect as that of the second generation example can be obtained.

[0046] In the example of FIG. 5, the depth image G2 is reduced, but the observation image G1 may be reduced, or the depth image G2 may be enlarged as necessary. The reduction or enlargement of these images may be made changeable by a user. For example, the user may perform an input operation such as a touch operation on the monitor 60, and the control device 50 may reduce or enlarge the images in accordance with the input operation.

[0047] (Fourth Generation Example) FIG. 6 is a diagram for explaining a fourth generation example of image generation according to the embodiment.

[0048] The fourth generation example is basically the same as the aforementioned second generation example, but as shown in FIG. 6, the display image G3 is provided with target site information A3 related to the target site A2. Specifically, the observation image G1, the depth image G2, and the target site information A3 are superimposed in the stated order to generate the display image G3. In the example of FIG. 6, the target site information A3 is provided around the target site A2, for example, above the target site A2.

[0049] The target site information A3 includes, for example, any one or more of numerical values such as the maximum size (maximum value) of the target site A2, the minimum size (minimum value) of the target site A2, the maximum depth (maximum value) of the target site A2, the minimum depth (minimum value) of the target site A2, and the area of the target site A2. The maximum size of the target site A2 is, for example, a two-dimensional size, but when the three-dimensional shape of the target site A2 can be obtained by processing such as AI, it may be a three-dimensional size. In this case, since the volume of the target site A2 can be obtained, the target site information A3 may include the volume of the target site A2.

[0050] The target site information A3 is generated by the depth estimation processing unit 53 using the aforementioned depth estimation method. Further, the display image G3 including the target site information A3, the observation image G1, and the depth image G2 is generated by the display image generation unit 54. For example, the display image generation unit 54 superimposes the observation image G1, the depth image G2, and the target site information A3 to generate the display image G3.

[0051] Normally, it is difficult for a user to grasp the actual size of the target site A2 from the observation image G1. For example, the user inserts a ruler or forceps into the patient to show it in the observation image G1, measures the size of the target site A2 with the ruler, or estimates the size of the target site A2 from the size of the forceps. Such work takes time. On the other hand, by displaying the target site information A3 on the monitor 60 as described above, the user can easily grasp information such as the size of the target site A2, so that the time can be reduced.

[0052] (Fifth Generation Example) Figure 7 is a diagram illustrating a fifth generation example of image generation according to the embodiment.

[0053] The fifth generation example is basically the same as the fourth generation example described above, and as shown in Figure 7, the displayed image G3 is provided with focus area information A3 related to the focus area A2. Specifically, the focus area information A3 is placed next to the observation image G1 and the depth image G2, and the displayed image G3 is generated. In the example in Figure 7, the focus area information A3 is provided below the color bar A1.

[0054] The area of ​​interest information A3 is generated by the depth estimation processing unit 53 using the depth estimation method described above, similar to the fourth generation example. The display image G3, which includes the area of ​​interest information A3, the observation image G1, and the depth image G2, is generated by the display image generation unit 54. For example, the display image generation unit 54 generates the display image G3 by placing the superimposed image of the observation image G1 and the depth image G2 and the area of ​​interest information A3 side by side. The fifth generation example can achieve the same effect as the fourth generation example.

[0055] <1-4. Image Generation Process Examples> An example of the image generation process according to the embodiment will be described with reference to Figures 8 and 9. Figure 8 is a flowchart showing a first image generation process according to the embodiment. Figure 9 is a flowchart showing a second image generation process according to the embodiment.

[0056] For example, if the endoscope system 1 (control device 50) is operating at 60 fps, the calculation process is executed 60 times per second. That is, the processes from start to end in Figures 8 and 9 are performed 60 times per second.

[0057] (First Processing Example) As shown in Figure 8, in the first processing example, the object detection processing unit 52 determines whether or not it has detected a tumor (an example of the area of ​​interest A2) (step S11). If the object detection processing unit 52 determines that it has not detected a tumor (step S11: No), the depth estimation processing unit 53 outputs depth information calculated to the distant view of the observation image G1 with a predetermined depth resolution to the display image generation unit 54 (step S12). Subsequently, the display image generation unit 54 superimposes the observation image G1 and the depth information of the tumor (step S13). This creates a display image G3 in which the observation image G1 and the depth information to the distant view (for example, depth image G2) are superimposed (see Figure 3).

[0058] On the other hand, if the object detection processing unit 52 determines that it has detected a tumor (step S11: Yes), the depth estimation processing unit 53 outputs depth information with increased depth resolution of the tumor in the captured image to the display image generation unit 54 (step S14). The control device 50 determines whether or not to display other information (tumor information) other than the tumor depth information (step S15).

[0059] If it is determined that no information other than the tumor depth information should be displayed (step S15: No), the display image generation unit 54 superimposes the observation image G1 and the tumor depth information (step S13). This creates a display image G3 in which the observation image G1 and the tumor depth information (for example, depth image G2) are superimposed (see Figure 4).

[0060] On the other hand, if it is determined that information other than the tumor depth information should be displayed (step S15: Yes), the depth estimation processing unit 53 adds other information about the tumor (for example, numerical values ​​such as maximum size, minimum size, maximum depth, minimum depth, and area) to the depth information and outputs it to the display image generation unit 54 (step S16). Subsequently, the display image generation unit 54 superimposes the observation image G1 with the tumor depth information and other information (step S13). This creates a display image G3 in which the observation image G1 and the tumor depth information (for example, depth image G2 and area of ​​interest information A3) are superimposed (see Figure 6).

[0061] Whether or not to display information other than the aforementioned tumor depth information (tumor information) is determined by the user, for example. Whether or not to display this other information (tumor information) may be determined in advance or determined on a case-by-case basis. For example, the user may perform an input operation such as touching the monitor 60, and the control device 50 may change whether or not to display the information in response to that input operation.

[0062] (Second Processing Example) As shown in Figure 9, in the second processing example, it is determined whether or not the touch sensor 61 has detected a user touch (step S21). If it is determined that the touch sensor 61 has not detected a user touch (step S21: No), the processing from steps S11 to S16 in Figure 8 is executed (step S22).

[0063] On the other hand, if the touch sensor 61 determines that it has detected a user touch (step S21: Yes), it determines whether there are three or more touch locations by the user (step S23). If it determines that there are three or more touch locations by the user (step S23: Yes), the touch sensor 61 processes each touch location as a touch error (step S24). Subsequently, the display image generation unit 54 superimposes the observed image G1 and the tumor depth information (step S25).

[0064] Furthermore, if the user touches three or more points, i.e., during touch error processing, each touch point is treated as an error, and depth information is not output from the depth estimation processing unit 53. Therefore, the previous value is retained as depth information.

[0065] On the other hand, if it is determined that the user's touch locations are not three or more points (step S23: No), it is determined whether the user's touch locations are two or more points (step S26). If it is determined that the user's touch locations are two or more points (step S26: Yes), the depth estimation processing unit 53 outputs length information between the two points of the user's touch locations and depth information with increased depth resolution at the median between the two points (step S27), and proceeds to step S25.

[0066] On the other hand, if it is determined that there are not two or more touch points by the user (step S26: No), the depth estimation processing unit 53 outputs depth information with increased depth resolution at the user's touch points (step S28), and proceeds to step S25.

[0067] The length information between two points includes, for example, the length of the straight line between the two touched points. If there are two touch points, the straight line defined by the two points functions as the area of ​​interest that the user focuses on, and the midpoint between the two points functions as the point of interest that the user focuses on. If there is one touch point, that single point functions as the point of interest that the user focuses on. The depth information is formed such that the depth resolution at the point of interest is higher than at other locations, for example.

[0068] The depth resolution at the median (center position) between two points is set by dividing the depth range of the straight line defined by the two points into a predetermined number of segments (e.g., 256). For example, the minimum and maximum depths of the straight line between the two points are calculated, and the depth range from the minimum to the maximum depth is divided into 256 segments, thereby increasing the depth resolution at the median between the two points.

[0069] The depth resolution at a single touch point is set by dividing a predetermined depth range into a predetermined number of segments (e.g., 256). The predetermined depth range is set to be narrower than the depth range before the single touch point (e.g., the reference depth range). Therefore, the depth resolution at a single touch point can be increased compared to the depth resolution before the single touch point.

[0070] <1-5. An Example of Depth Estimation Method> An example of a depth estimation method according to the embodiment will be described with reference to Figures 10 and 11. Figures 10 and 11 are diagrams illustrating an example of a depth estimation method according to the embodiment.

[0071] The example in Figure 10 shows illumination conditions using a point light source (LS), which is common in the field of monocular endoscopy. There is a correlation between image brightness and depth. For example, in an illumination environment using a point light source (LS), brightness decay proportional to the square of the distance is observed based on the inverse square measurement of illuminance. The correlation between brightness and depth can change depending on the illumination conditions. Depth information is estimated from the RAW signal (e.g., the acquired image) based on the pre-learned correlation between brightness and depth.

[0072] In the example shown in Figure 11, an object OB (SB) is photographed under lighting conditions using a point light source LS. The object OB (SB) has colors such as red, green, and blue. Although the brightness differs for each color depending on the wavelength characteristics (light source spectrum, transmission characteristics of the color filter, spectral reflectance characteristics of the object OB (SB), etc.), depth information can be obtained from the brightness.

[0073] The aforementioned depth estimation processing unit 53 uses a monocular depth estimation model to acquire depth information of the subject from the brightness image, which is an image captured by a monocular camera. The monocular camera is an example of the aforementioned imaging device 30. The monocular depth estimation model is a learned model that has learned the correlation between brightness and depth from, for example, a RAW signal (e.g., an image). The RAW signal has not undergone significant adjustments to color or brightness, as would be done in development processing. The monocular depth estimation model is an example of a pre-trained model using machine learning.

[0074] The monocular depth estimation model is trained using a set of images obtained under lighting conditions consistent with the actual shooting environment as training data. Training can be, for example, supervised learning or self-supervised learning. When using RAW signals that have undergone correction processing such as tone curve correction or frequency correction as input to the monocular depth estimation model, a set of images that have undergone the same correction processing can be used as training data.

[0075] Methods for acquiring depth information include those using structural light and triangulation techniques, and those using Lidar sensors. While these methods can acquire highly accurate depth information, they require specialized optical systems and dedicated sensors, resulting in large and complex equipment. This not only increases costs but also eliminates the possibility of system configurations like the Scopist Robot, which utilizes existing endoscopes. Depth estimation using stereo image matching, which does not require special lighting, can also be used, but this is limited to stereo endoscopes. Depth estimation processing using AI, as mentioned above (for example, depth estimation processing using a monocular depth estimation model), makes it possible to obtain accurate depth information that reflects the brightness information contained in the RAW signal, and to acquire highly accurate depth information (depth information) without using specialized equipment configurations.

[0076] Furthermore, by taking depth (depth) into account the shape of the object, the accurate shape of the observed object may be reconstructed. For example, the external shape of the observed object projected onto a two-dimensional plane from a predetermined direction may be obtained. Normally, obtaining an accurate shape that takes depth (depth) into account requires the experience of a physician or the effort of actually creating an object, inserting it into the affected area, and reshaping it. The method of this embodiment makes it possible to derive an accurate shape without requiring such experience or effort from a physician.

[0077] <1-6. Overview> As described above, the endoscope system 1 according to the embodiment (for example, the control device 50) includes an observation image generation unit 51 that generates an observation image G1 from an acquired image, a depth estimation processing unit 53 that generates depth information related to the depth of the acquired image so that the depth resolution of the area of ​​interest A2 in the acquired image is higher than that of other areas other than the area of ​​interest A2, and a display image generation unit 54 that generates a display image G3 including the observation image G1 and the depth information (see Figure 2). As a result, depth information with increased depth resolution of the area of ​​interest A2 is generated, and a display image G3 including that depth information is formed, thereby improving the visibility of the area of ​​interest A2 in the depth direction.

[0078] Furthermore, the display image generation unit 54 may generate a depth image G2 indicating the depth of the captured image based on depth information, and generate an image G3 that includes the observation image G1 and the depth image G2 (see Figures 4 and 5). This makes it possible to reliably improve the visibility of the area of ​​interest A2 in the depth direction.

[0079] Furthermore, the display image generation unit 54 may generate the aforementioned image (display image G3) by superimposing the depth image G2 onto the observation image G1 (see Figure 4). This reliably improves the visibility of the area of ​​interest A2 in the depth direction.

[0080] Furthermore, the display image generation unit 54 may generate the aforementioned image (display image G3) by placing the observation image G1 and the depth image G2 side by side (see Figure 5). This reliably improves the visibility of the area of ​​interest A2 in the depth direction.

[0081] Furthermore, the depth image G2 may be a heatmap or a meshmap (see Figures 4 and 5). This ensures improved visibility of the area of ​​interest A2 in the depth direction.

[0082] Furthermore, the depth estimation processing unit 53 may estimate depth information from images captured by a monocular camera using a pre-trained model obtained through machine learning (for example, a monocular depth estimation model) (see Figures 2, 10, and 11). This reliably improves the visibility of the area of ​​interest A2 in the depth direction.

[0083] Furthermore, the depth estimation processing unit 53 may generate focus area information A3 related to the focus area A2, and the display image generation unit 54 may generate an image G3 as a display image G3 that includes the focus area information A3, the observation image G1, and depth information (for example, the depth image G2) (see Figures 6 and 7). This allows the user to grasp the focus area information A3.

[0084] Furthermore, the display image generation unit 54 may generate the aforementioned image (display image G3) by overlaying the area of ​​interest information A3 onto the observed image G1 (see Figure 6). This allows the user to reliably grasp the area of ​​interest information A3.

[0085] Furthermore, the display image generation unit 54 may generate the aforementioned image (display image G3) by placing the observation image G1 and the area of ​​interest information A3 side by side (see Figure 7). This allows the user to reliably grasp the area of ​​interest information A3.

[0086] Furthermore, the area of ​​interest information A3 may include one or more of the following: the maximum size of area of ​​interest A2, the minimum size of area of ​​interest A2, the maximum depth of area of ​​interest A2, the minimum depth of area of ​​interest A2, and the area of ​​area of ​​interest A2 (see Figures 6 and 7). This ensures that the user can reliably grasp the area of ​​interest information A3.

[0087] Furthermore, the endoscope system 1 according to the embodiment (for example, the control device 50) may further include an object detection processing unit 52 that detects the area of ​​interest A2 from the captured image (see Figure 2). This makes it possible to detect the area of ​​interest A2 from the captured image, and reliably improves the visibility of the area of ​​interest A2 in the depth direction.

[0088] Furthermore, the object detection processing unit 52 may detect the area of ​​interest A2 using YOLO (You Only Look Once) or SSD (Single Shot multiBox Detector) (see Figure 2). This ensures reliable detection of the area of ​​interest A2 from the captured image.

[0089] Furthermore, if the object detection processing unit 52 does not detect the area of ​​interest A2, the depth estimation processing unit 53 may generate depth information with a predetermined depth resolution (see Figures 3 and 8). This makes it possible to obtain depth information even when the area of ​​interest A2 is not present.

[0090] Furthermore, when the object detection processing unit 52 detects the area of ​​interest A2, the depth estimation processing unit 53 may generate depth information such that the depth resolution of the area of ​​interest A2 is higher than that of other areas (see Figures 4, 5, and 8). This ensures that the visibility of the area of ​​interest A2 in the depth direction is improved when the area of ​​interest A2 is present.

[0091] Furthermore, if the object detection processing unit 52 detects the area of ​​interest A2, the depth estimation processing unit 53 may generate area of ​​interest information A3 related to the area of ​​interest A2 (see Figures 6, 7, and 8). This ensures that area of ​​interest information A3 is reliably obtained when the area of ​​interest A2 is present.

[0092] Furthermore, the depth estimation processing unit 53 may generate depth information such that the depth resolution at a location on the observation image G1 is higher than that at other locations, in response to the user specifying a location on the observation image G1 through input operations (see Figure 9). This improves the visibility in the depth direction at the location of interest that the user is focusing on.

[0093] Furthermore, the depth estimation processing unit 53 may measure the distance between two points on the observation image G1 in response to the user specifying the positions of two points on the observation image G1 through input operations (see Figure 9). This allows the user to obtain the distance between two points of interest (for example, the size of the area of ​​interest A2).

[0094] Furthermore, the depth estimation processing unit 53 may generate depth information such that the depth resolution at the central position of the two points on the observation image G1 is higher than that at other positions (for example, positions other than between the two points of interest to the user) in response to the user specifying the positions of two points on the observation image G1 through input operations (see Figure 9). This reliably improves the visibility in the depth direction between the two points of interest to the user (for example, the area of ​​interest A2).

[0095] <2. Other Embodiments> The configurations and processes described in the above-described embodiments (including examples and modifications) may be implemented in various other forms besides those described above. For example, the configurations and processes may be in various forms, not limited to the examples described above. Also, for example, the configurations, processing procedures, specific names, and information including various data and parameters shown in the above document and drawings may be changed at will unless otherwise specified.

[0096] Furthermore, the configurations and processes described in the above-mentioned embodiments (including examples and modifications) do not necessarily have to be physically configured as shown in the figures. In other words, the specific forms of distribution and integration of each configuration and process are not limited to those shown in the figures, and all or part of them may be functionally or physically distributed and integrated in any unit depending on various loads and usage conditions.

[0097] Furthermore, the various configurations and processes described in the above-mentioned embodiments (including examples and modifications) may be combined as appropriate. For example, at least a part of one embodiment may be combined with at least a part of another embodiment as appropriate. Also, the effects described in the embodiments are merely illustrative and not limiting, and other effects may also occur.

[0098] Furthermore, in the embodiments described above, the control device 50 is, for example, a CCU, but is not particularly limited. That is, the control device 50 is not limited to a CCU, but may be, for example, a general-purpose computer, a dedicated computer, or a mobile terminal (for example, a notebook PC (personal computer)). In addition, the control device 50 may be a relay device that relays signals from the imaging device 30 to the CCU, or it may be a recording device or a display device.

[0099] Furthermore, the endoscope system 1 may be connected to a server via a network (e.g., the Internet). The server on the network may perform processing (e.g., image processing) that the control device 50 executes. The server on the network may also perform AI-based processing. The server on the network may then transmit the analysis results to the endoscope system 1 (e.g., the control device 50).

[0100] Furthermore, the imaging device 30 may be capable of imaging both normal light and special light. Special light is light in a predetermined wavelength band different from normal light, which is the light used for normal light observation. For example, special light is at least one of near-infrared light, infrared light, blue light, and ultraviolet light. Normal light is, for example, white light or green light.

[0101] In this embodiment, the system refers to a collection of multiple components (e.g., devices, modules, etc.), regardless of whether all components are located in the same enclosure. For example, multiple devices housed in separate enclosures and connected via a network, as well as a single device containing multiple modules within a single enclosure, are all considered systems. Furthermore, a cloud computing configuration in which a single function is shared and collaboratively processed by multiple devices via a network may also be used.

[0102] <3. Application Examples> The endoscope system 1 according to the embodiment has been described above. Below, the endoscope systems 1000 and 5000 to which the above embodiment can be applied will be described in more detail.

[0103] <3-1. First Application Example> First, as a first application example of the endoscope system 1, an endoscope system 1000 equipped with a flexible endoscope will be described with reference to Figures 12 to 14.

[0104] Figure 12 shows an example configuration of an endoscope system 1000 according to the first application example. As shown in Figure 12, the endoscope system 1000 comprises an endoscope 200, a light source device 300, a CCU (Camera Control Unit) 400, a display device 500, a treatment device 600, a recording device 700, and an output device 800.

[0105] The endoscope 200 is a device that captures images of the inside of a subject's body by inserting its tip into the subject's body. The endoscope 200 comprises a flexible, elongated insertion section 210, an operation section 220 connected to the proximal end of the insertion section 210 that receives input of various operation signals, and a universal cord 230 that houses various cables from the operation section 220.

[0106] The insertion portion 210 comprises a tip portion 240, a curved portion 250, and a flexible portion 260. The tip portion 240 includes one or more imaging units (for example, imaging unit 243) having an image sensor configured in which pixels that perform photoelectric conversion of received light to generate pixel signals are arranged in a two-dimensional matrix. The curved portion 250 is configured to be flexible by being composed of a plurality of curved pieces. The flexible portion 260 is connected to the base end side of the curved portion 250 and is configured to be a flexible, elongated shape.

[0107] The insertion section 210 is inserted into the body cavity of the subject and uses an image sensor to image subjects such as biological tissue located in a position where external light cannot reach. In addition to the imaging section 243, the tip section 240 is equipped with an illumination section 242 that illuminates the subject with illumination light for imaging. Furthermore, the insertion section 210 (for example, the tip section 240) may be equipped with a forceps channel 244 for inserting a treatment instrument 620, a water inlet 245 for supplying water into the subject, and an air inlet 246 for supplying air into the subject.

[0108] The operating unit 220 operates the insertion unit 210. The operating unit 220 comprises a curved knob 221, a treatment instrument insertion unit 222, and a plurality of switches 223. The curved knob 221 curves the curved unit 250 in the vertical and horizontal directions. The treatment instrument insertion unit 222 inserts the treatment instrument 620 into the body cavity of the subject. The switches 223 are operation input units that input operation instruction signals for peripheral devices.

[0109] The treatment device 600 includes a treatment instrument operating section 610 and a flexible treatment instrument 620 extending from the treatment instrument operating section 610. Examples of the treatment instrument 620 include a therapeutic light irradiation device, biopsy forceps, an electrosurgical unit, and an examination probe. The treatment instrument operating section 610 operates the treatment instrument 620. The treatment instrument 620 is inserted through the treatment instrument insertion section 222 and emerges from the forceps channel 244 at the tip 240.

[0110] The universal cord 230 incorporates at least an optical fiber (for example, the optical fiber 340 shown in Figure 13) for transmitting light to the illumination unit 242 and a bundled cable including the signal line for the imaging unit 243 (for example, the cable 211 shown in Figure 13). The universal cord 230 branches at the end opposite to the end connected to the operation unit 220. The optical fiber 340 of the universal cord 230 is connected to the light source device 300. The bundled cable of the universal cord 230 is connected to the CCU 400. In this embodiment, it is described as transmitting electrical signals using signal lines, but it may also transmit optical signals, or signals may be transmitted between the endoscope 200 and the CCU 400 by wireless communication.

[0111] The light source device 300 emits light for illumination. The light emitted by the light source device 300 is transmitted to the illumination unit 242 via the optical fiber 340. The optical fiber 340 is also called a light guide.

[0112] The CCU 400 is a control device that comprehensively controls the connected endoscope 200 and light source device 300. The CCU 400 may also comprehensively control the connected display device 500, recording device 700, and output device 800. For example, the CCU 400 controls the irradiation timing, irradiation intensity, and type of light source of the light source device 300. The CCU 400 also performs image processing, such as development (e.g., demosaicing) and correction, on the pixel signals output from the endoscope 200, and outputs the processed pixel signals (e.g., images) to external devices such as the display device 500. Furthermore, the CCU 400 transmits control signals to the endoscope 200 to control its operation. These control signals may include information regarding imaging conditions, such as the magnification and focal length of the imaging unit 243. The CCU 400 may also have an image downconversion function, allowing it to simultaneously output high-resolution (e.g., 4K) images to the display device 500 and low-resolution (e.g., HD) images to the recording device 700. Furthermore, the CCU 400 and the light source device 300 may be integrated and housed in the same housing. Alternatively, the CCU 400 and the light source device 300 may be equipped with interfaces that can connect to both rigid and flexible endoscopes, and the necessary functions for both rigid and flexible endoscopes may be arranged within a common housing.

[0113] The display device 500 is a device capable of displaying images, such as a display monitor. The display device 500 displays a display image based on pixel signals acquired from the CCU 400. The display device 500 may also function as an input device that enables eye-tracking, voice recognition, and gesture-based instruction input by equipping it with a camera and microphone.

[0114] The recording device 700 is a device that records pixel signals (e.g., images) acquired from the CCU 400, and is, for example, a recorder. The recording device 700 records the images acquired from the CCU 400 onto an HDD, SSD, or optical disc. The recording device 700 may be connected to a hospital network and made accessible from equipment outside the operating room. The recording device 700 may also have an image down-conversion or up-conversion function.

[0115] The output device 800 is a device that outputs information acquired from the CCU 400, and is, for example, a printer. The output device 800 prints a print image on paper based on the pixel signals acquired from the CCU 400.

[0116] The right side of Figure 12 shows an example configuration of the tip section 240. An illumination section 242, an imaging section 243, a forceps channel 244, a water inlet 245, and an air inlet 246 can be arranged on the end face of the tip section 240. Note that the tip section 240 in the figure shows an example in which three illumination sections 242 are arranged. These illumination sections 242 are preferably positioned to surround the imaging section 243, as this suppresses the generation of shadows on the subject. A protective cover glass can be placed on the illumination section 242. In addition to the image sensor 248, the imaging section 243 is equipped with an optical system 249, such as a lens that collects light from the subject.

[0117] Water supplied from the water inlet 245 can be used to clean the surface of the imaging unit 243 and to wash away foreign matter to clarify the field of view. Air supplied from the air inlet 246 can be used to inflate the affected organ to secure the field of view. The forceps channel 244 can also be used for aspirating mucus and other fluids.

[0118] Figure 13 is a block diagram showing an endoscope system 1000 according to the first application example. Figure 13 shows an endoscope 200, a light source device 300, a CCU 400, a display device 500, and a treatment device 600.

[0119] The curved knob 221 of the operating section 220 pushes and pulls the guide wire 224 located inside the insertion section 210, allowing the tip section 240 to move up, down, left, and right.

[0120] The light source device 300 comprises a light source 310, an illumination control unit 320, and an optical system 330. The light source 310 emits illumination light based on the control of the illumination control unit 320. The light source 310 in the figure has an LD (Laser Diode) 311 and a phosphor 312. The LD 311 emits laser light. For example, the LD 311 emits a blue laser. The phosphor 312 converts the wavelength of the laser light from the LD 311. The phosphor 312 emits fluorescence when excited by the laser light, and generates light with a different wavelength from the laser light that is the excitation light. For example, a material that generates yellow fluorescence when excited by a blue laser can be used for the phosphor 312. A portion of the blue laser light from the LD 311 is converted into yellow light by the phosphor 312. These blue laser light and yellow light are mixed to generate white light. This white light is emitted from the light source 310. Note that the configuration of the light source device 300 is not limited to this example. For example, the light source 310 can consist only of an LD 311, with a phosphor 312 placed in the illumination section 242 of the tip 240. Alternatively, a light-emitting diode (LED) or a xenon lamp can be used as the light source 310.

[0121] The optical system 330 focuses the light emitted from the light source 310 onto the end of the optical fiber 340. The other end of the optical fiber 340 reaches the illumination section 242 of the tip 240, and the light emitted from the light source device 300 is transmitted to the tip 240 via the optical fiber 340 and emitted from the end face 241. In this case, the end of the optical fiber 340 constitutes the illumination section 242.

[0122] The imaging unit 243 comprises an image sensor 248 and an optical system 249. The optical system 249 guides light from the subject to the image sensor 248. The optical system 249 can be composed of, for example, multiple lenses. The image sensor 248 acquires the reflected light from the subject that is irradiated from the illumination unit 242, performs imaging, generates a pixel signal, and outputs it. For example, a CMOS (Complementary Metal Oxide Semiconductor) type image sensor can be used for the image sensor 248. The image sensor 248 performs imaging based on control signals from the CCU 400. The image sensor 248 also outputs the generated pixel signal to the CCU 400. These control signals and pixel signals are transmitted by a bundled cable located in the insertion unit 210. The CCU 400 generates an image from the image sensor 248 based on the pixel signal and outputs it to the display device 500.

[0123] The treatment device 600 comprises a treatment instrument operating section 610 and a treatment instrument 620. The treatment instrument 620 is inserted into the insertion section 210, and its end protrudes from the forceps channel 244 of the tip section 240. A water inlet 245 and an air inlet 246 are further arranged on the tip section 240.

[0124] Figure 14 is a block diagram showing an example configuration of a Camera Control Unit (CCU) 400 according to the first application example. The CCU 400 is an information processing device having an FPGA 401, a CPU 402, a RAM 403, a ROM 404, a GPU 405, and an I / F 406. The FPGA 401, CPU 402, and GPU 405 are processors that process various types of information. The RAM 403 is a read / write memory, and the ROM 404 is a read-only memory. The I / F 406 is, for example, a connection interface for an endoscope 200. For example, the I / F 406 is a connection port for a connector on the cable 211 of the endoscope 200.

[0125] The endoscope system 1000 corresponds to, for example, the endoscope system 1 described above. The imaging unit 243 corresponds to the imaging device 30 described above. The light source device 300 corresponds to, for example, the light source device 10 described above. The CCU 400 corresponds to, for example, the control device 50 described above. The display device 500 corresponds to, for example, the monitor 60 described above. The optical fiber 340 corresponds to, for example, the light guide unit 20 described above. The cable 211 corresponds to, for example, the signal line 40 described above.

[0126] <3-2. Second Application Example> Next, as a second application example of the endoscope system 1, an endoscope system 5000 equipped with a rigid endoscope will be described with reference to Figures 15 and 16.

[0127] (Endoscopic System) Figure 15 shows an example configuration of the endoscopic system 5000 according to the second application example. Figure 16 shows an example configuration of the endoscope 5001 and CCU 5039 according to the second application example. Figure 15 illustrates a surgeon (for example, a doctor) 5067, who is a participant in the surgery, performing surgery on a patient 5071 on a patient bed 5069 using the endoscopic system 5000.

[0128] As shown in Figure 15, the endoscopic system 5000 consists of an endoscope 5001 which is a medical imaging device, a CCU 5039, a light source device 5043, a recording device 5053, an output device 5055, and a support device 5027 that supports the endoscope 5001.

[0129] In endoscopic surgery, an insertion aid called a trocca 5025 is inserted into the patient 5071. Then, via the trocca 5025, the scope 5003 connected to the endoscope 5001 and surgical instruments 5021 are inserted into the body of the patient 5071. The surgical instruments 5021 include, for example, energy devices such as electrosurgical units or forceps.

[0130] Surgical images, which are medical images of the inside of the patient's body (5071) taken by the endoscope (5001), are displayed on the display device (5041). The surgeon (5067) performs the procedure on the surgical target using the surgical instruments (5021) while viewing the surgical images displayed on the display device (5041). Note that the medical images are not limited to surgical images; they may also be diagnostic images taken during the diagnosis.

[0131] (Endoscope) The endoscope 5001 is an imaging unit that images the inside of the patient 5071's body. For example, as shown in Figure 16, it is a camera 5005 that includes a focusing optical system 50051 that focuses incident light, a zoom optical system 50052 that changes the focal length of the imaging unit to enable optical zoom, a focusing optical system 50053 that changes the focal length of the imaging unit to enable focus adjustment, and a light-receiving element 50054. The endoscope 5001 generates a pixel signal by focusing light onto the light-receiving element 50054 via the connected scope 5003 and outputs the pixel signal to the CCU 5039 via a transmission system. The scope 5003 has an objective lens at its tip and is an insertion unit that guides light from the connected light source device 5043 into the patient 5071's body. The scope 5003 is, for example, a rigid scope in the case of a rigid endoscope, and a flexible scope in the case of a flexible endoscope. The scope 5003 may be a straight-viewing endoscope or an oblique-viewing endoscope. Furthermore, the pixel signal can be any signal based on the signal output from the pixel, such as a RAW signal or an image signal. Alternatively, the transmission system connecting the endoscope 5001 and the CCU 5039 may be equipped with memory to store parameters related to the endoscope 5001 and the CCU 5039. The memory may be located, for example, at the connection point of the transmission system or on the cable. For example, the factory settings of the endoscope 5001 and parameters that change during power-up may be stored in the transmission system's memory, and the operation of the endoscope may be modified based on the parameters read from the memory. The endoscope and transmission system may also be referred to as a set. The photodetector 50054 is a sensor that converts received light into a pixel signal, and is, for example, a CMOS (Complementary Metal Oxide Semiconductor) type image sensor. Preferably, the photodetector 50054 is a color image sensor having a Bayer array. Furthermore, the light-receiving element 50054 is preferably an image sensor having a number of pixels corresponding to a resolution of, for example, 4K (3840 horizontal pixels × 2160 vertical pixels), 8K (7680 horizontal pixels × 4320 vertical pixels), or square 4K (3840 or more horizontal pixels × 3840 or more vertical pixels). The light-receiving element 50054 may be a single sensor chip or multiple sensor chips.For example, a prism may be provided to separate the incident light into predetermined wavelength bands, and each wavelength band may be imaged by a different photodetector. Alternatively, multiple photodetectors may be provided for stereoscopic viewing. The photodetector 50054 may be a sensor that includes an image processing circuit within its chip structure, or it may be a Time of Flight (ToF) sensor. The transmission system may be, for example, an optical fiber cable or wireless transmission. Wireless transmission is only required if the pixel signals generated by the endoscope 5001 can be transmitted. For example, the endoscope 5001 and the CCU 5039 may be wirelessly connected, or the endoscope 5001 and the CCU 5039 may be connected via a base station in the operating room. In this case, the endoscope 5001 may simultaneously transmit not only the pixel signals but also information related to the pixel signals (for example, the processing priority of the pixel signals or synchronization signals). The endoscope may integrate the scope and camera, or a photodetector may be provided at the tip of the scope.

[0132] (CCU) The CCU 5039 is a control device that comprehensively controls the connected endoscope 5001 and light source device 5043. For example, as shown in Figure 16, it is an information processing device having an FPGA 50391, CPU 50392, RAM 50393, ROM 50394, GPU 50395, and I / F 50396. The CCU 5039 may also comprehensively control the connected display device 5041, recording device 5053, and output device 5055. For example, the CCU 5039 controls the irradiation timing, irradiation intensity, and type of light source of the light source device 5043. The CCU 5039 also performs image processing such as development processing (e.g., demosaicing) and correction processing on the pixel signals output from the endoscope 5001, and outputs the processed pixel signals (e.g., images) to an external device such as the display device 5041. Furthermore, the CCU 5039 transmits control signals to the endoscope 5001 to control its operation. The control signals include, for example, information regarding imaging conditions such as the magnification and focal length of the imaging unit. The CCU 5039 may also have an image downconversion function and be configured to simultaneously output high-resolution (e.g., 4K) images to the display device 5041 and low-resolution (e.g., HD) images to the recording device 5053.

[0133] Furthermore, the CCU 5039 may be connected to external devices (e.g., recording devices, display devices, output devices, support devices) via an IP converter that converts signals to a predetermined communication protocol (e.g., IP (Internet Protocol)). The connection between the IP converter and the external devices may consist of a wired network, or part or all of the network may be constructed as a wireless network. For example, the IP converter on the CCU 5039 side may have a wireless communication function and transmit the received video to an IP switcher or output-side IP converter via a wireless communication network such as a fifth-generation mobile communication system (5G) or a sixth-generation mobile communication system (6G).

[0134] (Light Source Device) The light source device 5043 is a device capable of irradiating light in a predetermined wavelength band, and includes, for example, a plurality of light sources and a light source optical system that guides the light from the plurality of light sources. The light sources are, for example, xenon lamps, LED light sources, and LD light sources. The light source device 5043 has, for example, LED light sources corresponding to each of the three primary colors R, G, and B, and emits white light by controlling the output intensity and output timing of each light source. In addition, the light source device 5043 may have a light source capable of irradiating special light used for special light observation, separate from the light source that irradiates normal light used for normal light observation. Special light is light in a predetermined wavelength band different from the normal light used for normal light observation, and includes, for example, near-infrared light (light with a wavelength of 760 nm or more), infrared light, blue light, and ultraviolet light. Normal light is, for example, white light or green light. In narrow-band light observation, which is a type of special light observation, by alternately irradiating with blue light and green light, predetermined tissues such as blood vessels on the surface of mucous membranes can be imaged with high contrast by utilizing the wavelength dependence of light absorption in body tissues. Furthermore, in fluorescence observation, a type of special light observation, excitation light is irradiated onto a drug injected into body tissue to excite it, and a fluorescence image is obtained by receiving the fluorescence emitted by the body tissue or the labeling drug. This makes it easier for the operator to visualize body tissues that are difficult to see with normal light. For example, in fluorescence observation using infrared light, infrared light having an excitation wavelength band is irradiated onto a drug such as indocyanine green (ICG) injected into body tissue, and the structure of the body tissue and the affected area can be made easier to visualize by receiving the fluorescence of the drug. In addition, in fluorescence observation, a drug that is excited by special light in the blue wavelength band and emits fluorescence in the red wavelength band (e.g., 5-ALA) may be used. The type of irradiation light of the light source device 5043 is set by the control of the CCU 5039. The CCU 5039 may have a mode in which normal light observation and special light observation are performed alternately by controlling the light source device 5043 and the endoscope 5001. In this case, it is preferable to superimpose information based on the pixel signals obtained by special light observation onto the pixel signals obtained by normal light observation. Furthermore, the special light observation may be infrared light observation, which involves irradiating with infrared light to view areas deeper than the organ surface, or multispectral observation utilizing hyperspectral spectroscopy. In addition, photodynamic therapy may be combined.

[0135] (Recording device) The recording device 5053 is a device that records pixel signals (e.g., images) acquired from the CCU 5039, and is, for example, a recorder. The recording device 5053 records the images acquired from the CCU 5039 onto an HDD, SSD, or optical disc. The recording device 5053 may be connected to a hospital network and made accessible from equipment outside the operating room. The recording device 5053 may also have an image down-conversion or up-conversion function.

[0136] (Display device) The display device 5041 is a device capable of displaying images, such as a display monitor. The display device 5041 displays a display image based on the pixel signals acquired from the CCU 5039. The display device 5041 may also function as an input device that enables eye-tracking, voice recognition, and gesture-based instruction input by equipping it with a camera and microphone.

[0137] (Output device) The output device 5055 is a device that outputs information acquired from the CCU 5039, and is, for example, a printer. The output device 5055 prints a print image on paper based on the pixel signals acquired from the CCU 5039.

[0138] (Support device) The support device 5027 is a multi-joint arm comprising a base portion 5029 having an arm control device 5045, an arm portion 5031 extending from the base portion 5029, and a holding portion 5032 attached to the tip of the arm portion 5031. The arm control device 5045 is composed of a processor such as a CPU and controls the driving of the arm portion 5031 by operating according to a predetermined program. The support device 5027 controls the position and orientation of the endoscope 5001 held by the holding portion 5032, for example, by controlling parameters such as the length of each link 5035 constituting the arm portion 5031 and the rotation angle and torque of each joint 5033 using the arm control device 5045. This allows the endoscope 5001 to be changed to a desired position or orientation, the scope 5003 to be inserted into the patient 5071, and the observation area inside the body to be changed. The support device 5027 functions as an endoscope support arm that supports the endoscope 5001 during surgery. This allows the support device 5027 to act as a substitute for the scopist, who is an assistant holding the endoscope 5001. The support device 5027 may also be a device that supports a microscope and can be called a medical support arm. The support device 5027 may be controlled autonomously by the arm control device 5045, or it may be controlled by the arm control device 5045 based on user input. For example, the control method may be a master-slave system in which the support device 5027, acting as a slave device (replica device) that is a patient cart, is controlled based on the movement of the master device (primary device), which is the operator console at the user's location. Furthermore, the support device 5027 may be controlled remotely from outside the operating room.

[0139] (Other) The endoscope 5001 corresponds to, for example, the endoscope system 1 described above. The camera 5005 corresponds to, for example, the imaging device 30 described above. The CCU 5039 corresponds to, for example, the control device 50 described above. The display device 5041 corresponds to, for example, the monitor 60 described above. The light source device 5043 corresponds to, for example, the light source device 10 described above.

[0140] <4. Hardware Configuration Example> A specific hardware configuration example of the control device 50 according to the above embodiment will be described. The control device 50 according to the embodiment may be implemented by a computer 2000 having the configuration shown in Figure 17, for example. Figure 17 is a diagram showing a hardware configuration example.

[0141] As shown in Figure 17, the computer 2000 has a CPU 2010, RAM 2020, ROM (Read Only Memory) 2030, HDD (Hard Disk Drive) 2040, a communication interface 2050, and an input / output interface 2060. The various parts of the computer 2000 are connected by a bus 2070.

[0142] The CPU 2010 operates based on programs stored in the ROM 2030 or HDD 2040, and controls various parts. For example, the CPU 2010 loads the programs stored in the ROM 2030 or HDD 2040 into the RAM 2020 and executes processing corresponding to various programs.

[0143] ROM2030 stores boot programs such as the BIOS (Basic Input Output System) that are executed by CPU2010 when the computer 2000 starts up, as well as programs that depend on the computer 2000's hardware.

[0144] HDD2040 is a recording medium readable by computer 2000 for non-temporarily recording programs executed by CPU 2010 and data used by such programs. Specifically, HDD2040 is a recording medium for recording program data 2041 (for example, an information processing program according to the embodiment).

[0145] The communication interface 2050 is an interface for the computer 2000 to connect to an external network 2080 (for example, the Internet). For example, the CPU 2010 receives data from other devices and transmits data it generates to other devices via the communication interface 2050.

[0146] The input / output interface 2060 is an interface for connecting the input / output device 2090 and the computer 2000. For example, the CPU 2010 receives data from input devices such as a keyboard or mouse via the input / output interface 2060. The CPU 2010 also sends data to output devices such as a display, speakers, or printer via the input / output interface 2060.

[0147] The input / output interface 2060 may also function as a media interface for reading programs and the like recorded on a predetermined recording medium (media). Examples of media include optical recording media such as DVDs (Digital Versatile Discs) and PDs (Phase Change Rewritable Disks), magneto-optical recording media such as MOs (Magneto-Optical disks), tape media, magnetic recording media, or semiconductor memory.

[0148] Here, for example, if computer 2000 functions as control device 50, the CPU 2010 of computer 2000 implements all or part of the functions of each part of the control device 50 by executing an information processing program loaded onto RAM 2020. The HDD 2040 stores the information processing program and data according to the embodiment. The CPU 2010 reads and executes the program data 2041 from HDD 2040, but as another example, these programs may be obtained from other devices via an external network 2080.

[0149] <5. Addendum> The technology may also be configured as follows: (1) A medical image processing device comprising: an observation image generation unit that generates a medical observation image from a medical imaging image; a depth estimation processing unit that generates depth information relating to the depth of the medical imaging image such that the depth resolution of the area of ​​interest in the medical imaging image is higher than that of other areas other than the area of ​​interest; and a display image generation unit that generates a display image including the medical observation image and the depth information. (2) The medical image processing device according to (1), wherein the display image generation unit generates a depth image indicating the depth of the medical imaging image based on the depth information, and generates an image including the medical observation image and the depth image as the display image. (3) The medical image processing device according to (2), wherein the display image generation unit generates the image by superimposing the depth image on the medical observation image. (4) The medical image processing device according to (2), wherein the display image generation unit generates the image by arranging the medical observation image and the depth image side by side. (5) The depth image is a heat map or a mesh map, as described in any one of (2) to (4). (6) The depth estimation processing unit estimates the depth information from the medical image captured by the monocular camera using a machine learning-trained model, as described in any one of (1) to (5). (7) The depth estimation processing unit generates focus area information relating to the focus area, and the display image generation unit generates an image as the display image including the focus area information, the medical observation image, and the depth information, as described in any one of (1) to (6). (8) The display image generation unit generates the image by overlaying the focus area information on the medical observation image, as described in (7). (9) The display image generation unit generates the image by placing the medical observation image and the focus area information side by side, as described in (7).(10) The medical image processing apparatus according to any one of (7) to (9), wherein the area of ​​interest information includes one or more of the maximum size of the area of ​​interest, the minimum size of the area of ​​interest, the maximum depth of the area of ​​interest, the minimum depth of the area of ​​interest, and the area of ​​the area of ​​interest. (11) The medical image processing apparatus according to any one of (1) to (10), further comprising an object detection processing unit for detecting the area of ​​interest from the medical image. (12) The medical image processing apparatus according to (11), wherein the object detection processing unit detects the area of ​​interest by YOLO (You Only Look Once) or SSD (Single Shot multiBox Detector). (13) The medical image processing apparatus according to (11) or (12), wherein the depth estimation processing unit generates depth information with a predetermined depth resolution when the object detection processing unit does not detect the area of ​​interest. (14) The depth estimation processing unit generates depth information such that the depth resolution of the area of ​​interest is higher than that of other areas when the object detection processing unit detects the area of ​​interest, according to any one of (11) to (13). (15) The depth estimation processing unit generates area of ​​interest information relating to the area of ​​interest when the object detection processing unit detects the area of ​​interest, according to any one of (11) to (14). (16) The depth estimation processing unit generates depth information such that the depth resolution at the location on the medical observation image is higher than that of other locations when the user specifies a location on the medical observation image by input operation, according to any one of (1) to (15). (17) The depth estimation processing unit measures the distance between two points on the medical observation image when the user specifies the locations of two points on the medical observation image by input operation, according to any one of (1) to (16).(18) The depth estimation processing unit generates depth information such that the depth resolution at the central position of the two points on the medical observation image is higher than that at other positions other than the central position, in response to the user specifying the positions of two points on the medical observation image by input operation, as described in any one of (1) to (17). (19) A medical image processing system comprising: an imaging device for acquiring a medical image; and a medical image processing device for processing the medical image, wherein the medical image processing device includes: an observation image generation unit for generating a medical observation image from the medical image; a depth estimation processing unit for generating depth information relating to the depth of the medical image such that the depth resolution of a point of interest in the medical image is higher than that of other regions other than the point of interest; and a display image generation unit for generating a display image including the medical observation image and the depth information. (20) A medical image processing method comprising: a medical image processing device generating a medical observation image from a medical imaging image; generating depth information relating to the depth of the medical imaging image such that the depth resolution of a region of interest in the medical imaging image is higher than that of other regions other than the region of interest; and generating a display image including the medical observation image and the depth information. (21) A medical image processing system comprising a medical image processing device according to any one of (1) to (18). (22) A medical image processing method for processing information using a medical image processing device according to any one of (1) to (18).

[0150] 1 Endoscope system 2 Object to be observed 3 Light 10 Light source device 20 Light guide unit 21 Light emission unit 30 Imaging device 40 Signal line 50 Control device 51 Observation image generation unit 52 Object detection processing unit 53 Depth estimation processing unit 54 Display image generation unit 60 Monitor 61 Touch sensor A1 Color bar A2 Area of ​​interest A3 Area of ​​interest information G1 Observation image G2 Depth image G3 Display image

Claims

1. A medical image processing device comprising: an observation image generation unit that generates a medical observation image from a medical imaging image; a depth estimation processing unit that generates depth information relating to the depth of the medical imaging image such that the depth resolution of the area of ​​interest in the medical imaging image is higher than that of other areas other than the area of ​​interest; and a display image generation unit that generates a display image including the medical observation image and the depth information.

2. The medical image processing apparatus according to claim 1, wherein the display image generation unit generates a depth image indicating the depth of the medical imaging image based on the depth information, and generates an image including the medical observation image and the depth image as the display image.

3. The medical image processing apparatus according to claim 2, wherein the display image generation unit generates the image by superimposing the depth image onto the medical observation image.

4. The medical image processing apparatus according to claim 2, wherein the display image generation unit generates the image by arranging the medical observation image and the depth image side by side.

5. The medical image processing apparatus according to claim 2, wherein the depth image is a heat map or a mesh map.

6. The medical image processing apparatus according to claim 1, wherein the depth estimation processing unit estimates the depth information from the medical image captured by the monocular camera using a pre-trained model obtained by machine learning.

7. The medical image processing apparatus according to claim 1, wherein the depth estimation processing unit generates information about the area of ​​interest, and the display image generation unit generates an image as the display image that includes the information about the area of ​​interest, the medical observation image, and the depth information.

8. The medical image processing apparatus according to claim 7, wherein the display image generation unit generates the image by superimposing the area of ​​interest information onto the medical observation image.

9. The medical image processing apparatus according to claim 7, wherein the display image generation unit generates the image by arranging the medical observation image and the area of ​​interest information side by side.

10. The medical image processing apparatus according to claim 7, wherein the area of ​​interest information includes one or more of the maximum size of the area of ​​interest, the minimum size of the area of ​​interest, the maximum depth of the area of ​​interest, the minimum depth of the area of ​​interest, and the area of ​​the area of ​​interest.

11. The medical image processing apparatus according to claim 1, further comprising an object detection processing unit for detecting the area of ​​interest from the medical image.

12. The medical image processing apparatus according to claim 11, wherein the object detection processing unit detects the area of ​​interest using YOLO (You Only Look Once) or SSD (Single Shot multiBox Detector).

13. The medical image processing apparatus according to claim 11, wherein the depth estimation processing unit generates depth information with a predetermined depth resolution when the object detection processing unit does not detect the area of ​​interest.

14. The medical image processing apparatus according to claim 11, wherein the depth estimation processing unit generates depth information such that the depth resolution of the area of ​​interest is higher than that of other areas when the object detection processing unit detects the area of ​​interest.

15. The medical image processing apparatus according to claim 11, wherein the depth estimation processing unit generates information about the area of ​​interest when the object detection processing unit detects the area of ​​interest.

16. The medical image processing apparatus according to claim 1, wherein the depth estimation processing unit generates depth information such that the depth resolution at the location on the medical observation image is higher than that at other locations, in response to the user specifying a location on the medical observation image by an input operation.

17. The medical image processing apparatus according to claim 1, wherein the depth estimation processing unit measures the distance between two points on the medical observation image in response to the user specifying the positions of two points on the medical observation image by an input operation.

18. The medical image processing apparatus according to claim 1, wherein the depth estimation processing unit generates depth information such that the depth resolution at the central position of the two points on the medical observation image is higher than that at other positions other than the central position, in response to the user specifying the positions of two points on the medical observation image by input operation.

19. A medical image processing system comprising: an imaging device for acquiring medical imaging images; and a medical image processing device for processing the medical imaging images, wherein the medical image processing device includes: an observation image generation unit for generating medical observation images from the medical imaging images; a depth estimation processing unit for generating depth information relating to the depth of the medical imaging images such that the depth resolution of a region of interest in the medical imaging images is higher than that of other regions other than the region of interest; and a display image generation unit for generating a display image including the medical observation image and the depth information.

20. A medical image processing method comprising: a medical image processing device generating a medical observation image from a medical imaging image; generating depth information relating to the depth of the medical imaging image such that the depth resolution of a region of interest within the medical imaging image is higher than that of other regions other than the region of interest; and generating a display image including the medical observation image and the depth information.

Images