Medical image processing device, medical control device, medical observation system, and medical image processing method

Abstract

This medical image processing device 932 comprises: an image acquisition unit that acquires a first captured image generated by imaging return light of first light from an observation object and a second captured image generated by imaging return light of second light from the observation object; and an interpolation image generation unit that generates an interpolation image that interpolates the second captured image. The interpolation image generation unit generates a first interpolation image, which is an interpolation image, on the basis of the second captured image, a third captured image that is a first captured image generated on the basis of the same synchronization signal as the second captured image, and a fourth captured image that is a first captured image generated later in time than the third captured image.

Description

Medical image processing apparatus, medical control apparatus, medical observation system, and medical image processing method

[0001] The present disclosure relates to a medical image processing apparatus, a medical control apparatus, a medical observation system, and a medical image processing method.

[0002] Conventionally, there has been known a medical observation system that irradiates an observation target (a subject such as a human) with visible light such as excitation light which is narrow-band light or white light which is wide-band light emitted from a light source device, and observes fluorescence emitted from a substance contained in the observation target by the irradiation of the excitation light (hereinafter referred to as observation target fluorescence) (see, for example, Patent Document 1). According to such fluorescence observation, it is possible to grasp a tissue state that is difficult to recognize through the observation target fluorescence. For this reason, fluorescence observation can be used for various purposes and applications such as identification of a lesion site.

[0003] Japanese Unexamined Patent Publication No. 2021-132695

[0004] Incidentally, as the observation target fluorescence, the fluorescence intensity may be weak. In such a case, for example, in order to capture more observation target fluorescence into an image sensor, by extending the exposure time of the image sensor, it is possible to generate a fluorescence image in which the brightness of the fluorescence region of the observation target fluorescence is increased. However, when the exposure time of the image sensor is extended, the frame rate of the fluorescence image decreases. That is, there are cases where fluorescence images cannot be sequentially generated in response to changes in the subject or changes in the angle of view. Therefore, there is a demand for a technique capable of generating an image suitable for observation.

[0005] The present disclosure has been made in view of the above, and an object thereof is to provide a medical image processing apparatus, a medical control apparatus, a medical observation system, and a medical image processing method that can generate an image suitable for observation.

[0006] To solve the above-mentioned problems and achieve the objective, the medical image processing apparatus according to the present disclosure comprises an image acquisition unit that acquires a first image generated by capturing the reflected light of a first light from an object to be observed, and a second image generated by capturing the reflected light of a second light from the object to be observed, and an interpolation image generation unit that generates an interpolation image to interpolate the second image, wherein the interpolation image generation unit generates a first interpolation image based on the second image, a third image which is the first image generated based on the same synchronization signal as the second image, and a fourth image which is the first image generated chronologically later than the third image.

[0007] Furthermore, the medical control device according to this disclosure comprises a medical image processing device that processes an input captured image, and a control unit that controls the operation of the medical image processing device, wherein the medical image processing device comprises an image acquisition unit that acquires a first captured image generated by capturing the reflected light of a first light from an object to be observed, and a second captured image generated by capturing the reflected light of a second light from the object to be observed, respectively, and an interpolation image generation unit that generates an interpolation image to interpolate the second captured image, wherein the interpolation image generation unit generates a first interpolation image which is the interpolation image based on the second captured image, a third captured image which is the first captured image generated based on the same synchronization signal as the second captured image, and a fourth captured image which is the first captured image generated chronologically later than the third captured image.

[0008] Furthermore, the medical observation system according to this disclosure comprises a light source device that emits a first light and a second light, respectively; an imaging device that generates a first image by imaging the reflected light of the first light from an object to be observed and generates a second image by imaging the reflected light of the second light from the object to be observed; and a medical image processing device that processes the first image and the second image. The medical image processing device comprises an image acquisition unit that acquires the first image and the second image, respectively, and an interpolation image generation unit that generates an interpolation image to interpolate the second image. The interpolation image generation unit generates a first interpolation image, which is the interpolation image, based on the second image, a third image, which is the first image generated based on the same synchronization signal as the second image, and a fourth image, which is the first image generated chronologically later than the third image.

[0009] Furthermore, the medical image processing method according to this disclosure acquires a first image generated by capturing the reflected light of a first light from an object to be observed, and a second image generated by capturing the reflected light of a second light from the object to be observed. Based on the second image, a third image which is the first image generated based on the same synchronization signal as the second image, and a fourth image which is the first image generated chronologically later than the third image, a first interpolated image is generated, which is an interpolated image obtained by interpolating the second image.

[0010] According to the medical image processing apparatus, medical control device, medical observation system, and medical image processing method relating to this disclosure, it is possible to generate images suitable for observation.

[0011] Figure 1 is a diagram illustrating the configuration of a medical observation system according to an embodiment. Figure 2 is a block diagram illustrating the configuration of a camera head and a control device. Figure 3 is a diagram illustrating a conventional problem. Figure 4 is a diagram illustrating a medical image processing method according to an embodiment. Figure 5 is a diagram illustrating a medical image processing method according to an embodiment. Figure 6 is a diagram illustrating a medical image processing method according to an embodiment. Figure 7 is a diagram illustrating a method for generating interpolated images. Figure 8 is a diagram illustrating modification 1 of the embodiment. Figure 9 is a diagram illustrating modification 1 of the embodiment. Figure 10 is a diagram illustrating modification 1 of the embodiment. Figure 11 is a diagram illustrating modification 2 of the embodiment. Figure 12 is a diagram illustrating modification 2 of the embodiment. Figure 13 is a diagram illustrating modification 3 of the embodiment. Figure 14 is a diagram illustrating modification 4 of the embodiment. Figure 15 is a diagram illustrating modification 4 of the embodiment. Figure 16 is a diagram illustrating modification 6 of the embodiment. Figure 17 is a diagram illustrating modification 7 of the embodiment. Figure 18 is a diagram illustrating modification 9 of the embodiment. Figure 19 is a diagram illustrating modification 10 of the embodiment. Figure 20 is a diagram illustrating modification 11 of the embodiment. Figure 21 is a diagram illustrating a modified example 11 of the embodiment.

[0012] The embodiments for implementing this disclosure (hereinafter referred to as "embodiments") will be described below with reference to the drawings. However, the embodiments described below do not limit this disclosure. Furthermore, the same parts are denoted by the same reference numerals in the drawings.

[0013] [Configuration of the Medical Observation System] Figure 1 shows the configuration of the medical observation system 1 according to this embodiment. In this embodiment, the medical observation system 1 is a medical endoscope system that uses an endoscope to observe an object (inside the body). As shown in Figure 1, the medical observation system 1 comprises an insertion unit 2, a light source device 3, a light guide 4, a camera head 5, a first transmission cable 6, a display device 7, a second transmission cable 8, a control device 9, and a third transmission cable 10.

[0014] In this embodiment, the insertion section 2 is made up of a rigid endoscope. That is, the insertion section 2 has an elongated shape that is either entirely rigid or partially flexible with other parts rigid, and is inserted into the object to be observed. Inside the insertion section 2, there is an optical system made up of one or more lenses that collects the reflected light (image of the subject) from the object to be observed.

[0015] Furthermore, an excitation light cut filter 22 (Figure 1) is provided at the base end (eyepiece 21) of the insertion section 2 to partially, substantially, or completely suppress the excitation light contained in the focused reflected light (subject image), which will be described later. Note that the excitation light cut filter 22 is not limited to the insertion section 2, but may also be provided inside the camera head 5.

[0016] One end of the light guide 4 is connected to the light source device 3. The light source device 3 then supplies light to the one end of the light guide 4 under the control of the control device 9. As shown in Figure 1, the light source device 3 comprises a first light source 31 and a second light source 32.

[0017] The first light source 31 emits first light (broadband light such as white light, or narrowband light such as red light, green light, or blue light) that includes at least a portion of the visible light wavelength band. Examples of the configuration of the first light source 31 include a configuration including a white LED (Light Emitting Diode), or a configuration including three light sources that emit red light, green light, and blue light respectively, and an optical element that combines the red light, green light, and blue light. The first light source 31 may be made of an LED or a semiconductor laser. The number of first light sources 31 may be one or more.

[0018] The second light source 32 emits excitation light that excites the substance contained in the object to be observed. This excitation light corresponds to the second light according to this disclosure. The second light source 32 may be composed of an LED or a semiconductor laser. There may be one or more second light sources 32. Furthermore, the second light source 32 may emit light that includes at least a portion of the visible light wavelength band, or it may emit light in the invisible light wavelength band such as infrared light or ultraviolet light.

[0019] In this embodiment, a configuration is provided in which two light sources (first and second light sources 31 and 32) are used to emit the first light and excitation light, respectively, but the invention is not limited to this configuration. For example, a configuration may be adopted in which only one light source is provided, and the wavelength is switched using a filter or the like to emit the first light and excitation light, respectively.

[0020] Examples of substances contained in the object being observed that are excited by the excitation light include drugs or fluorescent dyes applied to the object, or fluorescent substances originating from the object itself.

[0021] Examples of the above-mentioned drugs applied to the subjects of observation include "5-ALA (PP-IX)", "ADS780WS", "ADS830WS", "aggregation-induced emission dots allophycocyanin (APC)", "boron-dipyrromethane (BODIPY)", "CLR 1502", "Flavins", "fluorescamine", "Fluorescein", "fluoro-gold", "green fluorescence protein", "ICG (indocyanine green)", "IRDye 78", "IR-PEG nanoparticles", "Isothiocyanate", "rose bengal", "SGM-101", and "trypan blue".

[0022] Furthermore, the fluorescent dyes mentioned above that can be applied to the object of observation include: "coumarine", "Cy3", "DyLight547", "GE3126", "metal nanoclusters", "oxacarbocyanine", "Rhodamine", "Riboflavin", "fluorescein", "AlexaFluor 488", "AlexaFluor660", "AlexaFluor680", "AlexaFluor700", "Cy5", "Cy5.5", "Dy677", "Dy682", "Dy752", "DyLight647", "HiLyte Fluor 647", "HiLyte Fluor 680", "IRDye 700DX", "methylene blue", "Porphyrins", "Porphysomes", "VivoTag-680", "VivoTag-S680", "AlexaFluor750", "AlexaFluor790", "carbocyanine", "conjugated copolymers", "CW800-CA", "Cy7", "Cy7.5", "cyanine dyes", "Dy780", and "HiLyte Examples include "Fluor 750", "Indocarbocyanine", "IR-786", "IRDye 800CW", "IRDye 800RS", "IRDye 800BK", "Nervelight", "OTL-38 (Pafolacianine)", "Polymethine", "VivoTag-S750", "ASP5354", "Xanthene", and "LUM-015".

[0023] Furthermore, examples of fluorescent substances derived from the observed object that constitute the observed object itself include "collagen," "elastin," and "NADH."

[0024] In this embodiment, the light source device 3 is configured separately from the control device 9, but this is not the only option; it may also be configured to be housed in the same housing as the control device 9. Alternatively, the control device 9 may be divided into two units, and the light source device 3 may be housed in the housing of one of the two control devices 9.

[0025] One end of the light guide 4 is detachably connected to the light source device 3. The other end of the light guide 4 is detachably connected to the insertion section 2. The light guide 4 transmits light (first light or excitation light) supplied from the light source device 3 from one end to the other and supplies it to the insertion section 2. The light (first light or excitation light) supplied to the insertion section 2 is emitted from the tip of the insertion section 2 and irradiates the object to be observed. When the object to be observed is irradiated with the first light, the reflected light of the first light from the object is focused by the optical system in the insertion section 2. When the object to be observed is irradiated with excitation light, the reflected light of the excitation light from the object is focused by the optical system in the insertion section 2. The reflected light of the excitation light includes not only the excitation light reflected by the object to be observed, but also fluorescence emitted from a substance contained in the object when the excitation light irradiates the object and the substance is excited (hereinafter referred to as "observed object fluorescence").

[0026] The camera head 5 is detachably connected to the eyepiece 21 of the insertion section 2. The camera head 5 corresponds to the imaging device according to this disclosure. Under the control of the control device 9, the camera head 5 captures the reflected light (reflected light of the first light and reflected light of the excitation light) from the observation target focused by the insertion section 2 and generates a pixel signal. For convenience of explanation, the pixel signal may be referred to as the captured image below. The detailed configuration of the camera head 5 will be described later in "Configuration of the Camera Head".

[0027] One end of the first transmission cable 6 is detachably connected to the control device 9 via connector CN1 (Figure 1). The other end of the first transmission cable 6 is detachably connected to the camera head 5 via connector CN2 (Figure 1). The first transmission cable 6 transmits captured images and the like output from the camera head 5 to the control device 9, and also transmits control signals, synchronization signals, clock signals, and power signals output from the control device 9 to the camera head 5.

[0028] Furthermore, the captured images and other data transmitted from the camera head 5 to the control device 9 via the first transmission cable 6 may be transmitted as optical signals or as electrical signals. The same applies to the transmission of control signals, synchronization signals, and clock signals from the control device 9 to the camera head 5 via the first transmission cable 6.

[0029] The display device 7 is composed of a display using liquid crystal or organic EL (Electro Luminescence), and under the control of the control device 9, it displays an image based on a video signal from the control device 9.

[0030] One end of the second transmission cable 8 is detachably connected to the display device 7. The other end of the second transmission cable 8 is detachably connected to the control device 9. The second transmission cable 8 transmits the video signal processed by the control device 9 to the display device 7.

[0031] The control device 9 corresponds to the medical control device described herein. This control device 9 is composed of a CPU (Central Processing Unit), an FPGA (Field-Programmable Gate Array), etc., and comprehensively controls the operation of the light source device 3, the camera head 5, and the display device 7. The detailed configuration of the control device 9 will be described later in "Configuration of the Control Device".

[0032] One end of the third transmission cable 10 is detachably connected to the light source device 3. The other end of the third transmission cable 10 is detachably connected to the control device 9. The third transmission cable 10 transmits control signals from the control device 9 to the light source device 3.

[0033] [Camera Head Configuration] Next, the configuration of the camera head 5 will be described. Figure 2 is a block diagram showing the configuration of the camera head 5 and the control device 9. As shown in Figure 2, the camera head 5 comprises a lens unit 51, a prism 52, an imaging unit 53, and a communication unit 54.

[0034] The lens unit 51 is composed of one or more lenses. The lens unit 51 then focuses the reflected light (subject image) from the object being observed, which has been focused in the insertion section 2, onto the imaging surfaces of the first and second image sensors 531 and 532, respectively.

[0035] The prism 52 separates the reflected light (subject image) from the object being observed via the lens unit 51 into light of two different wavelength bands: first and second. The first wavelength band of light is light of a wavelength band excluding at least a portion of the wavelength band of the observed fluorescence, and includes at least a portion of the visible light wavelength band. Hereinafter, the light of the first wavelength band will be referred to as the normal subject image. The second wavelength band of light is light of a wavelength band excluding at least a portion of the wavelength band of the visible light, and includes at least a portion of the wavelength band of the observed fluorescence. Hereinafter, the light of the second wavelength band that includes at least a portion of the wavelength band of the observed fluorescence will be referred to as the fluorescent subject image. The prism 52 then propagates the normal subject image toward the first image sensor 531. The prism 52 also propagates the fluorescent subject image toward the second image sensor 532.

[0036] The imaging unit 53 captures an image of the object under the control of the control device 9. As shown in Figure 2, the imaging unit 53 comprises a first image sensor 531, a second image sensor 532, and a signal processing unit 533.

[0037] The number of image sensors constituting the first image sensor 531 may be one or more. Similarly, the number of image sensors constituting the second image sensor 532 may be one or more.

[0038] The first and second image sensors 531 and 532 receive the subject image and convert it into an electrical signal (analog signal). Examples of these first and second image sensors 531 and 532 include CMOS (Complementary Metal Oxide Semiconductor), which is a rolling shutter type image sensor in which multiple pixels are arranged in a two-dimensional manner in units of horizontal lines, and CCD (Charge Coupled Device), which is a global shutter type image sensor.

[0039] Here, the first image sensor 531, although not shown in detail in the illustration, is composed of an invalid region where the output signal is not used to generate the captured image, an optical black region (OB region), and an effective pixel region where the normal subject image formed by the lens unit 51 is converted into a pixel signal and output. Similarly, the second image sensor 532 is composed of an invalid region, an optical black region (OB region), and an effective pixel region.

[0040] The first image sensor 531 then captures a normal subject image via the prism 52 under the control of the control device 9. That is, the first image sensor 531 captures light that includes at least a portion of the visible light wavelength band. For the sake of explanation, the image generated by capturing a normal subject image will be referred to as the background image below. This background image corresponds to the first image image according to this disclosure.

[0041] Furthermore, the second image sensor 532 captures a fluorescent subject image via the prism 52 under the control of the control device 9. That is, the second image sensor 532 captures light that includes at least a portion of the wavelength band of the fluorescence to be observed. For the sake of explanation, the image generated by capturing the fluorescent subject image will be referred to as a fluorescent image below. This fluorescent image corresponds to the second image image according to this disclosure.

[0042] The number of pixels in the background image and the number of pixels in the fluorescence image may be different or the same.

[0043] The signal processing unit 533, under the control of the control device 9, performs signal processing on the captured images (analog signals) generated by the first and second image sensors 531 and 532 and outputs captured images (digital signals). For example, the signal processing unit 533 performs signal processing on the captured images (analog signals) generated by the first and second image sensors 531 and 532, such as removing reset noise, multiplying the analog signal by an analog gain to amplify it (hereinafter referred to as analog gain adjustment), and A / D conversion.

[0044] The communication unit 54 functions as a transmitter that sequentially transmits captured images output from the imaging unit 53 to the control device 9 via the first transmission cable 6. For example, the communication unit 54 is configured with a high-speed serial interface that communicates captured images at a transmission rate of 1 Gbps or higher with the control device 9 via the first transmission cable 6.

[0045] Note that the communication unit 54 may sequentially transmit a background image and a fluorescence image to the control device 9, or may transmit these two images simultaneously.

[0046] [Configuration of Control Device] Next, the configuration of the control device 9 will be described with reference to FIG. 2. As shown in FIG. 2, the control device 9 includes a communication unit 91, an image memory 92, a processing module 93, a control unit 94, an input unit 95, an output unit 96, and a storage unit 97.

[0047] The communication unit 91 functions as a receiver that sequentially receives captured images transmitted from the camera head 5 (communication unit 54) via the first transmission cable 6. For example, the communication unit 91 is configured with a high-speed serial interface that communicates captured images at a transmission rate of 1 Gbps or higher with the communication unit 54.

[0048] The image memory 92 is configured of, for example, a DRAM (Dynamic Random Access Memory) or the like. The image memory 92 is capable of temporarily storing a plurality of frames of captured images sequentially output from the camera head 5 (communication unit 54).

[0049] The processing module 93 processes captured images sequentially transmitted from the camera head 5 (communication unit 54) and received by the communication unit 91 under the control of the control unit 94. As shown in FIG. 2, the processing module 93 includes a memory controller 931, an image processing unit 932, and a display control unit 933.

[0050] The memory controller 931 controls writing of captured images to the image memory 92 and reading of the captured images from the image memory 92. The captured image read by the memory controller 931 is input to the image processing unit 932.

[0051] The image processing unit 932 corresponds to a medical image processing device according to this disclosure. This image processing unit 932 performs image processing on the input captured image. Examples of such image processing include optical black subtraction (clamping), white balance adjustment, demosaicing, color correction matrix processing, gamma correction, YC processing which converts RGB signals into luminance chromatic difference signals (Y, Cb / Cr signals), digital gain adjustment which multiplies the digital gain, noise reduction, and filtering which enhances structure.

[0052] Furthermore, the image processing performed on the background image and the image processing performed on the fluorescence image may be different processes, or they may be the same process.

[0053] Furthermore, the image processing unit 932 has the functions of an image acquisition unit and an interpolated image generation unit according to this disclosure. The functions of the image acquisition unit and the interpolated image generation unit in the image processing unit 932 will be explained later in "Medical Image Processing Method".

[0054] The display control unit 933 generates a display image (video signal) for display on the display device 7 based on the captured image after image processing has been performed by the image processing unit 932, under the control of the control unit 94. The display control unit 933 then outputs the video signal to the display device 7 via the second transmission cable 8.

[0055] The control unit 94 is implemented by a controller such as a CPU or MPU (Micro Processing Unit) executing various programs stored in the memory unit 97. It controls the operation of the light source device 3, the camera head 5, and the display device 7, as well as the operation of the entire control device 9. The control unit 94 is not limited to a CPU or MPU; it may also be composed of an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA. The functions of the control unit 94 will be explained later in the sections "Conventional Problems" and "Medical Image Processing Method."

[0056] The input unit 95 is configured using operating devices such as a mouse, keyboard, and touch panel, and accepts user operations from a user such as a surgeon. The input unit 95 then outputs an operation signal corresponding to the user operation to the control unit 94.

[0057] The output unit 96 is configured using a speaker, printer, etc., and outputs various types of information.

[0058] The memory unit 97 stores programs executed by the control unit 94, information necessary for processing by the control unit 94, and so on.

[0059] [Conventional Problems] Before describing the medical image processing method according to this embodiment, conventional problems will be explained. Figure 3 is a diagram illustrating conventional problems. Here, Figure 3(a) shows a synchronization signal. For example, in the NTSC system, the synchronization signal is a signal with a period of 1 / 60 [s] (frame period). Also, for example, in the PAL system, the synchronization signal is a signal with a period of 1 / 50 [s] (frame period). Figure 3(b) shows a background image F1. Here, in Figure 3(b), for the sake of explanation, the forceps Tt, which is the subject included in the background image F1, is represented by a dashed line, and a dot is placed within the dashed line. Figure 3(c) shows a fluorescence image F2. Here, in Figure 3(c), a dot is placed within a region (hereinafter referred to as the fluorescence region ArF) composed of pixels whose brightness (luminance value) in the fluorescence image F2 is above a certain brightness. Figure 3(d) shows a superimposed image F3 obtained by superimposing the background image F1 and the fluorescence image F2. Figure 3(e) shows the exposure time of the first image sensor 531. Figure 3(f) shows the exposure time of the second image sensor 532.

[0060] The control unit 94 controls the operation of the first image sensor 531, and when the interval between adjacent synchronization signals is set to one unit, it causes the first image sensor 531 to capture the reflected light of the first light, using that one unit as the exposure time, and sequentially generates the background image F1 (Figures 3(b) and 3(e)). In Figure 3(e), for the sake of explanation, the words "1 unit exposure time" are written within the rectangle indicating the exposure time of the first image sensor 531.

[0061] Furthermore, the control unit 94 controls the operation of the second image sensor 532, and, assuming that the interval between adjacent synchronization signals is 1 unit, sets the exposure period to 4 units and causes the second image sensor 532 to capture the fluorescence to be observed, thereby sequentially generating fluorescence images F2 (Figures 3(c) and 3(f)). In other words, by generating fluorescence images F2 through long exposure, even if the fluorescence intensity of the fluorescence to be observed is weak, fluorescence images F2 with a bright fluorescence region ArF can be obtained. Note that in Figure 3(f), for the sake of explanation, the words "4 units of exposure time" are written within the rectangle indicating the exposure time of the second image sensor 532.

[0062] In other words, the background image F1 is captured at a higher frame rate than the fluorescence image F2. While the background image F1 is captured at four times the frame rate of the fluorescence image F2, it is not limited to this; any frame rate higher than that of the fluorescence image F2 is acceptable, such as two, three, or even five times higher.

[0063] Furthermore, the image processing unit 932 performs a superposition process to generate a superimposed image F3 by superimposing a fluorescence image F2 onto the background image F1. Examples of this superposition process include the first and second superposition processes shown below. The first superposition process replaces the region in the background image F1 that has the same pixel position as the fluorescence region ArF with the image of the fluorescence region ArF in the fluorescence image F2. The second superposition process changes the brightness of the fluorescence-indicating color applied to each pixel in the region in the background image F1 that has the same pixel position as the fluorescence region ArF, according to the brightness value of each pixel in the fluorescence region ArF of the fluorescence image F2 (so-called alpha blending process).

[0064] The display control unit 933 then generates a video signal for displaying the generated superimposed image F3 on the display device 7 and outputs it to the display device 7. As a result, the superimposed image F3 is displayed sequentially on the display device 7 (Figure 3(d)).

[0065] Here, as mentioned above, the background image F1 has a higher imaging frame rate than the fluorescence image F2. Therefore, if the frame rate is matched to that of the fluorescence image F2, the background image F1 and the fluorescence image F2 obtained by actual imaging cannot be superimposed unless the period is 4 frames. Conventionally, in order to generate and display the superimposed image F3 every frame, as shown by the dashed line in Figure 3(c), the same image as the immediately preceding fluorescence image F2 obtained by actual imaging (fluorescence image F2 shown by the solid line in Figure 3(c)) is used for the frames of fluorescence image F2 that have not been obtained by actual imaging.

[0066] However, as mentioned above, when generating the superimposed image F3, the fluorescence image F2 is not used to account for changes in the position and size of the subject, as well as changes in the field of view. Therefore, the superimposed image F3 cannot be considered suitable for observation.

[0067] [Regarding the medical image processing method] Next, the medical image processing method according to this embodiment will be described. Figures 4 to 6 are diagrams illustrating the medical image processing method according to this embodiment. Specifically, Figure 4(a), Figure 5(a), and Figure 6(a) correspond to Figure 3(a) and show the synchronization signal. Figure 4(b), Figure 5(b), and Figure 6(b) correspond to Figure 3(b) and show the background image F1. Figure 4(c), Figure 5(c), and Figure 6(c) correspond to Figure 3(c) and show the fluorescence image F2. Figure 4(d), Figure 5(d), and Figure 6(d) show the interpolated image F4. Figure 4(e), Figure 5(e), and Figure 6(e) correspond to Figure 3(d) and show the superimposed image F3.

[0068] The control of the operation of the first and second image sensors 531 and 532 by the control unit 94 is the same as the control described in "Conventional Problems" above (Figures 4(b), 4(c), 5(b), 5(c), 6(b), and 6(c)).

[0069] Here, the image processing unit 932 performs the following operations, which are different from the operations described in "Regarding the Conventional Problems" above.

[0070] The image processing unit (image acquisition unit) 932 acquires the background image F1 and the fluorescence image F2, respectively.

[0071] Then, as shown by the arrows in Figure 4, the image processing unit (interpolation image generation unit) 932 generates an interpolated image F4 (F41) that interpolates the fluorescence image F2 based on the fluorescence image F2, a background image F1 (F11) generated based on the same synchronization signal as the fluorescence image F2, and a background image F1 (F12) generated chronologically immediately after the background image F11. Here, the background image F11 corresponds to the third captured image according to this disclosure. The background image F12 corresponds to the fourth captured image according to this disclosure. Furthermore, the interpolated image F41 corresponds to the first interpolated image according to this disclosure. The specific method for generating the interpolated image F41 will be explained later in "Method for Generating the Interpolated Image".

[0072] Furthermore, the image processing unit (interpolated image generation unit) 932 generates an interpolated image F4 (F42) that interpolates the fluorescence image F2 based on the fluorescence image F2, the background image F11, and the background image F1 (F13) that is generated chronologically immediately after the background image F12, as shown by the arrows in Figure 5. Here, the background image F13 corresponds to the fifth captured image according to this disclosure. The interpolated image F42 corresponds to the second interpolated image according to this disclosure. The specific method for generating the interpolated image F42 will be explained later in "Method for generating the interpolated image".

[0073] Furthermore, the image processing unit (interpolated image generation unit) 932 generates an interpolated image F4 (F43) that interpolates the fluorescence image F2 based on the fluorescence image F2, the background image F11, and the background image F1 (F14) that is generated chronologically immediately after the background image F13, as shown by the arrows in Figure 6. Here, the background image F14 corresponds to the fifth captured image according to this disclosure. The interpolated image F43 corresponds to the second interpolated image according to this disclosure. The specific method for generating the interpolated image F43 will be explained later in "Method for generating the interpolated image".

[0074] Furthermore, the image processing unit 932 sequentially executes the following superposition processes: superimposing a fluorescence image F2 onto the background image F11 to generate a superimposed image F3; superimposing an interpolated image F41 onto the background image F12 to generate a superimposed image F3; superimposing an interpolated image F42 onto the background image F13 to generate a superimposed image F3; and superimposing an interpolated image F43 onto the background image F14 to generate a superimposed image F4.

[0075] The display control unit 933 then generates a video signal for displaying the generated superimposed image F3 on the display device 7 and outputs it to the display device 7. As a result, the superimposed image F3 is displayed sequentially on the display device 7 (Figures 4(e), 5(e), and 6(e)).

[0076] [Regarding the method for generating interpolated images] Next, the method for generating the interpolated image F4 will be explained. Figure 7 is a diagram illustrating the method for generating the interpolated image F4. Specifically, Figure 7(a) shows a background image F1 (for example, background image F11) generated at a specific timing. Figure 7(b) shows a background image F1 (for example, any of background images F12 to F14) generated chronologically later than the background image F1 shown in Figure 7(a). Figure 7(c) shows a fluorescence image F2 generated based on the same synchronization signal as the background image F1 shown in Figure 7(a). Figure 7(d) shows an interpolated image F4 (for example, any of interpolated images F41 to F43) generated based on the background image F1 shown in Figure 7(a), the background image F1 shown in Figure 7(b), and the fluorescence image F2 shown in Figure 7(c).

[0077] The image processing unit (interpolation image generation unit) 932 calculates a motion vector based on two background images F1 arranged in time series, and generates an interpolated image F4 by processing the fluorescence image F2 based on the motion vector.

[0078] Specifically, the image processing unit (interpolation image generation unit) 932 generates an interpolated image F4 as shown below. First, the image processing unit (interpolation image generation unit) 932 divides the background image F1 and the fluorescence image F2 into certain mesh regions, as shown in Figures 7(a), 7(b), and 7(c). The mesh region is not limited to being composed of a group of pixels that are combined from multiple pixels, but may also be composed of only a single pixel.

[0079] Next, the image processing unit (interpolation image generation unit) 932 compares the background image F1 shown in Figure 7(a) with the background image F1 shown in Figure 7(b) and identifies mesh regions that are highly correlated with each other. The image processing unit (interpolation image generation unit) 932 also calculates the movement (vector) from the background image F1 shown in Figure 7(a) to the background image F1 shown in Figure 7(b) as a motion vector for these highly correlated mesh regions. In Figures 7(a) and 7(b), the highly correlated mesh regions are designated as mesh regions Ar1 and Ar1', and the motion vectors are represented by arrows.

[0080] Furthermore, known methods such as block matching or gradient methods can be used to calculate the motion vector.

[0081] Next, as shown in Figure 7(c), the image processing unit (interpolation image generation unit) 932 identifies mesh region Ar2, which corresponds to mesh region Ar1 identified as a mesh region with a high correlation between the two background images F1 arranged in time series in the fluorescence image F2.

[0082] Then, as shown in Figure 7(d), the image processing unit (interpolation image generation unit) 932 generates an interpolation image F4 by deforming the fluorescence image F2 so that the mesh region Ar2 moves to the mesh region Ar2' based on the calculated motion vector.

[0083] Although the example given assumes that the field of view (number of pixels) of the background image F1 and the fluorescence image F2 are the same, if the field of view (number of pixels) differs, the difference in field of view (difference in number of pixels) should be taken into consideration when calculating the motion vector used to generate the interpolated image.

[0084] Furthermore, the image processing unit (interpolation image generation unit) 932 generates an interpolated image F4 in which the brightness of the fluorescence image F2 is changed based on the change in brightness of two background images F1 arranged in a time series.

[0085] Specifically, the image processing unit (interpolation image generation unit) 932 generates an interpolated image F4 as shown below. First, the image processing unit (interpolation image generation unit) 932 calculates the ratio of the change in brightness of corresponding specific regions in the background image F1 shown in Figure 7(a) and the background image F1 shown in Figure 7(b). An example of such a specific region is a region composed of a group of pixels formed by combining multiple pixels. As for the brightness, an example of such brightness is the average value, median value, or integrated value of the brightness (e.g., luminance value) of all pixels in the specific region.

[0086] Then, the image processing unit (interpolated image generation unit) 932 generates an interpolated image F4 by increasing or decreasing the brightness of the fluorescence image F2 shown in Figure 7(c) by the ratio of the change in brightness calculated as described above.

[0087] The embodiment described above provides the following effects. The image processing unit 932 in this embodiment generates an interpolated image F41 that interpolates the fluorescence image F2 based on the fluorescence image F2 and background images F11 and F12. The image processing unit 932 also generates an interpolated image F42 that interpolates the fluorescence image F2 based on the fluorescence image F2 and background images F11 and F13. Furthermore, the image processing unit 932 generates an interpolated image F43 that interpolates the fluorescence image F2 based on the fluorescence image F2 and background images F11 and F14. Therefore, even if the frame rate of the background image F1 is higher than that of the fluorescence image F2, an interpolated image F4 that corresponds to changes in the position and size of the subject, as well as changes in the field of view, can be generated. Consequently, by superimposing the background image F1 and the interpolated image F4, a superimposed image F3 suitable for observation can be generated.

[0088] Furthermore, the image processing unit 932 calculates a motion vector based on two background images F1 arranged in time series, and generates an interpolated image F4 by processing the fluorescence image F2 based on the motion vector. Therefore, it is possible to efficiently generate an interpolated image F4 that corresponds to changes in the position and size of the subject, as well as changes in the field of view.

[0089] Incidentally, if the brightness of the background image F1 fluctuates, it is assumed that observation conditions such as the subject and subject distance have changed. When the frame rates for capturing the background image F1 and the fluorescence image F2 are the same, the brightness of the background image F1 and the fluorescence image F2 changes at the same time, thus maintaining the visibility of the fluorescent areas and fluorescence intensity. On the other hand, when the frame rate for capturing the background image F1 is higher than that of the fluorescence image F2, for example, when the subject distance becomes closer, the subject will be captured brighter until the brightness adjustment is made, but the background image F1 will become brighter, while the fluorescence image F2 will remain at the brightness it was at when it was captured earlier. When superimposed on the background image F1, the fluorescence image F2 will be relatively dark. This results in a problem where the visibility of the fluorescent areas and fluorescence intensity decreases. In response to this, the image processing unit 932 generates an interpolated image F4 in which the brightness of the fluorescence image F2 is changed according to the change in brightness of the two background images F1 arranged in time series. Therefore, the above problem can be solved. Furthermore, the brightness of the interpolated image F4 may be kept constant (maintaining the brightness at the time of imaging) without tracking fluctuations in the brightness of the background image F1.

[0090] (Other Embodiments) While embodiments for implementing this disclosure have been described so far, this disclosure should not be limited to the embodiments described above. In the embodiments described above, a configuration for 3D observation using a stereo system may be adopted. In this case, an interpolated image should be generated for each eye.

[0091] In the above-described embodiment, the following modified examples 1 to 11 may also be adopted.

[0092] (Modification 1) Figures 8 to 10 illustrate modification 1 of the embodiment. Specifically, Figures 8 to 10 correspond to Figure 4. In the embodiment described above, the images shown in Figures 8 to 10 may be used as the images used when generating the interpolated image F4.

[0093] Specifically, the image processing unit (interpolated image generation unit) 932 generates an interpolated image F4 (F41) that interpolates the fluorescence image F2, based on the fluorescence image F2, a background image F1 (F11) generated based on the same synchronization signal as the fluorescence image F2, and a background image F1 (F12) generated immediately after the background image F11 in chronological order, as shown by the arrows in Figure 8, similar to the embodiment described above (Figure 4). The method for generating the interpolated image F41 is the same as in the embodiment described above.

[0094] Furthermore, the image processing unit (interpolation image generation unit) 932 generates an interpolation image F4 (F42) that interpolates the fluorescence image F2 based on the interpolation image F41, the background image F12, and the background image F1 (F13) that is generated immediately after the background image F12 in chronological order, as shown by the arrows in Figure 9. The method for generating the interpolation image F42 is the same as in the embodiment described above.

[0095] Furthermore, the image processing unit (interpolation image generation unit) 932 generates an interpolation image F4 (F43) that interpolates the fluorescence image F2 based on the interpolation image F42, the background image F13, and the background image F1 (F14) that is generated immediately after the background image F13 in chronological order, as shown by the arrows in Figure 10. The method for generating the interpolation image F43 is the same as in the embodiment described above.

[0096] According to the modified example 1 described above, in addition to the same effects as the embodiment described above, the following effects are achieved. In this modified example 1, when generating the interpolated image F4, images captured at a more recent time point are used. Therefore, when the movement of the object being observed is large, when the movement is fast due to pulsation, or when there are large or rapid fluctuations in the subject, such as camera vibrations caused by the operator, it is possible to generate an interpolated image F4 with high detection accuracy for such fluctuations.

[0097] (Modification 2) Figures 11 and 12 illustrate Modification 2 of the embodiment. Specifically, Figure 11 corresponds to Figure 1 and shows the configuration of the medical observation system 1 according to Modification 2. Figure 12(a) corresponds to Figure 3(a) and shows the synchronization signal. Figure 12(b) shows the exposure time of the first image sensor 531. Figure 12(c) shows the emission time of the first light source 31. Figure 12(d) shows the exposure time of the second image sensor 532. Figure 12(e) shows the emission times of the second and third light sources 32 and 33. Figure 12(f) shows the generation timing of the interpolated image. Figure 12(g) shows the image used to generate the superimposed image.

[0098] As shown in Figure 11, the light source device 3 in this modified example 2 has a third light source 33 added to the light source device 3 described in the above-described embodiment. The third light source 33 emits excitation light that excites the substance contained in the object to be observed. For the sake of explanation, in order to distinguish it from the excitation light emitted by the second light source 32, the excitation light emitted by the second light source 32 will be referred to as fluorescence A excitation light, and the excitation light emitted by the third light source 33 will be referred to as fluorescence B excitation light. Furthermore, the fluorescence of the object to be observed corresponding to fluorescence A excitation light will be referred to as fluorescence A, and the fluorescence of the object to be observed corresponding to fluorescence B excitation light will be referred to as fluorescence B. Here, both fluorescence A excitation light and fluorescence B excitation light contained in the reflected light from the object to be observed are partially, substantially, or completely suppressed by the excitation light cut filter 22. Furthermore, both fluorescence A and fluorescence B are propagated toward the second image sensor 532 by the prism 52. Furthermore, the fluorescence image generated by imaging the target fluorescence A with the second image sensor 532 is referred to as the fluorescence A image, and the fluorescence image generated by imaging the target fluorescence B with the second image sensor 532 is referred to as the fluorescence B image. The third light source 33 may be an LED or a semiconductor laser. The number of third light sources 33 may be one or more. Furthermore, the third light source 33 may emit light that includes at least a part of the visible light wavelength band, or it may emit light in the invisible light wavelength band such as infrared light or ultraviolet light.

[0099] Here, the control unit 94 controls the operation of the first light source 31 to continuously emit the first light (white light) (Figure 12(c)). In Figure 12(c), for the sake of explanation, the words "White light emission" are written within the rectangle indicating the emission time of the first light source 31. The control unit 94 also controls the operation of the first image sensor 531, and when the interval between adjacent synchronization signals is considered as one unit, the first image sensor 531 is made to capture the reflected light of the first light, using this unit as the exposure time, thereby sequentially generating background images (Figure 12(b)). In Figure 12(b), for the sake of explanation, the words "Background image exposure" are written within the rectangle indicating the exposure time of the first image sensor 531. In addition, the numbers (1) to (4) are written after the background images to indicate the chronological order. Hereafter, the background images will also be referred to as background image (1) to background image (4) to indicate the chronological order.

[0100] Furthermore, the control unit 94 controls the operation of the second and third light sources 32 and 33, and when the interval between adjacent synchronization signals is considered as one unit, it causes the fluorescence A excitation light and the fluorescence B excitation light to be emitted alternately in a time-division manner for each of these units (Figure 12(e)). For the sake of explanation, in Figure 12(e), the words "Fluorescence A excitation light emission" are written in the rectangle indicating the emission time of the second light source 32, and the words "Fluorescence B excitation light emission" are written in the rectangle indicating the emission time of the third light source 33. The control unit 94 also controls the operation of the second image sensor 531, and when the interval between adjacent synchronization signals is considered as one unit, it causes the second image sensor 532 to alternately capture the target fluorescence A and target fluorescence B in a time-division manner, using this unit as the exposure time, and generates the fluorescence A image and the fluorescence B image alternately and sequentially (Figure 12(d)). In Figure 12(d), for the sake of explanation, the words "Fluorescence A image exposure" are written within the rectangle indicating the exposure time of the observed fluorescence A in the second image sensor 532, and the words "Fluorescence B image exposure" are written within the rectangle indicating the exposure time of the observed fluorescence B in the same second image sensor 532. In addition, the letters (1) to (4) are written after the fluorescence A image and fluorescence B image to indicate the chronological order. Hereafter, to indicate the chronological order, the fluorescence A images will be referred to as fluorescence A images (1) and (3), and the fluorescence B images will be referred to as fluorescence B images (2) and (4).

[0101] In other words, the background image is captured at a higher frame rate than the fluorescence A image and the fluorescence B image. Specifically, in this modified example 2, two drugs corresponding to the two excitation lights, fluorescence A excitation light and fluorescence B excitation light, are used, so the background image is captured at twice the frame rate of the fluorescence A image and the fluorescence B image.

[0102] The image processing unit 932 then performs the following operations.

[0103] The image processing unit (image acquisition unit) 932 acquires the background image, the fluorescence A image, and the fluorescence B image, respectively.

[0104] The image processing unit (interpolation image generation unit) 932 then generates an interpolated image (hereinafter referred to as "fluorescent A interpolated image (2)") that interpolates the fluorescent A image based on the fluorescent A image (1), a background image (1) generated based on the same synchronization signal as the fluorescent A image (1), and a background image (2) generated chronologically immediately after the background image (1). In Figure 12(f), the words "fluorescent A interpolated image (2) generation" are written within a rectangle indicating the interpolation image generation period. Here, background image (1) corresponds to the third captured image according to this disclosure. Background image (2) corresponds to the fourth captured image according to this disclosure. Furthermore, fluorescent A interpolated image (2) corresponds to the first interpolated image according to this disclosure. The method for generating the fluorescent A interpolated image (2) is the same as in the embodiment described above.

[0105] Furthermore, the image processing unit (interpolation image generation unit) 932 generates an interpolated image (hereinafter referred to as "fluorescent B interpolated image (3)") that interpolates the fluorescent B image based on the fluorescent B image (2), a background image (2) generated based on the same synchronization signal as the fluorescent B image (2), and a background image (3) generated chronologically immediately after the background image (2). In Figure 12(f), the words "fluorescent B interpolated image (3) generation" are written within a rectangle indicating the interpolation image generation period. Here, the background image (2) corresponds to the third captured image according to this disclosure. The background image (3) corresponds to the fourth captured image according to this disclosure. Furthermore, the fluorescent B interpolated image (3) corresponds to the first interpolated image according to this disclosure. The method for generating the fluorescent B interpolated image (3) is the same as in the embodiment described above.

[0106] Furthermore, the image processing unit (interpolation image generation unit) 932 generates an interpolated image (hereinafter referred to as "fluorescent A interpolated image (4)") that interpolates the fluorescent A image based on the fluorescent A image (3), a background image (3) generated based on the same synchronization signal as the fluorescent A image (3), and a background image (4) generated immediately after the background image (3) in chronological order. In Figure 12(f), the words "fluorescent A interpolated image (4) generation" are written within a rectangle indicating the interpolation image generation period. Here, the background image (3) corresponds to the third captured image according to this disclosure. The background image (4) corresponds to the fourth captured image according to this disclosure. Furthermore, the fluorescent A interpolated image (4) corresponds to the first interpolated image according to this disclosure.

[0107] Furthermore, the image processing unit 932 sequentially executes the following superposition processes: superimposing the fluorescent A image (1) onto the background image (1) to generate a superimposed image; superimposing the fluorescent A interpolated image (1) and the fluorescent B image (1) onto the background image (2) to generate a superimposed image; superimposing the fluorescent A image (3) and the fluorescent B interpolated image (3) onto the background image (3) to generate a superimposed image; and superimposing the fluorescent A interpolated image (4) and the fluorescent B image (4) onto the background image (4) to generate a superimposed image.

[0108] The display control unit 933 then generates a video signal for displaying the superimposed image on the display device 7 and outputs it to the display device 7. As a result, the superimposed image is displayed sequentially on the display device 7.

[0109] Even when adopting the configuration of the modified example 2 described above, the same effects as those of the embodiment described above are achieved.

[0110] (Modification 3) Figure 13 is a diagram illustrating modification 3 of the embodiment. Specifically, Figure 13 corresponds to Figure 12. In modification 2 described above, the exposure timing when generating the background image and the exposure timing when generating the fluorescence A image and fluorescence B image were the same, but this is not limited to this, and the timings may be shifted from each other. In the example of Figure 13, when the interval between adjacent synchronization signals is considered as 1 unit, the exposure timing when generating the fluorescence A image and fluorescence B image is shifted by 1 / 2 unit relative to the exposure timing when generating the background image.

[0111] Even when adopting the configuration of the modified example 3 described above, the same effects as those of the embodiment and modified example 2 described above are achieved.

[0112] (Modification 4) Figures 14 and 15 illustrate Modification 4 of the embodiment. Specifically, Figure 14 corresponds to Figure 1 and shows the configuration of the medical observation system 1 according to Modification 4. Figure 15(a) corresponds to Figure 3(a) and shows the synchronization signal. Figure 15(b) shows the exposure time of the first image sensor 531. Figure 15(c) shows the light emission time of the first light source 31. Figure 15(d) shows the exposure time of the second image sensor 532. Figure 15(e) shows the light emission times of the second to fourth light sources 32 to 34. Figure 15(f) shows the generation timing of the interpolated image. Figure 15(g) shows the image used to generate the superimposed image.

[0113] As shown in Figure 14, the light source device 3 in this modified example 4 has a fourth light source 34 added to the light source device 3 described in the modified example 2 above. The fourth light source 34 emits excitation light that excites the substance contained in the object to be observed. For the sake of explanation, the excitation light emitted by the fourth light source 34 will be referred to as the fluorescence C excitation light below. The fluorescence of the object to be observed corresponding to the fluorescence C excitation light will be referred to as the object to be observed fluorescence C. Here, the fluorescence C excitation light contained in the light reflected from the object to be observed, along with the fluorescence A excitation light and the fluorescence B excitation light, is partially, substantially, or completely suppressed by the excitation light cut filter 22. The object to be observed fluorescence C, along with the object to be observed fluorescence A and the object to be observed fluorescence B, is propagated toward the second image sensor 532 by the prism 52. Furthermore, the fluorescence image generated by imaging the object to be observed fluorescence C with the second image sensor 532 will be referred to as the fluorescence C image. The fourth light source 34 may be composed of an LED or a semiconductor laser. Furthermore, the number of the fourth light source 34 may be one or more. In addition, the fourth light source 34 may emit light that includes at least a portion of the visible light wavelength band, or it may emit light in the invisible light wavelength band such as infrared light or ultraviolet light.

[0114] Here, the control unit 94 controls the operation of the first light source 31 and the first image sensor 531, similar to the modified example 2 described above (Figures 15(b) and 15(c)). In Figure 15(b), for the sake of explanation, the words "background image exposure" are written within the rectangle indicating the exposure time of the first image sensor 531. Also, to indicate the chronological order, the numbers (1) to (6) are written after the background images. Hereafter, to indicate the chronological order, the background images will be referred to as background image (1) to background image (6).

[0115] Furthermore, the control unit 94 controls the operation of the second to fourth light sources 32 to 34, and when the interval between adjacent synchronization signals is considered as one unit, it causes the fluorescence A excitation light, fluorescence B excitation light, and fluorescence C excitation light to be emitted sequentially in a time-division manner for each of these units (Figure 15(e)). For the sake of explanation, in Figure 15(e), the words "Fluorescence C excitation light emission" are written within the rectangle indicating the emission time of the fourth light source 34. The control unit 94 also controls the operation of the second image sensor 531, and when the interval between adjacent synchronization signals is considered as one unit, it causes the second image sensor 532 to sequentially capture the observed fluorescence A, observed fluorescence B, and observed fluorescence C in a time-division manner, using this unit as the exposure time, and sequentially generates fluorescence A images, fluorescence B images, and fluorescence C images (Figure 15(d)). In Figure 15(d), for the sake of explanation, the words "Fluorescent C Image Exposure" are written within the rectangle indicating the exposure time of the observed fluorescent C in the second image sensor 532. Also, to indicate the chronological order, the letters (1) to (6) are written after the fluorescent A image, fluorescent B image, and fluorescent C image. Hereafter, to indicate the chronological order, the fluorescent A images will be referred to as fluorescent A images (1) and (4), the fluorescent B images as fluorescent B images (2) and (5), and the fluorescent C images as fluorescent C images (3) and (6).

[0116] In other words, the background image has a higher imaging frame rate than the fluorescence A image, fluorescence B image, and fluorescence C image. Specifically, in this modified example 4, three drugs corresponding to the three excitation lights, fluorescence A excitation light, fluorescence B excitation light, and fluorescence C excitation light, are used, so the background image has twice the imaging frame rate of the fluorescence A image, fluorescence B image, and fluorescence C image.

[0117] The image processing unit 932 then performs the following operations.

[0118] The image processing unit (image acquisition unit) 932 acquires the background image, the fluorescence A image, the fluorescence B image, and the fluorescence C image, respectively.

[0119] The image processing unit (interpolation image generation unit) 932 then generates an interpolated image (hereinafter referred to as "fluorescent A interpolated image (2)") that interpolates the fluorescent A image based on the fluorescent A image (1), a background image (1) generated based on the same synchronization signal as the fluorescent A image (1), and a background image (2) generated immediately after the background image (1) in chronological order. In Figure 15(f), the words "fluorescent A interpolated image (2) generation" are written within a rectangle indicating the interpolation image generation period. Here, background image (1) corresponds to the third captured image according to this disclosure. Background image (2) corresponds to the fourth captured image according to this disclosure. Furthermore, fluorescent A interpolated image (2) corresponds to the first interpolated image according to this disclosure. The method for generating the fluorescent A interpolated image (2) is the same as in the embodiment described above.

[0120] Furthermore, the image processing unit (interpolation image generation unit) 932 generates an interpolated image (hereinafter referred to as "fluorescent A interpolated image (3)") that interpolates the fluorescent A image based on the fluorescent A image (1), the background image (1), and the background image (3) which is generated chronologically immediately after the background image (2). In Figure 15(f), the words "fluorescent A interpolated image (3) generation" are written within a rectangle indicating the interpolation image generation period. Here, the background image (3) corresponds to the fifth captured image according to this disclosure. Also, the fluorescent A interpolated image (3) corresponds to the second interpolated image according to this disclosure. Here, in generating the fluorescent A interpolated image (3), the fluorescent A interpolated image (2), the background image (2), and the background image (3) may be used, similar to the modified example 1 described above. The method for generating the fluorescent A interpolated image (3) is the same as in the embodiment described above.

[0121] Furthermore, the image processing unit (interpolation image generation unit) 932 generates an interpolated image (hereinafter referred to as "fluorescent B interpolated image (3)") that interpolates the fluorescent B image based on the fluorescent B image (2), a background image (2) generated based on the same synchronization signal as the fluorescent B image (2), and a background image (3) generated chronologically immediately after the background image (2). In Figure 15(f), the words "fluorescent B interpolated image (3) generation" are written within a rectangle indicating the interpolation image generation period. Here, background image (2) corresponds to the third captured image according to this disclosure. Background image (3) corresponds to the fourth captured image according to this disclosure. Furthermore, fluorescent B interpolated image (3) corresponds to the first interpolated image according to this disclosure. The method for generating the fluorescent B interpolated image (3) is the same as in the embodiment described above.

[0122] Furthermore, the image processing unit (interpolation image generation unit) 932 generates an interpolated image (hereinafter referred to as "fluorescent B interpolated image (4)") that interpolates the fluorescent B image based on the fluorescent B image (2), the background image (2), and the background image (4) that is generated chronologically immediately after the background image (3). In Figure 15(f), the words "fluorescent B interpolated image (4) generation" are written within a rectangle indicating the interpolation image generation period. Here, the background image (4) corresponds to the fifth captured image according to this disclosure. Also, the fluorescent B interpolated image (4) corresponds to the second interpolated image according to this disclosure. Here, in generating the fluorescent B interpolated image (4), the fluorescent B interpolated image (3), the background image (3), and the background image (4) may be used, similar to the modified example 1 described above. The method for generating the fluorescent B interpolated image (4) is the same as in the embodiment described above.

[0123] Furthermore, the image processing unit (interpolation image generation unit) 932 generates an interpolated image (hereinafter referred to as "fluorescent C interpolated image (4)") that interpolates the fluorescent C image based on the fluorescent C image (3), a background image (3) generated based on the same synchronization signal as the fluorescent C image (3), and a background image (4) generated chronologically immediately after the background image (3). In Figure 15(f), the words "Fluorescent C interpolated image (4) generation" are written within a rectangle indicating the interpolation image generation period. Here, the background image (3) corresponds to the third captured image according to this disclosure. The background image (4) corresponds to the fourth captured image according to this disclosure. Furthermore, the fluorescent C interpolated image (4) corresponds to the first interpolated image according to this disclosure. The method for generating the fluorescent C interpolated image (4) is the same as in the embodiment described above.

[0124] Furthermore, the image processing unit (interpolation image generation unit) 932 generates an interpolated image (hereinafter referred to as the fluorescent C interpolated image (5)) that interpolates the fluorescent C image based on the fluorescent C image (3), the background image (3), and the background image (5) that is generated chronologically immediately after the background image (4). In Figure 15(f), the words "Fluorescent C interpolated image (5) generation" are written within a rectangle indicating the interpolation image generation period. Here, the background image (5) corresponds to the fifth captured image according to this disclosure. The fluorescent B interpolated image (5) corresponds to the second interpolated image according to this disclosure. Here, in generating the fluorescent C interpolated image (5), the fluorescent C interpolated image (4), the background image (4), and the background image (5) may be used, similar to the modified example 1 described above. The method for generating the fluorescent C interpolated image (5) is the same as in the embodiment described above.

[0125] Since the fluorescent A interpolated image (5), the fluorescent A interpolated image (6), and the fluorescent B interpolated image (6) are generated by the same method as described above, a detailed explanation of the generation method is omitted.

[0126] Furthermore, the image processing unit 932 sequentially executes the following superposition processes: superimposing the fluorescent A image (1) onto the background image (1) to generate a superimposed image; superimposing the fluorescent A interpolated image (1) and the fluorescent B image (2) onto the background image (2) to generate a superimposed image; superimposing the fluorescent A interpolated image (3), the fluorescent B interpolated image (3), and the fluorescent C image (3) onto the background image (3) to generate a superimposed image; superimposing the fluorescent A image (4), the fluorescent B interpolated image (4), and the fluorescent C interpolated image (4) onto the background image (4) to generate a superimposed image; superimposing the fluorescent A interpolated image (5), the fluorescent B image (5), and the fluorescent C interpolated image (5) onto the background image (5) to generate a superimposed image; and superimposing the fluorescent A interpolated image (6), the fluorescent B interpolated image (6), and the fluorescent C image (6) onto the background image (6) to generate a superimposed image.

[0127] The display control unit 933 then generates a video signal for displaying the superimposed image on the display device 7 and outputs it to the display device 7. As a result, the superimposed image is displayed sequentially on the display device 7.

[0128] Even when adopting the configuration of the modified example 4 described above, the same effects as those of the embodiment and modified examples 2 and 3 described above are achieved.

[0129] (Modification 5) The configuration described in the above-described embodiment and Modifications 2 to 4 may also be applied to surgeries, etc., using multiple drugs as shown in Table 1 below. In Table 1, drugs that are used are marked with "○" and drugs that are not used are marked with "×".

[0130]

[0131] Here, fluorescein emits a target fluorescence of approximately 520 nm when irradiated with excitation light in the wavelength range of approximately 470-480 nm. ALM-488 emits a target fluorescence of approximately 530 nm when irradiated with excitation light in the wavelength range of approximately 488 nm. Furthermore, LUM-015 emits a target fluorescence of approximately 675 nm when irradiated with excitation light in the wavelength range of approximately 650 nm. In addition, 5-ALA emits a target fluorescence of approximately 530-630 nm when irradiated with excitation light in the wavelength range of approximately 405 nm. Furthermore, ICG emits a target fluorescence of approximately 830 nm when irradiated with excitation light in the wavelength range of approximately 805 nm.

[0132] For example, when performing lymph node dissection during lobectomy for lung cancer, ALM-488 and ICG are used, as shown in Table 1. Specifically, ALM-488 is used to visualize the left recurrent laryngeal nerve, and ICG is used to visualize the lymph nodes. By visualizing the left recurrent laryngeal nerve and lymph nodes in this way, it is possible to distinguish between the left recurrent laryngeal nerve and the lymph nodes, thereby reducing recurrent laryngeal nerve paralysis during lobectomy. In this case, since two drugs, ALM-488 and ICG, are used, one of the two drugs is excited by one of the excitation lights, either fluorescence A excitation light or fluorescence B excitation light, and the other drug is excited by the other excitation light. The physician performing the lobectomy can then determine the location of the left recurrent laryngeal nerve and lymph nodes from the image displayed on the display device 7.

[0133] Furthermore, for example, when performing lateral lymph node dissection during rectal resection for colorectal cancer, ALM-488 and ICG are used as shown in Table 1. That is, ALM-488 is used to visualize neurovascular bundles, and ICG is used to visualize lymph nodes. By visualizing neurovascular bundles and lymph nodes in this way, it is possible to distinguish between neurovascular bundles and lymph nodes, thereby reducing urinary dysfunction, defecation dysfunction, male sexual dysfunction, or motor dysfunction caused by nerve damage during rectal resection. In this case, since two drugs, ALM-488 and ICG, are used, one of the two drugs is excited by one of the excitation lights, either fluorescence A excitation light or fluorescence B excitation light, and the other drug is excited by the other excitation light. The physician performing the rectal resection can then understand the location of neurovascular bundles and lymph nodes from the image displayed on the display device 7.

[0134] Furthermore, for example, in malignant tumor resection surgery for pediatric brain tumors, ALM-488 and ICG are used when performing malignant tumor resection, as shown in Table 1. That is, nerves are visualized by using ALM-488, and blood vessels (blood flow) are visualized by using ICG. By visualizing nerves and blood vessels in this way, it is possible to distinguish malignant tumors from nerves and blood vessels, prevent cancer recurrence after malignant tumor resection surgery, and further prevent damage to nerves and blood vessels during malignant tumor resection surgery. In this case, since two drugs, ALM-488 and ICG, are used, one of the two drugs is excited by one of the excitation lights, fluorescence A excitation light and fluorescence B excitation light, and the other drug is excited by the other excitation light. The physician performing the malignant tumor resection surgery can then grasp the location of nerves and blood vessels from the image displayed on the display device 7.

[0135] Furthermore, for example, in mastectomy for breast cancer, when removing malignant tumors or dissecting axillary lymph nodes, ALM-488, LUM-015, and ICG are used as shown in Table 1. Specifically, ALM-488 is used to visualize nerves such as the intercostal brachial nerve and thoracodorsal nerve, LUM-015 is used to visualize malignant tumors, and ICG is used to visualize lymph nodes. By visualizing nerves, malignant tumors, and lymph nodes in this way, it is possible to distinguish between malignant tumors, nerves, and lymph nodes, and reduce nerve damage during mastectomy. In this case, since three drugs, ALM-488, LUM-015, and ICG, are used, one of the three drugs is excited by one of the excitation lights (fluorescence A excitation light, fluorescence B excitation light, and fluorescence C excitation light), one of the other two drugs is excited by one of the other two excitation lights, and the remaining drug is excited by the remaining excitation light. Furthermore, the surgeon performing the mastectomy can determine the locations of nerves, malignant tumors, and lymph nodes from the image displayed on the display device 7.

[0136] Furthermore, for example, when performing brain tumor treatment using photodynamic diagnosis (PDD), ALM-488 and 5-ALA (5-aminolevulinic acid) are used, as shown in Table 1. Specifically, the area near the tumor is visualized by fluorescein, the fluorescent dye of ALM-488, and the tumor cells are visualized by protoporphyrin (PpIX), which is biosynthesized from 5-ALA. By visualizing tumor cells, including the area near the tumor, it is possible to prevent damage during brain tumor treatment and safely remove the tumor. In this case, since two drugs, ALM-488 and 5-ALA, are used, one of the two drugs is excited by one of the excitation lights, either fluorescent A excitation light or fluorescent B excitation light, and the other drug is excited by the other excitation light. The physician performing the brain tumor treatment can then determine the location of the tumor cells and the area near the tumor from the image displayed on the display device 7.

[0137] Furthermore, for example, when performing transurethral cystectomy in the surgical treatment of superficial urinary tract cell carcinoma using photodynamic diagnosis, fluorescein and 5-ALA are used as shown in Table 1. That is, the tumor is visualized by PpIX, which is biosynthesized from 5-ALA. However, PpIX has the drawback of visualizing not only the tumor but also highly metabolic inflammatory tissue present in the bladder. Therefore, false positives of highly metabolic inflammatory tissue caused by PpIX are reduced by fluorescein. In this case, since two drugs, fluorescein and 5-ALA, are used, one of the two drugs is excited by one of the excitation lights, fluorescence A excitation light and fluorescence B excitation light, and the other drug is excited by the other excitation light. Then, the physician performing the transurethral cystectomy can determine the location of the tumor and the false positives of highly metabolic inflammatory tissue caused by PpIX from the image displayed on the display device 7.

[0138] Even when adopting the configuration of the modified example 5 described above, the same effects as those of the embodiment and modified examples 2 to 4 described above are achieved.

[0139] (Modification 6) Figure 16 is a diagram illustrating modification 6 of the embodiment. In the embodiment described above, the control unit 94 may be provided with a function as a mode switching unit according to the present disclosure that switches between an interpolation mode in which the image processing unit 932 generates an interpolated image and a non-interpolation mode in which the image processing unit 932 does not generate an interpolated image.

[0140] Specifically, the control unit (mode switching unit) 94 determines whether or not it is set to automatically switch modes according to the brightness of the fluorescent region ArF in the fluorescence image F2 (step S1).

[0141] If "Yes" is determined in step S1, the control unit (mode switching unit) 94 detects the brightness of the fluorescent region ArF (step S2). Examples of this brightness include the average value, median value, or integrated value of the brightness (e.g., luminance value) of all pixels in the fluorescent region ArF.

[0142] After step S2, the control unit (mode switching unit) 94 determines whether the brightness of the fluorescent region ArF is above a certain brightness (step S3).

[0143] Then, if "Yes" is determined in step S3, the control unit (mode switching unit) 94 switches to interpolation mode (step S4). In this interpolation mode, for example, an overlaid image F3 is generated as explained in Figures 4 to 6.

[0144] On the other hand, if "No" is determined in step S3, the control unit (mode switching unit) 94 switches to non-interpolation mode (interpolation mode OFF) (step S5). In this non-interpolation mode, for example, a superimposed image F3 is generated as explained in Figure 3.

[0145] Furthermore, if "No" is determined in step S1, the control unit (mode switching unit) 94 maintains the current mode (interpolation mode or non-interpolation mode) (step S6).

[0146] As described above, Modification 6 provides the same effects as the embodiment described above, as well as the following effects. When the brightness of the fluorescent region ArF is weak, it is difficult to discern the displacement of the fluorescent region ArF. In other words, in this case, even if the superimposed image F3 as described in Figure 3 is generated and displayed, the effect on the observer's observation is small. On the other hand, when the brightness of the fluorescent region ArF is above a certain brightness, it is easy to discern the displacement of the fluorescent region ArF. In other words, in this case, if the superimposed image F3 as described in Figure 3 is generated and displayed, the effect on the observer's observation is large. In contrast, in Modification 6, the control unit 94 switches to interpolation mode when the brightness of the fluorescent region ArF in the fluorescent image F2 is above a certain brightness, and switches to non-interpolation mode when the brightness of the fluorescent region ArF is below that specific brightness. Therefore, the interpolation mode and non-interpolation mode can be appropriately switched, and an image suitable for observation by the observer can be displayed.

[0147] (Modification 7) Figure 17 is a diagram illustrating modification 7 of the embodiment. In the embodiment described above, the control unit 94 may be provided with a function as a mode switching unit according to the present disclosure that switches between an interpolation mode in which an interpolated image is generated by the image processing unit 932 and a non-interpolation mode in which an interpolated image is not generated by the image processing unit 9332.

[0148] Specifically, the control unit (mode switching unit) 94 determines whether or not it is set to automatically switch modes according to the size of the fluorescent region ArF in the fluorescence image F2 (step S1').

[0149] If "Yes" is determined in step S1', the control unit (mode switching unit) 94 detects the size of the fluorescent region ArF (for example, the number of pixels) (step S2').

[0150] After step S2', the control unit (mode switching unit) 94 determines whether the size of the fluorescent region ArF is greater than or equal to a specific size (step S3').

[0151] Then, if "Yes" is determined in step S3', the control unit (mode switching unit) 94 switches to interpolation mode (step S4). In this interpolation mode, for example, an overlaid image F3 is generated as explained in Figures 4 to 6.

[0152] On the other hand, if "No" is determined in step S3', the control unit (mode switching unit) 94 switches to non-interpolation mode (interpolation mode OFF) (step S5). In this non-interpolation mode, for example, a superimposed image F3 is generated as explained in Figure 3.

[0153] Furthermore, if "No" is determined in step S1, the control unit (mode switching unit) 94 maintains the current mode (interpolation mode or non-interpolation mode) (step S6).

[0154] As described above, Modification 7 provides the same effects as the embodiment described above, as well as the following effects. When the size of the fluorescent region ArF is less than a certain size, the displacement of the fluorescent region ArF is difficult to discern. In other words, in this case, even if the superimposed image F3 as described in Figure 3 is generated and displayed, the effect on the observer's observation is small. On the other hand, when the size of the fluorescent region ArF is greater than or equal to a certain size, the displacement of the fluorescent region ArF is easy to discern. In other words, in this case, if the superimposed image F3 as described in Figure 3 is generated and displayed, the effect on the observer's observation is large. In contrast to this, in Modification 7, the control unit 94 switches to interpolation mode when the size of the fluorescent region ArF in the fluorescent image F2 is greater than or equal to a certain size, and switches to non-interpolation mode when the size of the fluorescent region ArF is less than the certain size. Therefore, the interpolation mode and non-interpolation mode can be appropriately switched, and an image suitable for observation by the observer can be displayed.

[0155] (Modification 8) In Modifications 6 and 7 described above, the switching between interpolation mode and non-interpolation mode was performed automatically. However, the system is not limited to this, and a configuration may be adopted in which the control unit 94 performs the switching in response to user operation on the input unit 95. Furthermore, in the above-described embodiment, a configuration may be adopted in which the control unit 94 switches between any of the following three images as the image to be displayed, for example, in response to user operation on the input unit 95: (1) the superimposed image F3 shown in Figures 4 to 6, (2) the fluorescence image F2 and interpolation image F4 shown in Figures 4 to 6, and (3) the fluorescence image F2 shown in Figures 4 to 6.

[0156] (Modification 9) The medical observation system according to Modification 9 is a medical observation system that uses a so-called videoscope (flexible endoscope) having an imaging unit on the tip side of the insertion part. For the sake of explanation, the medical observation system 1 according to Modification 9 will be referred to as medical observation system 1B below.

[0157] Figure 18 is a diagram illustrating a modified example 9 of the embodiment. As shown in Figure 18, the medical observation system 1B includes an endoscope 300B that captures an internal image of the observation site by inserting an insertion part 2B into the body and outputs the captured image, a light source device 3 that emits a first light and excitation light from the tip of the endoscope 300B, a control device 9 that processes the captured image output from the endoscope 300B, and a display device 7 that is connected to the control device 9 via a second transmission cable 8 and displays an image based on the video signal processed by the control device 9.

[0158] As shown in Figure 18, the endoscope 300B comprises an insertion section 2B having a flexible, elongated shape, an operating section 301 connected to the base end of the insertion section 2B and receiving various operations, and a universal cord 302 extending from the operating section 301 in a direction different from the direction in which the insertion section 2B extends, and containing various cables connected to the light source device 3 and the control device 9. As shown in Figure 18, the insertion section 2B comprises a tip section 24, a flexible bending section 25 connected to the base end of the tip section 24 and composed of a plurality of bending pieces, and a flexible, elongated flexible tube section 26 connected to the base end of the bending section 25.

[0159] The tip portion 24 incorporates a configuration substantially similar to that of the camera head 5 described in the above-described embodiment, although a specific illustration is omitted. The captured image captured by the tip portion 24 (image sensor) is output to the control device 9 via the operation unit 301 and the universal code 302.

[0160] Even when adopting the configuration of the modified example 9 described above, the same effects as those of the embodiment described above are achieved.

[0161] (Modification 10) The medical observation system according to Modification 10 is a medical observation system that uses a surgical microscope to magnify and image a predetermined field of view of the inside (in vivo) or surface (biological surface) of the subject being observed. For the sake of explanation, the medical observation system 1 according to Modification 3 will be referred to as medical observation system 1C below.

[0162] Figure 19 is a diagram illustrating a modified example 10 of the embodiment. As shown in Figure 19, the medical observation system 1C comprises a surgical microscope 12 that captures images for observing a subject and outputs the captured images, a control device 9 that processes the captured images output from the surgical microscope 12, and a display device 7 connected to the control device 9 via a second transmission cable 8 and which displays images based on the video signals processed by the control device 9.

[0163] As shown in Figure 19, the surgical microscope 12 comprises a microscope unit 121 that magnifies and images minute parts of a subject and outputs the captured image, a support unit 122 connected to the base end of the microscope unit 121 and including an arm that rotatably supports the microscope unit 121, and a base unit 123 that rotatably holds the base end of the support unit 122 and is movable on the floor. The control device 9 is installed on the base unit 123 as shown in Figure 19. Although not specifically shown in the illustration, a light source device 3 that emits first light and excitation light from the surgical microscope 12 to the object being observed is also installed on the base unit 123. Note that the base unit 123 may be fixed to the ceiling or wall to support the support unit 122, rather than being movably installed on the floor.

[0164] Although not shown in detail in the illustration, the microscope unit 121 incorporates a configuration substantially similar to that of the camera head 5 described in Embodiment 2 above. The image captured by the microscope unit 121 (image sensor) is output to the control device 9 via the first transmission cable 6, which is wired along the support unit 122.

[0165] Even when adopting the configuration of the modified example 10 described above, the same effects as those of the embodiment described above are achieved.

[0166] (Modification 11) Figures 20 and 21 illustrate modification 11 of the embodiment. Specifically, Figure 20 is a side view of the ring light 15. Figure 21 is a front view of the ring light 15 (left side in Figure 20). In this modification 11, in addition to the insertion part 2 described in the above embodiment, the ring light 15 shown in Figures 20 and 21 is detachably connected to the camera head 5. That is, depending on the user's usage, the camera head 5 may be connected to either the insertion part 2 or the ring light 15, as shown in Figure 20.

[0167] Unlike the insertion unit 2, the ring light 15 is not inserted into the object of observation, but rather supplies first light and excitation light to the surgical unit and captures second light (subject image), which is the reflected light of the first light and excitation light from the surgical unit. As shown in Figures 20 and 21, the ring light 15 comprises an illumination unit 151 and a subject image capture unit 152 that captures the subject image.

[0168] As shown in Figures 20 and 21, the illumination unit 151 comprises a housing 1511 and a plurality of illumination lenses 1512. The housing 1511 has an annular shape centered on the optical axis Ax. The other end of the light guide 4 is detachably connected to the housing 1511.

[0169] As shown in Figure 21, the multiple illumination lenses 1512 are arranged at predetermined intervals along the circumferential direction centered on the optical axis Ax on the front end face of the housing 1511. The multiple illumination lenses 1512 each irradiate the surgical area with the first light supplied from the light source device 3 and introduced into the housing 1511 via the light guide 4.

[0170] The subject image acquisition section 152 extends along the optical axis Ax. Within the subject image acquisition section 152, there is an optical system composed of one or more lenses that focuses the second light (subject image) illuminated from multiple illumination lenses 1512 and transmitted through the surgical section. Furthermore, a connection section 1521 is provided at the base end (right side in Figure 20) of the subject image acquisition section 152. This connection section 1521 is designed (shaped) to be compatible with the eyepiece section 21 in the insertion section 2 and is detachably connected to the camera head 5.

[0171] Even when adopting the configuration of the modified example 11 described above, the same effects as those of the embodiment described above are achieved.

[0172] The following configurations also fall within the technical scope of this disclosure: (1) A medical image processing apparatus comprising: an image acquisition unit that acquires a first image generated by capturing the reflected light of a first light from an object to be observed, and a second image generated by capturing the reflected light of a second light from the object to be observed; and an interpolation image generation unit that generates an interpolation image to interpolate the second image, wherein the interpolation image generation unit generates a first interpolation image which is the interpolation image based on the second image, a third image generated which is the first image generated based on the same synchronization signal as the second image, and a fourth image generated which is the first image generated later in time with respect to the third image. (2) The medical image processing apparatus according to (1), wherein the first light includes at least a portion of the visible light wavelength band. (3) The medical image processing apparatus according to (1) or (2), wherein the second light is excitation light that excites a substance contained in the object to be observed, and the second image is a fluorescence image obtained by capturing fluorescence which is the reflected light of the second light emitted from the object to be observed due to irradiation with the excitation light. (4) The medical image processing apparatus according to any one of (1) to (3), wherein the interpolation image generation unit generates a second interpolation image which is a later interpolation image in time with respect to the first interpolation image, based on the second image, the third image, and the fifth image, which is the first image generated later in time with respect to the fourth image. (5) The medical image processing apparatus according to any one of (1) to (3), wherein the interpolation image generation unit generates a second interpolation image which is a later interpolation image in time with respect to the first interpolation image, based on the first interpolation image, the fourth image, and the fifth image, which is the first image generated later in time with respect to the fourth image. (6) The medical image processing apparatus according to any one of (1) to (5) above, wherein the interpolation image generation unit calculates a motion vector based on the third captured image and the fourth captured image, and generates the first interpolation image by processing the second captured image based on the motion vector.(7) The medical image processing apparatus according to any one of (1) to (6), wherein the interpolation image generation unit generates the first interpolation image by changing the brightness of the second image in accordance with the change in brightness from the third image to the fourth image. (8) The medical image processing apparatus according to any one of (1) to (7), wherein the first image is captured at a higher frame rate than the second image. (9) The medical image processing apparatus according to any one of (1) to (7), wherein the second light includes a first excitation light that excites a substance contained in the object of observation and a second excitation light that excites a substance contained in the object of observation, and the second image is an image produced by time-division imaging the return light of the first excitation light from the object of observation and the return light of the second excitation light from the object of observation, and comprises a first fluorescence image obtained by imaging the return light of the first excitation light and a second fluorescence image obtained by imaging the return light of the second excitation light, wherein the first image is obtained by imaging a frame rate higher than that of the first fluorescence image and the second fluorescence image. (10) A medical image processing device for processing an input image, and a control unit for controlling the operation of the medical image processing device, wherein the medical image processing device comprises an image acquisition unit that acquires a first image generated by capturing the reflected light of a first light from an object to be observed, and a second image generated by capturing the reflected light of a second light from the object to be observed, and an interpolation image generation unit that generates an interpolation image to interpolate the second image, wherein the interpolation image generation unit generates a first interpolation image which is the interpolation image based on the second image, a third image generated which is the first image generated based on the same synchronization signal as the second image, and a fourth image generated which is the first image generated later in time with respect to the third image, the first interpolation image.(11) The medical control device according to (10), wherein the second light is excitation light that excites a substance contained in the object to be observed, the second image is a fluorescence image captured of the fluorescence which is the reflected light of the second light emitted from the object to be observed due to the irradiation of the excitation light, the control unit comprises a mode switching unit that switches between an interpolation mode in which the interpolation image generation unit generates the interpolation image and a non-interpolation mode in which the interpolation image generation unit does not generate the interpolation image, the mode switching unit switches to the interpolation mode when the brightness of the fluorescence region in the fluorescence image is above a certain brightness, and switches to the non-interpolation mode when the brightness of the fluorescence region is below the certain brightness. (12) The medical control device according to (10), wherein the second light is excitation light that excites a substance contained in the object to be observed, the second image is a fluorescence image obtained by capturing fluorescence which is the reflected light of the second light emitted from the object to be observed due to irradiation with the excitation light, the control unit comprises a mode switching unit that switches between an interpolation mode in which the interpolation image generation unit generates the interpolation image and a non-interpolation mode in which the interpolation image generation unit does not generate the interpolation image, the mode switching unit switches to the interpolation mode when the size of the fluorescence region in the fluorescence image is greater than or equal to a specific size, and switches to the non-interpolation mode when the size of the fluorescence region is less than the specific size. (13) A medical observation system comprising: a light source device that emits a first light and a second light, respectively; an imaging device that generates a first image by imaging the reflected light of the first light from an object to be observed and generates a second image by imaging the reflected light of the second light from the object to be observed; and a medical image processing device that processes the first image and the second image, wherein the medical image processing device comprises: an image acquisition unit that acquires the first image and the second image, respectively; and an interpolation image generation unit that generates an interpolation image that interpolates the second image, wherein the interpolation image generation unit generates a first interpolation image that is the interpolation image based on the second image, a third image which is the first image generated based on the same synchronization signal as the second image, and a fourth image which is the first image generated later in time with respect to the third image.(14) A medical image processing method that acquires a first image image generated by capturing the reflected light of a first light from an object to be observed, and a second image image generated by capturing the reflected light of a second light from the object to be observed, and generates a first interpolated image which is an interpolated image obtained by interpolating the second image image based on the second image image, a third image image which is the first image image generated based on the same synchronization signal as the second image image, and a fourth image image which is the first image image generated chronologically later than the third image image.

[0173] 1, 1B, 1C Medical observation system 2, 2B Insertion section 3 Light source device 4 Light guide 5 Camera head 6 First transmission cable 7 Display device 8 Second transmission cable 9 Control device 10 Third transmission cable 12 Surgical microscope 15 Ring light 21 Eyepiece section 22 Excitation light cut filter 24 Tip section 25 Bending section 26 Flexible tube section 31 First light source 32 Second light source 33 Third light source 34 Fourth light source 51 Lens unit 52 Prism 53 Imaging section 54 Communication section 91 Communication section 92 Image memory 93 Processing module 94 Control section 95 Input section 96 Output section 97 Storage section 121 Microscope section 122 Support section 123 Base section 151 Illumination section 152 Subject image capture section 300B Endoscope 301 Control Unit 302 Universal Code 531 First Image Sensor 532 Second Image Sensor 533 Signal Processing Unit 931 Memory Controller 932 Image Processing Unit 933 Display Control Unit 1511 Housing 1512 Illumination Lens 1521 Connection Unit Ar1, Ar1', Ar2, Ar2' Region ArF Fluorescence Region Ax Optical Axis CN1, CN2 Connector F1, F11-F14 Background Image F2 Fluorescence Image F3 Superimposed Image F4, F41-F43 Interpolated Image Tt Forceps

Claims

1. A medical image processing apparatus comprising: an image acquisition unit that acquires a first image generated by capturing the reflected light of a first light from an object to be observed, and a second image generated by capturing the reflected light of a second light from the object to be observed; and an interpolation image generation unit that generates an interpolation image to interpolate the second image, wherein the interpolation image generation unit generates a first interpolation image, which is the interpolation image, based on the second image, a third image, which is the first image generated based on the same synchronization signal as the second image, and a fourth image, which is the first image generated chronologically later than the third image.

2. The medical image processing apparatus according to claim 1, wherein the first light comprises at least a portion of the visible light wavelength band.

3. The medical image processing apparatus according to claim 1, wherein the second light is excitation light that excites a substance contained in the object to be observed, and the second image is a fluorescence image obtained by capturing fluorescence which is the reflected light of the second light emitted from the object to be observed upon irradiation with the excitation light.

4. The medical image processing apparatus according to claim 1, wherein the interpolation image generation unit generates a second interpolation image which is 5. The medical image processing apparatus according to claim 1, wherein the interpolation image generation unit generates a second interpolation image which is a later interpolation image in time with respect to the first interpolation image, based on the first interpolation image, the fourth imaging image, and a fifth imaging image which is the first imaging image generated later in time with respect to the fourth imaging image.

6. The medical image processing apparatus according to claim 1, wherein the interpolation image generation unit calculates a motion vector based on the third captured image and the fourth captured image, and generates the first interpolation image by processing the second captured image based on the motion vector.

7. The medical image processing apparatus according to claim 1, wherein the interpolation image generation unit generates the first interpolation image in which the brightness of the second image is changed in accordance with the change in brightness from the third image to the fourth image.

8. The medical image processing apparatus according to claim 1, wherein the first captured image has a higher frame rate than the second captured image.

9. The medical image processing apparatus according to claim 1, wherein the second light comprises a first excitation light that excites a substance contained in the object of observation, and a second excitation light that excites a substance contained in the object of observation, and the second image is an image produced by time-division imaging the return light of the first excitation light from the object of observation and the return light of the second excitation light from the object of observation, and comprises a first fluorescence image obtained by imaging the return light of the first excitation light and a second fluorescence image obtained by imaging the return light of the second excitation light, and the first image has a higher imaging frame rate than the first fluorescence image and the second fluorescence image.

10. A medical image processing device for processing an input captured image, and a control unit for controlling the operation of the medical image processing device, wherein the medical image processing device comprises an image acquisition unit that acquires a first captured image generated by capturing the reflected light of a first light from an object to be observed, and a second captured image generated by capturing the reflected light of a second light from the object to be observed, and an interpolation image generation unit that generates an interpolation image to interpolate the second captured image, wherein the interpolation image generation unit generates a first interpolation image which is the interpolation image based on the second captured image, a third captured image which is the first captured image generated based on the same synchronization signal as the second captured image, and a fourth captured image which is the first captured image generated chronologically later than the third captured image.

11. The medical control device according to claim 10, wherein the second light is excitation light that excites a substance contained in the object to be observed, the second image is a fluorescence image captured of the fluorescence which is the reflected light of the second light emitted from the object to be observed due to irradiation with the excitation light, the control unit comprises a mode switching unit that switches between an interpolation mode in which the interpolation image generation unit generates the interpolation image and a non-interpolation mode in which the interpolation image generation unit does not generate the interpolation image, and the mode switching unit switches to the interpolation mode when the brightness of the fluorescence region in the fluorescence image is above a certain brightness, and switches to the non-interpolation mode when the brightness of the fluorescence region is below the certain brightness.

12. The medical control device according to claim 10, wherein the second light is excitation light that excites a substance contained in the object to be observed, the second image is a fluorescence image captured of the fluorescence which is the reflected light of the second light emitted from the object to be observed due to irradiation with the excitation light, the control unit comprises a mode switching unit that switches between an interpolation mode in which the interpolation image generation unit generates the interpolation image and a non-interpolation mode in which the interpolation image generation unit does not generate the interpolation image, and the mode switching unit switches to the interpolation mode when the size of the fluorescence region in the fluorescence image is greater than or equal to a specific size, and switches to the non-interpolation mode when the size of the fluorescence region is less than the specific size.

13. A medical observation system comprising: a light source device that emits a first light and a second light, respectively; an imaging device that generates a first image by imaging the reflected light of the first light from an object to be observed, and generates a second image by imaging the reflected light of the second light from the object to be observed; and a medical image processing device that processes the first image and the second image, wherein the medical image processing device comprises: an image acquisition unit that acquires the first image and the second image, respectively; and an interpolation image generation unit that generates an interpolation image that interpolates the second image, wherein the interpolation image generation unit generates a first interpolation image that is the interpolation image based on the second image, a third image which is the first image generated based on the same synchronization signal as the second image, and a fourth image which is the first image generated chronologically later than the third image.

14. A medical image processing method that acquires a first image generated by capturing the reflected light of a first light from an object to be observed, and a second image generated by capturing the reflected light of a second light from the object to be observed, and generates a first interpolated image which is an interpolated image obtained by interpolating the second image based on the second image, a third image generated based on the same synchronization signal as the second image, which is the first image, and a fourth image generated chronologically later than the third image, which is the first image.

Images