Image processing device, image processing method, and program

Abstract

Provided are an image processing device, an image processing method, and a program capable of improving extraction accuracy of a fine structure and a fine blood vessel of a mucous membrane surface layer. The image processing device comprises a processor. The processor: acquires an input image captured under illumination light including blue-violet light; estimates a plurality of illumination light components from the input image by smoothing filter processing with different parameters; extracts a plurality of reflectance components on the basis of the input image and the plurality of illumination light components; generates a base component on the basis of the plurality of illumination light components; generates a detail component on the basis of the plurality of reflectance components; and generates an output image on the basis of the base component and the detail component.

Description

Image processing device, image processing method, and program

[0001] This disclosure relates to an image processing apparatus, an image processing method, and a program.

[0002] Conventionally, in the medical field, a technique has been disclosed in which the illumination light component is estimated by applying a smoothing filter to an input image based on an imaging signal captured by an endoscope under illumination light containing blue-violet light, and the fine structure and microvessels of the mucosal surface in living organisms are extracted based on the reflectance component extracted by dividing this estimated illumination light component by the input image.

[0003] International Publication No. 2024 / 158040

[0004] However, while Patent Document 1, mentioned above, extracts the fine structure and microvessels of the mucosal surface from the input image, there are cases where the local contrast value of the fine structure and the local contrast value of the microvessels are extracted at the same value. This makes it difficult to determine whether something corresponds to a fine structure or a microvessel by comparing it with a preset reference value, leaving room for further improvement, and a technology that can extract the fine structure and microvessels of the mucosal surface with greater accuracy has been desired.

[0005] This disclosure has been made in view of the above, and aims to provide an image processing apparatus, an image processing method, and a program that can improve the accuracy of extracting the fine structure and microvessels of the mucosal surface.

[0006] To solve the above-mentioned problems and achieve the objectives, the image processing apparatus according to the present disclosure is an image processing apparatus comprising a processor, the processor acquires an input image captured under illumination light including blue-violet light, estimates a plurality of illumination light components from the input image by smoothing filter processing with different parameters, extracts a plurality of reflectance components based on the input image and the plurality of illumination light components, generates a base component based on the plurality of illumination light components, generates a detail component based on the plurality of reflectance components, and generates an output image based on the base component and the detail component.

[0007] Furthermore, the image processing method according to this disclosure is an image processing method performed by an image processing apparatus equipped with a processor, wherein the processor acquires an input image captured under illumination light including blue-violet light, estimates a plurality of illumination light components from the input image by smoothing filter processing with different parameters, extracts a plurality of reflectance components based on the input image and the plurality of illumination light components, generates a base component based on the plurality of illumination light components, generates a detail component based on the plurality of reflectance components, and generates an output image based on the base component and the detail component.

[0008] Furthermore, the program relating to this disclosure is a program executed by an image processing device equipped with a processor, which causes the processor to acquire an input image captured under illumination light including blue-violet light, estimate a plurality of illumination light components from the input image by smoothing filter processing with different parameters, extract a plurality of reflectance components based on the input image and the plurality of illumination light components, generate a base component based on the plurality of illumination light components, generate a detail component based on the plurality of reflectance components, and generate an output image based on the base component and the detail component.

[0009] According to this disclosure, the accuracy of extracting the fine structure and microvessels of the mucosal surface can be improved.

[0010] Figure 1 is a schematic diagram of a medical system according to one embodiment of the present disclosure. Figure 2 is a block diagram showing the functional configuration of the main part of the medical system according to one embodiment of the present disclosure. Figure 3 is a flowchart showing an overview of the processing performed by the medical system according to one embodiment of the present disclosure. Figure 4 is a diagram illustrating an overview of the processing performed by the image processing unit according to one embodiment of the present disclosure. Figure 5 is a diagram showing an example of an image generated by the image processing unit according to one embodiment of the present disclosure. Figure 6 is a diagram showing an example of components selected by the base component generation unit and the detail component generation unit according to one embodiment of the present disclosure. Figure 7 is a diagram showing the image processing content of an image to be displayed on a display device by the display control unit according to one embodiment of the present disclosure. Figure 8 is a diagram showing an example of the display on a display device by the display control unit according to one embodiment of the present disclosure. Figure 9 is a diagram showing another example of the display on a display device by the display control unit according to one embodiment of the present disclosure. Figure 10 is a diagram showing another example of the display on a display device by the display control unit according to one embodiment of the present disclosure. Figure 11 is a diagram showing another example of the display on a display device by the display control unit according to one embodiment of the present disclosure. Figure 12 is a diagram showing the image processing content of an image to be displayed on a display device by the display control unit according to Modification 1 of one embodiment of the present disclosure. Figure 13 is a diagram showing an example of the display of a display device by a display control unit according to Modification 1 of one embodiment of the present disclosure. Figure 14 is a diagram showing another example of the display of a display device by a display control unit according to Modification 1 of one embodiment of the present disclosure. Figure 15 is a diagram showing another example of the display of a display device by a display control unit according to Modification 1 of one embodiment of the present disclosure. Figure 16 is a diagram showing another example of the display of a display device by a display control unit according to Modification 1 of one embodiment of the present disclosure. Figure 17 is a diagram showing another image processing content for an image displayed on a display device by a display control unit according to Modification 1 of one embodiment of the present disclosure. Figure 18 is a diagram showing an example of an image resulting from the image processing content of Figure 17. Figure 19 is a diagram showing the image processing content for an image displayed on a display device by a display control unit according to Modification 2 of one embodiment of the present disclosure. Figure 20 is a diagram showing the image processing content for an image displayed on a display device by a display control unit according to Modification 3 of one embodiment of the present disclosure.Figure 21 is a diagram showing another image processing content for an image displayed on a display device by a display control unit according to Modification 3 of one embodiment of the present disclosure. Figure 22 is a diagram showing another image processing content for an image displayed on a display device by a display control unit according to Modification 3 of one embodiment of the present disclosure. Figure 23 is a diagram showing another image processing content for an image displayed on a display device by a display control unit according to Modification 3 of one embodiment of the present disclosure. Figure 24 is a diagram showing an example of display on a display device by a display control unit according to Modification 3 of one embodiment of the present disclosure. Figure 25 is a diagram showing an example of an image. Figure 26 is a diagram showing an example of an image. Figure 27 is a diagram showing an example of display on a display device by a display control unit according to Modification 4 of one embodiment of the present disclosure. Figure 28 is a diagram showing the image processing content for an image displayed on a display device 4 by a display control unit according to Modification 4 of one embodiment of the present disclosure. Figure 29 is a diagram showing the image processing content for an image displayed on a display device by a display control unit according to Modification 4 of one embodiment of the present disclosure. Figure 30 is a diagram showing the image processing content for an image displayed on a display device by a display control unit according to Modification 5 of one embodiment of the present disclosure. Figure 31 is a diagram showing the image processing content for an image displayed on a display device by a display control unit according to Modification 6 of one embodiment of the present disclosure. Figure 32 is a diagram showing the image processing content of an image to be displayed on a display device by a display control unit according to a modified example 7 of one embodiment of the present disclosure.

[0011] The embodiments for implementing this disclosure will be described in detail below with reference to drawings. However, this disclosure is not limited to the embodiments described below. Furthermore, the figures referenced in the following description only provide a general overview of the shape, size, and positional relationships to the extent that the content of this disclosure can be understood. That is, this disclosure is not limited to the shapes, sizes, and positional relationships exemplified in the figures. In addition, the same parts will be denoted by the same reference numerals in the drawings. Furthermore, as an example of a medical system relating to this disclosure, an endoscopic system equipped with a flexible endoscope will be described.

[0012] [Configuration of the Medical System] Figure 1 is a schematic diagram of the medical system according to one embodiment. Figure 2 is a block diagram showing the functional configuration of the main parts of the medical system according to one embodiment. The medical system 1 shown in Figures 1 and 2 is inserted into the body of a subject such as a patient and displays a display image based on the image signal (image data) generated by imaging the inside of the subject's body. Users such as doctors can examine the display image to check for bleeding sites, tumor sites, and abnormal sites, and measure their size. In one embodiment, the medical system 1 is described as an endoscopic system using a flexible endoscope as shown in Figure 1, but it is not limited to this, and may be a medical system equipped with a rigid endoscope, for example. Furthermore, the medical system 1 can also be a medical microscope or a medical surgical robot system, etc., which displays an output image (display image) based on the image signal (image data) captured by the endoscope on a display device while performing surgery or procedures.

[0013] The medical system 1 shown in Figure 1 comprises an endoscope device 2, a light source device 3, a display device 4, and a control device 5.

[0014] [Configuration of the Endoscope Device] First, the configuration of the endoscope device 2 will be explained. The endoscope device 2 is inserted into the subject and generates image signals (RAW data) by imaging the inside of the subject's body, and outputs these generated image signals to the control device 5. The endoscope device 2 comprises an insertion unit 21, an operating unit 22, and a universal cord 23.

[0015] The insertion portion 21 has a flexible, elongated shape. The insertion portion 21 includes a tip portion 24 that houses the imaging portion 244, which will be described later, a flexible curved portion 25 composed of a plurality of curved pieces, and a flexible, elongated flexible tube portion 26 connected to the base end of the curved portion 25.

[0016] The tip portion 24 is constructed using glass fiber or the like. The tip portion 24 includes a light guide 241 that forms a light guide path for light supplied from the light source device 3, an illumination lens 242 provided at the tip of the light guide 241, an optical system 243 that focuses at least one of the reflected light and the backlight from the subject, and an imaging unit 244 positioned at the imaging position of the optical system 243.

[0017] The illumination lens 242 is composed of one or more lenses and emits light supplied from the light guide 241 to the outside.

[0018] The optical system 243 is composed of one or more lenses and focuses the light returned from the subject and the reflected light reflected from the subject to form an image of the subject on the imaging plane of the imaging unit 244. The optical system 243 may also have a structure that allows the focal position (focus position) to be changed by moving along the optical axis L1 under the drive of an actuator (not shown). Of course, the optical system 243 may also have a zoom lens group in which the focal length can be changed by moving multiple lenses along the optical axis L1.

[0019] The imaging unit 244 is configured using an image sensor such as a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor), and generates an image signal (RAW data) by capturing images at a predetermined frame rate, and outputs this image signal to the control device 5. In one embodiment, the imaging unit 244 functions as an imaging device.

[0020] The operating unit 22 includes a bending knob 221 for bending the curved section 25 in the vertical and horizontal directions, a treatment instrument insertion section 222 for inserting treatment instruments such as biological forceps, laser scalpels, and examination probes into the body cavity, and a plurality of switches 223 that receive input of operation instruction signals for peripheral equipment such as air supply means, water supply means, and gas supply means, in addition to the light source device 3 and control device 5, as well as a pre-freeze signal that instructs the imaging unit 244 to take still images. Treatment instruments inserted from the treatment instrument insertion section 222 emerge from an opening (not shown) via a treatment instrument channel (not shown) at the tip 24.

[0021] The universal cord 23 incorporates at least a light guide 241 and a light-gathering cable that bundles one or more cables together. The bundled cable is a signal line that transmits and receives signals between the endoscope device 2 and the light source device 3 and the control device 5, and includes signal lines for transmitting and receiving setting data (signal data), signal lines for transmitting and receiving image signals (image data), signal lines for transmitting and receiving drive clock signals for driving the imaging unit 244, etc. The universal cord 23 has a connector portion 27 that can be attached to and detached from the light source device 3. The connector portion 27 has a coiled cable 27a extending from it, and the extended end of the coiled cable 27a has a connector portion 28 that can be attached to and detached from the control device 5.

[0022] [Configuration of the Light Source Device] Next, the configuration of the light source device 3 will be described. The light source device 3 supplies illumination light to irradiate the subject from the tip 24 of the endoscope device 2. The light source device 3 comprises a light source unit 31, a light source driver 32, and an illumination control unit 33.

[0023] The light source unit 31 irradiates the subject with white light including light in the red wavelength band, light in the green wavelength band, and light in the blue wavelength band, and at least one of special light. The light source unit 31 includes a focusing lens 311, a first light source 312, a second light source 313, a third light source 314, a fourth light source 315, and a fifth light source 316.

[0024] The focusing lens 311 is composed of one or more lenses. The focusing lens 311 focuses the light emitted by each of the first light source 312, second light source 313, third light source 314, fourth light source 315, and fifth light source 316 and emits it to the light guide 241.

[0025] The first light source 312 is configured using a red LED (Light Emitting Diode) lamp. The first light source 312 emits light in the red wavelength range (610 nm to 750 nm) (hereinafter simply referred to as "R light") based on the current supplied from the light source driver 32.

[0026] The second light source 313 is configured using a green LED lamp. The second light source 313 emits light in the green wavelength range (500 nm to 560 nm) (hereinafter simply referred to as "G light") based on the current supplied from the light source driver 32.

[0027] The third light source 314 is configured using a blue LED lamp. The third light source 314 emits light in the blue wavelength band (435 nm to 480 nm) (hereinafter simply referred to as "B light") based on the current supplied from the light source driver 32.

[0028] The fourth light source 315 is configured using a purple LED lamp. Based on the current supplied from the light source driver 32, the fourth light source 315 emits narrowband light (hereinafter simply referred to as "V light") in the blue-violet wavelength range (for example, 400 nm to 435 nm).

[0029] The fifth light source 316 is configured using a green LED lamp and a transmission filter that transmits a predetermined wavelength band. Based on the current supplied from the light source driver 32, the fifth light source 316 emits narrowband light (530 nm to 550 nm) in a predetermined wavelength band (hereinafter simply referred to as "NG light").

[0030] The light source driver 32, under the control of the illumination control unit 33, supplies current to the first light source 312, the second light source 313, the third light source 314, the fourth light source 315, and the fifth light source 316, thereby emitting light corresponding to the observation mode set in the medical system 1. Specifically, under the control of the illumination control unit 33, the light source driver 32 emits white light (hereinafter simply referred to as "W light") by emitting the first light source 312, the second light source 313, and the third light source 314 when the observation mode set in the medical system 1 is the normal observation mode. Furthermore, under the control of the illumination control unit 33, the light source driver 32 emits special light (hereinafter simply referred to as "S light") that enables narrow-band imaging (NBI) by emitting the fourth light source 315 and the fifth light source 316 when the observation mode set in the medical system 1 is the special light observation mode.

[0031] The lighting control unit 33 controls the timing of the illumination of the light source device 3 based on instruction signals received from the control device 5. Specifically, the lighting control unit 33 causes the first light source 312, the second light source 313, and the third light source 314 to emit light at predetermined intervals. The lighting control unit 33 is configured using a CPU (Central Processing Unit) and the like. Furthermore, when the observation mode of the medical system 1 is the normal observation mode, the lighting control unit 33 controls the light source driver 32 to cause the first light source 312, the second light source 313, and the third light source 314 to emit W light. Furthermore, when the observation mode of the medical system 1 is the special light observation mode, the lighting control unit 33 controls the light source driver 32 to combine the fourth light source 315 and the fifth light source 316 to emit S light. The lighting control unit 33 may, depending on the observation mode of the medical system 1, control the light source driver 32 to emit light from two or more of the first light source 312, second light source 313, third light source 314, fourth light source 315, and fifth light source 316 in combination.

[0032] [Configuration of the Display Device] Next, the configuration of the display device 4 will be described. The display device 4 displays a display image based on image data generated by the endoscope device 2 received from the control device 5. The display device 4 displays various information related to the medical system 1. The display device 4 is constructed using a display panel such as a liquid crystal or organic EL (Electro Luminescence).

[0033] [Configuration of the control device] Next, the configuration of the control device 5 will be described. The control device 5 receives image data generated by the endoscope device 2, performs predetermined image processing on the received image data, and outputs it to the display device 4. The control device 5 also comprehensively controls the operation of the entire medical system 1. The control device 5 comprises an image processing unit 51, an input unit 52, a recording unit 53, and a control unit 54.

[0034] The image processing unit 51, under the control of the control unit 54, acquires the image signal generated by the endoscope device 2, performs predetermined image processing on the acquired image signal, and outputs it to the display device 4. The image processing unit 51 is configured using a memory and a processor having hardware such as a GPU (Graphics Processing Unit), DSP (Digital Signal Processing), or FPGA (Field Programmable Gate Array). The image processing unit 51 includes an acquisition unit 511, an estimation unit 512, an extraction unit 513, a base component generation unit 514, a detail component generation unit 515, a determination unit 516, a correction unit 517, a synthesis unit 518, an image generation unit 519, and a display control unit 520. In one embodiment, the image processing unit 51 functions as an image processing device.

[0035] The acquisition unit 511 acquires an input image corresponding to the image signal (RAW data) from the imaging unit 244 of the endoscope device 2.

[0036] The estimation unit 512 processes the input image P acquired by the acquisition unit 511. IN Multiple illumination light components are estimated from the input image by applying smoothing filter processing with different parameters. Specifically, the estimation unit 512 estimates multiple illumination light components by performing multiple bilateral filter processing of different sizes as multiple smoothing filter processing. More specifically, the estimation unit 512 performs bilateral filter processing of a first kernel size, bilateral filter processing of a second kernel size, and bilateral filter processing of a third kernel size on the input image to estimate the first illumination light component, the second illumination light component, and the third illumination light component, and generates first illumination light component images, second illumination light component images, and third illumination light component images.

[0037] The extraction unit 513 extracts multiple reflectance components and generates multiple reflectance component images based on the input image acquired by the acquisition unit 511 and the multiple illumination light components estimated by the estimation unit 512. Specifically, the extraction unit 513 extracts the first reflectance component, the second reflectance component, and the third reflectance component by dividing the pixel value of each pixel in the input image by the respective pixel values ​​of the first illumination light component, the second illumination light component, and the third illumination light component at the corresponding pixel address, thereby generating the first reflectance component image, the second reflectance component image, and the third reflectance component image.

[0038] The base component generation unit 514 generates a base component based on multiple illumination light components. Specifically, the base component generation unit 514 generates a base component by selecting one illumination light component from the first illumination light component, the second illumination light component, and the third illumination light component based on the object of interest and object of non-interest determined by the determination unit 516.

[0039] The detail component generation unit 515 generates detail components based on a plurality of reflectance components. Specifically, the detail component generation unit 515 generates detail components by selecting one reflectance component from the first reflectance component, the second reflectance component, and the third reflectance component based on the object of interest and the object of non-interest determined by the determination unit 516.

[0040] The determination unit 516 determines the object of focus, the object of non-focus, and correction parameters based on information about the object of focus in the image input from the input unit 52, as the user operates the input unit 52. Specifically, the object of focus includes information about the object of focus and the object of non-focus. More specifically, the objects of focus or non-focus are the microvessels of the mucosal surface and the microstructure of the mucosal surface.

[0041] The correction unit 517 performs correction processing on the detail components generated by the detail component generation unit 515 based on the correction parameters determined by the determination unit 516. Specifically, the correction unit 517 generates a corrected detail component image corresponding to the detail component corrected by the correction parameters determined by the determination unit 516, based on the detail component image of the detail component generated by the detail component generation unit 515.

[0042] The synthesis unit 518 generates a composite image by combining the base component generated by the base component generation unit 514 and the detail component corrected by the correction unit 517.

[0043] The image generation unit 519 generates an illumination light image corresponding to the illumination light based on the input image acquired by the acquisition unit 511. Specifically, the image generation unit 519 generates a narrowband light image or a white light image.

[0044] The display control unit 520 causes the illumination light image generated by the image generation unit 519 and the composite image combined by the synthesis unit 518 to be displayed on the display device 4.

[0045] The input unit 52 receives instruction signals that instruct the operation of the medical system 1 and instruction signals that instruct the observation mode of the medical system 1, and outputs these received instruction signals to the control unit 54. The input unit 52 is configured using switches, buttons, a touch panel, etc.

[0046] The recording unit 53 records various programs executed by the medical system 1, data being executed by the medical system 1, and image data generated by the endoscope device 2. The recording unit 53 is configured using volatile memory, non-volatile memory, and a memory card, etc. The recording unit 53 has a program recording unit 531 that records various programs executed by the medical system 1.

[0047] The control unit 54 has memory and a processor consisting of at least one hardware component such as an FPGA or CPU. The control unit 54 controls each component that makes up the medical system 1.

[0048] [Medical System Processing] Next, the processing performed by the medical system 1 will be explained. Figure 3 is a flowchart showing an overview of the processing performed by the medical system 1. Figure 4 is a diagram illustrating an overview of the processing performed by the image processing unit 51. Figure 5 is a diagram showing an example of an image generated by the image processing unit 51.

[0049] As shown in Figure 3, first, the control unit 54 controls the illumination control unit 33 to cause the fourth light source 315 and the fifth light source 316 of the light source device 3 to emit illumination light of blue-violet and green narrowband light (V light + NG light) onto the biological tissue (step S101). Alternatively, the control unit 54 may also cause the first light source 312, the second light source 313, and the third light source 314 of the light source device 3 to emit illumination light of white light onto the biological tissue. In other words, the control unit 54 may sequentially irradiate the light source device 3 with illumination light while intermittently switching between narrowband light and white light.

[0050] Next, the control unit 54 causes the imaging unit 244 to capture the reflected light from the biological tissue (step S102), and causes the imaging unit 244 to generate an image signal (step S103).

[0051] Subsequently, the acquisition unit 511 acquires an input image corresponding to the image signal (RAW data) from the imaging unit 244 of the endoscope device 2 (step S104). Specifically, as shown in Figures 4 and 5, the acquisition unit 511 acquires an input image corresponding to the image signal (RAW data) from the imaging unit 244 of the endoscope device 2 in order to input it to the image processing unit 51. Here, the input image P IN This is an image of the subject's microstructure and biological tissue, including microvessels, captured by the endoscope device 2 under illumination light. This input image consists of pixels in a two-dimensional matrix of M rows and N columns (M = an integer greater than or equal to 1, N = an integer greater than or equal to 1). Each pixel has pixel values ​​(R, G, B).

[0052] Next, the estimation unit 512 processes the input image P acquired by the acquisition unit 511. IN By applying smoothing filter processing with different parameters to the input image P, INa plurality of illumination light components are estimated from (step S105). Specifically, as shown in FIG. 4 and FIG. 5, the estimating unit 512 estimates a plurality of illumination light components by a plurality of smoothing filter processes, that is, a first smoothing filter process, a second smoothing filter process and a third smoothing filter process, using a plurality of bilateral filter processes with different sizes. More specifically, as shown in FIG. 4 and FIG. 5, in estimation of illumination light components based on Retinex theory, the estimating unit 512 receives the input image P IN , performs bilateral filter processing with a first kernel size, bilateral filter processing with a second kernel size, and bilateral filter processing with a third kernel size on to obtain a first illumination light component (R B1 , G B1 , B B1 ), a second illumination light component (R B2 , G B2 , B B2 ), and a third illumination light component (R B3 , G B3 , B B3 ) to generate a first illumination light component image P B1 , a second illumination light component image P B2 and a third illumination light component image P B3 . That is, in Retinex theory, a composite image (output image) described later is expressed as a product of illumination light illuminating an object (biological tissue) and the reflectance of the object. Furthermore, each of the plurality of illumination light components represents the influence of light from a light source in an image. This indicates the intensity of light that varies across the entire image.

[0053] Also, the first illumination light component image P B1 's first illumination light component (R B1 , G B1 , B B1 ), the second illumination light component image P B2 's second illumination light component (R B2 , G B2 , B B2 ) and the third illumination light component image P B3 's third illumination light component (R B3 , G B3 , B B3 ) each has an illumination light component value that is the value of the input image P INIt is expressed in the range of 0-255, similar to the pixel values. That is, the first illumination light component image P B1、 Image P of the second illumination light component B2 and the third illumination light component image P B3 The input image P IN Similarly, it is composed of pixels that make up a two-dimensional matrix of M rows and N columns.

[0054] Furthermore, the first kernel size, second kernel size, and third kernel size each indicate the range of pixels used when applying the bilateral filter. Specifically, the first kernel size is smaller than the second kernel size, and the second kernel size is smaller than the third kernel size (first kernel size < second kernel size < third kernel size). Bilateral filtering allows for the extraction of illumination components while preserving edges, thus removing the effects of illumination while maintaining edge detail.

[0055] Furthermore, the estimation unit 512 may perform Gaussian filtering as a smoothing filter process. In this case, the estimation unit 512 may reduce the detail components of the edges by performing Gaussian filtering, but it can smooth and extract the illumination light component. In addition, the estimation unit 512 may perform low-pass filtering as a smoothing filter process. In this case, the estimation unit 512 may reduce the detail components of the edges by performing low-pass filtering, but it can remove high-frequency components and extract the illumination light component. Also, the estimation unit 512 processes the input image P according to the observation mode of the endoscope device 2 input by the user via the input unit 52. INThe type of smoothing filter applied to the signal may be selected from bilateral filtering, Gaussian filtering, and low-pass filtering. The estimation unit 512 may also select Gaussian filtering and low-pass filtering as the type of smoothing filter, in addition to bilateral filtering. Furthermore, when the estimation unit 512 generates multiple illumination light components, such as a first illumination light component, a second illumination light component, and a third illumination light component, it uses bilateral filters with different kernel sizes. However, it may also generate multiple illumination light components using bilateral filters with different smoothing parameters (the magnitude of the variance of the Gaussian function in the bilateral filter equation).

[0056] Returning to Figure 3, we will continue the explanation from step S106 onwards. In step S106, the extraction unit 513 extracts the input image P acquired by the acquisition unit 511. IN The estimation unit 512 estimates the first illumination light component (R B1 , G B1 , B B1 ), second illumination light component (R B2 , G B2 , B B2 ) and the third illumination light component (R B3 , G B3 , B B3 Image P of the first illumination light component corresponding to each of the following: B1 , second illumination light component image P B2 and the third illumination light component image P B3 Based on this, multiple reflectance components are extracted to generate multiple reflectance component images. Specifically, as shown in Figures 4 and 5, the extraction unit 513 extracts multiple reflectance components from the input image P IN The pixel value of each pixel is the first illumination light component (R) of the corresponding pixel address. B1 , G B1 , B B1 ), second illumination light component (R B2 , G B2 , B B2 ) and the third illumination light component (R B3 , G B3 , B B3By dividing by each pixel value of ), the first reflectance component (R D1 , G D1 , B D1 ), the second reflectance component (R D2 , G D2 , B D2 ) and the third reflectance component (R D3 , G D3 , B D3 ) is extracted to create the first reflectance component image P D1 , second reflectance component image P D2 and the third reflectance component image P D3 It generates R. More specifically, the extraction unit 513 determines a predetermined pixel value in the reflectance component image corresponding to the Nth (N=1, 2, 3) reflectance component. （m，n） , input image ＩＮ１ I （m，n） L is the pixel value of the corresponding pixel address in the illumination light component image corresponding to the Nth (N=1, 2, 3) illumination light component. （m，n） In this case, the R of each pixel in the reflectance component is calculated using the following equation (1). （m，n） Calculate R （m，n） = I （m，n） / L （m，n） (1) The reflectance component represents the color and brightness of the object itself. This indicates how much light the object reflects. In other words, the reflectance component can be calculated and generated using the above-mentioned equation (1).

[0057] Next, the determination unit 516 determines the object of focus, the object of non-focus, and the correction parameters based on the information about the object of focus in the image input from the input unit 52, as the user operates the input unit 52 (step S107). Specifically, the object of focus includes information about the object of focus and the object of non-focus. More specifically, the objects to be focused on or not focused on are the microvessels of the mucosal surface and the microstructure of the mucosal surface. That is, the object of focus is the object to be enhanced, and the object of non-focus is the object to be suppressed. Specifically, the information about the object of focus and the object of non-focus includes the content of an observation mode that specifies one or more of the following: microvessels of the mucosal surface as the object of enhancement, microvessels of the mucosal surface as the object of suppression, microstructure of the mucosal surface as the object of enhancement, and microstructure of the mucosal surface as the object of suppression, and the correction parameters that the correction unit 517 corrects. The observation modes include ultrastructure / microvessel enhancement mode, microvessel enhancement mode, microvessel suppression mode, ultrastructure enhancement mode, ultrastructure suppression mode, ultrastructure enhancement / microvessel suppression mode, microvessel enhancement / ultrastructure suppression mode, etc.

[0058] Here, we will explain the correction parameters determined by the determination unit 516. The detail component (R) is a reflectance component (≒ local contrast information) which will be described later. D , G D , B D Pixels where the value of ) is greater than the reference value are called fine structures, and detail components (R) which are the reflectance components (≒ local contrast information) are called fine structures. D , G D , B D Assuming that pixels whose detail component takes a value smaller than the reference value are microvessels, the determination unit 516 determines a correction coefficient that moves the value of the detail component (≒local contrast information), which is the reflectance component, away from 1 for pixels whose detail component takes a small value (a value less than 1) less than the reference value, when the observation mode of the endoscope device 2, which is input by the user via the input unit 52 as INPUT2, is selected as the microvessel enhancement mode that emphasizes microvessels. In other words, when the microvessel enhancement mode is selected as the observation mode, the determination unit 516 determines a coefficient of 1 or less as the correction coefficient.

[0059] Furthermore, when the observation mode of the endoscope apparatus 2 input by the user via the input unit 52 as INPUT2 selects the microvessel suppression mode for suppressing microvessels, the determining unit 516 determines a correction coefficient that brings the value of the detail component (≈ local contrast information), which is a reflectance component, closer to 1 for pixels whose detail component takes a small value less than a reference value (a value less than 1). That is, when the microvessel suppression mode is selected as the observation mode, the determining unit 516 determines a correction coefficient of 1 or greater.

[0060] Furthermore, when the observation mode of the endoscope apparatus 2 input by the user via the input unit 52 as INPUT2 selects the microstructure enhancement mode for enhancing microstructures, the determining unit 516 determines a correction coefficient that moves the value of the detail component (≈ local contrast information), which is a reflectance component, away from 1 for pixels whose detail component takes a value larger than a reference value (a value larger than 1). That is, when the microstructure enhancement mode is selected as the observation mode, the determining unit 516 determines a correction coefficient of 1 or greater.

[0061] Furthermore, when the observation mode of the endoscope apparatus 2 input by the user via the input unit 52 as INPUT2 selects the microstructure suppression mode for suppressing microstructures, the determining unit 516 determines a correction coefficient that brings the value of the detail component (≈ local contrast information), which is a reflectance component, closer to 1 for pixels whose detail component takes a value larger than a reference value (a value larger than 1). That is, when the microstructure suppression mode is selected as the observation mode, the determining unit 516 determines a correction coefficient of 1 or less.

[0062] Thereafter, the base component generating unit 514 generates the first illumination light component (R B1 , G B1 , B B1 ), the second illumination light component (R B2 , G B2 , B B2 ) and the third illumination light component (R B3 , G B3 , B B3), a base component is generated by selecting one illumination light component (step S108). Specifically, as shown in Fig. 4 and Fig. 5, the base component generation unit 514 generates the first illumination light component (R B1 , G B1 , B B1 ), the second illumination light component (R B2 , G B2 , B B2 ) and the third illumination light component (R B3 , G B3 , B B3 ), a base component (R B , G B , B B ) is selected by selecting one illumination light component, to generate a base component image P B .

[0063] Subsequently, based on the target object and non-target object determined by the determination unit 516, the detail component generation unit 515 generates the first reflectance component (R D1 , G D1 , B D1 ), the second reflectance component (R D2 , G D2 , B D2 ) and the third reflectance component (R D3 , G D3 , B D3 ), a detail component is generated by selecting one reflectance component (step S109). Specifically, as shown in Fig. 4 and Fig. 5, the detail component generation unit 515 generates the first reflectance component (R D1 , G D1 , B D1 ), the second reflectance component (R D2 , G D2 , B D2 ) and the third reflectance component (R D3 , G D3 , B D3 ), a detail component (R D , G D , B D ) is selected by selecting one reflectance component, to generate a detail component image P D .

[0064] Here, we will explain the components that the base component generation unit 514 and the detail component generation unit 515 select based on the focus and non-focus determined by the determination unit 516. Figure 6 shows an example of the components selected by the base component generation unit 514 and the detail component generation unit 515.

[0065] As shown in Table T1 of Figure 6, when microstructure and microvessels are selected as the object of interest (when the microvessel-enhancing / microstructure-enhancing mode is selected), the base component generation unit 514 selects the first illumination light component and generates it as the base component, and the detail component generation unit 515 selects the first reflectance component and generates it as the detail component. Furthermore, when the object of interest is microvessels, the base component generation unit 514 selects the third illumination light component and generates the base component, and the detail component generation unit 515 selects the third reflectance component and generates the detail component. In addition, when the object of interest is microstructure, the base component generation unit 514 selects the second illumination light component and generates the base component, and the detail component generation unit 515 selects the second reflectance component and generates the detail component. Furthermore, when the object of interest is microstructure and the non-object of interest is microvessels, the base component generation unit 514 selects the third illumination light component and generates the base component, and the detail component generation unit 515 selects the third reflectance component and generates the detail component. When the object of interest is microvessels and the object of non-interest is a microstructure, the base component generation unit 514 selects a second illumination light component to generate the base component, and the detail component generation unit 515 selects a third reflectance component to generate the detail component.

[0066] In this way, the base component generation unit 514 and the detail component generation unit 515 generate base components and detail components by selecting desired illumination light components and reflectance components from a plurality of illumination light components and a plurality of reflectance components based on the object of interest and non-object of interest (type of observation mode) determined by the determination unit 516.

[0067] Returning to Figure 3, we will continue the explanation from step S110 onward. In step S110, the correction unit 517 performs correction processing on the detail components generated by the detail component generation unit 515 based on the correction parameters determined by the determination unit 516. Specifically, the correction unit 517 performs correction processing on the detail components (R) generated by the detail component generation unit 515. D , G D , B D ) Detail component image P D In contrast, the detail component (R) corrected by the correction parameter determined by the determination unit 516. D´ , G D´ , B D´ ) Corrected detail component image P D´ This generates the detail component (R), which is the reflectance component (≒ local contrast information). D , G D , B D When the value is greater than 1, and when it is less than or equal to 1, it represents different states. Specifically, the detail component (R), which is the reflectance component (≒ local contrast information), represents different states. D , G D , B D If the reflectance component is greater than 1, it indicates that the object is reflecting more light than the illumination light component. For example, in microstructures, very bright or white objects reflect strong light, indicating that the surface of the object is highly reflective, and this corresponds to microstructures. Therefore, in microstructures, the reflectance component is greater than 1, indicating that the surface of the object is highly reflective. In contrast, in microvessels, the detail component (R), which is the reflectance component (≒ local contrast information), is greater than 1. D , G D , B D If the value is 1 or less, it indicates that the object is reflecting the same amount of light as or less than the illumination light component, meaning that the surface of the object does not reflect or absorb much light, which corresponds to microvessels. In this way, the correction unit 517 corrects the detail component image P D In contrast, the detail component (R) corrected by the parameters determined by the determination unit 516 D´ , G D´ , B D´ ) corresponding detail component image P D´Generates.

[0068] Next, the synthesis unit 518 processes the base component (R) generated by the base component generation unit 514. B , G B , B B ) and the detail component corrected by the correction unit 517 (R D´ , G D´ , B D´ ) and a composite image is generated by combining them (step S111). Specifically, the synthesis unit 518 combines the base component (R B , G B , B B ) Base component image P B Then, the detail component (R) corrected by the correction unit 517 D´ , G D´ , B D´ ) Detail component image P D´ Based on this, composite image P B＋D´ It generates the base component image P. Specifically, the synthesis unit 518 generates the base component image P. B And, detail component image P D´ By multiplying the pixel values ​​of the pixels at their corresponding pixel addresses, a composite image P is created. B＋D´ It generates the output image P. In detail, the synthesis unit 518 generates the output image P. B＋D´ The pixel value of O （m，n） , base ingredients L （m，n） , detail component R （m，n） In that case, the composite image P is obtained by the following equation (2) B＋D´ O of the pixel value （m，n） Calculate O （m，n） = L （m，n） ×R （m，n） ... (2)

[0069] Subsequently, the image generation unit 519 processes the input image P acquired by the acquisition unit 511. IN Based on this, an illumination light image corresponding to the illumination light is generated (step S112). Specifically, the image generation unit 519 generates a narrowband light image or a white light image.

[0070] Next, the display control unit 520 causes the illumination light image generated by the image generation unit 519 and the composite image combined by the synthesis unit 518 to be displayed on the display device 4 (step S113).

[0071] Figure 7 shows the image processing details of the image to be displayed on the display device 4 by the display control unit 520. Figure 8 shows an example of the display on the display device 4 by the display control unit 520. Figure 9 shows another example of the display on the display device 4 by the display control unit 520. Figure 10 shows another example of the display on the display device 4 by the display control unit 520. Figure 11 shows another example of the display on the display device 4 by the display control unit 520.

[0072] The display control unit 520, referring to Table T2 in Figure 7, displays the NBI image as the illumination light image on the main screen P100 and the composite image generated by the synthesis unit 518 on the sub-screen P101, as shown in Figure 8, on the display device 4. In this case, as shown in Figure 9, the display control unit 520 may also display the composite image on the display device 4 on the sub-screen P102, which has a smaller display area than the main screen P100 on which the NBI image is displayed. Furthermore, as shown in Figure 10, if the medical system 1 is further equipped with a display device 6 in addition to the display device 4, the NBI image may be displayed on the main screen P100 of the display device 4 and the composite image may be displayed on the sub-screen P101 of the display device 6. In addition, as shown in Figure 11, the display control unit 520 may switch the display timing of the NBI image and the composite image according to a switching signal input from the input unit 52 and display them on the display device 4. Of course, the display control unit 520 may also swap the display positions, display area sizes, and display timings of the main screen P100 and the sub-screen P101 to display the NBI image and the composite image. Note that "2" in Table T2 of Figure 7 indicates the fine structure enhancement mode, and if it is neither an enhancement target nor a suppression target, it is written as extraction.

[0073] According to the embodiment described above, the synthesis unit 518 generates a composite image by combining the base component generated by the base component generation unit 514 and the detail component corrected by the correction unit 517, thereby improving the extraction accuracy of the fine structure of the mucosal surface and the microvessels.

[0074] In one embodiment, the estimation unit 512 estimated multiple illumination light components by performing multiple bilateral filter processes of different sizes as a first smoothing filter process, a second smoothing filter process, and a third smoothing filter process. However, it is not limited to this, and for example, at least two of the first smoothing filter process, the second smoothing filter process, and the third smoothing filter process may be performed to estimate two illumination lights.

[0075] (Modification 1) Next, Modification 1 of one embodiment of the present disclosure will be described. Figure 12 is a diagram showing the image processing content of an image to be displayed on the display device 4 by the display control unit 520 according to Modification 1 of one embodiment of the present disclosure. Figure 13 is a diagram showing an example of the display on the display device 4 by the display control unit 520 according to Modification 1 of one embodiment of the present disclosure. Figure 14 is a diagram showing another example of the display on the display device 4 by the display control unit 520 according to Modification 1 of one embodiment of the present disclosure. Figure 15 is a diagram showing another example of the display on the display device 4 by the display control unit 520 according to Modification 1 of one embodiment of the present disclosure. Figure 16 is a diagram showing another example of the display on the display device 4 by the display control unit 520 according to Modification 1 of one embodiment of the present disclosure.

[0076] The display control unit 520, referring to Table T3 in Figure 12, sets the NBI image as the illumination light image as the main screen P200, and displays two composite images with different focus or non-focus areas, generated by the synthesis unit 518, as sub-screens P201 and P202 on the display device 4, as shown in Figure 13. In this case, as shown in Figure 14, the display control unit 520 may also display two composite images with different image processing contents on the display device 4, each of the sub-screens P301 and P302, which have a smaller display area than the main screen P100 on which the NBI image is displayed. Furthermore, as shown in Figure 15, if the medical system 1 is further equipped with a display device 6 in addition to the display device 4, the NBI image may be displayed on the main screen P100 of the display device 4, and two composite images with different image processing contents may be displayed on the sub-screens P101 and P102 of the display device 6. Furthermore, as shown in Figure 16, the display control unit 520 may switch the display timing of the NBI image and two composite images with different image processing content according to a switching signal input from the input unit 52 and display them on the display device 4. Of course, the display control unit 520 may also swap the display position, display area size, and display timing of the main screen P200 and sub-screens P201 and P202 to display the NBI image and two composite images with different image processing content. Note that Table T3 in Figure 12 stores the possible combinations of base components, detail components, and correction parameters for the composite image displayed on the sub-screens (sub-screen 1 and sub-screen 2).

[0077] Figure 17 shows another image processing content for an image displayed on the display device 4 by the display control unit 520 according to Modification 1 of one embodiment of the present disclosure. Figure 18 shows an example of an image resulting from the image processing content of Figure 17.

[0078] The display control unit 520 can display images of further subdivided image processing results on the sub-screen of the display device 4, as shown in display T4 of Figure 17, from the image processing content of "4" and "5" in Table T3 of Figure 12 performed by the image processing unit 51. Specifically, the display control unit 520 can display NBI images P in response to the operation of the input unit 52 by a user such as a doctor. NBI A composite image P of "4" generated from 「4」 and composite image P of "5"「5」 In contrast, multiple composite images were generated using correction parameters that suppressed Cy (cyan) and Br (brown) blood vessels, i.e., composite image P 「4-2」 , composite image P 「4-3」 , composite image P 「5-2」 , composite image P 「5-3」 This is displayed on the secondary screen of the display device 4.

[0079] According to the modified example 1 described above, a physician can simultaneously or while switching between NBI images or WLI images (which are white light images), composite images of microvessels processed with enhancement, normal processing, or suppression, and composite images of fine structures processed with enhancement, normal processing, or suppression.

[0080] (Modification 2) Next, Modification 2 of one embodiment of the present disclosure will be described. Figure 19 is a diagram showing the image processing content of the image that the display control unit 520 displays on the display device 4 according to Modification 2 of one embodiment of the present disclosure.

[0081] As shown in Table T5 of Figure 19, the display control unit 520 processes the microvessels generated by the image processing unit 51 using normal processing or suppression processing, and displays the composite image with enhanced or normally processed microstructures on the main and sub-screens of the display device 4. The display control unit 520 may also display NBI images or the like on the sub-screen of the display device 4.

[0082] According to the modified example 2 described above, users such as doctors can compare and observe composite images of microvessels and microstructures that have been processed differently.

[0083] (Modification 3) Next, Modification 3 of one embodiment of the present disclosure will be described. Figure 20 is a diagram showing the image processing content of the image that the display control unit 520 displays on the display device 4 according to Modification 3 of one embodiment of the present disclosure.

[0084] As shown in display T6 of Figure 20, the image processing unit 51, under the control of the control unit 54, changes the content of image processing for the NBI image as an input image based on endoscope identification information that identifies the type of endoscope device 2 connected to the control device 5. Specifically, under the control of the control unit 54, if the endoscope device 2 connected to the control device 5 is for the upper gastrointestinal tract, the display control unit 520 controls the image processing unit 51 so that the NBI image is displayed on the main screen, a composite image with enhanced fine structure is displayed on sub-screen 1, and a composite image with enhanced fine vessels is displayed on sub-screen 2. In contrast, if the endoscope device 2 connected to the control device 5 is for the lower gastrointestinal tract, the display control unit 520 controls the image processing unit 51 so that the NBI image is displayed on the main screen, a composite image with enhanced fine vessels is displayed on sub-screen 1, and a composite image with enhanced fine structure is displayed on sub-screen 2.

[0085] Figure 21 is a diagram showing another image processing content of the image that the display control unit 520 causes to be displayed on the display device 4, according to Modification 3 of one embodiment of the present disclosure.

[0086] The display control unit 520 may display images of further subdivided image processing results on the sub-screen 1 or sub-screen 2 of the display device 4, as shown in display T7 of Figure 21. In this case, "5-4", "6-4", and "7-4" are images processed by the image processing unit 51 with parameters that result in a higher sharpness of microvessels compared to "5-5", "6-5", and "7-5".

[0087] Figure 22 is a diagram showing another image processing content for an image displayed on the display device 4 by the display control unit 520 according to Modification 3 of one embodiment of the present disclosure.

[0088] The display control unit 520 may display images of further subdivided image processing results on the sub-screen 1 or sub-screen 2 of the display device 4, as shown in display T8 of Figure 22. In this case, "2-4", "3-4", and "4-4" are images processed by the image processing unit 51 with parameters that have a higher degree of fine structure sharpness compared to "2-5", "3-5", and "4-5".

[0089] Figure 23 is a diagram showing another image processing content of an image displayed on the display device 4 by the display control unit 520 according to Modification 3 of one embodiment of the present disclosure. Figure 24 is a diagram showing an example of the display on the display device 4 by the display control unit 520 according to Modification 3 of one embodiment of the present disclosure. Figure 25 is a diagram showing an example of an image. Figure 26 is a diagram showing an example of an image.

[0090] As shown in Table T9 of Figure 23 and Figure 24, the display control unit 520 can display images of further subdivided image processing results from the image processing content of "3" and "6" in Table T6 of Figure 20 on the display device 4 in the order of main screen, sub-screen 1, sub-screen 2, and sub-screen 3, switching sequentially with each operation on the input unit 52. Specifically, the display control unit 520 can process the NBI image P in response to the operation of the input unit 52 by a user such as a doctor. NBI Multiple composite images were generated from the composite image "6" generated from Cy (cyan) and Br (brown) using correction parameters that emphasize blood vessels, i.e., composite image P as shown in Figures 25 and 26. 「6-6」 , composite image P 「6-7」 This is displayed on the sub-screen 3 of the display device 4. In this case, "3-4" is an image processed by the image processing unit 51 with parameters that provide a higher level of sharpness of the fine structure compared to "3-5". Also, "6-4" is an image processed by the image processing unit 51 with parameters that provide a higher level of sharpness of the microvessels compared to "6-5". Similarly, "6-6" is an image processed by the image processing unit 51 with parameters that provide a higher level of sharpness of the Cy microvessels compared to "6-7", and it is possible to improve the diagnostic ability based on neoplastic vessels that may exhibit a Cy-like appearance.

[0091] According to the modified example 3 described above, images corresponding to the image processing required for observation of the upper gastrointestinal tract endoscopic device 2 and the lower gastrointestinal tract endoscopic device 2 can be automatically switched and displayed, thereby improving the usability for users such as doctors.

[0092] (Modification 4) Next, Modification 4 of one embodiment of the present disclosure will be described. Figure 27 is a diagram showing an example of the display of the display device 4 by the display control unit 520 according to Modification 4 of one embodiment of the present disclosure. Figure 28 is a diagram showing the image processing content of the image that the display control unit 520 causes to be displayed on the display device 4 according to Modification 4 of one embodiment of the present disclosure. Figure 29 is a diagram showing the image processing content of the image that the display control unit 520 causes to be displayed on the display device 4 according to Modification 4 of one embodiment of the present disclosure.

[0093] As shown in Figure 27, when a user such as a doctor operates the input unit 52 or long-presses the switch 223, the display control unit 520 may switch from a combination of two screens as shown in Table T10 in Figure 28 (see Figure 27(a)) to a combination of four screens as shown in Table T11 in Figure 29 (see Figure 27(b)). At this time, each time a user such as a doctor operates the input unit 52 or long-presses the switch 223, the display control unit 520 switches the image to be displayed and displays it on the display device 4.

[0094] According to the modified example 4 of the embodiment described above, images corresponding to the image processing required for observation of the upper gastrointestinal tract endoscopic device 2 and the lower gastrointestinal tract endoscopic device 2 can be switched and displayed, thereby improving the usability for users such as doctors.

[0095] (Modification 5) Next, Modification 5 of one embodiment of the present disclosure will be described. Figure 30 is a diagram showing the image processing content of the image that the display control unit 520 displays on the display device 4 according to Modification 5 of one embodiment of the present disclosure.

[0096] As shown in Table T12 of Figure 30, the display control unit 520, under the control of the control unit 54, displays images generated by the image processing unit 51 on the display device 4 using a combination of image processing content pre-set according to the user, such as a doctor, or the subject of examination or symptoms. The contents of this combination setting are recorded in the recording unit 53, and when the user, such as a doctor, operates the input unit 52, the image processing unit 51 retrieves the image processing content of the desired combination of names from the recording unit 53 and generates each image.

[0097] According to the modified example 5 of the embodiment described above, the image processing unit 51 can generate an image using a combination of pre-set image processing content, thereby improving usability for users such as doctors.

[0098] (Modification 6) Next, Modification 6 of one embodiment of the present disclosure will be described. Figure 31 is a diagram showing the image processing content of the image that the display control unit 520 displays on the display device 4 according to Modification 6 of one embodiment of the present disclosure.

[0099] As shown in Table T13 of Figure 31, when a user such as a doctor presses the input unit 52 to perform a freeze operation, the display control unit 520 receives an instruction signal from the input unit 52 to freeze the still image of the observation mode during video observation (white light image, NBI image, RDI image (Red Dichromatic)). The Imaging is displayed on the display device 4. When a user such as a doctor operates the input unit 52 to switch observation modes, the display control unit 520 refers to the operation signal input from the input unit 52 and the display T13 in Figure 31, and sequentially switches between still images of multiple modes generated by the image processing unit 51 and displays them on the display device 4 (see Figure 24). Subsequently, when a user such as a doctor presses the input unit 52 to perform a release or record operation, and an instruction signal to release or record is input from the input unit 52, the control unit 54 records the still image or video of the current video observation mode and the image or video generated by the image processing unit 51 with the image processing content of the display T13 in Figure 31 in the recording unit 53.

[0100] According to the modified example 6 of the embodiment described above, when a user such as a doctor presses the input unit 52 to perform a freeze operation, the image processing unit 51 can generate an image using a combination of pre-set image processing contents generated by the image processing unit 51, thereby improving the usability for users such as doctors.

[0101] (Modification 7) Next, Modification 7 of one embodiment of the present disclosure will be described. Figure 32 is a diagram showing the image processing content of the image that the display control unit 520 displays on the display device 4 according to Modification 7 of one embodiment of the present disclosure.

[0102] As shown in Table T14 of Figure 32, when a user such as a doctor operates the input unit 52 to select from multiple presets in which images are set to be displayed on multiple sub-screens, the display control unit 520 sequentially switches and displays the images generated by the image processing unit 51 on the display device 4 in the order of the sub-screens corresponding to the selected preset. After the examination is completed, the control unit 54 records the usage information in the recording unit 53, associating the preset selected by the user such as a doctor with the endoscope identification information of the endoscope device 2 connected to the control device 5. Subsequently, when another user selects a preset from among the multiple presets, the display control unit 520 displays the presets most frequently used by the endoscope device 2 connected to the control device 5 on the display device 4 based on the usage information recorded in the recording unit 53, so that they are displayed higher up in the selection. Alternatively, the user may operate the input unit 52 not before the start of the examination, but during the examination to display multiple presets. In this case, when the display control unit 520 receives an instruction signal from the input unit 52 to display a preset, it displays multiple presets (Table T14 in Figure 32) on the display device 4. Of course, the control unit 54 may sequentially switch between multiple presets in response to the user's operation of the input unit 52 and set them in the image processing unit 51.

[0103] According to the modified example 7 of the embodiment described above, the display order of frequently used sub-screens can be presented according to the endoscope device 2, and the content of image processing by the image processing unit 51 can be switched with simple operation, thereby improving the usability for users such as doctors.

[0104] (Other Embodiments) Various inventions can be formed by appropriately combining the multiple components disclosed in the medical system according to one embodiment of the present disclosure described above. For example, some components may be removed from all the components described in the medical system according to one embodiment of the present disclosure described above. Furthermore, the components described in the endoscopic system according to one embodiment of the present disclosure described above may be appropriately combined.

[0105] Furthermore, in the medical system according to one embodiment of this disclosure, the "parts" described above can be replaced with "means" or "circuits," etc. For example, the control unit can be replaced with control means or control circuit.

[0106] Furthermore, the program to be executed by the medical system according to one embodiment of this disclosure is provided as installable or executable file data recorded on a computer-readable recording medium such as a CD-ROM, flexible disk (FD), CD-R, DVD (Digital Versatile Disk), USB medium, or flash memory.

[0107] Furthermore, the program to be executed by the medical system according to one embodiment of this disclosure may be configured to be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network.

[0108] In this specification, while the flowchart descriptions use expressions such as "first," "then," and "followed by" to indicate the sequence of processes between steps, the order of processes necessary to carry out the present invention is not uniquely determined by these expressions. That is, the order of processes in the flowcharts described herein can be changed within a reasonable range. Furthermore, the program is not limited to simple branching processes; it may also branch by comprehensively evaluating a larger number of decision items.

[0109] Although some embodiments of this application have been described in detail above with reference to the drawings, these are illustrative examples, and the present invention can be implemented in various other forms with modifications and improvements based on the knowledge of those skilled in the art, starting with the embodiments described in the disclosure section of the present invention.

[0110] 1 Medical system 2 Endoscope device 3 Light source device 4 Display device 5 Control device 21 Insertion unit 22 Operation unit 31 Light source unit 51 Image processing unit 52 Input unit 53 Recording unit 54 Control unit 244 Imaging unit 312 First light source 313 Second light source 314 Third light source 315 Fourth light source 316 Fifth light source 511 Acquisition unit 512 Estimation unit 513 Extraction unit 514 Base component generation unit 515 Detail component generation unit 516 Determination unit 517 Correction unit 518 Synthesis unit 519 Image generation unit 520 Display control unit 531 Program recording unit

Claims

1. An image processing apparatus comprising a processor, wherein the processor acquires an input image captured under illumination light including blue-violet light, estimates a plurality of illumination light components from the input image by smoothing filter processing with different parameters, extracts a plurality of reflectance components based on the input image and the plurality of illumination light components, generates a base component based on the plurality of illumination light components, generates a detail component based on the plurality of reflectance components, and generates an output image based on the base component and the detail component.

2. An image processing apparatus according to claim 1, wherein the smoothing filter processing with different parameters comprises at least a first smoothing filter processing, a second smoothing filter processing, and a third smoothing filter processing, wherein the first smoothing filter processing is a bilateral filter processing with a first kernel size, the second smoothing filter processing is a bilateral filter processing with a second kernel size, and the third smoothing filter processing is a bilateral filter processing with a third kernel size.

3. An image processing apparatus according to claim 2, wherein the first kernel size is smaller than the second kernel size, and the second kernel size is smaller than the third kernel size.

4. An image processing apparatus according to claim 1, wherein the detail component is calculated by dividing the pixel value of each pixel of the input image by the corresponding base component.

5. An image processing apparatus according to claim 1, wherein the output image is generated by multiplying a base component and a detail component.

6. An image processing apparatus according to claim 3, further comprising an input interface, wherein the input interface receives input of attention information, which is information relating to an object of interest in an image.

7. An image processing apparatus according to claim 6, wherein the information of interest includes information relating to an object of interest and an object not of interest.

8. An image processing apparatus according to claim 7, wherein the object of interest or the object not of interest includes microvessels of the mucosal surface and microstructures of the mucosal surface.

9. An image processing apparatus according to claim 8, wherein the processor generates a base component by selecting one illumination light component from the plurality of illumination light components based on the attention information, and generates a detail component by selecting one reflectance component from the plurality of reflectance components.

10. An image processing apparatus according to claim 8, wherein the plurality of illumination light components are a first illumination light component, a second illumination light component, and a third illumination light component, the plurality of reflectance components are a first reflectance component, a second reflectance component, and a third reflectance component, and the processor, when the object of interest is a microvessel and a microstructure, generates the first illumination light component as the base component from the plurality of illumination light components, and generates the first reflectance component as the detail component from the plurality of reflectance components.

11. An image processing apparatus according to claim 8, wherein the plurality of reflectance components are a first illumination light component, a second illumination light component, and a third illumination light component, and the processor generates the third illumination light component as a base component from the plurality of illumination light components, and generates the third reflectance component as a detail component from the plurality of reflectance components when the object of interest is a microvessel.

12. An image processing apparatus according to claim 8, wherein the plurality of reflectance components are a first illumination light component, a second illumination light component, and a third illumination light component, and the processor generates the second illumination light component as a base component from the plurality of illumination light components, and generates the second reflectance component as a detail component from the plurality of reflectance components when the object of interest is a microstructure.

13. An image processing apparatus according to claim 8, wherein the plurality of reflectance components are a first illumination light component, a second illumination light component, and a third illumination light component, and the processor generates the third illumination light component as a base component from the plurality of illumination light components, and generates the second reflectance component as a detail component from the plurality of reflectance components, when the object of interest is a microstructure and the object of non-interest is a microvessel.

14. An image processing apparatus according to claim 8, wherein the plurality of reflectance components are a first illumination light component, a second illumination light component, and a third illumination light component, and the processor generates the second illumination light component as a base component from the plurality of illumination light components, and generates the third reflectance component as a detail component from the plurality of reflectance components when the object of interest is a microvessel and the object of non-interest is a microstructure.

15. An image processing apparatus according to claim 6, wherein the processor applies a correction process to the detail component based on the information of interest.

16. An image processing apparatus according to claim 15, wherein the processor, when the object of interest includes microvessels, applies a correction process to pixels in which the detail component has a value of less than 1, by multiplying the detail component of the pixel by a coefficient of 1 or less.

17. An image processing apparatus according to claim 15, wherein the information of interest includes information relating to an object of interest and an object not of interest, and the processor applies a correction process to pixels in which the detail component has a value of less than 1, by multiplying the detail component of the pixel by a coefficient of 1 or more, if the object not of interest includes microvessels.

18. An image processing apparatus according to claim 15, wherein the processor, when the object of interest includes a fine structure, applies a correction process to pixels in which the detail component takes a value greater than 1, by multiplying the detail component of the pixel by a coefficient of 1 or more.

19. An image processing apparatus according to claim 15, wherein the processor, when the object of interest includes a fine structure, applies a correction process to pixels whose detail component is greater than 1, by multiplying the detail component of the pixel by a coefficient of 1 or less.

20. An image processing apparatus according to claim 1, wherein the input image is an image of a subject illuminated by a first illumination light containing blue-violet light and a second illumination light containing green light.

21. An image processing method performed by an image processing apparatus comprising a processor, wherein the processor acquires an input image captured under illumination light including blue-violet light, estimates a plurality of illumination light components from the input image by smoothing filter processing with different parameters, extracts a plurality of reflectance components based on the input image and the plurality of illumination light components, generates a base component based on the plurality of illumination light components, generates a detail component based on the plurality of reflectance components, and generates an output image based on the base component and the detail component.

22. A program to be executed by an image processing apparatus equipped with a processor, which causes the processor to: acquire an input image captured under illumination light including blue-violet light; estimate a plurality of illumination light components from the input image by smoothing filter processing with different parameters; extract a plurality of reflectance components based on the input image and the plurality of illumination light components; generate a base component based on the plurality of illumination light components; generate a detail component based on the plurality of reflectance components; and generate an output image based on the base component and the detail component.

Images