Control device, endoscope system, and processing method

Abstract

A control device (10) comprises: a processor (100) capable of executing one or more enhancement processes; and a memory (20) that stores a trained model (22) trained on training images to classify images into classes corresponding to enhancement levels of the respective enhancement processes according to endoscope observation states. The processor (100) acquires an endoscopic image, causes the trained model (22) to classify the endoscopic image into a class, and performs an enhancement level change process for changing each of the enhancement levels of the one or more enhancement processes on the basis of the class into which the endoscopic image is classified.

Description

Control Device, Endoscope System, and Processing Method

[0001] The present invention relates to a control device, an endoscope system, a processing method, and the like.

[0002] In inspections using an endoscope system including a light source device, an imaging device, etc., by setting appropriate light source settings and imaging conditions according to the situation, improvements in the efficiency, accuracy, etc. of inspections are required. Patent Document 1 discloses a method of determining an imaging scene from an endoscope image and setting imaging conditions based on the determined scene.

[0003] International Publication No. 2018 / 159083

[0004] In recent years, since a large number of types of enhancement processing for improving the visibility of endoscope images and types of irradiation light used in light source devices have been proposed, the combinations of imaging conditions have become enormous. Therefore, it is difficult for users at the training stage to select an optimal combination of enhancement processing according to the situation. Patent Document 1 does not propose a specific method for setting the parameters of the enhancement processing from an endoscope image.

[0005] One aspect of the present disclosure relates to a control device including a processor capable of executing one or more enhancement processes and a memory storing a learned model learned to classify classes corresponding to the enhancement levels of each of the enhancement processes according to the endoscope observation state from learning images, wherein the processor acquires an endoscope image, uses the learned model to classify the classes from the endoscope image, and performs an enhancement level change process of changing the enhancement levels of one or more of the enhancement processes based on the classified classes.

[0006] Another aspect of the present disclosure relates to an endoscope system including the above-described control device and an endoscope that captures the endoscope image.

[0007] Another aspect of this disclosure relates to a processing method that uses a trained model capable of performing one or more enhancement processes, which is trained to classify classes from training images according to the enhancement level of each enhancement process according to the endoscopic observation state, wherein the method acquires an endoscopic image, uses the trained model to classify the classes from the endoscopic image, and performs an enhancement level modification process to change the enhancement level of one or more enhancement processes based on the classified classes.

[0008] A diagram illustrating an example configuration of the control device and endoscope system. A diagram illustrating an example dataset of input / output data for machine learning. A flowchart illustrating an example of processing in this embodiment. A flowchart illustrating an example of processing related to the determination of enhancement processing. A diagram illustrating the first table. Another diagram illustrating the first table. Another diagram illustrating the first table. Another diagram illustrating the first table. Another diagram illustrating the first table. A diagram illustrating another example configuration of the control device. A flowchart illustrating another example of processing related to the determination of enhancement processing. A diagram illustrating the second table. Another diagram illustrating the second table. Another diagram illustrating the third table. A diagram illustrating the fourth table. Another diagram illustrating the fourth table. Another diagram illustrating the fourth table. Another diagram illustrating the fourth table. Another diagram illustrating the fifth table. A flowchart illustrating another example of processing related to the determination of enhancement processing. A flowchart illustrating an example of processing for determining the endoscopic observation state.

[0009] The following describes this embodiment. Note that the embodiment described below does not unduly limit the scope of the present invention as described in the claims. Furthermore, not all of the configurations described in this embodiment are necessarily essential components of the present invention.

[0010] Figure 1 is a block diagram illustrating the endoscope system 1 of this embodiment. The endoscope system 1 of this embodiment includes an endoscope 3 and a control device 10. The control device 10 includes at least a processor 100.

[0011] The endoscope 3 is specifically a flexible endoscope, and it is not particularly important whether it is operated manually by a user or by a robot. The various components of the endoscope 3, such as the tip, bending section, insertion section, operating section, and universal cord, can be broadly adapted from those known in the field of flexible endoscopes, and therefore, specific illustrations are omitted. Hereafter, the side of the endoscope 3 that is inserted into the patient's lumen will be referred to as the "tip side," and the side of the endoscope 3 that is attached to the control device 10, etc., will be referred to as the "proximal end side." Furthermore, the following description shows an example of applying the method of this embodiment to upper endoscopy, lower endoscopy, etc. (hereinafter, these collectively referred to simply as "endoscopic examination") using the endoscope 3 as a flexible endoscope, but this does not preclude applying some or all of the method of this embodiment to surgical procedures using a flexible or rigid endoscope. For example, in endoscopic submucosal dissection, the enhancement processing described later is performed, making it useful to apply at least some of the method of this embodiment.

[0012] Endoscope 3 captures endoscopic images. For example, although not shown in the diagram, the tip of endoscope 3 includes an imager and an illumination lens. The imager includes an objective optical system and an image sensor. The image sensor is, for example, a CMOS sensor, CCD, etc. A light guide is attached to the base end of the illumination lens, and the light guide is inserted through the insertion section, operating section, universal cord, etc., and connected to a light source device (hereinafter simply referred to as "light source device") which is not shown. The light source device includes a predetermined light source, a rotating filter, a motor that rotates the rotating filter, and a motor driver that controls the motor. The predetermined light source is, for example, a xenon lamp. The rotating filter includes a plurality of predetermined color transmission filters through which light in different wavelength regions passes. More specifically, the predetermined color transmission filters include a first color transmission filter that passes light in the wavelength region of 400 nm to 500 nm, a second color transmission filter that passes light in the wavelength region of 500 nm to 600 nm, and a third color transmission filter that passes light in the wavelength region of 600 nm to 700 nm. Illumination light passing through the first color transmission filter is red (hereinafter sometimes simply referred to as "R"), illumination light passing through the second color transmission filter is green (hereinafter sometimes simply referred to as "G"), and illumination light passing through the third color transmission filter is blue (hereinafter sometimes simply referred to as "B").

[0013] The illumination light emitted from the light source device is separated in time series into the R, G, and B wavelength regions by the aforementioned rotating filter and input to the incident end of the light guide, passing through the illumination lens to illuminate the subject. The image of the subject illuminated by this illumination light is then converted into an image signal by the objective optical system, which forms an image on the photoelectric conversion surface of the image sensor. The converted image signal is input to the control device 10 via a cable inserted through the insertion section, etc. The image signal input to the control device 10 is processed by an amplifier, A / D converter, etc. (not shown) to create an endoscopic image. The imaged endoscopic image is displayed, for example, on a display device (not shown) connected to the control device 10. The light source device may include a first light-emitting element that emits light corresponding to the R wavelength region, a second light-emitting element that emits light corresponding to the G wavelength region, and a third light-emitting element that emits light corresponding to the B wavelength region. The first, second, and third light-emitting elements are, for example, LEDs. Furthermore, an endoscopic image may be generated by sequentially illuminating a subject with the first, second, and third light-emitting elements in a time series.

[0014] The processor 100 of this embodiment is composed of the following hardware. The hardware may include at least one of a circuit that processes digital signals and a circuit that processes analog signals. For example, the hardware may consist of one or more circuit devices or one or more circuit elements mounted on a circuit board. One or more circuit devices are, for example, ICs. One or more circuit elements are, for example, resistors, capacitors, etc.

[0015] Furthermore, the control device 10 of this embodiment, for example, as shown in Figure 1, may include a memory 20 and a processor 100 that operates based on the information stored in the memory 20. This allows the processor 100 to function as an enhancement processing decision unit 120 and an image processing unit 130. The information includes, for example, a program and various types of data. The program is, for example, a trained model 22, which will be described later. The processor 100 can be a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), etc. The memory 20 may be a semiconductor memory such as an SRAM (Static Random Access Memory) or a DRAM (Dynamic Random Access Memory), a register, a magnetic storage device such as a hard disk drive, or an optical storage device such as an optical disk drive. For example, the memory 20 stores instructions that can be read by a computer, and when these instructions are executed by the processor 100, the functions of the enhancement processing decision unit 120 and the image processing unit 130 are realized as processing. The instructions referred to here may be instructions from the instruction set that constitutes the program, or instructions that instruct the hardware circuit of the processor 100 to perform an action. The memory 20 is also called a storage device. For the sake of explanation, unless otherwise specified, the main entity performing the processing related to the method of this embodiment is the processor 100. Although Figure 1 shows a single processor 100, the control device 10 may include multiple processors 100, and the processing related to the method described below may be performed separately by each of them.

[0016] Furthermore, the above-mentioned program can be stored in a non-temporary information storage medium, such as a computer-readable medium. This information storage medium can be implemented as, for example, an optical disc, memory card, HDD, or semiconductor memory. Semiconductor memory can be, for example, ROM or non-volatile memory.

[0017] The processor 100 of this embodiment is capable of performing one or more enhancement processes. Enhancement processing in this embodiment refers to a method of making a specific object on an endoscopic image stand out or identify it using an image processing algorithm, and can be more narrowly called digital enhancement processing.

[0018] Specifically, the enhancement processing in this embodiment includes, for example, structural enhancement processing. Structural enhancement processing is image processing that enhances desired frequency band components in an endoscopic image, and a predetermined spatial filtering is performed on the endoscopic image. In other words, in this embodiment, the control device 10 includes a filtering circuit (not shown), and the processor 100 controls the filtering circuit and can process the image signal. As a result, for example, by performing structural enhancement processing on an endoscopic image showing mucosa, the fine structure on the surface of the mucosa can be grasped in more detail. Alternatively, for example, multiple spatial filters with different filter coefficients may be stored in the memory 20 or the like. As a result, the user can set the enhancement level in multiple stages by selecting a suitable spatial filter from a predetermined number of spatial filters, and the user can perform structural enhancement processing corresponding to the desired enhancement level. In the following, eight stages will be given as an example of multiple stages, but the number of stages is not limited to eight. Alternatively, for example, the frequency band components to be enhanced may be classified into multiple "enhancement modes," and the user may be able to select the desired enhancement mode. More specific illustrations and other details are disclosed in Japanese Patent Publication No. 08-313823, etc., and are therefore omitted here. However, for example, by performing spatial filtering using a spatial filter with different coefficients depending on the frequency band, structural enhancement processing according to the selected enhancement mode can be performed. Furthermore, in the following, two types of enhancement modes consisting of Mode A and Mode B will be given as examples of multiple enhancement modes, but there may be three or more types of enhancement modes. In addition, the frequency band that can be enhanced in Mode B is wider than the frequency band that can be enhanced in Mode A.

[0019] Furthermore, the enhancement processing in this embodiment may include, for example, color enhancement processing. More specifically, the color enhancement processing in this embodiment is also called hemoglobin index (hereinafter referred to as "IHb") color enhancement processing. IHb color enhancement processing is a process that makes the colors at positions where the IHb of the mucous membrane covering the surface of the subject is higher than the average value redder, and the colors at positions where the IHb of the mucous membrane is lower than the average value whiter. More specific illustrations and other details are disclosed in Japanese Patent Publication No. 06-335451, etc., so they are omitted here, but for example, in this embodiment, the control device 10 includes a predetermined image processing unit that performs color enhancement processing, and the processor 100 controls the image processing unit and can perform desired signal processing on the image signal. IHb is calculated by utilizing the difference between the absorption coefficient in the R wavelength region of hemoglobin and the absorption coefficient in the G wavelength region of hemoglobin, and calculating the ratio of the brightness levels of the R image signal and the G image signal. To highlight a desired location in red, for example, by multiplying the R value of the pixel relating to the desired location by a predetermined coefficient. This allows for greater emphasis on the color of areas where blood flow has increased in the endoscopic image. This enables the user to observe in more detail lesions where blood flow has increased due to increased cellular activity, for example. Alternatively, multiple predetermined coefficients with different values ​​may be stored in memory 20, etc. This allows the user to perform color enhancement processing corresponding to a desired enhancement level. While there are, for example, five types of coefficients, the number of types is not limited to five.

[0020] In the following, IHb color enhancement processing will be used as an example to explain color enhancement processing, but color enhancement processing is not limited to IHb color enhancement processing and may include various processes that enhance colors in an image. For example, color enhancement processing may be a process that enhances the saturation of an unspecified color component, or a process that enhances the saturation or brightness of a specific color component, or a process that changes the hue of a specific color component. Color enhancement processing may also be a process that enhances a specific color component by suppressing the saturation or brightness of color components other than a specific color component. Color enhancement processing may also be a process that enhances the hue difference or brightness difference between a specific first color component and a specific second color component. Furthermore, color enhancement processing may be an image process that acts only on a specific area in the image, or it may be an image process that acts on the entire image.

[0021] Furthermore, the enhancement processing in this embodiment is not limited to the above, and may also be, for example, structural color enhancement processing (Texture and Color Enhancement Imaging). Structural color enhancement processing is disclosed in, for example, International Publication No. 2018 / 037676 and International Publication No. 2019 / 130834. As described above, various methods have been proposed for the enhancement processing in this embodiment, and at least multiple levels of enhancement can be set, but it may also be possible to set multiple types of enhancement modes, as in structural enhancement processing. In addition, although it has been described that structural enhancement processing and color enhancement processing are performed on separate hardware, it may also be possible to perform both structural enhancement processing and color enhancement processing on a single image processing unit, for example.

[0022] Furthermore, as shown in Figure 1, the memory 20 of this embodiment stores a trained model 22. The trained model 22 is trained to classify classes corresponding to the enhancement level of each enhancement process according to the endoscopic observation state. In this embodiment, the endoscopic observation state refers to the observation state when performing an endoscopic examination, etc., and is conveniently classified from various viewpoints. For example, the endoscopic observation state can be classified into a first observation state or a second observation state depending on the tasks that the user mainly focuses on. For example, in a lower endoscopy, this includes the process of inserting the endoscope 3 so that the tip of the endoscope 3 reaches the cecum, and the process of withdrawing it from the cecum afterward. Since the user observes the lumen, etc. in detail during the process of withdrawing from the cecum, the endoscopic observation state related to this assumption is mainly observation and can be conveniently called the first observation state. On the other hand, the period until the tip of the endoscope 3 reaches the cecum is considered to be a process mainly of insertion. Being mainly insertion does not mean that no observation is performed at all, but rather that the user only roughly looks at the endoscopic image. However, even when insertion is the primary focus, if an unusual area is discovered in the endoscopic image, it may be necessary to observe it in detail. Therefore, the endoscopic observation state related to the process of primarily inserting the insertion part of the endoscope 3 can be conveniently referred to as the second observation state. Furthermore, the endoscopic observation states related to upper endoscopy can also be classified into first and second observation states. For example, in upper endoscopy, the endoscopic observation state related to the process of inserting the endoscope 3 so that its tip reaches the duodenum can be treated as the second observation state, and the endoscopic observation state related to the process of withdrawing the endoscope from the pylorus to the cardia along a predetermined path can be treated as the first observation state. Note that the processes related to the first and second observation states are not limited to those described above and can be determined by the user as appropriate. For example, in upper endoscopy, if a user is accustomed to the process of observing in detail along a predetermined path from the cardia to the pylorus, then the endoscopic observation state related to that process can be designated as the first observation state.

[0023] Furthermore, the endoscopic observation state may be treated as a combination of various states. More specifically, the endoscopic observation state may be classified by considering, for example, which part is being observed, which type of illumination light is being used, and how close the tip of the endoscope 3 is to the subject. Details will be described later in Figure 10 and subsequent figures.

[0024] In this embodiment, "learning" specifically refers to "machine learning," or more precisely, "deep learning," but hereafter it will be referred to as "machine learning" or simply "learning." At least a portion of the trained model 22 in this embodiment includes a neural network. Although not shown in the diagram, the neural network has an input layer into which data is input, an intermediate layer that performs calculations based on the output from the input layer, and an output layer that outputs data based on the output from the intermediate layer. Nodes in a given layer are connected to nodes in adjacent layers, and each connection is assigned a weighting coefficient. Each node multiplies the output of the preceding node by the weighting coefficient and calculates the sum of the multiplication results. Furthermore, each node adds a bias to the sum and applies an activation function to the sum result to obtain the output of that node. By sequentially executing this process from the input layer to the output layer, the output of the neural network can be obtained. Various functions such as the sigmoid function and the ReLU function are known as activation functions, and these can be widely applied in this embodiment.

[0025] The neural network in this embodiment is more specifically a CNN (Convolutional Neural Network), and the following explanation will use the CNN as an example, but this does not prevent the application of other models to the method of this embodiment. The CNN, although not shown in the diagram, includes a convolutional layer and a pooling layer that perform convolution operations. The convolutional layer is a layer that performs filtering. The pooling layer is a layer that performs pooling operations to reduce the size in the vertical and horizontal directions. More specifically, the CNN obtains an output by performing operations by a fully connected layer after performing operations by the convolutional layer and the pooling layer multiple times. A fully connected layer is a layer that performs operations when all nodes of the previous layer are mapped to the nodes of a given layer.

[0026] The machine learning in this embodiment is supervised learning. The training data in supervised learning is a dataset that associates input data with correct labels. In the learning stage of this embodiment, the relationship between input data and output is as shown in Figure 2. Although detailed illustrations are omitted, the output layer of the neural network has N nodes. The first node is information representing the likelihood that the class corresponding to the input endoscopic image belongs to class 1. The second to the Nth nodes are similar, with each node representing information representing the likelihood that the input endoscopic image belongs to class 2 to class N. For example, if the output layer is a known softmax layer, the N outputs are a set of probability data that sum to 1. In other words, probability data relating to the likelihood of the class related to each node is output from each node.

[0027] For example, when a user performs an endoscopic examination using endoscope 3, endoscopic images are acquired as training images. For instance, if one endoscopic examination takes 15 minutes, 20 endoscopic images can be acquired per minute, and the number of endoscopic examinations performed is 1000, then 300,000 endoscopic images can be acquired as training images. Note that the training images do not need to be endoscopic images from 1000 examinations performed by a single user. In other words, there can be multiple users, and endoscopic images can be acquired from multiple examination facilities.

[0028] The user then repeatedly examines the acquired endoscopic images, classifies them, and assigns the corresponding class as the correct label to the endoscopic image, for each endoscopic image. During the learning phase, a learning device (not shown) performs machine learning based on a dataset consisting of training images and the classes assigned to those training images, thereby generating or updating the trained model 22.

[0029] It is desirable that the training images be endoscopic images that have not undergone the enhancement processing described above. However, for example, an image obtained by applying a transformation process that produces the opposite effect of the transformation process related to enhancement processing to an endoscopic image that has undergone enhancement processing may be used as a training image. More specifically, for example, if the enhancement processing is structural enhancement processing, an endoscopic image to which structural enhancement processing based on a predetermined enhancement mode and enhancement level has been applied can be further subjected to spatial filtering processing based on frequency characteristics that are the opposite of the structural enhancement processing, thereby obtaining an endoscopic image that has not undergone structural enhancement processing in effect.

[0030] Furthermore, when applying a method using the generated or updated trained model 22 to the control device 10, the control device 10 may be configured as shown in Figure 1. Specifically, for example, the control device 10 includes a memory 20 for storing the trained model 22, an input unit 40, an output unit 50, and a processor 100. The input unit 40 is an image data interface for receiving endoscopic images. The output unit 50 is a data interface for outputting data related to the classified classes. In the inference stage, the processor 100 reads the trained model 22 from the memory 20. The processor 100 then inputs the endoscopic images to the trained model 22 via the input unit 40 and outputs the classes classified by inference via the output unit 50. In Figure 1, the destination of the classes as output data is shown to be outside the control device 10, but it may also be inside the control device 10 and can be determined as appropriate. The inference may be performed during the execution of a procedure using the endoscopic system 1, which includes the endoscope 3 and the control device 10.

[0031] Although Figure 1 shows one trained model 22 stored in memory 20, multiple trained models 22 may be stored in memory 20, and details will be described later.

[0032] Figure 3 is a flowchart illustrating an example of processing according to the method of this embodiment. The processor 100 determines whether or not an endoscopic image has been acquired (step S10). If the processor 100 determines that an endoscopic image has been acquired (YES in step S10), it classifies the image (step S20). On the other hand, if the processor 100 determines that an endoscopic image has not been acquired (NO in step S10), it repeats step S10. In other words, steps S20 onward in Figure 3 are performed at intervals in which the processor 100 acquires an endoscopic image.

[0033] In step S20, the processor 100 reads the trained model 22 from the memory 20, performs inference processing using the endoscopic image as input data, and performs class classification. Then the processor 100 decides on the enhancement process (step S100). Details of step S100 will be described later. After that, the processor 100 outputs the acquired endoscopic image and an image based on the determined enhancement process (step S200).

[0034] Subsequently, the processor 100 performs a process (step S300) to determine whether the procedure is complete or not. If it determines that the procedure is not complete (NO in step S300), it repeats step S10. On the other hand, if the processor 100 determines that the procedure is complete (YES in step S300), it terminates the flow shown in Figure 3. In other words, the flow shown in Figure 3 continues until the procedure is complete.

[0035] Figure 4 is a flowchart illustrating step S100 in more detail. The processor 100 determines the enhancement level of the enhancement process based on the classified class (step S110). The processor 100 then performs an enhancement level change process (step S190). In other words, the processor 100 functions as an image processing unit 130 and performs image processing on the endoscopic image acquired in step S10 based on the content determined in step S110.

[0036] A specific example of step S110 in Figure 4 will be explained in more detail. The processor 100 determines the emphasis level of the emphasis processing using, for example, the tables shown in Figures 5, 6, 7, 8, and 9 (hereinafter referred to as the first tables for convenience). In the first tables in Figures 5 to 9, the classes classified in step S20 are associated with the emphasis mode of the first emphasis processing, the emphasis level of the first emphasis processing, and the emphasis level of the second emphasis processing.

[0037] In the first table of Figures 5 to 9, the first enhancement process does not have any specific enhancement process as long as it has multiple enhancement modes, and the second enhancement process does not have any specific enhancement process as long as it has a single enhancement mode. However, hereafter, the first enhancement process will be exemplified as a structural enhancement process, and the second enhancement process will be exemplified as a color tone enhancement process. In this embodiment, the first enhancement process (i.e., structural enhancement process) has two types of enhancement modes, Mode A and Mode B, and the enhancement level of the first enhancement process can be changed in eight steps. In addition, the enhancement level of the second enhancement process (i.e., color tone enhancement process) can be changed in five steps.

[0038] Furthermore, in the control device 10 of this embodiment, the processor 100 can automatically control the on / off state of the first enhancement processing. More specifically, for example, by controlling the on / off state of the filtering function, the processor 100 can automatically switch between a state in which images are created based on image signals processed by the aforementioned filtering circuit and a state in which images are created based on image signals that were not input to the filtering circuit, thereby realizing the on / off state of the first enhancement processing. Similarly, in the control device 10 of this embodiment, the processor 100 can automatically control the on / off state of the second enhancement processing. More specifically, for example, by controlling the on / off state of the image processing function of the image processing unit, the processor 100 can automatically switch between a state in which images are created based on image signals processed by the aforementioned image processing unit and a state in which images are created based on image signals that were not input to the image processing unit, thereby realizing the on / off state of the second enhancement processing. In other words, if it is determined in step S110 of Figure 4 that the first enhancement processing is off and the second enhancement processing is off, the processor 100 modifies the enhancement processing in step S190 so that no enhancement processing is performed on the endoscopic image.

[0039] Based on the above, there are 102 possible combinations as shown in the first table, depending on the emphasis mode of the first emphasis processing, the emphasis level of the first emphasis processing, the emphasis level of the second emphasis processing, the on / off status of the first emphasis processing, and the on / off status of the second emphasis processing. Specifically, for example, in Figure 5, classes 0, 1, 2, 3, 4, and 5 are classified as classes in which the first emphasis processing is changed to off. Furthermore, class 0 is a class in which the second emphasis processing is also changed to off, and class 1 is a class in which the emphasis level of the second emphasis processing is changed to level 1. Similarly, class 2 is a class in which the emphasis level of the second emphasis processing is changed to level 2, class 3 is a class in which the emphasis level of the second emphasis processing is changed to level 3, class 4 is a class in which the emphasis level of the second emphasis processing is changed to level 4, and class 5 is a class in which the emphasis level of the second emphasis processing is changed to level 5.

[0040] Furthermore, in Figure 5, classes 6, 7, 8, 9, 10, and 11 are classes in which the emphasis mode of the first emphasis processing is changed to mode A and the emphasis level of the first emphasis processing is changed to level 1. In addition, class 6 is a class in which the second emphasis processing is changed to off, and class 7 is a class in which the emphasis level of the second emphasis processing is changed to level 1. Similarly, class 8 is a class in which the emphasis level of the second emphasis processing is changed to level 2, class 9 is a class in which the emphasis level of the second emphasis processing is changed to level 3, class 10 is a class in which the emphasis level of the second emphasis processing is changed to level 4, and class 11 is a class in which the emphasis level of the second emphasis processing is changed to level 5.

[0041] Furthermore, in Figure 5, classes 12, 13, 14, 15, 16, and 17 are classes in which the emphasis mode of the first emphasis processing is changed to mode A and the emphasis level of the first emphasis processing is changed to level 2. In addition, class 12 is a class in which the second emphasis processing is changed to off, and class 13 is a class in which the emphasis level of the second emphasis processing is changed to level 1. Similarly, class 14 is a class in which the emphasis level of the second emphasis processing is changed to level 2, class 15 is a class in which the emphasis level of the second emphasis processing is changed to level 3, class 16 is a class in which the emphasis level of the second emphasis processing is changed to level 4, and class 17 is a class in which the emphasis level of the second emphasis processing is changed to level 5.

[0042] Furthermore, in Figure 5, classes 18, 19, 20, 21, 22, and 23 are classes in which the emphasis mode of the first emphasis processing is changed to mode A and the emphasis level of the first emphasis processing is changed to level 3. In addition, class 18 is a class in which the second emphasis processing is changed to off, and class 19 is a class in which the emphasis level of the second emphasis processing is changed to level 1. Similarly, class 20 is a class in which the emphasis level of the second emphasis processing is changed to level 2, class 21 is a class in which the emphasis level of the second emphasis processing is changed to level 3, class 22 is a class in which the emphasis level of the second emphasis processing is changed to level 4, and class 23 is a class in which the emphasis level of the second emphasis processing is changed to level 5.

[0043] Furthermore, in Figure 6, classes 24, 25, 26, 27, 28, and 29 are classes in which the emphasis mode of the first emphasis processing is changed to mode A and the emphasis level of the first emphasis processing is changed to level 4. In addition, class 24 is a class in which the second emphasis processing is changed to off, and class 25 is a class in which the emphasis level of the second emphasis processing is changed to level 1. Similarly, class 26 is a class in which the emphasis level of the second emphasis processing is changed to level 2, class 27 is a class in which the emphasis level of the second emphasis processing is changed to level 3, class 28 is a class in which the emphasis level of the second emphasis processing is changed to level 4, and class 29 is a class in which the emphasis level of the second emphasis processing is changed to level 5.

[0044] Also, in FIG. 6, classes 30, 31, 32, 33, 34, and 35 are classes in which the enhancement mode of the first enhancement process is changed to mode A and the enhancement level of the first enhancement process is changed to level 5. Further, class 30 is a class in which the second enhancement process is changed to off, and class 31 is a class in which the enhancement level of the second enhancement process is changed to level 1. Similarly, class 32 is a class in which the enhancement level of the second enhancement process is changed to level 2, class 33 is a class in which the enhancement level of the second enhancement process is changed to level 3, class 34 is a class in which the enhancement level of the second enhancement process is changed to level 4, and class 35 is a class in which the enhancement level of the second enhancement process is changed to level 5.

[0045] Also, in FIG. 6, classes 36, 37, 38, 39, 40, and 41 are classes in which the enhancement mode of the first enhancement process is changed to mode A and the enhancement level of the first enhancement process is changed to level 6. Further, class 36 is a class in which the second enhancement process is changed to off, and class 37 is a class in which the enhancement level of the second enhancement process is changed to level 1. Similarly, class 38 is a class in which the enhancement level of the second enhancement process is changed to level 2, class 39 is a class in which the enhancement level of the second enhancement process is changed to level 3, class 40 is a class in which the enhancement level of the second enhancement process is changed to level 4, and class 41 is a class in which the enhancement level of the second enhancement process is changed to level 5.

[0046] Furthermore, in Figure 6, classes 42, 43, 44, 45, 46, and 47 are classes in which the emphasis mode of the first emphasis processing is changed to mode A and the emphasis level of the first emphasis processing is changed to level 7. In addition, class 42 is a class in which the second emphasis processing is changed to off, and class 43 is a class in which the emphasis level of the second emphasis processing is changed to level 1. Similarly, class 44 is a class in which the emphasis level of the second emphasis processing is changed to level 2, class 45 is a class in which the emphasis level of the second emphasis processing is changed to level 3, class 46 is a class in which the emphasis level of the second emphasis processing is changed to level 4, and class 47 is a class in which the emphasis level of the second emphasis processing is changed to level 5.

[0047] Furthermore, in Figure 7, classes 48, 49, 50, 51, 52, and 53 are classes in which the emphasis mode of the first emphasis processing is changed to mode A and the emphasis level of the first emphasis processing is changed to level 8. In addition, class 48 is a class in which the second emphasis processing is changed to off, and class 49 is a class in which the emphasis level of the second emphasis processing is changed to level 1. Similarly, class 50 is a class in which the emphasis level of the second emphasis processing is changed to level 2, class 51 is a class in which the emphasis level of the second emphasis processing is changed to level 3, class 52 is a class in which the emphasis level of the second emphasis processing is changed to level 4, and class 53 is a class in which the emphasis level of the second emphasis processing is changed to level 5.

[0048] Also, in FIG. 7, classes 54, 55, 56, 57, 58, and 59 are classes in which the enhancement mode of the first enhancement process is changed to mode B and the enhancement level of the first enhancement process is changed to level 1. Further, class 54 is a class in which the second enhancement process is changed to off, and class 55 is a class in which the enhancement level of the second enhancement process is changed to level 1. Similarly, class 56 is a class in which the enhancement level of the second enhancement process is changed to level 2, class 57 is a class in which the enhancement level of the second enhancement process is changed to level 3, class 58 is a class in which the enhancement level of the second enhancement process is changed to level 4, and class 59 is a class in which the enhancement level of the second enhancement process is changed to level 5.

[0049] Also, in FIG. 7, classes 60, 61, 62, 63, 64, and 65 are classes in which the enhancement mode of the first enhancement process is changed to mode B and the enhancement level of the first enhancement process is changed to level 2. Further, class 12 is a class in which the second enhancement process is changed to off, and class 60 is a class in which the enhancement level of the second enhancement process is changed to level 1. Similarly, class 61 is a class in which the enhancement level of the second enhancement process is changed to level 2, class 62 is a class in which the enhancement level of the second enhancement process is changed to level 3, class 63 is a class in which the enhancement level of the second enhancement process is changed to level 4, and class 64 is a class in which the enhancement level of the second enhancement process is changed to level 5.

[0050] Furthermore, in Figure 7, classes 66, 67, 68, 69, 70, and 71 are classes in which the emphasis mode of the first emphasis processing is changed to mode B and the emphasis level of the first emphasis processing is changed to level 3. In addition, class 66 is a class in which the second emphasis processing is changed to off, and class 67 is a class in which the emphasis level of the second emphasis processing is changed to level 1. Similarly, class 68 is a class in which the emphasis level of the second emphasis processing is changed to level 2, class 69 is a class in which the emphasis level of the second emphasis processing is changed to level 3, class 70 is a class in which the emphasis level of the second emphasis processing is changed to level 4, and class 71 is a class in which the emphasis level of the second emphasis processing is changed to level 5.

[0051] Furthermore, in Figure 8, classes 72, 73, 74, 75, 76, and 77 are classes in which the emphasis mode of the first emphasis processing is changed to mode B and the emphasis level of the first emphasis processing is changed to level 4. In addition, class 72 is a class in which the second emphasis processing is changed to off, and class 73 is a class in which the emphasis level of the second emphasis processing is changed to level 1. Similarly, class 74 is a class in which the emphasis level of the second emphasis processing is changed to level 2, class 75 is a class in which the emphasis level of the second emphasis processing is changed to level 3, class 76 is a class in which the emphasis level of the second emphasis processing is changed to level 4, and class 77 is a class in which the emphasis level of the second emphasis processing is changed to level 5.

[0052] Furthermore, in Figure 8, classes 78, 79, 80, 81, 82, and 83 are classes in which the emphasis mode of the first emphasis processing is changed to mode B and the emphasis level of the first emphasis processing is changed to level 5. In addition, class 78 is a class in which the second emphasis processing is changed to off, and class 79 is a class in which the emphasis level of the second emphasis processing is changed to level 1. Similarly, class 80 is a class in which the emphasis level of the second emphasis processing is changed to level 2, class 81 is a class in which the emphasis level of the second emphasis processing is changed to level 3, class 82 is a class in which the emphasis level of the second emphasis processing is changed to level 4, and class 83 is a class in which the emphasis level of the second emphasis processing is changed to level 5.

[0053] Furthermore, in Figure 8, classes 84, 85, 86, 87, 88, and 89 are classes in which the emphasis mode of the first emphasis processing is changed to mode B and the emphasis level of the first emphasis processing is changed to level 6. In addition, class 84 is a class in which the second emphasis processing is changed to off, and class 85 is a class in which the emphasis level of the second emphasis processing is changed to level 1. Similarly, class 86 is a class in which the emphasis level of the second emphasis processing is changed to level 2, class 87 is a class in which the emphasis level of the second emphasis processing is changed to level 3, class 88 is a class in which the emphasis level of the second emphasis processing is changed to level 4, and class 89 is a class in which the emphasis level of the second emphasis processing is changed to level 5.

[0054] Furthermore, in Figure 8, classes 90, 91, 92, 93, 94, and 95 are classes in which the emphasis mode of the first emphasis processing is changed to mode B and the emphasis level of the first emphasis processing is changed to level 7. In addition, class 90 is a class in which the second emphasis processing is changed to off, and class 91 is a class in which the emphasis level of the second emphasis processing is changed to level 1. Similarly, class 92 is a class in which the emphasis level of the second emphasis processing is changed to level 2, class 93 is a class in which the emphasis level of the second emphasis processing is changed to level 3, class 94 is a class in which the emphasis level of the second emphasis processing is changed to level 4, and class 95 is a class in which the emphasis level of the second emphasis processing is changed to level 5.

[0055] Furthermore, in Figure 9, classes 96, 97, 98, 99, 100, and 101 are classes in which the emphasis mode of the first emphasis processing is changed to mode B and the emphasis level of the first emphasis processing is changed to level 8. In addition, class 96 is a class in which the second emphasis processing is changed to off, and class 97 is a class in which the emphasis level of the second emphasis processing is changed to level 1. Similarly, class 98 is a class in which the emphasis level of the second emphasis processing is changed to level 2, class 99 is a class in which the emphasis level of the second emphasis processing is changed to level 3, class 100 is a class in which the emphasis level of the second emphasis processing is changed to level 4, and class 101 is a class in which the emphasis level of the second emphasis processing is changed to level 5.

[0056] Thus, the classes in the first table are based on the enhancement level and enhancement mode themselves in the enhancement processing. Therefore, the processor 100 completes the processing in step S100 by performing step S190 based on the enhancement level and enhancement mode in the enhancement processing determined in step S110. When performing step S100 using such a first table, the user performs the task of classifying the endoscopic images, which are the learning images, while taking into account the endoscopic observation state. In other words, the user looks at the endoscopic image and, assuming that it is in the first or second observation state, determines which class in the first table it belongs to and classifies it. To put it another way, the classes in the first table correspond to the enhancement levels of each enhancement processing according to the endoscopic observation state.

[0057] For example, suppose a user performs a lower endoscopy and inserts endoscope 3 into the body. Regardless of what is shown in the endoscopic image, they define the period until the tip of endoscope 3 reaches the cecum as the second observation state and do not use enhancement processing. They define the period after the tip of endoscope 3 reaches the cecum as the first observation state and use enhancement processing. In this case, the user looks at the endoscopic images from past lower endoscopies and assigns class 0 to all endoscopic images up to the point before reaching the cecum, because they represent the second observation state. In this case, class 0 can be said to correspond to the enhancement level (off) of the first enhancement processing according to the second observation state and the enhancement level (off) of the second enhancement processing according to the second observation state. In other words, classes 1 to 101 can be said to correspond to the enhancement level of the first enhancement processing according to the first observation state and the enhancement level of the second enhancement processing according to the first observation state.

[0058] Furthermore, let's assume that there is a user who, as a default setting when the tip of the endoscope 3 reaches the cecum, sets the enhancement mode of the first enhancement processing to mode A, the enhancement level of the first enhancement processing to level 3, and the enhancement level of the second enhancement processing to level 2. In this case, the user assigns class 20 to the endoscopic image when the tip of the endoscope 3 reaches the cecum. Let's also assume that there is a user who, upon discovering an unusual location, first brings the tip of the endoscope 3 closer to that location, and after bringing the tip of the endoscope 3 closer to that location, sets the enhancement mode of the first enhancement processing to mode B and the enhancement level of the first enhancement processing to level 5. In this case, the user assigns class 80 to the endoscopic image when the tip of the endoscope 3 is close to that location.

[0059] By using the trained model 22 thus learned, during lower endoscopy, the endoscopic image acquired in step S10 until the tip of the endoscope 3 reaches the cecum is classified as class 0 in step S20, and in step S100, it is determined that both the first and second enhancement processing are turned off. In step S190, the processor 100 controls the image signal related to the endoscopic image so that it is not subjected to signal processing by the spatial filtering circuit and image processing unit, and visualizes the image signal as an endoscopic image. As a result, in step S200, the endoscopic image without enhancement processing is displayed on a display device or the like.

[0060] Subsequently, when the tip of the endoscope 3 reaches the cecum, the endoscopic image acquired in step S10 is classified into class 20 in step S20, and in step S100, it is decided to change the enhancement process so that the first enhancement process is performed with enhancement mode A and enhancement level 3, and the second enhancement process is performed with enhancement level 2. Then, in step S190, the processor 100 controls the spatial filtering circuit and the image processing unit so that the above-described first enhancement process and second enhancement process are performed on the image signal relating to the acquired endoscopic image. As a result, in step S200, the endoscopic image that has undergone the first enhancement process with enhancement mode A and enhancement level 3, and the second enhancement process with enhancement level 2, is automatically displayed on the display device or the like. Subsequently, when an unusual location is discovered and the tip of the endoscope 3 is brought close to that location, the endoscope image is classified into class 80 in step S20, and in step S100, it is decided to change the enhancement processing so that the first enhancement processing is set to enhancement mode B and enhancement level 5, and the second enhancement processing is set to enhancement level 3. Then, in step S190, the processor 100 controls the spatial filtering circuit and the image processing unit so that the above-described first enhancement processing and second enhancement processing are performed on the image signal relating to the acquired endoscope image. As a result, in step S200, the endoscope image in which the first enhancement processing (enhancement mode B and enhancement level 5) has been performed and the second enhancement processing (enhancement level 3) has been performed is automatically displayed on the display device or the like.

[0061] Furthermore, the first table shown in Figures 5 to 9 may associate a class with, for example, a first enhancement process having a single enhancement mode and multiple enhancement levels, and a second enhancement process having multiple enhancement levels. In this case, the classes in the first table are based on the respective enhancement levels themselves, and step S190 in Figure 4 can be considered to be performed based on the enhancement level determined in step S110. Also, it is not essential that the first table associates two or more enhancement processes with classes; either the first enhancement process or the second enhancement process may be associated with a class.

[0062] Furthermore, when performing step S100 using the first table, multiple trained models 22 may be stored in memory 20. For example, separate trained models 22 may be prepared for upper endoscopy and lower endoscopy.

[0063] For example, suppose a user performs an upper endoscopy and inserts endoscope 3 into the body. Regardless of what is shown in the endoscopic image, the period until the tip of endoscope 3 reaches the duodenum is considered the second observation state, and default enhancement processing is used. The period after the tip of endoscope 3 is withdrawn from the pylorus is considered the first observation state. In this case, the default enhancement processing is set to Mode A for the first enhancement processing (structural enhancement processing), Level 3 for the first enhancement processing, and Level 2 for the second enhancement processing. In this case, the user looks at endoscopic images from past upper endoscopies and assigns Class 20 to any endoscopic image up to before reaching the duodenum, because it represents the second observation state. In this case, Class 20 can be said to correspond to the enhancement level of the first enhancement processing according to the second observation state and the enhancement level of the second enhancement processing according to the second observation state. In other words, Classes 0 to 19 and Classes 21 to 101 can be said to correspond to the enhancement level of the first enhancement processing according to the first observation state and the enhancement level of the second enhancement processing according to the first observation state.

[0064] Furthermore, for example, in the first observation state of an upper endoscopy, endoscopic images lacking sharpness may be classified into class 27 by step S20. This results in step S110 determining that the enhancement mode of the first enhancement processing is mode A, the enhancement level of the first enhancement processing is level 4, and the enhancement level of the second enhancement processing is level 3. Also, for example, in the first observation state of an upper endoscopy, endoscopic images that have detected a finding and are attempting to approach that finding may be classified into one of classes 78 to 83 by step S20. This results in step S110 determining that the enhancement mode of the first enhancement processing is mode B and the enhancement level of the first enhancement processing is level 5. Furthermore, for example, in the first observation state of an upper endoscopy, endoscopic images that are attempting to focus on the mucosal surface may be classified into one of classes 30 to 35 by step S20. This results in step S110 determining that the enhancement mode of the first enhancement processing is mode A and the enhancement level of the first enhancement processing is level 5. Furthermore, for example, in the first observation state of an upper endoscopy, when observing the mucosa of the pharynx or esophagus at close range, the acquired endoscopic image may be classified into class 88 by step S20. As a result, by step S110, it is determined that the enhancement mode of the first enhancement process is mode B, the enhancement level of the first enhancement process is level 6, and the enhancement level of the second enhancement process is level 4.

[0065] Furthermore, the first observation state may include the state related to the process of discovering and diagnosing other lesions after completing a series of observations in an endoscopic examination. For example, if a predetermined lesion that has changed in color is discovered, the endoscopic image surrounding the predetermined lesion may be classified as class 35 in step S20. As a result, in step S110, the enhancement mode of the first enhancement processing is determined to be mode A, the enhancement level of the first enhancement processing is level 3, and the enhancement level of the second enhancement processing is determined to be level 5. This allows the enhancement level of the second enhancement processing to be increased, making it easier to discover similar lesions.

[0066] Furthermore, if the control device 10 includes a light source device capable of emitting multiple types of irradiation light, a separate trained model 22 may be prepared for each observation method based on the irradiation light used in the same endoscopic examination. Multiple types of irradiation light include, for example, white light, but may also include special light. White light is also called normal light and refers to light that has intensity in a wide wavelength band, as exemplified by xenon lamps, and observation methods using white light are also called WLI (White Light Imaging). Special light refers to light that has intensity in a narrower wavelength band compared to white light. An example of an observation method using special light is NBI (registered trademark). NBI is an abbreviation for Narrow Band Imaging. NBI uses narrowband light included in the blue wavelength band and narrowband light included in the green wavelength band. This makes it possible to observe blood vessels and glandular surface structures located relatively shallowly in the mucosa more clearly. Details of the NBI method are disclosed in, for example, Japanese Patent Publication No. 2002-095635, so further explanation is omitted in this specification. Furthermore, observation techniques using special light may include, for example, RDI (Red Dichromatic Imaging). RDI uses narrowband light contained in the amber wavelength band, narrowband light contained in the green wavelength band, and narrowband light contained in the red wavelength band. This makes it possible to observe blood vessels located relatively deep within the mucosa more clearly. Details of the RDI technique are disclosed in publications such as International Publication No. 2012 / 081297, so further explanation is omitted in this specification.

[0067] Furthermore, observation techniques using special light are not limited to those described above, and may also include TXI (Texture and Color Enhancement Imaging), IRI (Infra Red Imaging), AFI (Auto Fluorescence Imaging), etc. Details of the TXI technique are disclosed, for example, in International Publication 2020 / 075227. IRI is a technique that involves intravenously injecting an infrared indicator drug (e.g., indocyanine green) that readily absorbs infrared light, and then sequentially irradiating the subject with infrared light based on the 790-820 nm wavelength band and infrared light based on the 905-970 nm wavelength band to observe blood vessels located relatively deep within the mucosa more clearly. AFI is a technique that involves irradiating the subject with excitation light to observe autofluorescence from biological tissue and light of a wavelength absorbed by hemoglobin in the blood, thereby highlighting neoplastic lesions and normal mucosa with different color tones.

[0068] Furthermore, although flowcharts and other diagrams are omitted, the processor 100 may perform a process to change the trained model 22 to be read when these illumination lights are switched. However, it is not necessary to always use a different trained model 22 for each observation method based on the illumination light used. For example, when the illumination light is switched to narrowband light corresponding to NBI, the acquired endoscopic image may be classified in step S20 into one of the following classes: class 0, class 6, class 12, class 18, class 24, class 30, class 36, class 42, class 48, class 54, class 60, class 66, class 72, class 78, class 84, class 90, or class 96. As a result, step S110 further determines that the second enhancement processing is turned off. This is because the second enhancement processing is an enhancement processing that is advantageous for observation with white light.

[0069] Furthermore, separate trained models 22 may be prepared for cases where a predetermined drug is sprayed on the subject and cases where the predetermined drug is not sprayed on the subject. In this case, the processor 100 may perform a process to change the trained model 22 to be read when the predetermined drug is sprayed on the subject. The predetermined drug is, for example, a dye, but it may also be a staining agent. The dye here refers to a drug that is not absorbed by the biological mucosa and remains stored in depressions on the surface, for example, indigo carmine. The staining agent here refers to a drug that is absorbed by biological tissue, for example, crystal violet, toluidine blue, Lugol's solution, etc. For example, in lower endoscopy, if it is desired to highlight and observe the fine irregularities of the subject, indigo carmine is sprayed as a dye. Also, when determining whether a lesion in the subject is benign or malignant, crystal violet is sprayed as a staining agent to stain the cell nuclei related to the lesion. Toluidine blue is used, for example, in upper endoscopy, when a defect is found in the esophageal epithelium, to stain the necrotic material blue so that the area can be highlighted. However, it is not necessary to always use different trained models 22 depending on whether or not a specific drug is sprayed. For example, if an endoscopic image is obtained in which indigo carmine has been sprayed on the stomach during an upper endoscopy, the endoscopic image may be classified into one of classes 48 to 53. As a result, in step S110, it is determined that the enhancement mode of the first enhancement process is mode A and the enhancement level of the first enhancement process is level 8.

[0070] Alternatively, a number of pre-trained models 22 and tables corresponding to the first table may be prepared for each possible combination of enhancement processing, and stored in memory 20. When performing an endoscopic examination, the user can then select a suitable pre-trained model 22 according to the specifications of the endoscopic system 1 used for the examination. For example, if a new enhancement processing method is proposed in the future, a new table combining the new enhancement processing method with the existing enhancement processing methods, and a new pre-trained model 22 corresponding to the output data in the classes of the new table, can be created and stored in memory 20.

[0071] Furthermore, different trained models 22 may be stored in memory 20 depending on the purpose of the endoscopic examination. For example, different trained models 22 may be stored in memory 20 for the purpose of screening examinations and for follow-up examinations after lesion resection. Follow-up examinations are intended, for example, to detect residual or recurrent lesions, or metachronous multiple lesions, after lesion resection by previous surgery.

[0072] As described above, the control device 10 of this embodiment includes a processor 100 capable of executing one or more enhancement processes, and a memory 20 that stores a trained model 22 that has been trained to classify classes corresponding to the enhancement level of each enhancement process according to the endoscopic observation state from training images. The processor 100 acquires endoscopic images, uses the trained model 22 to classify classes from the endoscopic images, and performs enhancement level modification processing to change the enhancement level of one or more enhancement processes based on the classified classes.

[0073] Thus, the control device 10 of this embodiment includes a memory 20 for storing the trained model 22 described above and a processor 100, enabling inference processing using the trained model 22. Furthermore, the processor 100 can perform the enhancement level change processing described above using the trained model 22, so that the enhancement level of one or more enhancement processes is automatically changed, and an endoscope system 1 can be constructed that can automatically acquire endoscope images with the changed enhancement processing. In addition, since the trained model 22 is trained as described above, it can perform classification so that appropriate enhancement processing can be applied to endoscope images acquired from the imager of the endoscope 3.

[0074] The scenes disclosed in International Publication No. 2018 / 159083 are determined based on information represented in the image captured by the endoscopic imager, such as the type of observation site and the presence or absence of residual fluid, and are not determined based on the endoscopic observation state. For example, in a situation where a finding that could be a candidate lesion is discovered during an endoscopic examination, and a more detailed differential diagnosis is performed on the discovered finding under the set enhancement processing, it is desirable that the set enhancement processing be continued without change. However, if the method disclosed in International Publication No. 2018 / 159083 is applied, for example, if residual fluid is captured in part of the endoscopic image, the enhancement processing that should be continued may be changed, and it may become impossible to continue the differential diagnosis appropriately. In this respect, by applying the method of this embodiment, since a trained model 22 that classifies classes corresponding to the enhancement level of each enhancement processing according to the endoscopic observation state is used, it is possible to obtain an endoscopic image in which appropriate enhancement processing has been automatically performed while appropriately continuing the endoscopic examination.

[0075] In endoscopic examinations, the user operates the insertion section of the endoscope 3 with one hand and the control unit with the other. Traditionally, the user has changed the enhancement processing settings by operating switches on the control unit with the other hand. However, accommodating diverse enhancement processing options complicates the control unit's specifications, reducing its usability. Furthermore, given the limited time available for each endoscopic examination, manually changing the enhancement level is a significant burden for the user. Users unfamiliar with selecting enhancement processing levels according to the endoscopic observation state require considerable practice to master the control unit. By applying the method of this embodiment, appropriate enhancement processing can be automatically applied to endoscopic images, thereby reducing the user's burden during endoscopic examinations.

[0076] Furthermore, in the control device 10 of this embodiment, the class may be based on the enhancement level of each enhancement process itself, and the processor 100 may perform enhancement level modification processing based on the classified class. In this way, the user can automatically acquire endoscopic images that have undergone enhancement processing based on an appropriate enhancement level.

[0077] Furthermore, the method of this embodiment may be implemented as an endoscope system 1. That is, the endoscope system 1 of this embodiment includes the control device 10 described above and an endoscope 3 for capturing endoscopic images. By doing so, the same effects as described above can be obtained.

[0078] Furthermore, the method of this embodiment may also be implemented as a processing method. In other words, the processing method of this embodiment is capable of executing one or more enhancement processes and relates to a processing method that uses a trained model 22 that has been trained to classify classes corresponding to the enhancement level of each enhancement process according to the endoscopic observation state from training images. The processing method of this embodiment acquires an endoscopic image, uses the trained model 22 to classify classes from the endoscopic image, and performs an enhancement level change process to change the enhancement level of one or more enhancement processes based on the classified classes. By doing so, the same effects as described above can be obtained.

[0079] Furthermore, the method of this embodiment may be implemented as a trained model 22. That is, this embodiment relates to a trained model 22 used in a control device 10 which includes a processor 100 capable of executing one or more enhancement processes, a memory 20 for storing the trained model 22, an input unit 40, and an output unit 50, and which is trained to classify classes from training images according to the enhancement level of each enhancement process according to the endoscopic observation state. The input unit 40 inputs the endoscopic image to the trained model 22. The processor 100 uses the trained model 22 to classify classes from the endoscopic image. The output unit 50 outputs the classified classes. The processor 100 performs an enhancement level change process to change the enhancement level of one or more enhancement processes based on the classified classes. By doing so, the same effects as described above can be obtained.

[0080] Furthermore, in the control device 10 of this embodiment, the processor 100 may change the breakdown of one or more enhancement processes applied to the endoscopic image, as well as the enhancement mode and enhancement level of the applied enhancement process, based on the classified class. In this way, a control device 10 can be constructed in which the breakdown of one or more enhancement processes applied to the endoscopic image, as well as the enhancement mode and enhancement level of the applied enhancement process, are automatically changed.

[0081] Furthermore, in the control device 10 of this embodiment, one or more enhancement processes may be either structural enhancement processes or color tone enhancement processes, or multiple enhancement processes including structural enhancement processes and color tone enhancement processes. In this way, a control device 10 can be constructed that automatically applies an enhancement process including at least one of structural enhancement processes and color tone enhancement processes to an endoscopic image.

[0082] Furthermore, in the processing method of this embodiment, one or more enhancement processes may be either a structural enhancement process or a color enhancement process, or a plurality of enhancement processes including a structural enhancement process and a color enhancement process. In this way, a processing method can be constructed that automatically applies an enhancement process including at least one of a structural enhancement process and a color enhancement process to an endoscopic image.

[0083] Furthermore, the spatial filtering circuit related to the first enhancement process (structural enhancement process) does not necessarily have to support multiple enhancement modes. In this case, the processor 100 determines the enhancement level of the first enhancement process and the enhancement level of the second enhancement process in step S110 of Figure 4, and in step S190, applies the first enhancement process based on the determined enhancement level and the second enhancement process based on the determined enhancement level to the endoscopic image, respectively. In this way, in the control device 10 of this embodiment, the processor 100 changes the breakdown of one or more enhancement processes applied to the endoscopic image and the enhancement level of the applied enhancement process, respectively, based on the classified class. By doing so, a control device 10 can be constructed in which the breakdown of one or more enhancement processes applied to the endoscopic image and the enhancement level of the applied enhancement process are automatically changed.

[0084] Furthermore, in the processing method of this embodiment, the breakdown of one or more enhancement processes applied to the endoscopic image and the enhancement level of the applied enhancement process are changed based on the classified class. In this way, a processing method can be constructed in which the breakdown of one or more enhancement processes applied to the endoscopic image, and the enhancement mode and enhancement level of the applied enhancement process are automatically changed.

[0085] Furthermore, the method of this embodiment is not limited to the above and can be implemented in various modified forms. For example, as shown in Figure 10, in the control device 10 of this embodiment, the processor 100 may further function as an endoscope observation state determination unit 110. Note that Figure 10 omits the illustration of configurations other than those that differ from those in Figure 1.

[0086] When applying the method of this embodiment using the control device 10 shown in Figure 10, the processing in step S100 may be carried out in more detail as shown in the flowchart in Figure 11. In Figure 11, the processor 100 functions as an endoscope observation state determination unit 110 and determines the endoscope observation state based on the class classified in step S20 (step S120), and determines the enhancement level of each enhancement process based on the determined endoscope observation state (step S130). After that, the processor 100 performs enhancement level change processing (step S190) in the same manner as in Figure 4.

[0087] Step S120 will be explained in more detail. For example, after performing step S20 to perform class classification, the processor 100 determines the endoscopic observation status using the table shown in Figures 12 and 13 (hereinafter referred to as the second table for convenience).

[0088] The second table associates the observation site group, observation distance group, irradiation light group, and dye spraying group with the classes classified in step S20. The observation site group is a grouping of the areas observed during endoscopic examination for convenience. In this embodiment, for example, the "pharynx" is classified as group 1, the "esophagus" as group 2, the "stomach" as group 3, and the "duodenum" as group 4.

[0089] It should be noted that the grouping of the areas to be observed is not limited to this, and each area may be classified in more detail. For example, the esophagus may be further classified into "esophagogastric junction" and "areas other than the esophagogastric junction." This is because there is value in treating the esophagogastric junction separately, as diseases specific to the esophagogastric junction, such as Barrett's esophagus and Barrett's adenocarcinoma, occur there. Similarly, the stomach may be further classified into the gastric fundus, cardia, gastric body, gastric angle, pylorus, etc.

[0090] Observation distance groups are convenient classifications of observation distances in endoscopic examinations. Observation distance grouping involves grouping the distance between the imager lens at the tip of the endoscope, the subject, and the user based on commonly used terminology. In this embodiment, for example, "distant view" is classified as observation distance group 1, "mid-to-far view" as observation distance group 2, and "close-up" as observation distance group 3. In other words, the user performing the annotation looks at the endoscopic image, which is the training image, and determines whether it is "distant view," "mid-to-far view," or "close-up" based on their own experience, and then classifies it.

[0091] The illumination light group is a convenient classification of the types of illumination light used to illuminate the subject during endoscopic examination. In the second table, for example, if the illumination light used is white light, it is classified into illumination light group 1, and if the illumination light used is narrowband light used for NBI, it is classified into illumination light group 2. For example, when a user looks at an endoscopic image as a training image, if it is an endoscopic image taken using white light, it is classified into illumination light group 1. Also, for example, when a user looks at an endoscopic image as a training image, if it is an endoscopic image taken during observation using NBI, it is classified into illumination light group 2.

[0092] The dye spraying group is a convenient grouping of subjects in endoscopic examinations based on whether or not dye is sprayed on them. In endoscopic examinations where dye spraying is not typically performed depending on the area being observed, it is not necessary to determine whether or not dye is sprayed. For example, if the user does not need to determine whether or not dye is sprayed from the area in the endoscopic image they are viewing, it is classified as group 0 of the dye spraying group. Furthermore, the user can classify areas where dye spraying is possible but not performed as group 1 of the dye spraying group, and areas where dye spraying is possible but actually performed as group 2 of the dye spraying group. In this embodiment, the second table specifically assumes only indigo carmine as the dye, but Lugol's solution and other dyes may be added. If Lugol's solution is added, the second table should be modified so that classes where the observation area group is classified as group 2 (esophagus) are classified as dye spraying group 1 or 2.

[0093] Although a detailed explanation is omitted, Figures 12 and 13 show the correspondence between classes and dye dispersal groups. However, the groups may be divided based on whether or not a staining agent has been dispersed, and then corresponded to the classes. Also, although not shown in the illustrations, the type of equipment being used as part of the endoscope system 1 may be included in the endoscopic observation state. This is because different equipment used will result in different resolutions, etc., which can lead to differences in the acquired endoscopic images. Thus, in the control device 10 of this embodiment, the endoscopic observation state includes states corresponding to the organs shown in the endoscopic image, the observation distance, the type of illumination light, and the presence or absence of dye or staining agent dispersed in the area shown in the endoscopic image. By doing so, the endoscopic observation state can be classified more clearly.

[0094] As shown in Table 2, endoscopic observation states are classified according to the part being observed, the observation distance, the type of illumination light being used, and whether dye is being sprayed. Combinations of these states are associated with classes. In other words, the classes shown in Table 2 are based on the endoscopic observation state itself. However, the classification of the endoscopic observation state by the first and second observation states described above may be further considered. More specifically, for example, the processing in Figure 11 may be combined with processing of the same nature as the processing described later in Figure 21.

[0095] In Figure 12, Classes 0, 1, 2, 3, 4, and 5 are classes in which the observation site group is classified as Group 1. Furthermore, Class 0 is a class in which both the observation distance group and the illumination light group are classified as Group 1. Class 1 is a class in which both the observation distance group and the illumination light group are classified as Group 2. Class 2 is a class in which both the observation distance group and the illumination light group are classified as Group 3. Furthermore, Class 3 is a class in which both the observation distance group and the illumination light group are classified as Group 2. Class 4 is a class in which both the observation distance group and the illumination light group are classified as Group 2. Class 5 is a class in which both the observation distance group and the illumination light group are classified as Group 3. Note that in all classes 0 to 5, the dye dispersal group is classified as Group 0.

[0096] In Figure 12, classes 6, 7, 8, 9, 10, and 11 are classes in which the observation site group is classified as Group 2. Furthermore, in class 6, the observation distance group is classified as Group 1 and the illumination light group is classified as Group 1. In class 7, the observation distance group is classified as Group 2 and the illumination light group is classified as Group 1. In class 8, the observation distance group is classified as Group 3 and the illumination light group is classified as Group 1. In class 9, the observation distance group is classified as Group 1 and the illumination light group is classified as Group 2. In class 10, the observation distance group is classified as Group 2 and the illumination light group is classified as Group 2. In class 11, the observation distance group is classified as Group 3 and the illumination light group is classified as Group 2. Note that in all classes 6 to 11, the dye dispersal group is classified as Group 0.

[0097] In Figure 13, classes 12, 13, 14, 15, 16, and 17 are classes in which the observation site group is classified as Group 3. Furthermore, in class 12, the observation distance group is classified as Group 1 and the illumination light group is classified as Group 1; in class 13, the observation distance group is classified as Group 2 and the illumination light group is classified as Group 1; in class 14, the observation distance group is classified as Group 3 and the illumination light group is classified as Group 1; in class 15, the observation distance group is classified as Group 1 and the illumination light group is classified as Group 2; in class 16, the observation distance group is classified as Group 2 and the illumination light group is classified as Group 2; and in class 17, the observation distance group is classified as Group 3 and the illumination light group is classified as Group 2. Note that in all classes 12 to 17, the dye dispersal group is classified as Group 1.

[0098] In Figure 13, classes 18, 19, 20, 21, 22, and 23 are all classes in which the observation site group is classified as group 3 and the dye spraying group is classified as group 2. Furthermore, class 18 is a class in which the observation distance group is classified as group 1 and the illumination light group is classified as group 1, class 19 is a class in which the observation distance group is classified as group 2 and the illumination light group is classified as group 1, class 20 is a class in which the observation distance group is classified as group 3 and the illumination light group is classified as group 1, class 21 is a class in which the observation distance group is classified as group 1 and the illumination light group is classified as group 2, class 22 is a class in which the observation distance group is classified as group 2 and the illumination light group is classified as group 2, and class 23 is a class in which the observation distance group is classified as group 3 and the illumination light group is classified as group 2.

[0099] In Figure 13, classes 24, 25, 26, 27, 28, and 29 are classes in which the observation site group is classified as Group 4. Furthermore, class 24 is a class in which the observation distance group is classified as Group 1 and the illumination light group is classified as Group 1, class 25 is a class in which the observation distance group is classified as Group 2 and the illumination light group is classified as Group 1, class 26 is a class in which the observation distance group is classified as Group 3 and the illumination light group is classified as Group 1, class 27 is a class in which the observation distance group is classified as Group 1 and the illumination light group is classified as Group 2, class 28 is a class in which the observation distance group is classified as Group 2 and the illumination light group is classified as Group 2, and class 29 is a class in which the observation distance group is classified as Group 3 and the illumination light group is classified as Group 2. Note that for all classes 24 to 29, the dye dispersal group is classified as Group 0.

[0100] By using this second table, the processor 100 performs a class classification of the input endoscopic image in step S120 based on the endoscopic observation state itself. Then, in step S130, the enhancement processing corresponding to the classified class is determined, and in step S190, image processing based on the determined content is performed on the endoscopic image. In other words, in the control device 10 of this embodiment, the class is based on the endoscopic observation state itself, and the processor 100 performs the enhancement level change processing based on the classified class. In this way, a method for classifying endoscopic images into classes based on the endoscopic observation state can be constructed. This eliminates the burden of updating the trained model 22 again, for example, when a new enhancement processing is proposed. In this case, the user only needs to update the third table, which will be described later.

[0101] Step S130 will now be explained in detail. For example, after performing class classification in step S120, the processor 100 determines the details of the collaborative processing based on the table shown in Figure 14 (hereinafter referred to as the third table for convenience). Note that the third table in Figure 14 is shown only for some classes, but the third table can be determined as appropriate for other classes as well.

[0102] In the third table shown in Figure 14, class 12 is associated with mode A as the emphasis mode of the first emphasis processing, level 2 as the emphasis level of the first emphasis processing, and level 3 as the emphasis level of the second emphasis processing. Similarly, class 13 is associated with mode A as the emphasis mode of the first emphasis processing, level 4 as the emphasis level of the first emphasis processing, and level 4 as the emphasis level of the second emphasis processing. Similarly, class 14 is associated with mode A as the emphasis mode of the first emphasis processing, level 7 as the emphasis level of the first emphasis processing, and level 5 as the emphasis level of the second emphasis processing. Similarly, class 15 is associated with mode A as the emphasis mode of the first emphasis processing, level 3 as the emphasis level of the first emphasis processing, and level 2 as the emphasis level of the second emphasis processing. Similarly, class 16 is associated with mode A as the emphasis mode of the first emphasis processing, level 5 as the emphasis level of the first emphasis processing, and level 3 as the emphasis level of the second emphasis processing. Furthermore, class 17 is associated with mode B as the enhancement mode for the first enhancement process, with level 8 as the enhancement level for the first enhancement process, and with level 4 as the enhancement level for the second enhancement process.

[0103] For example, in an endoscopic examination where the stomach is the observation site, when the processor 100 acquires an endoscopic image that is observed from a distance using white light and without the scattering of dye, the processor 100 classifies the endoscopic image into class 12 in step S20. Then, in step S130, the processor 100 decides to set the enhancement mode of the first enhancement processing to mode A, the enhancement level of the first enhancement processing to level 2, and the enhancement level of the second enhancement processing to level 3, based on the third table in Figure 14. Then, in step S190, the processor 100 controls the aforementioned spatial filtering circuit, image processing unit, etc., based on the content determined in step S130. For example, if the user discovers a finding in the endoscopic image, they try to move the tip of the endoscope 3 closer to the finding in order to observe it in more detail. In this case, the observation distance group of the acquired endoscopic image is classified into group 3, so the processor 100 classifies the acquired endoscopic image into class 14 in step S20. Then, in step S130, the processor 100 refers to the third table in Figure 14 and decides to set the enhancement mode of the first enhancement processing to mode A, the enhancement level of the first enhancement processing to level 7, and the enhancement level of the second enhancement processing to level 5. After that, the processor 100 controls the aforementioned spatial filtering circuit, image processing unit, etc., based on the content decided in step S130. In other words, as the observation distance changes from a distant view to a close view, the enhancement level of the first enhancement processing automatically increases from 2 to 7. Thus, in the control device 10 of this embodiment, the endoscopic observation state includes at least a state corresponding to the observation distance, and the processor 100 increases the enhancement level of the structural enhancement processing when classified into a class corresponding to a close view observation distance compared to when classified into a class corresponding to a distant view observation distance. In this way, the user can concentrate more on observing the findings without having to perform operations related to structural enhancement processing. This makes endoscopic examinations smoother.

[0104] Furthermore, the same method may be applied to the processing method of this embodiment. That is, in the processing method of this embodiment, the endoscopic observation state includes at least a state corresponding to the observation distance, and the processor 100 strengthens the emphasis level of the structure enhancement processing when classified into a class corresponding to a close observation distance compared to when classified into a class corresponding to a distant observation distance. By doing so, the same effect as described above can be obtained.

[0105] Such changes in the enhancement level of enhancement processing occur reversibly. That is, when the observation distance changes to a distant observation distance in an endoscopic observation state classified as class 14, the enhancement level of the first enhancement processing weakens from level 7 to level 2, and the enhancement level of the second enhancement processing weakens from level 5 to level 2. Thus, in the control device 10 of this embodiment, the endoscopic observation state includes at least a state corresponding to the observation distance. When the processor 100 is classified as a class corresponding to a distant observation distance, it weakens the enhancement level of the structural enhancement processing and the enhancement level of the color tone enhancement processing compared to when it is classified as a class corresponding to a close observation distance. In this way, the burden on the user to manually adjust the settings related to structural enhancement processing and color tone enhancement processing when the observation distance is changed to a distance corresponding to a distant view can be reduced. This makes endoscopic examinations smoother.

[0106] Furthermore, comparing Class 12 and Class 15 in the second table of Figure 13, the difference is that in Class 12, the irradiation light group is associated with Group 1, while in Class 15, the irradiation light group is associated with Group 2; otherwise, they are the same. Similarly, comparing Class 13 and Class 16, the difference is that in Class 13, the irradiation light group is associated with Group 1, while in Class 16, the irradiation light group is associated with Group 2; otherwise, they are the same. Similarly, comparing Class 14 and Class 17, the difference is that in Class 14, the irradiation light group is associated with Group 1, while in Class 17, the irradiation light group is associated with Group 2; otherwise, they are the same.

[0107] Furthermore, comparing class 12 and class 15 in the third table of Figure 14, class 12 has an emphasis level of level 2 for the first emphasis processing and an emphasis level of level 3 for the second emphasis processing, whereas class 15 has an emphasis level of level 3 for both the first and second emphasis processing. Also, comparing class 13 and class 16 in the third table of Figure 14, class 13 has an emphasis level of level 4 for both the first and second emphasis processing, while class 15 has an emphasis level of level 5 for the first emphasis processing and an emphasis level of level 3 for the second emphasis processing. Furthermore, comparing class 14 and class 17 in the third table of Figure 14, class 14 has an emphasis level of level 7 for the first emphasis processing and an emphasis level of level 5 for the second emphasis processing, while class 17 has an emphasis level of level 8 for the first emphasis processing and an emphasis level of level 4 for the second emphasis processing.

[0108] In other words, the following can be derived from the relationship between Class 12 and Class 15. For example, when an endoscopic image is input into the trained model under observation conditions where the observation site is the stomach, no dye is scattered, white light is used as the illumination light, and the observation is performed from a distance, it is classified into Class 12 in step 20, and the endoscopic observation conditions are determined in step S120. Then, in step S130, the processor 100 determines the enhancement levels of the enhancement processing based on the third table, such that the enhancement level of the first enhancement processing is level 2 and the enhancement level of the second enhancement processing is level 3. Then, in step S190, the processor 100 controls the aforementioned spatial filtering circuit, image processing unit, etc., to change the enhancement levels based on the content determined in step S130. After that, suppose the user changes the illumination light from white light to narrowband light used for NBI. As a result, the appearance of the captured endoscopic image changes, the input data input to the trained model 22 changes, and it is classified into Class 15 in step S20. Then, in steps S120, S130, and S190, the enhancement levels of the enhancement processes are determined such that the enhancement level of the first enhancement process is level 3 and the enhancement level of the second enhancement process is level 2. The spatial filtering circuit, image processing unit, etc. are then controlled to achieve the determined enhancement levels. In this way, by changing the illumination light from white light to narrowband light used for NBI in an endoscopy observation state classified as class 12, the endoscopy observation state changes to class 15, the enhancement level of the first enhancement process automatically increases from 2 to 3, and the enhancement level of the second enhancement process automatically decreases from 3 to 2. Similarly, by changing the illumination light from white light to narrowband light used for NBI in an endoscopy observation state classified as class 13, the endoscopy observation state changes to class 16, the enhancement level of the first enhancement process automatically increases from 4 to 5, and the enhancement level of the second enhancement process automatically decreases from 4 to 3.Similarly, in an endoscopic observation state classified as Class 14, changing the illumination light from white light to narrowband light used for NBI changes the endoscopic observation state to Class 17, the enhancement level of the first enhancement processing automatically increases from 7 to 8, and the enhancement level of the second enhancement processing automatically decreases from 5 to 4. From the above, it can be seen that in the control device 10 of this embodiment, the endoscopic observation state includes at least a state corresponding to the type of illumination light. When the processor 100 is classified as a class corresponding to observation using narrowband light as the illumination light, it increases the enhancement level of the structure enhancement processing and decreases the enhancement level of the color enhancement processing compared to when it is classified as a class corresponding to observation using white light as the illumination light.

[0109] Furthermore, if the imager of the endoscope 3 also includes a magnification observation function, the processor 100 may perform step S120 using the table shown in Figures 15, 16, 17, 18, and 19 (hereinafter conveniently referred to as the fourth table) instead of the second table. The magnification observation function refers to the function of magnifying and observing a subject by moving the position of the lens of the imager at the tip of the endoscope 3, and the magnification is determined according to the amount of displacement of the moved lens, etc. The fourth table differs from the second table in that the magnification groups are further associated with classes. The magnification groups are grouped based on the user's experience, etc., regarding the magnification used when observing a subject, and it is not necessarily required to group based on specific magnification numbers. In this embodiment, for example, "no magnification" is classified as group 1, "weak magnification" as group 2, and "high magnification" as group 3. In other words, the user performing annotation judges whether the endoscopic image, which is a learning image, is "no magnification," "weak magnification," or "high magnification" based on the user's own experience. Note that "non-magnified" refers to a magnification of 1x, but it may also include cases where the image is slightly magnified, and it is sufficient for the user to make an appropriate judgment and perform annotation. Thus, in the control device 10 of this embodiment, the endoscopic observation state further includes states corresponding to the magnification. By doing so, the endoscopic observation state of the endoscope 3, which has a magnification observation function, can be classified more accurately.

[0110] In the fourth table of Figure 15, classes 0, 1, 2, 3, 4, 5, 6, 7, and 8 are classes in which the observation site group is classified as group 1, the irradiation light group is classified as group 1, and the dye dispersal group is classified as group 0. Furthermore, in class 0, the observation distance group is classified as group 1, and the magnification group is classified as group 1. In class 1, the observation distance group is classified as group 1, and the magnification group is classified as group 2. In class 2, the observation distance group is classified as group 1, and the magnification group is classified as group 3. In class 3, the observation distance group is classified as group 2, and the magnification group is classified as group 1. In class 4, the observation distance group is classified as group 2, and the magnification group is classified as group 2. In class 5, the observation distance group is classified as group 2, and the magnification group is classified as group 3. In class 6, the observation distance group is classified as group 3, and the magnification group is classified as group 1. Furthermore, Class 7 is classified as Group 3 for observation distance and Group 2 for magnification. Similarly, Class 8 is classified as Group 3 for observation distance and Group 3 for magnification.

[0111] In the fourth table of Figure 15, classes 9, 10, 11, 12, 13, 14, 15, 16, and 17 are classes in which the observation site group is classified as group 1, the irradiation light group is classified as group 2, and the dye dispersal group is classified as group 0. Furthermore, in class 9, the observation distance group is classified as group 1 and the magnification group is classified as group 1. In class 10, the observation distance group is classified as group 1 and the magnification group is classified as group 2. In class 11, the observation distance group is classified as group 1 and the magnification group is classified as group 3. In class 12, the observation distance group is classified as group 2 and the magnification group is classified as group 1. In class 13, the observation distance group is classified as group 2 and the magnification group is classified as group 2. In class 14, the observation distance group is classified as group 2 and the magnification group is classified as group 3. Furthermore, Class 15 is classified as Group 3 for observation distance and Group 1 for magnification. Also, Class 16 is classified as Group 3 for observation distance and Group 2 for magnification. Also, Class 17 is classified as Group 3 for observation distance and Group 3 for magnification.

[0112] In the fourth table of Figure 16, classes 18, 19, 20, 21, 22, 23, 24, 25, and 26 are classes in which the observation site group is classified as group 2, the irradiation light group is classified as group 1, and the dye dispersal group is classified as group 0. Furthermore, in class 18, the observation distance group is classified as group 1 and the magnification group is classified as group 1. In class 19, the observation distance group is classified as group 1 and the magnification group is classified as group 2. In class 20, the observation distance group is classified as group 1 and the magnification group is classified as group 3. In class 21, the observation distance group is classified as group 2 and the magnification group is classified as group 1. In class 22, the observation distance group is classified as group 2 and the magnification group is classified as group 2. In class 23, the observation distance group is classified as group 2 and the magnification group is classified as group 3. Furthermore, class 24 is classified as Group 3 for observation distance and Group 1 for magnification. Also, class 25 is classified as Group 3 for observation distance and Group 2 for magnification. Also, class 26 is classified as Group 3 for observation distance and Group 3 for magnification.

[0113] In the fourth table of Figure 16, classes 27, 28, 29, 30, 31, 32, 33, 34, and 35 are classes in which the observation site group is classified as group 2, the irradiation light group is classified as group 2, and the dye dispersal group is classified as group 0. Furthermore, in class 27, the observation distance group is classified as group 1, and the magnification group is classified as group 1. In class 28, the observation distance group is classified as group 1, and the magnification group is classified as group 2. In class 29, the observation distance group is classified as group 1, and the magnification group is classified as group 3. In class 30, the observation distance group is classified as group 2, and the magnification group is classified as group 1. In class 31, the observation distance group is classified as group 2, and the magnification group is classified as group 2. In class 32, the observation distance group is classified as group 2, and the magnification group is classified as group 3. Furthermore, class 33 is classified as Group 3 for observation distance and Group 1 for magnification. Similarly, class 34 is classified as Group 3 for observation distance and Group 2 for magnification. And class 35 is classified as Group 3 for observation distance and Group 3 for magnification.

[0114] In the fourth table of Figure 17, classes 36, 37, 38, 39, 40, 41, 42, 43, and 44 are classes in which the observation site group is classified as group 3, the irradiation light group is classified as group 1, and the dye dispersal group is classified as group 1. Furthermore, in class 36, the observation distance group is classified as group 1, and the magnification group is classified as group 1. In class 37, the observation distance group is classified as group 1, and the magnification group is classified as group 2. In class 38, the observation distance group is classified as group 1, and the magnification group is classified as group 3. In class 39, the observation distance group is classified as group 2, and the magnification group is classified as group 1. In class 40, the observation distance group is classified as group 2, and the magnification group is classified as group 2. In class 41, the observation distance group is classified as group 2, and the magnification group is classified as group 3. Furthermore, class 42 is classified as Group 3 for observation distance and Group 1 for magnification. Also, class 43 is classified as Group 3 for observation distance and Group 2 for magnification. Also, class 44 is classified as Group 3 for observation distance and Group 3 for magnification.

[0115] In the fourth table of Figure 17, classes 45, 46, 47, 48, 49, 50, 51, 52, and 53 are classes in which the observation site group is classified as group 3, the irradiation light group as group 2, and the dye dispersal group as group 1. Furthermore, in class 45, the observation distance group is classified as group 1 and the magnification group as group 1. In class 46, the observation distance group is classified as group 1 and the magnification group as group 2. In class 47, the observation distance group is classified as group 1 and the magnification group as group 3. In class 48, the observation distance group is classified as group 2 and the magnification group as group 1. In class 49, the observation distance group is classified as group 2 and the magnification group as group 2. In class 50, the observation distance group is classified as group 2 and the magnification group as group 3. Furthermore, class 51 is classified as Group 3 for observation distance and Group 1 for magnification. Also, class 52 is classified as Group 3 for observation distance and Group 2 for magnification. Also, class 53 is classified as Group 3 for observation distance and Group 3 for magnification.

[0116] In the fourth table of Figure 18, classes 54, 55, 56, 57, 58, 59, 60, 61, and 62 are classes in which the observation site group is classified as group 3, the irradiation light group is classified as group 1, and the dye dispersal group is classified as group 2. Furthermore, in class 54, the observation distance group is classified as group 1 and the magnification group is classified as group 1. In class 55, the observation distance group is classified as group 1 and the magnification group is classified as group 2. In class 56, the observation distance group is classified as group 1 and the magnification group is classified as group 3. In class 57, the observation distance group is classified as group 2 and the magnification group is classified as group 1. In class 58, the observation distance group is classified as group 2 and the magnification group is classified as group 2. In class 59, the observation distance group is classified as group 2 and the magnification group is classified as group 3. Furthermore, class 60 is classified as Group 3 for observation distance and Group 1 for magnification. Also, class 61 is classified as Group 3 for observation distance and Group 2 for magnification. Also, class 62 is classified as Group 3 for observation distance and Group 3 for magnification.

[0117] In the fourth table of Figure 18, classes 63, 64, 65, 66, 67, 68, 69, 70, and 71 are classes in which the observation site group is classified as group 3, the irradiation light group is classified as group 2, and the dye dispersal group is classified as group 2. Furthermore, in class 63, the observation distance group is classified as group 1, and the magnification group is classified as group 1. In class 64, the observation distance group is classified as group 1, and the magnification group is classified as group 2. In class 65, the observation distance group is classified as group 1, and the magnification group is classified as group 3. In class 66, the observation distance group is classified as group 2, and the magnification group is classified as group 1. In class 67, the observation distance group is classified as group 2, and the magnification group is classified as group 2. In class 68, the observation distance group is classified as group 2, and the magnification group is classified as group 3. Furthermore, Class 69 is classified as Group 3 for observation distance and Group 1 for magnification. Also, Class 70 is classified as Group 3 for observation distance and Group 2 for magnification. Also, Class 71 is classified as Group 3 for observation distance and Group 3 for magnification.

[0118] In the fourth table of Figure 19, classes 72, 73, 74, 75, 76, 77, 78, 79, and 80 are classes in which the observation site group is classified as group 4, the irradiation light group as group 1, and the dye dispersal group as group 0. Furthermore, in class 72, the observation distance group is classified as group 1 and the magnification group as group 1. In class 73, the observation distance group is classified as group 1 and the magnification group as group 2. In class 74, the observation distance group is classified as group 1 and the magnification group as group 3. In class 75, the observation distance group is classified as group 2 and the magnification group as group 1. In class 76, the observation distance group is classified as group 2 and the magnification group as group 2. In class 77, the observation distance group is classified as group 2 and the magnification group as group 3. Furthermore, Class 78 is classified as Group 3 for observation distance and Group 1 for magnification. Also, Class 79 is classified as Group 3 for observation distance and Group 2 for magnification. Also, Class 80 is classified as Group 3 for observation distance and Group 3 for magnification.

[0119] In the fourth table of Figure 19, classes 81, 82, 83, 84, 85, 86, 87, 88, and 89 are classes in which the observation site group is classified as group 4, the irradiation light group as group 2, and the dye dispersal group as group 0. Furthermore, in class 81, the observation distance group is classified as group 1 and the magnification group as group 1. In class 82, the observation distance group is classified as group 1 and the magnification group as group 2. In class 83, the observation distance group is classified as group 1 and the magnification group as group 3. In class 84, the observation distance group is classified as group 2 and the magnification group as group 1. In class 85, the observation distance group is classified as group 2 and the magnification group as group 2. In class 86, the observation distance group is classified as group 2 and the magnification group as group 3. Furthermore, class 87 is classified as Group 3 for observation distance and Group 1 for magnification. Also, class 88 is classified as Group 3 for observation distance and Group 2 for magnification. Also, class 89 is classified as Group 3 for observation distance and Group 3 for magnification.

[0120] Furthermore, for example, after performing step S130 to classify the classes, the processor 100 determines the details of the collaborative processing based on the table shown in Figure 20 (hereinafter referred to as the fifth table for convenience). Note that Figure 20 shows only some classes, but the fifth table can be determined appropriately for other classes as well.

[0121] In the fifth table shown in Figure 20, class 36 is associated with mode A as the emphasis mode for the first emphasis processing, level 3 as the emphasis level for the first emphasis processing, and level 2 as the emphasis level for the second emphasis processing. Also, class 37 is associated with mode A as the emphasis mode for the first emphasis processing, level 5 as the emphasis level for the first emphasis processing, and level 2 as the emphasis level for the second emphasis processing. Also, class 38 is associated with mode A as the emphasis mode for the first emphasis processing, level 8 as the emphasis level for the first emphasis processing, and level 2 as the emphasis level for the second emphasis processing. Class 39 is associated with mode A as the emphasis mode for the first emphasis processing, level 3 as the emphasis level for the first emphasis processing, and level 3 as the emphasis level for the second emphasis processing. Also, class 40 is associated with mode A as the emphasis mode for the first emphasis processing, level 5 as the emphasis level for the first emphasis processing, and level 3 as the emphasis level for the second emphasis processing. Furthermore, class 38 is associated with mode A as the emphasis mode of the first emphasis processing, with level 8 as the emphasis level of the first emphasis processing, and with level 2 as the emphasis level of the second emphasis processing. Furthermore, class 42 is associated with mode B as the emphasis mode of the first emphasis processing, with level 3 as the emphasis level of the first emphasis processing, and with level 2 as the emphasis level of the second emphasis processing. Furthermore, class 43 is associated with mode B as the emphasis mode of the first emphasis processing, with level 5 as the emphasis level of the first emphasis processing, and with level 3 as the emphasis level of the second emphasis processing. Furthermore, class 44 is associated with mode B as the emphasis mode of the first emphasis processing, with level 8 as the emphasis level of the first emphasis processing, and with level 5 as the emphasis level of the second emphasis processing.

[0122] In the fifth table shown in Figure 20, classes 45, 46, 47, 48, 49, 50, 51, 52, and 53 are associated with the second enhancement processing being off. In other words, if the processor 100 classifies the endoscopic image into classes 45 to 53 in step S20, the light source device (not shown) is controlled in step S140 of Figure 11 to turn off the module related to the second enhancement processing. Also, in the fifth table of Figure 20, class 45 is associated with mode A as the enhancement mode of the first enhancement processing and with level 3 as the enhancement level of the first enhancement processing. Class 46 is associated with mode A as the enhancement mode of the first enhancement processing and with level 5 as the enhancement level of the first enhancement processing. Class 47 is associated with mode A as the enhancement mode of the first enhancement processing and with level 8 as the enhancement level of the first enhancement processing. Class 48 is associated with Mode A as the emphasis mode of the first emphasis processing and with Level 3 as the emphasis level of the first emphasis processing. Also, Class 49 is associated with Mode A as the emphasis mode of the first emphasis processing and with Level 5 as the emphasis level of the first emphasis processing. Also, Class 50 is associated with Mode A as the emphasis mode of the first emphasis processing and with Level 8 as the emphasis level of the first emphasis processing. Also, Class 51 is associated with Mode B as the emphasis mode of the first emphasis processing and with Level 3 as the emphasis level of the first emphasis processing. Also, Class 52 is associated with Mode B as the emphasis mode of the first emphasis processing and with Level 5 as the emphasis level of the first emphasis processing. Also, Class 53 is associated with Mode B as the emphasis mode of the first emphasis processing and with Level 8 as the emphasis level of the first emphasis processing.

[0123] A specific example of performing step S130 using the fourth and fifth tables will be explained. For example, when the observation site is the stomach, no dye is scattered, white light is used as the illumination light, the observation is at a close distance, and no magnification is applied, an endoscopic image in this observation state is input to the trained model, and in step 20 it is classified as class 42, and in step S120 the endoscopic observation state is determined. Then, in step S130, the processor 100 determines the enhancement levels of the enhancement processing based on the fifth table, so that the enhancement level of the first enhancement processing is level 3 and the enhancement level of the second enhancement processing is level 2. Then, in step S190, the processor 100 controls the aforementioned spatial filtering circuit, image processing unit, etc. to change the enhancement level based on the content determined in step S130. After that, suppose the user changes the magnification from no magnification to high magnification. As a result the appearance of the captured endoscopic image changes, the input data input to the trained model 22 changes, and in step S20 it is classified as class 44. Then, in steps S120, S130, and S190, the enhancement levels of the enhancement processes are determined such that the enhancement level of the first enhancement process is level 8 and the enhancement level of the second enhancement process is level 5. The spatial filtering circuit, image processing unit, etc. are then controlled to achieve the determined enhancement levels. In this way, by changing the magnification from non-magnification to high magnification in an endoscopic observation state classified as class 42, the endoscopic observation state changes to class 44, the enhancement level of the first enhancement process automatically increases from 3 to 8, and the enhancement level of the second enhancement process automatically increases from 2 to 5.

[0124] More specifically, when the magnification changes from no magnification to low magnification during endoscopic observation, the class changes from 42 to 43, the enhancement level of the first enhancement processing increases from level 3 to level 5, and the enhancement level of the second enhancement processing increases from level 2 to level 3. Furthermore, when the magnification changes from low magnification to high magnification during endoscopic observation, the class changes from 43 to 44, the enhancement level of the first enhancement processing increases from level 5 to level 8, and the enhancement level of the second enhancement processing increases from level 3 to level 5. Thus, in the control device 10 of this embodiment, the endoscopic observation state includes at least a state corresponding to the magnification. When the processor 100 is classified into a class corresponding to a second magnification higher than the first magnification, it increases the enhancement level of the structural enhancement processing compared to when it is classified into a class corresponding to the first magnification. In this way, the burden on the user to manually adjust the enhancement level of the structural enhancement processing when performing endoscopic observation at a higher magnification can be reduced. This allows for smoother endoscopic observation.

[0125] Furthermore, for example, if the observation site is the stomach, no dye is scattered, white light is used as the illumination light, and the observation is performed at a distant observation distance without magnification, an endoscopic image in this observation state is input to the trained model, and in step 20 it is classified into class 36, and in step S120 the endoscopic observation state is determined. Then, in step S130 the processor 100 determines the enhancement levels of the enhancement processing based on the fifth table, so that the enhancement level of the first enhancement processing is level 3 and the enhancement level of the second enhancement processing is level 2. Then, in step S190 the processor 100 controls the aforementioned spatial filtering circuit, image processing unit, etc. to change the enhancement level based on the content determined in step S130. After that, suppose the user changes the illumination light from white light to narrowband light used for NBI. As a result the appearance of the captured endoscopic image changes, the input data input to the trained model 22 changes, and in step S20 it is classified into class 45. Then, in steps S120, S130, and S190, the image processing unit and the like are controlled so that the second enhancement processing is turned off. In this way, by changing the illumination light from white light to narrowband light used for NBI, the class changes from class 36 to class 45, and the second enhancement processing is turned off. The same applies when the class changes from class 37 to class 46, from class 38 to class 47, from class 39 to class 48, from class 40 to class 49, from class 41 to class 50, from class 42 to class 51, from class 43 to class 52, and from class 44 to class 53. From the above, it can be seen that in the control device 10 of this embodiment, the endoscopic observation state includes at least a state corresponding to the type of illumination light. When the processor 100 is classified as a class that corresponds to observation using narrowband light as the illumination light, it increases the enhancement level of the structure enhancement processing and turns off or decreases the enhancement level of the color enhancement processing compared to when it is classified as a class that corresponds to observation using white light as the illumination light.This reduces the burden on the user of having to adjust the enhancement levels of both the structure enhancement and color enhancement processing when the illumination light is changed to narrowband light. As a result, endoscopic examinations can be performed more smoothly.

[0126] Alternatively, for example, step S100 in Figure 4 may be performed as shown in the flowchart in Figure 21. In Figure 21, the processor 100 performs a process (step S140) to determine whether the endoscopic observation state is either the first observation state or the second observation state. If the endoscopic observation state is the first observation state (YES in step S150), the processor 100 performs an enhancement level change process (step S190) and terminates the flow. On the other hand, if the endoscopic observation state is the second observation state (NO in step S150), the processor 100 terminates the flow. In this way, in the control device 10 of this embodiment, the processor 100 determines whether the endoscopic observation state is the first observation state or the second observation state, and if it determines that it is the first observation state, it performs an enhancement level change process, and if it determines that it is the second observation state, it does not perform an enhancement level change process. In this way, it is possible to choose whether to perform an enhancement level change process or not depending on the endoscopic observation state.

[0127] In Figure 21, the flow is terminated if the result is NO in step S150. However, for example, the processor 100 may perform the enhancement processing set by default on the endoscopic image after step S150.

[0128] The user can determine, as appropriate, whether the endoscopic observation state is the first observation state or the second observation state. For example, in the lower endoscopy described above, a user who believes it is preferable not to change the enhancement processing until the tip of the endoscope 3 reaches the cecum can designate the period until the tip of the endoscope 3 reaches the cecum as the second observation state, and the period thereafter as the first observation state. Similarly, in the upper endoscopy, a user who believes it is preferable not to change the enhancement processing until the tip of the endoscope 3 reaches the duodenum can designate the period until the tip of the endoscope 3 reaches the duodenum as the second observation state, and the period thereafter as the first observation state.

[0129] In this case, for example, if a lower endoscopy is being performed, the processor 100 may use the aforementioned image recognition method such as CNN to determine whether or not to recognize the cecum from the endoscopic image, and determine that the period until the cecum is recognized from the endoscopic image is the second observation state, and after the cecum is recognized from the endoscopic image it is the first observation state. Alternatively, for example, an endoscope shape observation device may be used to acquire shape information of the insertion part of the endoscope 3, and position information of the tip of the endoscope 3 may be acquired from the acquired shape information, and the first or second observation state may be determined based on the acquired position information of the tip. Note that the method related to the endoscope shape observation device is well known, so its explanation will be omitted. The processor 100 may also allow the user to recognize the first and second observation states by, for example, operating a predetermined control unit. Thus, in the control device 10 of this embodiment, the second observation state is mainly the state in which the insertion part of the endoscope 3 is inserted, and if the processor 100 determines that the endoscopic observation state is the second observation state (NO in step S150), it does not perform the enhancement level change process. For example, in the lower endoscopy described above, for users who prefer not to change the enhancement processing until the tip of the endoscope 3 reaches the cecum, the automatic change in enhancement processing during the second observation state could hinder the smooth progress of the endoscopic examination. In this respect, by using the flowchart shown in Figure 21, it is possible to intentionally create a state where the change in enhancement processing can be restricted, thus enabling a smoother endoscopic examination.

[0130] Furthermore, for example, step S140 may be performed in more detail as shown in the flowchart in Figure 22. The processor 100 acquires the amount of image change (step S142) and determines whether the magnitude of the amount of image change is greater than or equal to a certain amount (step S144). If the processor 100 determines that the magnitude of the amount of image change is not greater than or equal to a certain amount (NO in step S144), it determines that it is in the first observation state (step S146) and terminates the flow. On the other hand, if the processor 100 determines that the magnitude of the amount of image change is greater than or equal to a certain amount (YES in step S144), it determines that it is in the second observation state (step S148) and terminates the flow.

[0131] For example, the processor 100 sequentially performs the following processes from the set of endoscopic images acquired in step S10: extracting endoscopic images at predetermined intervals, and comparing the M-th acquired endoscopic image with the (M-1)th (M>2) acquired endoscopic image. The processor 100 then calculates the amount of image change, which is the sum of the differences in each pixel value included in the endoscopic image, and determines whether the amount of image change is greater than or equal to a certain amount. The endoscopic images related to step S144 only need to be at least one of the endoscopic images related to R, G, and B. Since the first observation state is mainly observation, the amount of change in the position of the tip of the endoscope 3 is small, and therefore the amount of image change in the captured endoscopic image is considered to be small. On the other hand, since the second observation state is mainly insertion of the insertion part, the amount of change in the position of the tip of the endoscope 3 is large, and therefore the amount of image change is considered to be large. Thus, in the control device 10 of this embodiment, the processor 100 determines whether it is the first observation state or the second observation state based on the amount of image change in the acquired endoscopic image. In this way, it is possible to determine whether the device is in the first or second observation state, without relying on the positional information of the tip of the endoscope 3.

[0132] Furthermore, step S140 is not limited to the above and can be modified in various ways. For example, the processor 100 may determine whether it is the first observation state or the second observation state based on whether the movement distance of the feature quantity is greater than or equal to a certain length using an image tracking method. Since the image tracking method is well known, a detailed explanation will be omitted. The processor 100 may also determine whether it is the first observation state or the second observation state based on whether the amount of movement of the tip of the endoscope 3 acquired by the shape observation device is greater than or equal to a certain amount. Furthermore, the processor 100 may determine that it is the first observation state if the tip of the endoscope 3 acquired by the shape observation device remains stationary for a certain period of time or longer, as this is considered to correspond to a state of detailed observation.

[0133] The method of this embodiment is not limited to the above and can be modified in various ways. For example, the classes in the fourth table in Figures 15 to 19 are based on all possible combinations, and the trained model 22 is trained accordingly. However, the fourth table may be created in a way that excludes classes that are unlikely to be classified. For example, in endoscopic examinations where a situation is not anticipated where the observation distance group is classified into group 1 and the magnified observation group is classified into group 3 (a situation where observation is performed at an observation distance corresponding to a distant view and at a magnification corresponding to high magnification), classes such as class 2 may be excluded from the fourth table before training the trained model 22. By doing so, the neural network included in the trained model 22 can be made to an appropriate size. Furthermore, for example, the user may manually change the enhancement level of an endoscopic image that has been automatically enhanced using the method of this embodiment by operating a switch or the like included in the control unit. By doing so, the convenience of the control device 10 can be further improved.

[0134] Although this embodiment has been described in detail above, it will be readily apparent to those skilled in the art that many modifications are possible without substantially departing from the novel aspects and effects of this embodiment. Therefore, all such modifications are included within the scope of this disclosure. For example, any term that appears at least once in the specification or drawings together with a broader or synonymous term may be replaced with that different term anywhere in the specification or drawings. Furthermore, all combinations of this embodiment and its modifications are also included within the scope of this disclosure. In addition, the configuration and operation of the control device, endoscope system, and processing method are not limited to those described in this embodiment, and various modifications are possible.

[0135] 1...Endoscope system, 3...Endoscope, 10...Control device, 20...Memory, 22...Trained model, 40...Input unit, 50...Output unit, 100...Processor, 110...Endoscope observation state determination unit, 120...Enhanced processing determination unit, 130...Image processing unit

Claims

1. A control device comprising: a processor capable of performing one or more enhancement processes; and a memory that stores a trained model trained to classify classes corresponding to the enhancement level of each enhancement process according to the endoscopic observation state from training images, wherein the processor acquires an endoscopic image, uses the trained model to classify the classes from the endoscopic image, and performs an enhancement level change process to change the enhancement level of one or more enhancement processes based on the classified classes.

2. The control device according to claim 1, wherein the class is based on the enhancement level of each of the enhancement processes, and the processor performs the enhancement level modification process based on the classified class.

3. A control device according to claim 1, wherein the class is based on the endoscopic observation state itself, and the processor performs the enhancement level change processing based on the classified class.

4. The control device according to claim 1, wherein the processor determines whether the endoscope observation state is a first observation state or a second observation state, performs the enhancement level change process if it is determined to be the first observation state, and does not perform the enhancement level change process if it is determined to be the second observation state.

5. The control device according to claim 4, wherein the second observation state is a state in which the insertion portion of the endoscope is inserted, and the processor determines that the endoscope observation state is the second observation state, and the control device is characterized in that it does not perform the enhancement level change process.

6. The control device according to claim 4, wherein the processor determines whether the state is the first observation state or the second observation state based on the amount of image change of the acquired endoscope image.

7. A control device according to claim 1, wherein the processor changes, based on the classified class, the breakdown of one or more enhancement processes applied to the endoscopic image and the enhancement level of the enhancement process to be applied.

8. A control device according to claim 1, wherein the processor changes, based on the classified class, the breakdown of one or more enhancement processes applied to the endoscopic image, the enhancement mode of the enhancement process to be applied, and the enhancement level.

9. The control device according to claim 1, characterized in that the endoscopic observation state includes states corresponding to the organs shown in the endoscopic image, the observation distance, the type of irradiation light, and the presence or absence of dyes or stains scattered in the area shown in the endoscopic image.

10. The control device according to claim 9, wherein the endoscopic observation state further includes a state corresponding to the magnification.

11. A control device according to claim 1, characterized in that one or more of the enhancement processes are either a structural enhancement process or a color tone enhancement process, or a plurality of enhancement processes including the structural enhancement process and the color tone enhancement process.

12. A control device according to claim 11, wherein the endoscopic observation state includes at least a state corresponding to the observation distance, and the processor is characterized in that, when classified into the class corresponding to the close observation distance, it increases the enhancement level of the structure enhancement processing compared to when classified into the class corresponding to the distant observation distance.

13. A control device according to claim 11, wherein the endoscopic observation state includes at least a state corresponding to the type of irradiation light, and the processor, when classified into the class corresponding to observation using narrowband light as the irradiation light, strengthens the enhancement level of the structure enhancement processing and turns off or weakens the color enhancement processing compared to when classified into the class corresponding to observation using white light as the irradiation light.

14. A control device according to claim 11, wherein the endoscopic observation state includes at least a state corresponding to the observation distance, and the processor is characterized in that, when classified into the class corresponding to the distant observation distance, it weakens the enhancement level of the structure enhancement processing and the enhancement level of the color enhancement processing compared to when classified into the class corresponding to the close observation distance.

15. A control device according to claim 11, wherein the endoscopic observation state includes at least a state corresponding to the magnification, and the processor is characterized in that, when classified into the class corresponding to a second magnification higher than the first magnification, it increases the emphasis level of the structure enhancement processing compared to when classified into the class corresponding to the first magnification.

16. An endoscopic system comprising: a control device according to claim 1; and an endoscope for capturing the endoscopic image.

17. A processing method that is capable of performing one or more enhancement processes and uses a trained model that has been trained to classify classes corresponding to the enhancement level of each enhancement process according to the endoscopic observation state from training images, characterized in that: an endoscopic image is acquired; the trained model is used to classify the classes from the endoscopic image; and an enhancement level changing process is performed to change the enhancement level of one or more enhancement processes based on the classified classes.

18. The processing method according to claim 17, characterized in that, based on the classified class, the breakdown of one or more enhancement processing applied to the endoscopic image and the enhancement level of the applied enhancement processing are changed, respectively.

19. The processing method according to claim 17, wherein one or more of the enhancement processing are either a structural enhancement processing or a color tone enhancement processing, or a plurality of enhancement processing including the structural enhancement processing and the color tone enhancement processing.

20. The processing method according to claim 19, wherein the endoscopic observation state includes at least a state corresponding to the observation distance, and the processing method is characterized in that when the state is classified into the class corresponding to the close observation distance, the enhancement level of the structure enhancement processing is increased compared to when the state is classified into the class corresponding to the distant observation distance.

Images