Endoscope system, focus control device for endoscope, and focus control method for endoscope

Abstract

An endoscope system (1) according to the present invention comprises: an imaging unit (26) that is provided at a distal end part (21a) of an endoscope (20) and images an object of interest; a treatment tool identification unit (14a) that identifies a treatment tool image in an image on the basis of image data acquired by the imaging unit; and a focus control unit (16) that controls the imaging unit to perform focus control. The focus control unit performs focus control for setting a focus target position by prioritizing a region including a specific site in the treatment tool image identified in the image.

Description

Endoscope system, endoscope focus control apparatus, and endoscope focus control method

[0001] The present invention relates to focus control of an imaging unit applied in an endoscope system, particularly in an endoscope included in an endoscope system, and is an invention relating to an endoscope system, an endoscope focus control apparatus, and an endoscope focus control method.

[0002] Conventionally, an endoscope system comprising: an endoscope that captures images of the inside of a subject to acquire image data; a processor that performs various types of image processing on image data acquired by the endoscope; a display device that displays the image data processed by the processor as a visible image or the like; and a storage device that records or stores the image data, is widely used, for example, in the medical field, industrial field, and the like.

[0003] Furthermore, in the medical field, various examinations and treatments targeting the inside of a living organ or the like using this type of endoscope system are widely performed. In an examination using a medical endoscope system (hereinafter referred to as endoscopic examination), for example, while advancing and retracting an insertion portion of an endoscope inserted into the lumen of a living organ or the like along the longitudinal direction of the lumen, imaging or observation of the inside of the lumen is performed by an imaging unit provided at the distal end portion of the insertion portion.

[0004] It should be noted that the imaging unit (imaging section) of an endoscope only needs to be provided near the distal end of a tubular insertion portion. That is, the imaging unit does not need to be strictly provided at the distal end of the tubular insertion portion, and the present invention can also be applied to an endoscope having an imaging unit provided at a position closer to the side surface near the distal end of the tube (such as a so-called side-viewing endoscope).

[0005] At this time, images based on endoscopic image data acquired by the imaging unit are sequentially displayed as moving images on a display device in chronological order in real time. At the same time, the moving image data can also be stored in a storage device.

[0006] It is generally known that when imaging or observing an object with the imaging optical system of an imaging unit in close proximity (hereinafter abbreviated as close-up imaging), the depth of field becomes shallower (the range of focus in the front-to-back direction along the optical axis of the imaging optical system becomes narrower).

[0007] Therefore, in close-up imaging, for example, if the object being observed has an extension in the front-to-back direction along the optical axis of the imaging optical system, or if an object with an uneven shape is observed from above, the range of focus becomes limited. Under such circumstances, it may not be possible to clearly image or observe the entire image of the object.

[0008] Furthermore, for example, when performing a prescribed procedure on a specific object using a treatment tool, the treatment tool will be captured within the image.

[0009] It is generally known that instruments and other objects appear as clearer objects (high-contrast images) than surfaces of biological tissues, etc. Therefore, for example, when observing with a focus target set on the surface of biological tissue (e.g., an area containing a specific object), if an instrument or other object enters the imaging frame, the instrument or object may be recognized as the object of focus. In this case, there is a possibility that the focus target setting may be changed.

[0010] In normal circumstances, even if a medical device enters the imaging frame, it is preferable to continue observing the specific object to which the medical device is intended to perform treatment until the device reaches that specific object.

[0011] Therefore, various endoscopic systems capable of simultaneously identifying a specific object and its surrounding area have been proposed, for example, in references 1 and 2.

[0012] Reference 1, etc., discloses an endoscopic system that identifies lesions in an image, generates a boundary line (demarcation line) for the lesion, generates a virtual incision line at a specified distance from the boundary line, and controls the system to superimpose the generated boundary line and virtual incision line onto the image.

[0013] Reference Document 2, etc., discloses an endoscopic system that screens for potentially diseased areas based on endoscopic images, controls the imaging direction based on the location of the screened potentially diseased areas, and displays the screened potentially diseased areas in magnified view.

[0014] Japanese Patent Publication No. 2023-5896 Japanese Patent Publication No. 2024-20868

[0015] However, the aforementioned cited document 1 only discloses the display of a virtual indicator line relating to the specific object to be treated, and does not consider the situation when performing focus adjustment control or focus control (hereinafter simply referred to as focus control).

[0016] However, considering that the focus range narrows when imaging at close range to a specific object, it is desirable to have a configuration that performs appropriate focus control in order to acquire a clear image even when imaging at close range.

[0017] Furthermore, the aforementioned cited document 2 only discloses how to efficiently locate specific objects, and does not consider the situation when treatment is performed using treatment tools, etc.

[0018] However, in recent years, medical procedures using endoscopes have expanded beyond mere observation to include various treatments for specific objects discovered.

[0019] Therefore, in endoscopic systems including endoscopes equipped with imaging units capable of focus control, when used in conjunction with treatment instruments, there is always a need to observe, in addition to properly observing the specific object being observed and treated, a clear image of a specific area of ​​the object and a specific part of the treatment instrument, depending on the type of treatment instrument acting on the object.

[0020] The present invention aims to provide an endoscope system including an endoscope equipped with an imaging unit capable of performing focus control, which allows for appropriate focus control according to the treatment instrument used on the specific object when observing a specific object and performing various treatments on the specific object, thereby enabling the acquisition or observation of a region including the desired area with a clear and good image at all times, as well as a focus control device for the endoscope and a method for controlling the focus of the endoscope.

[0021] To achieve the above objective, an endoscope system according to one aspect of the present invention comprises an imaging unit provided at the tip of the endoscope for imaging an object, a treatment instrument identification unit for identifying a treatment instrument image in the image based on image data acquired by the imaging unit, and a focus control unit for controlling the imaging unit to perform focus control, wherein the focus control unit performs focus control to set a focus target position prioritizing a region including a specific part of the treatment instrument image identified in the image.

[0022] A focus control device for an endoscope according to one aspect of the present invention comprises: a treatment instrument identification unit that identifies a treatment instrument image in an image based on image data acquired by an imaging unit provided at the tip of the endoscope for imaging an object; and a focus control unit that controls the relative position between the imaging optical system and the image sensor of the imaging unit to perform focus control, wherein the focus control unit performs focus control to set a focus target position prioritizing a region including a specific part of the treatment instrument identified in the image.

[0023] A first focus control method for an endoscope according to one aspect of the present invention involves imaging an object to acquire image data, identifying a treatment instrument image within the image based on the image data, and performing focus control to set a focus target position prioritizing a region containing a specific part of the treatment instrument image identified within the image.

[0024] A second focus control method for an endoscope according to one aspect of the present invention comprises the steps of: acquiring image data from an imaging unit provided at the tip of the endoscope from which a treatment instrument can protrude and which images an object; identifying a treatment instrument from the image patterns contained in the image data to identify the presence or absence of treatment instrument protrusion and the treatment instrument state that can be classified as when the treatment instrument is in operation; and performing focus control by controlling the relative position of the imaging optical system and the image sensor of the imaging unit, wherein the focus control step prioritizes the region containing the treatment instrument tip image pattern when the treatment instrument is in operation in the image data where the treatment instrument is in operation, compared with image data where the treatment instrument is not protruding, and sets the focus target position.

[0025] A third focus control method for an endoscope according to one aspect of the present invention comprises the steps of: acquiring image data sequentially in a time series from an imaging unit provided at the tip of the endoscope from which a treatment instrument can be extended and which images an object; detecting the process state of the treatment instrument being extended or retracted by using the sequentially obtained time series image data to determine a protruding image change pattern in which the image data protrudes from the edge of the screen toward the center of the screen and spreads in the direction of said protrusion, but spreads less in the direction perpendicular to the direction of said protrusion; detecting a treatment instrument extension stopped state in which the position change of the tip of the treatment instrument has decreased; and performing focus control by controlling the relative position of the imaging optical system and the image sensor of the imaging unit, wherein the focus control step compares the treatment instrument extension stopped state with the process state of the treatment instrument being extended or retracted in the image and performs focus control by prioritizing the tip of the treatment instrument in the image to set the focus target position.

[0026] A fourth focus control method for an endoscope according to one aspect of the present invention comprises the steps of: acquiring image data sequentially in a time series from an imaging unit provided at the tip of the endoscope from which a treatment instrument can protrude and which images an object; detecting whether or not a treatment instrument is protruding by determining the change in the distribution of bright spots in a time series using the sequentially obtained image data; identifying an image pattern related to a treatment instrument that can be detected in at least one of the sequentially obtained image data; and performing focus control by controlling the relative position of the imaging optical system and the image sensor of the imaging unit, wherein the focus control step prioritizes the region containing the image pattern of the tip of the treatment instrument when the treatment instrument is in action in the image data where the treatment instrument is in action, compared with image data where the treatment instrument is not protruding, and sets the focus target position.

[0027] A fifth focus control method for an endoscope according to one aspect of the present invention comprises the steps of: acquiring image data sequentially in a time series from an imaging unit provided at the tip of the endoscope from which a treatment instrument can be extended and which images an object; detecting the process of retracting the treatment instrument by determining the change in the distribution of bright spots in a time series using the sequentially obtained image data; identifying an image pattern related to the treatment instrument that can be detected in at least one of the sequentially obtained image data; and performing focus control by controlling the relative position of the imaging optical system and the image sensor of the imaging unit, wherein the focus control step maintains the result of setting a focus target position by prioritizing the region that includes the image pattern of the tip of the treatment instrument at the time of treatment instrument operation in the image data detected as the retraction process of the treatment instrument extension in the image.

[0028] According to the present invention, an endoscope system including an endoscope equipped with an imaging unit capable of performing focus control is provided, which allows for appropriate focus control according to the treatment instrument used on the specific object when observing a specific object and performing various treatments on the specific object, and enables imaging or observation of a region including the part to be observed with a clear and good image at all times. The present invention also provides a focus control device for an endoscope and a focus control method for an endoscope.

[0029] Figure 1 shows a schematic diagram of an endoscope system according to one embodiment of the present invention, a block diagram showing the detailed configuration of the processor in the endoscope system according to one embodiment of the present invention, an example of the display screen of the display device in the endoscope system of Figure 1, a first schematic diagram showing several typical types of treatment tools used in the endoscope system of Figure 1 and their respective usage situations, a second schematic diagram showing several typical types of treatment tools used in the endoscope system of Figure 1 and their respective usage situations, a schematic diagram showing the procedure for endoscopic mucosal resection (EMR) performed using the endoscope system of Figure 1, a schematic diagram showing the procedure for endoscopic submucosal dissection (ESD) performed using the endoscope system of Figure 1, a schematic diagram showing typical auxiliary tools used in the endoscope system of Figure 1, a flowchart explaining the operation of the first form of the endoscope system according to one embodiment of the present invention, a diagram showing specific examples of the operating areas of various treatment tools used in the endoscope system of Figure 1 in tabular form, a flowchart explaining the operation of the second form of the endoscope system according to one embodiment of the present invention, and specific examples of the operating areas of various treatment tools used in the endoscope system of Figure 1. Figures shown in table format, schematic diagrams showing the operating area or site of action identified by the instruments used in the endoscopic system of Figure 1 (examples of biopsy forceps or hemostatic forceps) and the corresponding focus targets, schematic diagrams showing the operating area or site of action identified by the instruments used in the endoscopic system of Figure 1 (example of hemostatic clip (separation type)) and the corresponding focus targets, another schematic diagram showing the operating area or site of action identified by the instruments used in the endoscopic system of Figure 1 (example of hemostatic clip (separation type)) and the corresponding focus targets, Figure 1 A schematic diagram showing the working area or site of action identified by the treatment instruments used in the endoscopic system (example: injection needle) and each corresponding focus target; a schematic diagram showing the working area or site of action identified by the treatment instruments used in the endoscopic system in Figure 1 (example: snare) and each corresponding focus target; a schematic diagram showing the working area or site of action identified by the treatment instruments used in the endoscopic system in Figure 1 (example: tip hood or transparent cap) and each corresponding focus target; and a flowchart showing the operation of creating an inference model in the inference model creation device.

[0030] The present invention will be described below with reference to the illustrated embodiments. The drawings used in the following description are schematic. Therefore, in these drawings, each component is shown at a size that is recognizable on the drawing. For this reason, the dimensional relationships and scales of each component may differ on the drawing. The present invention is not limited to the illustrated forms with respect to the quantity, shape, size ratio, relative positional relationships, etc., of each component shown in each drawing. [One Embodiment]

[0031] The schematic configuration of an endoscope system according to one embodiment of the present invention will be described below with reference to Figures 1 and 2. Figure 1 is a diagram showing the schematic configuration of an endoscope system according to one embodiment of the present invention. Figure 2 is a block diagram showing the detailed configuration of the processor in the endoscope system according to one embodiment of the present invention.

[0032] First, the general configuration of the endoscope system 1 of this embodiment will be described. As shown in the figure, the endoscope system 1 is configured to include a processor 10, an endoscope 20, a display device 30, a light source device 40, a storage device 50, an inference model creation device 60, and the like.

[0033] The endoscopic system 1 illustrated in this embodiment exemplifies a general configuration of an endoscopic system used for endoscopic examinations performed to observe the inside of organs such as the upper digestive tract (esophagus, stomach, duodenum, etc.) or lower digestive tract (large intestine, etc.) of a subject (patient, etc.) such as a living organism.

[0034] The endoscope 20 is an imaging inspection device comprising an insertion section 21, an operating section 22, a universal cable 23, and the like.

[0035] The insertion portion 21 is a component that is inserted into a subject such as a living organism. The insertion portion 21 is formed by continuously connecting a tip portion 21a, a curved portion 21b, and a flexible tube portion 21c in order from the tip side.

[0036] The insertion section 21 is formed in a generally elongated tube shape. The insertion section 21 has an internal instrument insertion channel 21d, which is a conduit for inserting endoscopic instruments, etc. (hereinafter simply referred to as "instruments, etc.") 27. This instrument insertion channel 21d is provided to extend from the tip to the base of the insertion section 21.

[0037] The tip portion 21a is a component unit provided at the very tip of the insertion portion 21. Various components such as an illumination unit 25 and an imaging unit 26 (not shown in Figure 1; see Figure 2) are provided inside the tip portion 21a.

[0038] The illumination unit 25 is a component unit that includes optical elements (illumination lens including illumination window 25b) that emit a light beam guided from the light source device 40 (described later) forward from the tip surface of the tip portion 21a to illuminate the observation target area, including specific objects (hereinafter referred to as "specific objects") such as lesions such as polyps or tumors within the subject. The illumination unit 25 is driven and controlled by the illumination control unit 17 (described later) of the processor 10.

[0039] The imaging unit 26 is an electronic device unit consisting of an optical lens (imaging optical system 26a including observation window 26aa; see Figure 2) that forms an optical image of a specific object to be observed inside the subject, and a photoelectric conversion element (image sensor 26b; see Figure 2) that generates image data (still image data and moving image data, etc.) based on the optical image.

[0040] The image data acquired by the imaging unit 26 is output to the image processing unit 11 (described later) of the processor 10, where various signal processing is performed.

[0041] Furthermore, the imaging optical system 26a of the imaging unit 26 is driven and controlled by the focus control unit 16 of the processor 10 to perform focus adjustment control (also called focus control).

[0042] Here, the imaging unit 26 is sometimes also called the imaging section. This imaging unit 26 includes an imaging optical system and an image sensor, and focus control is performed by controlling the relative position in the optical axis direction between them (the imaging optical system and the light-receiving surface of the image sensor).

[0043] A distal opening 21da of a treatment instrument insertion channel 21d, an illumination window 25b, an observation window 26aa and the like are provided on the distal end surface of the distal end portion 21a.

[0044] Here, the distal opening 21da of the treatment instrument insertion channel 21d is an opening through which the distal end side of a treatment instrument 27 (to be described in detail later) inserted into the treatment instrument insertion channel 21d protrudes. The illumination window 25b is a window member that emits illumination light from an illumination unit 25. The observation window 26aa is a window member that transmits an incident port to an imaging unit 26. An auxiliary tool 28 (transparent cap) may be attached to the distal end portion 21a as needed (see FIG. 1). Here, as the auxiliary tool 28, there are various forms such as a distal hood, a transparent cap, and the like, for example.

[0045] The bending portion 21b is a tubular portion configured to be actively bendable by a bending operation mechanism (not shown) that operates in conjunction with the operation of a bending operation member 22b described later.

[0046] The flexible tube portion 21c is a tubular member extending from the distal end of the operation portion 22 and connected to the proximal end of the bending portion 21b. The insertion portion 21 formed in this manner has the operation portion 22 connected to the proximal end side thereof.

[0047] The operation portion 22 is a constituent unit connected to the proximal end of the insertion portion 21. The operation portion 22 includes an operation portion main body 22a, the bending operation member 22b, a plurality of operation members 22c, a treatment instrument insertion port 22d and the like.

[0048] The operation portion main body 22a has a substantially box shape as a whole, and constitutes a grip portion gripped by a user (such as a doctor) of the endoscope 20 with fingers. Various constituent units such as a bending operation mechanism (not shown) are provided inside the operation portion main body 22a. As described above, the insertion portion 21 extends from the operation portion main body 22a.

[0049] The bending operation member 22b and the plurality of operation members 22c are operation members for performing various operations of the endoscope 20. The bending operation member 22b and the plurality of operation members 22c are respectively provided at predetermined positions on the outer surface of the operation portion main body 22a.

[0050] The instrument insertion port 22d is provided at a predetermined position near the tip of the operating unit body 22a. The instrument insertion port 22d is the proximal end opening of the instrument insertion channel 21d of the insertion section 21. The instrument insertion channel 21d is connected at the tip end to the tip opening 21da of the tip section 21a.

[0051] With this configuration, the treatment instrument 27 inserted through the treatment instrument insertion port 22d is configured to protrude outward and forward from the tip opening 21da of the tip portion 21a after being inserted through the treatment instrument insertion channel 21d.

[0052] The universal cable 23 is a connection cord for connecting the endoscope 20 to the light source device 40 and the processor 10. For this purpose, the universal cable 23 consists of a tubular member extending from the side of the operating unit body 22a of the operating unit 22. A scope connector 23a is provided at the tip of the universal cable 23. This scope connector 23a is connected to the front panel of the light source device 40.

[0053] An electrical cable 23b extends from the scope connector 23a. A connector 23c is provided at the end of this electrical cable 23b. This connector 23c is connected to the front panel of the processor 10. Various signal transmission cables and optical fiber cables (not shown) are inserted into the universal cable 23.

[0054] The light source device 40 is a device that supplies illumination light to an illumination unit 25 (see Figure 2) located inside the tip 21a of the insertion section 21 of the endoscope 20. The illumination light emitted from the light source device 40 is transmitted to the illumination unit 25 in the tip 21a via an optical fiber cable (not shown) inserted from the scope connector 23a through the universal cable 23, the operating section 22, and the insertion section 21. The illumination light then passes through an illumination lens (not shown) included in the illumination unit 25 in the tip 21a and is irradiated toward the observation target area in front of the tip 21a. The light source device 40 is driven and controlled by the illumination control unit 17 of the processor 10.

[0055] The processor 10 is a control device, signal processing device, or circuit unit that includes control circuits and signal processing circuits that control the entire endoscope system 1.

[0056] The control circuit included in the processor 10 receives, for example, an operation instruction signal from the operating member 22c of the operating section 22 of the endoscope 20 and outputs various control signals for driving and controlling the imaging unit 26, the light source device 40, or the illumination unit 25, etc. The signal processing circuit included in the processor 10 also receives, for example, an imaging signal from the imaging unit 26 and performs predetermined image signal processing, etc.

[0057] To this end, the processor 10 and the imaging unit 26 are electrically connected by a signal transmission cable (not shown). This signal transmission cable is routed from connector 23c through electrical cable 23b, scope connector 23a, universal cable 23, operation unit 22, insertion unit 21 to the imaging unit 26 at the tip 21a.

[0058] With this configuration, control signals output from the processor 10 and imaging signals output from the imaging unit 26 are transmitted between the imaging unit 26 and the processor 10 via the signal transmission cable. One form of the signal transmission cable is, for example, a composite cable in which multiple cables are bundled together and covered with an outer shield, outer tube, etc.

[0059] The display device 30 receives image signals and the like output from the processor 10 and displays endoscopic images in a predetermined format and various types of information. For this purpose, the display device 30 and the processor 10 are electrically connected using a video cable 24. The display device 30 is driven and controlled by the display control unit 12 (described later) of the processor 10. The display device 30 can be, for example, a display device made using a general liquid crystal panel.

[0060] The storage device 50 is a storage device that stores (records or stores) storage image data generated for storage by the processor 10 after various processes have been performed on the image information generated by the imaging unit 26.

[0061] As shown in Figure 1, the storage device 50 in the endoscope system 1 of this embodiment is disposed inside the processor 10 and is configured integrally with the processor 10. Inside the storage device 50, as shown in Figure 2, a temporary storage unit 51 is provided separately from the main memory unit (not shown) of the storage device 50. This temporary storage unit 51 is configured, for example, with semiconductor memory and functions as a temporary memory area that temporarily stores predetermined image data output from the image processing unit 11.

[0062] In the example configuration shown in Figure 1, the storage device 50 is shown as being integrally disposed inside the casing of the processor 10, but the configuration is not limited to this. For example, the storage device 50 may be configured as an external storage device using a separate casing from the processor 10.

[0063] Furthermore, although the example shown in Figure 2 illustrates a configuration in which the temporary storage unit 51 is located inside the storage device 50, the configuration is not limited to this. The temporary storage unit 51 can also be configured to be integrated inside, for example, the storage control unit 13 (see Figure 2), which will be described later.

[0064] Furthermore, the processor 10 and the light source device 40 are not limited to being separate components, as illustrated in Figure 1. For example, the processor 10 and the light source device 40 may be integrated into a single housing.

[0065] The configuration of the lighting unit 25 is not limited to the above-described configuration (a configuration in which illumination light from the light source device 40 is transmitted to the tip portion 21a via an optical fiber cable or the like). Other configurations of the lighting unit 25 include, for example, a configuration in which a light-emitting element such as an LED (Light Emitting Diode) is provided inside the tip portion 21a as an illumination light source, and the power supply to the illumination light source (LED) and the control of its emission are controlled by a predetermined control circuit included in the processor 10.

[0066] Furthermore, the endoscope system 1 of this embodiment includes an inference model creation device 60. In the endoscope system 1 of this embodiment, the inference model creation device 60 is configured as an external device and is shown as being provided separately from the housing of the processor 10 and the like.

[0067] The inference model creation device 60 is a device and circuit unit for creating an inference model to identify what kind of procedure is being performed and what kind of procedure it is, based on information about various procedures (hereinafter abbreviated as endoscopic procedures) performed using the procedure instruments 27, information about the type of procedure instruments, and information about the operating area or site of action of the procedure instruments when they are used.

[0068] The inference model creation device 60 has an endoscopic procedure DB 61 (where DB means database; the same applies hereinafter). In this embodiment, the endoscopic procedure DB 61 is shown as being included inside the inference model creation device 60. However, the endoscopic procedure DB 61 is not limited to the example configuration. For example, the endoscopic procedure DB 61 may be configured as an external database (DB) device separate from the inference model creation device 60, and the endoscopic procedure DB 61 and the inference model creation device 60 may be connected by a connecting cable, wireless communication, or a network. The operation of the inference model creation device 60 will be described later (see Figure 19).

[0069] Furthermore, if the endoscopic treatment DB61 is configured as a separate external DB device, the external DB device may also include the treatment instrument DB14x described later.

[0070] The other components of the endoscope system 1, configured as described above, are basically the same as those of a conventional endoscope system of the same type. Therefore, detailed explanations and illustrations of the other components are omitted. [Processor]

[0071] Next, the internal configuration of the processor 10 in the endoscope system 1 will be described in detail using Figure 2.

[0072] The processor 10 is configured to include an image processing unit 11, a display control unit 12, a storage control unit 13, an image identification unit 14, a focus control unit 16, an illumination control unit 17, and a storage device 50 including a temporary storage unit 51.

[0073] The image processing unit 11 is a component unit or circuit unit that receives image information output from the imaging unit 26 and performs various information processing based on said image information. The image processing performed here includes, for example, normal image information processing, display image data processing, storage image data processing, and various other image processing.

[0074] Normal image information processing includes AD conversion processing, which receives the analog image signal output from the image sensor 26b and converts it into a digital image signal, and is basic image information processing that is normally performed on image information. Normal image information processing is assumed to be the application of conventionally common processing methods, and a detailed explanation thereof is omitted.

[0075] Display image data processing is the process of generating image data for normal display. Display image data is, for example, the process of generating image data representing an endoscopic image to be displayed in a predetermined display area on the display screen of the display device 30.

[0076] Figure 3 shows an example of the display screen of the display device in the endoscope system of this embodiment. In Figure 3, reference numeral 100 indicates the display screen frame. Reference numeral 101 in Figure 3 is the display area of ​​the endoscope image 101a. Reference numeral 103 in Figure 3 is an information display area for displaying information other than the image (for example, text information or icon information). Reference numeral 104 in Figure 3 is an information display area that is displayed at appropriate timings to display necessary predetermined information (for example, notification display, warning display, status notification display, etc.). Note that the information display area 104 is normally hidden.

[0077] The example shown in Figure 3 is a typical endoscopic image observed during an examination screening of a subject's body cavity using the endoscopic system 1. In the endoscopic image 101a illustrated in Figure 3, a dark area 101c is present in the central region. This dark area 101c is the lumen of an organ, etc. 300, and corresponds to the anterior region of the endoscope 20.

[0078] Generally, when the endoscope 20 is in a screening state, a lumen often extends in the anterior region of the endoscope 20. In this case, there is an area far from the anterior region of the endoscope 20 where the illumination light from the illumination unit 25 does not reach sufficiently.

[0079] Therefore, in such situations, a dark area 101c often exists within a predetermined range in the central region of the endoscopic image 101a. In addition, a bright area 101d exists in the area surrounding the dark area 101c in the endoscopic image 101a. Here, the bright area 101d is the wall surface inside the body cavity. Capillaries 101e, for example, can be observed on this wall surface inside the body cavity. In the example of displaying the endoscopic image 101a in Figure 3, it is shown that a specific object image 101b exists within a part of the range of the bright area 101d. The endoscopic image 101a shown in Figure 3 is an example of displaying such a situation. Furthermore, the memory image data processing is the process of generating image data for recording, etc.

[0080] Various image processing techniques are applied to regular display image data as appropriate. Specific examples of these image processing techniques include standard image adjustments such as brightness adjustment and white balance adjustment, as well as image enhancements such as edge enhancement and texture and color enhancement (TXI; Texture and Color Enhancement Imaging).

[0081] The display control unit 12 is a component unit or circuit unit that controls the display device 30 so that it displays in an appropriate display format according to the display image data output from the image processing unit 11. In addition to controlling the display of the display image data (various settings such as display position, display area, and display size), the display control unit 12 also controls the display of various information other than image information (text information, icon information, etc.). Here, the various information other than image information includes examination date and time information, subject (patient) information, medical record information, and various information that is displayed as appropriate during an endoscopic examination (notification display, alert display, status display, etc.).

[0082] The memory control unit 13 is a component unit or circuit unit that controls the storage device 50 (including the temporary storage unit 51) so that storage (recording or memory) is performed in an appropriate storage format according to the storage image data output from the image processing unit 11.

[0083] The memory control unit 13 performs memory control to permanently store image data representing the endoscopic images acquired by the endoscope 20 in the memory area of ​​the memory device 50.

[0084] The image identification unit 14 is a component unit or circuit unit that detects an image region containing a predetermined object (e.g., a treatment instrument) from an endoscopic image based on image data acquired by the endoscope 20, and identifies the predetermined object. In this embodiment, the image identification unit 14 is shown as an example configuration having, for example, a treatment instrument identification unit 14a, a treatment instrument DB 14x, and an inference unit 14z.

[0085] Of these, the treatment instrument identification unit 14a detects an image region (hereinafter referred to as the treatment instrument image) in which a treatment instrument 27 is captured from the endoscopic image based on image data, and identifies the treatment instrument 27 included in the image region.

[0086] In this case, the treatment tool identification unit 14a, for example, performs similar image recognition by referring to the treatment tool DB 14x, or uses the inference unit 14z to perform data analysis and detect an image region containing the treatment tool image.

[0087] The treatment tool DB14x is a database that stores various information (for example, type and detailed specifications) about multiple types of treatment tools, etc. 27.

[0088] The inference unit 14z is a component unit or circuit unit that performs image data analysis using deep learning and machine learning to detect images of medical devices.

[0089] In recent years, AI (Artificial Intelligence) technology has been developed to detect objects by applying machine learning to endoscopic images acquired by endoscopes, thereby obtaining desired inference results (for example, detection of specific objects such as tumors or treatment instruments).

[0090] Machine learning learns the features, time-series information, spatial information, etc., of known input information, and then performs inferences based on the learning results to obtain inference results about unknown things. In other words, in machine learning, a trained model (also called an inference model) is first obtained that makes it possible to infer a determinable output result from specific input information.

[0091] In this case, when generating a trained model, a large amount of information with known input-output relationships is used as training data to obtain highly reliable inference results. For example, in deep learning, the network is designed using a large amount of training data so that the expected output can be obtained for known inputs. A trained model obtained through such a process can be used independently of the network that performed the training.

[0092] Therefore, by providing the processor 10 with this type of inference model, it becomes possible to accurately and quickly perform various judgments or detections, such as the detection of treatment instrument images in the image recognition unit 14, as well as the determination of the operating status of treatment instruments during endoscopic examinations or processing.

[0093] Deep learning is a multi-layered structure of the machine learning process using neural networks. A typical example is the "feedback neural network," which sends information from front to back to make a decision. In its simplest form, this requires only three layers: an input layer consisting of N1 neurons, a hidden layer consisting of N2 neurons given by parameters, and an output layer consisting of N3 neurons corresponding to the number of classes to be classified. Each neuron in the input layer and the hidden layer, and the hidden layer and the output layer, are connected by connection weights, and a bias value is added between the hidden layer and the output layer, making it easy to form logic gates. Three layers are sufficient for simple classification, but by using many hidden layers, it becomes possible to learn how to combine multiple features during the machine learning process. In recent years, networks with 9 to 152 layers have become practical due to their relationship with learning time, classification accuracy, and energy consumption.

[0094] Various publicly known networks can be used as the network for machine learning. For example, R-CNN (Regions with CNN features) or FCN (Fully Convolutional Networks), which utilize CNN (Convolutional Neural Network), can be used. These involve a process called "convolution" to compress image features, operate with minimal processing, and are strong in pattern recognition. Furthermore, to handle more complex information and to support information analysis where the meaning changes depending on the order and sequence, a "recurrent neural network" (fully connected recurrent neural network), where information flows bidirectionally, can be used.

[0095] To realize these technologies, conventional general-purpose computing circuits such as CPUs and FPGAs can be used, but since much of the processing in neural networks involves matrix multiplication, specialized GPUs (Graphics Processing Units) and Tensor Processing Units (TPUs) are sometimes used. In recent years, these artificial intelligence (AI) dedicated hardware, called "Neural Network Processing Units (NPUs)," have been designed to be integrated and embedded together with CPUs and other circuits, and are sometimes part of the processing circuit.

[0096] Furthermore, inference models may be obtained using various publicly known machine learning techniques, not limited to deep learning. For example, there are methods such as support vector machines and support vector regression. The learning here involves calculating the classifier weights, filter coefficients, and offsets, but there are also methods that utilize logistic regression. When a machine is to make a judgment, a human needs to teach the machine how to make the judgment. In this example, a method was adopted to derive image judgment using machine learning, but other rule-based methods that apply rules acquired by humans through experience and heuristics may also be used to make specific judgments.

[0097] The focus control unit 16 shown in Figure 2 is a component unit or circuit unit that drives and controls the imaging optical system 26a of the imaging unit 26. The focus control unit 16 drives and controls the imaging optical system 26a based on the identification result of the treatment instrument identification unit 14a of the image identification unit 14, and switches to an appropriate focus control according to the identified treatment instrument, etc. 27.

[0098] Normally, when automatic focus control (hereinafter referred to as autofocus control) is performed, the focus control is generally performed targeting an object surrounded by a focus frame (not shown), which is an indicator that shows the focus target position or the in-focus position and is displayed in a predetermined area approximately in the center of the imaging screen (display screen frame 100; see Figure 3).

[0099] In addition to this autofocus control, in the endoscope system 1 of this embodiment, if the displayed imaging screen includes an image of, for example, a treatment instrument, the system focuses on the treatment instrument, identifies its type, and prioritizes setting a focus frame (not shown) in a predetermined area according to the operating status of the treatment instrument to perform autofocus control (details will be described later).

[0100] The lighting control unit 17 is a component unit or circuit unit that drives and controls the light source device 40 and the lighting unit 25. The lighting control unit 17 controls the light source device 40 or the lighting unit 25 to switch between light source types (white light, special light, etc.) according to operation instruction signals from various operating members 22c provided on the operation unit 22, or the identification result of the treatment tool identification unit 14a of the image identification unit 14 (type of treatment tool, operating status, etc.).

[0101] Here, the types of light sources include, for example, white light irradiated onto the subject during normal observation, as well as specific special lights irradiated onto the subject when performing image-enhanced observation such as narrow-band imaging (NBI) or red dichromatic imaging (RDI).

[0102] Note that the processor 10 includes hardware in whole or in part. Here, the processor 10 is composed of a well-known configuration and peripheral devices, such as a central processing unit (CPU), RAM (Random Access Memory), ROM (Read Only Memory), non-volatile memory, non-volatile storage, and a non-transitory computer-readable medium.

[0103] ROM, non-volatile memory, and non-volatile storage devices pre-store software programs executed by the CPU, as well as fixed data such as data tables. The CPU reads the software programs stored in the ROM, loads them into RAM, and executes them. The functions of the processor 10 are realized when these software programs appropriately refer to various data.

[0104] Furthermore, the processor 10 may be composed of semiconductor chips such as FPGAs (Field Programmable Gate Arrays). In addition, the processor 10 may be composed of electronic circuits.

[0105] Furthermore, the software program may be recorded or stored in whole or in part as a computer program product on portable disk media such as flexible disks, CD-ROMs, DVD-ROMs, or non-transitory computer-readable media such as card-type memory, HDDs (Hard Disk Drives), and SSDs (Solid State Drives). The program is then read by a computer, and all or part of its operation is executed. Alternatively, the program, in whole or in part, can be distributed or provided via a communication network. Users can easily implement the endoscopic system of the present invention by downloading and installing the program on their computer via the communication network, or by installing it on their computer from a recording medium. [Procedure tools, etc.]

[0106] Next, the types and specifications of the treatment instruments and other items used in the endoscope system 1 of this embodiment will be described below, along with simple illustrations.

[0107] Figures 4 and 5 are schematic diagrams illustrating several typical treatment instruments used in endoscopic systems and their respective usage scenarios. Here, Figure 4 is the first schematic diagram, and Figure 5 is the second schematic diagram.

[0108] In Figures 4 and 5, reference numeral 27 indicates a treatment instrument, etc., regardless of type. Reference numeral 200 in Figures 4 and 5 indicates biological tissue, etc. Reference numeral 201 in Figure 4 indicates, for example, a specific target object (polyp, etc.). Reference numeral 202 in Figure 4 indicates a foreign object, etc., that has entered a body cavity (for example, a button battery, etc.). Reference numeral 203 in Figure 4 indicates a bleeding site. Reference numeral 204 in Figure 5 indicates lithotomy, etc. Reference numeral 205 in Figure 5 indicates a specific target object (lesion, etc.). Reference numeral 206 in Figure 5 indicates saline solution or medication after injection. Reference numeral 200a in Figure 5 indicates the mucosal layer on the surface of biological tissue, etc. 200. Reference numeral 200b in Figure 5 indicates the submucosa. Reference numeral 200c in Figure 5 indicates the muscularis propria. Reference numeral 207 in Figure 5 indicates marking with a high-frequency knife. Reference numeral 208 in Figure 5 indicates traces of incision or dissection with a high-frequency knife.

[0109] Generally, the treatment instruments 27 that can be used in the endoscopic system 1 (endoscope 20) include, for example, pinch type, annular hook type, capture type, high-frequency knife type, needle type, and the like.

[0110] Pinch-type instruments are used to grasp or hold biological tissue, etc., for collection and retrieval. They can also be used to pinch specific objects and cauterize their bases. Alternatively, they can be used to cauterize bleeding areas to stop bleeding.

[0111] Specifically, pinch-type instruments include biopsy forceps (Figure 4(A)), hemostatic forceps (non-separable type; high frequency) (Figure 4(A)), grasping forceps (Figure 4(B)), and hemostatic clips (separable type; Figure 4(C)).

[0112] The biopsy forceps (Figure 4(A)) are used to grasp or pinch a specific target object 201, such as a polyp, on the living tissue, and while maintaining that position, the tissue is thermally coagulated using high-frequency current.

[0113] Grasping forceps (Figure 4 (B)) are used to grasp or pinch and remove foreign objects that have entered a body cavity.

[0114] In a detachable hemostatic clip (Figure 4(C)), for example, the clip 27a grasps the bleeding site 203 of the living tissue 200, maintains the grasping state, detaches the clip 27a, and places this clip 27a on the mucous membrane of the living tissue 200. The placed clip 27a is usually expelled naturally within a few days.

[0115] In the case of non-separable hemostatic forceps (Figure 4(A)), for example, they grasp the bleeding site 203 in the biological tissue 200 in much the same way as a separable hemostatic clip (see Figure 4(C)). Then, with non-separable hemostatic forceps, the tissue is thermally coagulated by high-frequency current while maintaining this grasping state.

[0116] In pinch-type surgical instruments, the part that acts on a specific object, primarily when pinching or gripping biological tissue, is an area that includes part or all of the specific object itself.

[0117] Furthermore, annular hook-type treatment devices use annular wires to hook onto the base of a target object, constrict it, and then burn or cut it off with high-frequency current. Alternatively, annular hook-type treatment devices use looped nylon resin threads to ligate the base of a target object and stop bleeding.

[0118] Annular hook type treatment instruments include snares (Figure 4 (D) column) and ligation devices (separation type). Snares are used for specific targets such as relatively large lesions that cannot be grasped with biopsy forceps. There are two types of snares: those that use high-frequency current and those that do not.

[0119] Snares are used, for example, in endoscopic mucosal resection (EMR) and polypectomy, which will be discussed later.

[0120] Capture-type instruments capture and collect excised biological tissue or crushed lithotripsy. Examples of capture-type instruments include basket-type grasping forceps (Figure 5(A)) and collection nets (not shown).

[0121] Needle-type instruments are used to inject saline solution or medication into the submucosal layer of the body. Alternatively, needle-type instruments are used to puncture living tissue and collect specimens.

[0122] Needle-type treatment instruments include local injection needles (local injection needles; Figure 5 (B) column) and aspiration biopsy needles. Local injection needles are used, for example, in endoscopic mucosal resection (EMR) and endoscopic submucosal dissection (ESD), which will be described later.

[0123] In surgical instruments such as snare-type annular hooks and needle-type instruments, the area that acts on a specific target on biological tissue is primarily the area near the boundary line (demarcation line) with the target.

[0124] High-frequency knife-type instruments use high-frequency current to excise specific targets in biological tissue (Figure 5(C)). High-frequency knife-type instruments come in various forms. High-frequency knives are used, for example, in endoscopic submucosal dissection (ESD), which will be described later.

[0125] In high-frequency knife-type treatment instruments, the area that acts on a specific target object on biological tissue is primarily near the boundary line (demarcation line) with the target object, and is slightly outside of it.

[0126] Here, Figure 6 is a schematic diagram illustrating the procedure for endoscopic mucosal resection (EMR). The symbols used in Figure 6 are the same as those in Figures 4 and 5.

[0127] In endoscopic mucosal resection (hereinafter abbreviated as EMR), various types of treatment instruments such as local injection needles, snares, and high-frequency knives are used sequentially as needed.

[0128] First, we assume that a flat lesion 205 originating in the mucosal layer 200a has been detected by endoscopic screening (Figure 6(A)). Generally, lesions 205 treated with EMR are those up to approximately 2 cm in size.

[0129] Next, saline solution 206 is injected into the submucosal layer 200b between the lesion 205 and the muscularis propria 200c using a needle-type treatment instrument (such as an injection needle). This causes the lesion 205 to become raised (Figure 6(B)). Subsequently, a snare (treatment instrument 27) is placed over the raised lesion 205 (Figure 6(C)).

[0130] Next, the base of the lesion 205 is constricted with a snare (treatment instrument 27) (Figure 6(D)). Subsequently, a high-frequency current is passed through the snare (treatment instrument 27) to burn away the lesion (Figure 6(E)). The lesion 205a that has been excised in this way is recovered using a treatment instrument 27 such as a capture type or grasping forceps, although this is not shown in the figure. The lesion 205a recovered after excision is subjected to pathological examination, etc.

[0131] Figure 7 is a schematic diagram illustrating the procedure for endoscopic submucosal dissection (ESD). The symbols used in Figure 7 are the same as those in Figures 4 and 5.

[0132] In endoscopic submucosal dissection (hereinafter abbreviated as ESD), various types of treatment instruments, such as high-frequency knives, local injection needles, grasping forceps, and hemostatic forceps, are used sequentially as needed.

[0133] First, it is assumed that a lesion 205 originating in the mucosal layer 200a has been detected by endoscopic screening. A high-frequency knife (treatment instrument, etc. 27) is used to mark the area to be removed from the discovered lesion 205 (Figure 7(A)). Lesions 205 treated with ESD are typically relatively large ones, exceeding approximately 2 cm in size.

[0134] Next, the drug 206 is injected into the submucosal layer 200b between the lesion 205 and the muscularis propria 200c using a needle-type treatment instrument (such as an injection needle) (local injection). This causes the lesion 205 to float (Figure 7(B)). Subsequently, the mucosal layer 200a surrounding the lesion 205 is cut out using a high-frequency cutting knife (treatment instrument 27) to cut around the marked area (Figure 7(C)).

[0135] Next, the lesion 205 is peeled off and completely detached using a high-frequency knife (treatment instrument 27) (Figure 7(D)). Subsequently, the excised lesion 205a is recovered using treatment instruments 27 such as grasping forceps (Figure 7(E)). The recovered lesion 205a is then subjected to pathological examination.

[0136] Finally, if bleeding occurs, hemostasis should be performed using hemostatic forceps or other treatment instruments 27 (Figure 7(F)).

[0137] Incidentally, during endoscopic examinations, there are auxiliary devices 28 that are attached to the tip of the end section 21a, for example, and are a type of accessory that assists the treatment instruments 27. Examples of these auxiliary devices 28 include tip hoods and transparent caps. Figure 8 is a schematic diagram showing typical auxiliary devices used when performing procedures with various treatment instruments.

[0138] Of the auxiliary devices 28, the tip hood is attached to the tip of the endoscope and is an auxiliary device that prevents images outside the endoscopic field of view from appearing, thereby ensuring a good field of view (Figure 8(A)). The transparent cap is attached to the tip of the endoscope and is an auxiliary device that ensures a good field of view by securing the distance between the tip surface of the endoscope and the object (Figure 8(B)). [Function]

[0139] The operation of the endoscope system 1 of this embodiment, configured in this manner, will be described below. Figure 9 is a flowchart illustrating the operation of the first embodiment of the endoscope system of one embodiment of the present invention.

[0140] The flowchart shown in Figure 9 outlines the general operation when performing endoscopic examinations and procedures using the endoscope system 1. This flowchart in Figure 9 shows an overview of the first focus control process, in which the focus control unit performs focus control on a specific target object, prioritizing the area including the operating area of ​​a treatment instrument as the focus target.

[0141] When the first focus control shown in Figure 9 starts, in step S11, the processor 10 first checks whether the imaging unit 26 is running. If the imaging unit 26 is not running, the same check process is repeated until it is confirmed to be running. Once it is confirmed that the imaging unit 26 is running, the process proceeds to the next step, S12.

[0142] In step S12, the processor 10 performs predetermined imaging operation processing, temporary storage processing, and image change analysis processing.

[0143] Here, the imaging operation process is the process of acquiring image data by driving and controlling the imaging unit 26. The acquired image data is then subjected to predetermined image processing by the image processing unit 11.

[0144] The temporary storage process involves storing the processed image data in a predetermined storage unit of the storage device 50 via the storage control unit 13, and also temporarily storing a predetermined amount of data in the temporary storage unit 51.

[0145] Image change analysis processing is a process that identifies various objects (e.g., medical instruments, etc.) within an image based on a series of image data that has been temporarily stored over time, and analyzes the changes in the identified objects (medical instruments, etc.) over time.

[0146] These processes (image acquisition, temporary storage, and image change analysis) are executed continuously throughout the endoscopic examination.

[0147] Next, in step S13, the processor 10 performs an identification process for the treatment instrument 27 or auxiliary instrument 28 (hereinafter abbreviated as "treatment instrument") based on the image data through the image identification unit 14. If the treatment instrument is identified, the process proceeds to the next step S14. If the treatment instrument is not identified, the process proceeds to step S17.

[0148] Generally, during the process of inserting an endoscope into a body cavity, the treatment instrument 27 is kept in a state where it does not protrude forward from the tip surface of the endoscope, taking into consideration the possibility of it interfering with the insertion operation. On the other hand, when performing a procedure, the treatment instrument 27 is extended forward from the tip surface of the endoscope. At this time, a part of the treatment instrument 27 that is protruding forward from the tip surface is captured in the endoscopic image. In this way, when performing a procedure, a part of the treatment instrument 27 is captured in the endoscopic image, so the operator can perform the desired procedure while viewing the image.

[0149] In other words, in step S13, the time-series changes of specific image features (color information, saturation, gloss, bright spots, pixel saturation, contrast, pattern, etc.) are determined by, for example, examining the time-series changes of image data (image frames) sequentially acquired from the imaging unit 26 of the endoscope, comparing image frames that are in an earlier or later period of time, or determining the differences in the display of specific parts within each image in multiple images. This makes it possible to determine in step S13 whether or not the treatment instrument 27 has entered the field of view of the imaging unit 26 of the endoscope, that is, whether or not the treatment instrument 27 is protruding.

[0150] Of course, it is also conceivable to apply pixel information to determine the presence or absence of a treatment tool, etc., by taking into account not only the color of the treatment tool, etc. 27 but also the color of the target object, and utilizing the material and color characteristics of the treatment tool, etc. 27 to differentiate it from the biological tissue, etc. that is the target object of treatment.

[0151] For example, by using sequentially acquired image data to determine changes in the distribution of bright spots over time, it is possible to detect whether or not the instrument protrudes from its tip. Then, based on the detection result, precise focus control becomes possible by controlling the relative position of the imaging optical system and the image sensor (and its tip) in the optical axis direction.

[0152] Furthermore, using a similar approach, it is possible to detect the protrusion state from the tip surface of the treatment instrument by using sequentially obtained image data to determine changes in the distribution of bright spots over time, and based on the detection results, to detect the process of the treatment instrument being retracted into the interior of the tip.

[0153] Furthermore, detecting time-series changes in such image data allows for determining, for example, that the point at which no further changes are detected indicates that the instrument has finished moving in the protruding direction, or that the tip of the instrument has been fully retracted into the interior.

[0154] Furthermore, many instruments and other devices are designed to protrude from the tip opening of the instrument insertion channel. When using these instruments, they appear to gradually protrude into the endoscope screen from a specific point within the screen.

[0155] In this case, focusing on the changes in each pixel of the image, for example, when a metal surface is illuminated by the illumination light from an endoscope, it reflects the illumination light. As a result, in the image, relatively bright pixel points (bright spots) appear as a group with a distribution of varying brightness at low contrast (bright spot distribution).

[0156] Furthermore, the treatment instrument has a relatively thin, wire-like form that emerges from an extremely narrow tip opening. As a result, the distribution of bright spots changes in a unique pattern: as the treatment instrument protrudes toward the treatment target, it spreads from the edge of the screen where it first appeared toward the center of the screen, but the spread is less pronounced in the direction perpendicular to the direction of protrusion of the treatment instrument.

[0157] In image detection of medical instruments, determining such unique pattern changes makes it possible to detect when a medical instrument appears in the image, as well as the tip of the instrument (equal to the tip in the direction of protrusion of the brightness distribution).

[0158] In step S13, the process involves identifying medical instruments based on image data, but the identification of medical instruments is not limited to this method. For example, the system may be configured so that a user, such as a physician, can manually input information about the type and specifications of medical instruments, etc., using a predetermined operating member (not shown) provided on the processor 10, thereby inputting information about medical instruments used in a procedure that is being performed or is about to be performed.

[0159] Next, in step S14, the processor 10 identifies the operating area of ​​the identified treatment instrument, etc. Here, the operating area of ​​the treatment instrument, etc. refers to the range of motion when the treatment instrument, etc. is in operation.

[0160] Furthermore, as mentioned above, by using sequentially obtained image data to determine changes in the distribution of bright spots over time, it is possible to detect whether or not there is a protrusion from the tip surface of the treatment instrument. Therefore, this allows detection from the start to the end of the protrusion from the tip surface of the treatment instrument (until the change in the movement of the bright spot in the direction of protrusion of the treatment instrument ceases). In this case, the range of positional change of the tip in the direction of protrusion of the bright spot can also be the range of motion of the treatment instrument. Other specific examples will be described later (see Figure 10).

[0161] Next, in step S15, the processor 10 checks whether the operating area of ​​the treatment instrument, etc., identified based on the results of the image change analysis process, has reached the area of ​​the specific object. If it is confirmed that the treatment instrument, etc., has reached the specific object via its operating area, the process proceeds to the next step S16.

[0162] Furthermore, as mentioned above, for example, by using sequentially obtained image data to determine changes in the distribution of bright spots over time, it is possible to detect whether or not the instrument is protruding from its tip surface. Therefore, it is possible to detect the period from the start to the end of the protrusion from the tip surface of the instrument (until the change in the movement of the bright spot in the direction of protrusion of the instrument ceases). In this case, it may be possible to determine whether the instrument has reached a specific target object based on the change in the position of the tip in the direction of protrusion of the bright spot.

[0163] Furthermore, if it is not confirmed that the operating area of ​​the treatment instrument or the like has reached the specific target object, the process proceeds to step S17.

[0164] Although the processing steps S13, S14, and S15 are shown as examples performed by image judgment, the system is not limited to this configuration. These judgments may be performed, for example, using an inference model based on training data using images of similar situations, or by matching with patterns in images of similar situations. Furthermore, the changes in the image features described above may be compared with specific conditions as appropriate (for example, determining specific bright spots or colors within the screen, or detecting changes such as extension or cessation in a specific direction), and if the specific conditions are met, these processing steps may be performed, or the logic to branch these steps in the "Y" direction may be programmed and executed.

[0165] In step S16, the processor 10 performs focus control on a specific object on biological tissue, for example, by setting a focus target to a part that includes the operating area of ​​a treatment tool or the like. After that, the process returns to step S11 and the subsequent processes are repeated.

[0166] In step S17, the processor 10 drives and controls the imaging optical system 26a through the focus control unit 16 to perform so-called pan-focus control or normal focus control with the central area of ​​the screen or a specific object as the focus target. After that, the process returns to step S11.

[0167] In this process, when performing a predetermined procedure on a specific object using various treatment tools, the treatment tools are operated while observing the object being treated. Eventually, the treatment tools enter the field of view. Then, the treatment tools act on a predetermined part of the specific object.

[0168] Thus, when applying a part of a medical device to a specific object, there is always a need to perform the procedure while clearly observing the area including the operating area of ​​the medical device. For example, it is clear that if the operating area of ​​a medical device cannot be observed during its operation, it is impossible to perform the procedure efficiently and reliably.

[0169] Therefore, in the first focus control process shown in Figure 9, focus control is performed on a specific object with respect to the part that includes the operating area of ​​a treatment tool, etc., as the priority area for the focus target. In this case, the part that includes the operating area of ​​a treatment tool, etc., on a specific object can be identified by identifying the type and specifications of the treatment tool, etc.

[0170] The operating range of a treatment instrument is, for example, the amount of protrusion of the instrument forward from the tip surface of the tip portion 21a, or the amount of movement in the width direction of the instrument (in a direction perpendicular to the protrusion direction). Such an operating range of a treatment instrument is determined by the type and specifications of the instrument (operating mode, dimensions of main parts, etc.).

[0171] For example, specific examples of the operating ranges of various treatment instruments are shown in the table in Figure 10. Note that the numerical values ​​for the operating ranges shown in Figure 10 are merely examples. Many other types of treatment instruments exist besides those shown in Figure 10. The types of these treatment instruments and their specifications (treatment instrument information) are stored, for example, in the treatment instrument DB 14x.

[0172] In this embodiment, the endoscope system 1 of the processor 10 identifies the type of treatment instrument, etc., from image data obtained by the imaging unit of the endoscope, etc., based on the image pattern and its changes, and identifies the operating area of ​​the treatment instrument, etc., by identifying specification information (treatment instrument information) that matches the identified type of treatment instrument. Then, it performs focus control that takes into account the operating area of ​​the identified treatment instrument, etc.

[0173] Generally, during an endoscopic examination, when a treatment instrument 27 protrudes from the tip of the endoscope, it is designed to be visible in the endoscopic image. The physician or other medical professional performs the procedure on the target area while viewing the endoscopic image.

[0174] In other words, during an endoscopic examination, time-series changes in specific image features (color, gloss, bright spots, contrast, patterns, etc.) are determined by examining the time-series changes in the images, comparing image frames that are out of or ahead of each other in time, and determining differences that appear at specific locations within multiple images, based on the image data (image frames) sequentially obtained from the imaging unit of the endoscope.

[0175] By performing various image detections like these, it is possible to determine whether or not a treatment instrument is within the field of view of the endoscope's imaging unit, that is, whether or not the treatment instrument is protruding.

[0176] For example, by using sequentially acquired image data to determine changes in the distribution of bright spots over time, it is possible to detect whether or not a treatment instrument is protruding. Furthermore, by using sequentially acquired image data to determine changes in the distribution of bright spots over time, it is possible to detect the process of changes in the state of a treatment instrument (from a protruding state to a retracted state). The treatment instrument identification unit 14a is capable of such detection.

[0177] Another example is a function that is learned by the flowchart in Figure 19, which will be described later, namely an inference model obtained by the inference model creation device 60, and which may have functions for identifying treatment tools and their operating areas, or for identifying the operating parts of treatment tools.

[0178] Next, Figure 11 is a flowchart illustrating the operation of a second embodiment of the endoscope system according to one embodiment of the present invention. The flowchart shown in Figure 11 outlines the operation when performing an endoscopic examination using the endoscope system 1. This flowchart in Figure 11 shows an overview of the second focus control process, in which the focus control unit performs focus control on a specific target object, with the part on which the treatment instrument or the like acts as the priority area for the focus target.

[0179] Note that the flowchart in Figure 11 (second focus control process) includes some processing steps similar to those in the flowchart in Figure 9 (first focus control process). Therefore, the same processing steps are denoted by the same reference numerals, and their explanations are omitted.

[0180] In the second focus control shown in Figure 11, the processes in steps S11 to S13 and step S17 are the same as those in the flowchart shown in Figure 9 above.

[0181] After the processing in step S13 of Figure 11, in the next step S14A, the processor 10 identifies the part of the specific object that the identified treatment tool, etc. acts upon (hereinafter referred to as the part of action of the treatment tool, etc.).

[0182] Next, in step S15A, the processor 10 checks whether the identified site of action of the treatment instrument, etc., identified based on the results of the image change analysis process, is identified in the observed image. If the site of action of the treatment instrument, etc. is identified, the process proceeds to the next step S16A. If the site of action of the treatment instrument, etc. is not identified, the process proceeds to step S18.

[0183] Note that while steps S13, S14A, and S15A are shown as examples performed by image judgment, the system is not limited to this configuration. These judgments may be performed, for example, using an inference model based on training data using images of similar situations, as in Figure 9, or by matching with patterns in images of similar situations. Furthermore, the changes in the image features described above may be compared with specific conditions as appropriate (for example, determining specific bright spots or colors within the screen, or detecting changes such as extension or cessation in a specific direction), and if the specific conditions are met, these processing steps may be performed, or the logic to branch these steps in the "Y" direction may be programmed and executed.

[0184] In step S16A, the processor 10 performs focus control, for example, on a specific object on biological tissue, with the area where the treatment instrument or the like is acting as the focus target. After that, the process proceeds to step S21.

[0185] In step S21, the processor 10 checks whether the working part of the treatment instrument, etc., has moved away from the specified object. If it is confirmed that the working part of the treatment instrument, etc., has moved away from the specified object, the process proceeds to the next step S22. If the working part of the treatment instrument, etc., has not moved away from the specified object, the process returns to step S16A.

[0186] The example shown for step S21 is performed by determining the image obtained from the imaging unit of the endoscope, but the configuration is not limited to this example. This determination may be made using an inference model based on training data using images (which may also be moving images) of each instrument when it is separated from the object, or it may be determined by matching the pattern of the image when each instrument is separated from the object. Furthermore, the changes in the image features described above may be appropriately compared with specific conditions (for example, a specific bright spot or color is determined in the screen, or the determined bright spot or color changes by extending in a specific direction, or the change stops), and if the specific condition is met, for example, in the case of a change in which a specific point extends, if it is detected that it has contracted after reaching the end in the direction of extension, the program may be programmed to branch to "Y".

[0187] Furthermore, when a treatment instrument separates from the object, it can be considered a process of changing from an extended state to a retracted state. For example, when removing an endoscope, if the treatment instrument is in an extended state, it may hinder the removal operation. Therefore, during the endoscope removal operation, the treatment instrument is returned to its retracted state within the endoscope to complete the series of operations.

[0188] Furthermore, for example, there are medical devices that are left inside the body before being stored, but even with such devices, changes in the image of the tip, for example, the movement of the tip portion in the storage direction after it has stopped, can be detected using the image judgment described above.

[0189] Next, in step S22, the processor 10 performs focus control, focusing on a site where it can be confirmed that the treatment instrument or the like has acted on a specific object. After that, it returns to the process in step S11 and repeats the subsequent processes.

[0190] On the other hand, in step S18, the processor 10 checks whether the tip portion of the identified treatment instrument or the like has been identified.

[0191] Generally, some medical instruments have tips that are, for example, needle-shaped (needle type) or have a sharp shape. When using these types of instruments during endoscopic examinations and procedures, it is naturally desirable to avoid situations where the tip of the instrument accidentally pierces living tissue during handling. Therefore, it is desirable that the tip of the instrument be constantly observable with a clear image during handling.

[0192] Taking these factors into consideration, in this embodiment, each of the processes in steps S18 to S19 is performed.

[0193] In other words, if the tip of a treatment instrument or the like is identified during the process in step S18, the process proceeds to the next step, S19.

[0194] The tip detection process in step S18 is shown as an example where it is performed by determining the image obtained from the imaging unit of the endoscope, but it is not limited to this configuration. This determination may be made using an inference model based on training data using images of the tips of each treatment instrument, or it may be determined by matching it with the image pattern of the tip of each treatment instrument. Furthermore, the changes in the image features described above may be compared with specific conditions as appropriate (e.g., a specific bright spot or color is determined in the screen, the determined specific point changes by extending in a specific direction, or stops), and if the specific condition is met, for example, in the case of a change where a specific point extends, if the tip in the direction of extension is detected as the tip, the program may be programmed to branch to "Y" and executed.

[0195] Then, in step S19, the processor 10 performs focus control, targeting the tip of the treatment instrument or the like as the focus target. After that, the process returns to step S15A and the subsequent processes are repeated.

[0196] On the other hand, if the tip of the treatment instrument or the like is not identified in step S18, the process proceeds to step S20.

[0197] Then, in step S20, the processor 10 ignores the presence of treatment tools and other equipment and performs normal focus control with the surface of biological tissue (for example, an area containing a specific object or the central area of ​​the observation screen) as the focus target. After that, it returns to the process in step S15A and repeats the subsequent processes.

[0198] Furthermore, there is a need to observe in particular how the treatment instruments act on the treatment site. In cases where such requests are met (for example, when the focus control is optimized for close range to clearly see patterns such as blood vessels only during treatment), pan-focus control may be used for all other special focus settings.

[0199] The concept of the present invention can also be applied to such processing. An example of specific processing is shown below.

[0200] For example, by sequentially acquiring image data over time, a protruding image change pattern is determined, which protrudes from the edge of the image towards the center of the image, and spreads in the direction of protrusion, but spreads less in the direction perpendicular to the direction of protrusion. This determination process makes it possible to detect the transition from a protruding state to a retracted state of a treatment instrument or similar object.

[0201] Similarly, it is also possible to detect a state where the protrusion of a treatment instrument has stopped when the change in the tip position of the instrument becomes small. In this state where the protrusion of the treatment instrument has stopped, a focus target position prioritizing the tip of the treatment instrument in the image should be set.

[0202] In the second focus control process shown in Figure 11, focus control is performed with the part of the treatment tool, etc. that acts on the specific object (the part of the treatment tool, etc. that acts on) as the priority part of the focus target. In this case, the part of the treatment tool, etc. that acts on the specific object can be identified by identifying the type and specifications of the treatment tool, etc.

[0203] For example, specific examples of the action sites of various treatment tools are shown in the table in Figure 12. Note that the action sites of the treatment tools shown in Figure 12 are merely examples. Many other types of treatment tools exist besides those shown in Figure 12. Information on the types of these treatment tools and their specifications (treatment tool information) is stored, for example, in the treatment tool database 14x.

[0204] Thus, in the endoscope system 1 of this embodiment, the treatment instrument identification unit 14a of the processor 10 identifies the type of treatment instrument, and identifies the working site of the treatment instrument by identifying specification information (treatment instrument information) that matches the identified treatment instrument type. Then, it performs focus control that takes into account the working site of the identified treatment instrument.

[0205] Figures 13 to 18 are schematic diagrams illustrating typical forms of medical instruments, showing the specified operating area or site of action for each instrument and the corresponding target area. Of these, Figure 13 is an example of a biopsy forceps or hemostatic forceps. Figures 14 and 15 are examples of hemostatic clips (separation type). Figure 16 is an example of an injection needle. Figure 17 is an example of a snare. Figure 18 is an example of a tip hood or transparent cap.

[0206] In Figures 13, 14, 16, 17, and 18, the diagrams shown in column (1) are schematic diagrams showing how treatment is being performed on a specific object using treatment instruments during endoscopic observation. In each of these figures, the diagrams shown in column (2) are schematic examples of the display screen of the endoscopic image corresponding to column (1).

[0207] In each of the figures, row (A) shows the state before the operating area or site of action of the treatment instrument reaches the area of ​​the specific object, or the state before the treatment instrument acts (see row (A) column (1)). In the endoscopic image at this time, a part of the treatment instrument is visible within the display range of the endoscopic image (see row (A) column (2)).

[0208] In each of the figures, row (B) indicates a state where the operating area or site of action of a treatment instrument reaches the area of ​​a specific object, or where the treatment instrument is in action (see row (B) column (1)). In this case, the endoscopic image shows the treatment instrument within the display range of the endoscopic image (see row (B) column (2)).

[0209] Figure 15 is an example showing the display screen over time when multiple hemostatic clip procedures are performed in a single treatment.

[0210] Furthermore, the reference numerals shown in Figures 13 to 18 are the same as those shown in Figures 4 to 8, etc., as described above. Here, only the reference numerals appearing for the first time in Figures 13 to 18 will be explained below.

[0211] In Figure 13, etc., the symbol FP indicates the target area or region of focus. In the same Figure 13, etc., the symbol FF is a display frame (focus frame) that is displayed on the endoscopic image display screen and indicates the focus area. In Figure 13, the symbol 201a indicates a polyp that has been grasped by a treatment instrument, etc. In Figure 14, the symbol 27a indicates a clip after separation in a separation type treatment instrument, etc. (hemostatic clip).

[0212] As shown in row (A) of Figures 13 to 18 (excluding Figure 15), when the operating area or part of action of a treatment instrument has not yet reached the area of ​​a specific object, or when the treatment instrument has not yet acted, and the treatment instrument is visible within the display range of the endoscopic image, pan-focus control or normal focus control with the central area of ​​the screen or a specific object as the focus target is performed.

[0213] On the other hand, as shown in row (B) of Figures 13 to 18 (excluding Figure 15), when the operating area or part of action of a treatment instrument reaches the area of ​​a specific target object, or when the treatment instrument is in action and the treatment instrument is visible within the display range of the endoscopic image, focus control is performed with the area or part including the operating area or part of action of the treatment instrument as the focus target.

[0214] Hereinafter, the creation process of the inference model creation device 60 for the inference model used in the process of step S14 in Figure 9 (processing to identify the operating area of ​​the treatment tool, etc.) and the process of step S14A in Figure 11 (processing to identify the operating part of the treatment tool, etc.) will be briefly explained below using the flowchart in Figure 19.

[0215] In step S31, the inference model creation device 60 performs a process to determine the corresponding site of action for each treatment instrument, etc. This process involves, for example, identifying the type of treatment instrument, etc., based on image (or video) data stored in the endoscopic treatment DB 61, and determining a specific site of action for each identified treatment instrument, etc.

[0216] Next, in step S32, the inference model creation device 60 classifies the endoscopic images, including the area (specific target area) where a treatment instrument or the like is acting on a specific target object.

[0217] In step S33, the inference model creation device 60 learns the association of endoscopic images with the corresponding site of action for each treatment instrument. Then, it checks whether the learning process has been completed. If it is confirmed that the learning process has been completed, the process proceeds to step S38. If the learning process has not been completed, the process proceeds to the next step, S34.

[0218] Next, in step S34, the inference model creation device 60 candidates endoscopic images including specific target areas associated with each treatment instrument, etc., as training data.

[0219] Next, in step S35, the inference model creation device 60 annotates the candidate training data images to specify a particular target area and performs machine learning. At this time, priority may be added to the candidate training data images.

[0220] In step S36, the inference model creation device 60 uses test data to verify whether highly reliable inference is possible. If it is confirmed that highly reliable inference is possible, the process proceeds to the next step S37. If highly reliable inference is not possible, the process proceeds to step S39.

[0221] In step S37, the inference model creation device 60 records the created inference model. Then, the process proceeds to step S38.

[0222] In step S38, the inference model creation device 60 confirms whether to continue learning about another treatment tool, etc. If learning is to be continued, the process returns to step S32.

[0223] Meanwhile, in step S39, the inference model creation device 60 performs machine learning using different training data. After that, it returns to the process in step S36.

[0224] As described above, the endoscope system 1 of the above embodiment includes a treatment instrument identification unit 14a that identifies an image region containing a treatment instrument 27 within the endoscope image shown based on image data acquired by the imaging unit 26 (imaging unit), and a focus control unit 16 that controls the imaging unit 26 (imaging unit) to perform focus control.

[0225] In this case, the focus control unit 16 performs focus control according to the identified treatment instrument 27. Specifically, the focus control unit 16 performs focus control to set the focus target position prioritizing the area that includes the operating area of ​​the treatment instrument 27 identified in the endoscopic image or the area where the treatment instrument 27 acts on a specific target object.

[0226] According to the endoscope system 1 of this embodiment with the configuration described above, focus control is performed according to the identified treatment instrument 27. Therefore, even when using the endoscope and during operations involving treatment using the treatment instrument 27, the focus target is set to the area desired by the user, such as a physician (the area to be observed, or the area where the treatment instrument 27 is to be applied). Consequently, the area desired by the user, such as a physician, can always be observed with a clear and good image.

[0227] Furthermore, the focus control unit 16 changes the setting of the focus target position while the treatment instrument 27 is being operated.

[0228] For example, if a treatment instrument or similar object enters the display screen of the endoscopic image while observing a specific object, and the focus target is changed to that instrument or similar object, the specific object that you want to continue observing may move out of the focus area. In this case, it may become impossible to continue observing the specific object clearly.

[0229] On the other hand, when performing a prescribed treatment on a specific object using a treatment tool, it is convenient if the specific object being treated and the area including the operating area or the part of action of the treatment tool applied to that specific object can be observed simultaneously.

[0230] In other words, in endoscopic images in such situations, focus control is desirable so that both a part of the specific object being observed and the site of action of the treatment instrument on that part of the specific object are clearly captured. By performing such focus control, it becomes possible to observe the operation of the treatment instrument and the effects of its action on the specific object in a good manner.

[0231] Therefore, in this embodiment, the focus control unit 16 of the endoscope system 1 performs focus control to set the focus target position prioritizing the specific target object during the period until the treatment instrument 27 is activated or until the treatment instrument 27 acts on the specific target object.

[0232] On the other hand, during the period after the treatment device 27 has been activated or after the treatment device 27 has acted on a specific object, focus control is performed to set the focus target position prioritizing the operating area of ​​the treatment device 27 or the part of the treatment device 27 that is acting on the specific object.

[0233] With this configuration, even if a treatment instrument 27 enters the display range of the endoscopic image during observation, it is always possible to continue observing the desired specific object as a clear and good image, even if the treatment instrument 27 is present in the endoscopic image during the period before the treatment instrument 27 starts its desired action.

[0234] Furthermore, for the period after the treatment device 27 begins its desired action, focus control is performed with the region including the area where the treatment device 27 acts on the specific object as the focus target. As a result, the specific object and the region including the area where the treatment device 27 acts on this specific object can always be observed as a clear and good image.

[0235] This allows for continuous observation of the area being treated with clear images throughout the procedure. Therefore, it can contribute to increased efficiency in the procedure and also improve the operability of endoscopes and other treatment instruments.

[0236] Furthermore, the treatment instrument identification unit 14a is configured to identify the types of treatment instruments 27 within the endoscopic image. The focus control unit 16 then performs focus control to set the focus target position according to the type and specifications of the identified treatment instruments 27.

[0237] This allows for appropriate focus control according to the different operating areas and target sites for each type and specification of the treatment instrument 27. Therefore, even when performing treatment while switching between multiple treatment instruments 27 of different types and specifications, clear and good images can always be used for observation and treatment.

[0238] Furthermore, the system is further equipped with a display control unit 12. This display control unit 12 superimposes an indicator representing the focus target position or the in-focus position set by the focus control unit 16 onto the endoscopic image. This allows users, such as doctors, to easily visually confirm the focus target position or the in-focus position within the endoscopic image.

[0239] As explained above, the block diagrams of each function, such as the hardware in each embodiment of this application, and the flowcharts illustrating how algorithms are controlled in conjunction with their respective functions, are designed to improve the user experience overall, such as providing control that is user-friendly and in line with the user's intentions. These improvements are brought about or created by the technical features of the organically linked various blocks. In other words, these demonstrate examples of how hardware processing speed can be increased, computational efficiency and execution effectiveness can be improved, power consumption can be reduced, and the overall cost performance of the system can be enhanced.

[0240] For example, using appropriate and easily obtainable data in the relevant application, or data appropriately selected from a wealth of data, and the efficient coordination of each function that processes this data, aims to improve the efficiency of various judgments, branching, calculations, and correct and desirable analysis and information input / output. This helps reduce the amount of data to be stored and retained, and also improves the computational efficiency and execution effectiveness of the hardware.

[0241] The present invention is not limited to the embodiments described above, and various modifications and applications can be implemented without departing from the spirit of the invention. Furthermore, the above embodiments include inventions at various stages, and various inventions can be extracted by appropriate combinations of the multiple components disclosed. For example, if the problem that the invention aims to solve can be solved and the effects of the invention can be obtained even if some components are deleted from all the components shown in one embodiment, then the configuration with these components deleted can be extracted as an invention. Furthermore, components from different embodiments may be combined as appropriate. This invention is not limited by any particular embodiment other than being limited by the appended claims.

[0242] 1...Endoscope system 10...Processor 10 11...Image processing unit 12...Display control unit 13...Memory control unit 14...Image identification unit 14a...Treatment instrument identification unit 14z...Inference unit 16...Focus control unit 17...Illumination control unit 20...Endoscope 21...Insertion unit 21a...Tip 21b...Bending unit 21c...Flexible tube 21d...Treatment instrument insertion channel 21da...Tip opening 22...Operation unit 22a...Operation unit body 22b...Bending operation member 22c...Operation member 22d...Treatment instrument insertion port 23...Universal cable 23a...Scope connector 23b...Electrical cable 23c...Connector 24...Video cable 25...Illumination unit 25b...Illumination window 26...Imaging unit 26a...Imaging optical system 26aa...Observation window 26b...Image sensor 27...Treatment instruments, etc. 28...Auxiliary devices 30...Display device 40...Light source device 50...Storage device 51...Temporary storage unit 60...Inference model creation device 100...Display screen frame 101a...Endoscopic image 101b...Specific target object image 200...Biological tissue, etc. 200a...Mucosal layer 200b...Submucosa 200c...Muscularis propria 201...Specific target object 206...Physiological saline, drugs 14x...Treatment tool DB 61...Endoscopic treatment DB FF...Focus frame FP...Focus target area (region)

Claims

1. An endoscope system comprising: an imaging unit provided at the tip of the endoscope for imaging an object; a treatment instrument identification unit for identifying a treatment instrument image in the image based on image data acquired by the imaging unit; and a focus control unit for controlling the imaging unit to perform focus control, wherein the focus control unit performs focus control to set a focus target position prioritizing a region containing a specific part of the treatment instrument image identified in the image.

2. The endoscope system according to claim 1, characterized in that the focus control unit performs focus control to set the focus target position prioritizing the region including the operating region of the treatment instrument.

3. The endoscope system according to claim 1, characterized in that the focus control unit performs focus control to set the focus target position prioritizing the region including the part on which the treatment instrument acts on a specific object.

4. The endoscope system according to claim 1, characterized in that the focus control unit changes the setting of the focus target position during operation of the treatment instrument.

5. The endoscope system according to claim 4, characterized in that the focus control unit performs focus control such that the setting of the focus target position differs between the period before the treatment instrument is activated and the period after the treatment instrument is activated.

6. The endoscope system according to claim 4, characterized in that the focus control unit performs focus control such that the setting of the focus target position differs between the period before the treatment instrument acts on the specific target object and the period after the treatment instrument acts on the specific target object.

7. The endoscope system according to claim 4, characterized in that the focus control unit performs focus control to set the focus target position prioritizing the specific object during the period until the treatment instrument is activated or until the treatment instrument acts on the specific object, and performs focus control to set the focus target position prioritizing the operating area of ​​the treatment instrument or the part on which the treatment instrument is acting on the specific object during the period after the treatment instrument is activated or after the treatment instrument has acted on the specific object.

8. The endoscope system according to claim 1, characterized in that the treatment instrument identification unit identifies the type of treatment instrument, and the focus control unit performs focus control to set the focus target position according to the type of treatment instrument identified.

9. The endoscopic system according to claim 2, characterized in that the instrument identification unit identifies the type of instrument, whether pinch-type or needle-type, the operating area for a pinch-type instrument includes the clamping, gripping, or capturing portion of the instrument, and the operating area for a needle-type instrument includes the needle tip of the instrument.

10. The endoscopic system according to claim 3, wherein the instrument identification unit identifies the types of instrument, namely, a separable type and a non-separable type, and when the instrument identification unit identifies a separable type instrument, the focus control unit, for the period until the instrument is separated, prioritizes focusing on the region including the part in which the instrument acts on a specific target site, and for the period after the instrument has been separated, prioritizes focusing on the region including the part in which the separable part of the instrument acts on the specific target site.

11. The endoscopic system according to claim 1, further comprising a display control unit, wherein the display control unit superimposes and displays an index representing the focus target position or focus position set by the focus control unit onto the endoscopic image.

12. A focus control device for an endoscope comprising: a treatment instrument identification unit that identifies a treatment instrument image in an image based on image data acquired by an imaging unit provided at the tip of the endoscope for imaging an object; and a focus control unit that controls the relative position between the imaging optical system and the image sensor of the imaging unit to perform focus control, wherein the focus control unit performs focus control to set a focus target position prioritizing a region including a specific part of the treatment instrument identified in the image.

13. A method for controlling the focus of an endoscope, characterized by: capturing an image of an object to acquire image data; identifying a treatment instrument image within the image based on the image data; and performing focus control to set a focus target position prioritizing a region containing a specific part of the treatment instrument image identified within the image.

14. A method for controlling the focus of an endoscope, comprising: acquiring image data from an imaging unit provided at the tip of an endoscope from which a treatment instrument can be extended and which images an object; identifying a treatment instrument from the image patterns contained in the image data, identifying the presence or absence of treatment instrument extension and the treatment instrument state that can be classified as when the treatment instrument is in operation; and performing focus control by controlling the relative position of the imaging optical system and the image sensor of the imaging unit, wherein the focus control step prioritizes the region containing the treatment instrument tip image pattern when the treatment instrument is in operation in the image data detected as when the treatment instrument is in operation, compared with image data detected as when the treatment instrument is not extended, and sets the focus target position.

15. The focus control method for an endoscope according to claim 14, characterized in that the focus control step performs focus control to set the focus target position by prioritizing the region in the direction of protrusion of the treatment instrument when the treatment instrument is in operation, compared with image data in which the treatment instrument is detected to be absent in the image and when the treatment instrument is not detected in the image data.

16. A method for controlling the focus of an endoscope, comprising: a step of sequentially acquiring image data in a time series from an imaging unit provided at the tip of an endoscope from which a treatment instrument can be extended, and which images an object; a step of detecting the process state of the treatment instrument being extended or retracted by using the sequentially acquired time series image data to determine a protruding image change pattern in which the image data protrudes from the edge of the screen toward the center of the screen and spreads in the direction of said protrusion, but spreads less in the direction perpendicular to the direction of said protrusion; a step of detecting a treatment instrument extension stopped state in which the position change of the tip of the treatment instrument has decreased; and a focus control step of controlling the relative position of the imaging optical system and the image sensor of the imaging unit to perform focus control, wherein the focus control step compares the treatment instrument extension stopped state with the process state of the treatment instrument being extended or retracted in the image, and performs focus control by prioritizing the tip of the treatment instrument in the image to set the focus target position.

17. The method for controlling the focus of an endoscope according to 16, characterized in that the protruding image change pattern, which protrudes from the edge of the image data toward the center of the screen and spreads in the direction of protrusion but has little spread in the direction perpendicular to the direction of protrusion, is a pattern that appears as a distribution of bright spots in the image data as a result of reflected light from the illumination unit of the endoscope being reflected by a treatment instrument and incident on the image sensor of the imaging unit of the endoscope.

18. The step of detecting a treatment instrument protrusion stop state in which the position change of the tip of the treatment instrument has decreased is characterized in that the position of the tip of the treatment instrument is determined by using the sequentially obtained time-series image data to determine the tip of the protrusion direction of the protrusion image change pattern which protrudes from the edge of the image toward the center of the image and spreads in the direction of protrusion but does not spread in the direction perpendicular to the direction of protrusion, thereby determining the position of the tip of the treatment instrument.

19. A method for controlling the focus of an endoscope, comprising: a step of sequentially acquiring image data in a time series from an imaging unit provided at the tip of an endoscope from which a treatment instrument can be extended and which images an object; a step of detecting whether or not a treatment instrument is extended by determining the change in the distribution of bright spots in a time series using the sequentially obtained image data; a treatment instrument identification step of identifying an image pattern related to a treatment instrument that can be detected in at least one of the sequentially obtained image data; and a focus control step of controlling the relative position of the imaging optical system and the image sensor of the imaging unit to perform focus control, wherein the focus control step prioritizes the region containing the image pattern of the tip of the treatment instrument when the treatment instrument is in action in the image data where the treatment instrument is in action, compared with image data where the treatment instrument is not extended, and sets the focus target position.

20. A method for controlling the focus of an endoscope, comprising: a step of sequentially acquiring image data in a time series from an imaging unit provided at the tip of an endoscope capable of extending a treatment instrument and imaging an object; a step of detecting the process of retracting the treatment instrument by determining the change in the distribution of bright spots in a time series using the sequentially acquired image data; a treatment instrument identification step of identifying an image pattern related to the treatment instrument that can be detected in at least one of the sequentially acquired image data; and a focus control step of controlling the relative position of the imaging optical system and the image sensor of the imaging unit to perform focus control, wherein the focus control step maintains the result of setting the focus target position by prioritizing the region that includes the image pattern of the tip of the treatment instrument at the time of treatment instrument operation in the image data detected as the retraction process of the treatment instrument extension in the image.

Images