Image processing device, control method for image processing device, and image processing program
Patent Information
- Application Number
- PCT/JP2026/011486
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-10-06
- Filing Date
- 2026-03-23
- Publication Date
- 2026-10-22
Smart Images
Figure JP2026011486_22102026_PF_FP_ABST
Abstract
Description
Image processing apparatus, control method for image processing apparatus, and image processing program
[0001] The present invention relates to an image processing apparatus that displays an observation range, a control method for the image processing apparatus, and an image processing program.
[0002] In recent years, endoscope apparatuses including an endoscope that is inserted into a subject to observe the inside of the subject or perform treatment using a treatment tool or the like have been widely spread. Further, Computer-Aided Diagnosis (CAD) and CADe (Computer-Aided Detection), which provide identification results such as differentiation of lesioned parts as support information by AI (Artificial Intelligence) based on images obtained by the endoscope apparatus, have also been developed. In such endoscopic observation using an endoscope, it is required to observe a target site without omission.
[0003] Unlike CT (Computed Tomography) and MRI (Magnetic Resonance Imaging), an endoscope cannot acquire images of a wide range, so it is necessary to make an effort to comprehensively observe an examination target. However, in the case of an organ with a complicated shape, oversight may occur even when comprehensive observation is attempted. Accordingly, a technique is known in which an organ model is displayed during endoscopic observation, and an already imaged region / an unimaged region is visualized on the organ model.
[0004] For example, Japanese Patent No. 7506264 (hereinafter referred to as Patent Document 1) discloses a technique that allows the position of an unimaged region to be easily and intuitively recognized by the display of the organ model by aligning the top-bottom orientation of an examination image with the top-bottom orientation of the organ model.
[0005] Patent No. 7506264
[0006] As described above, in an endoscopic examination, a doctor three-dimensionally divides the organ of a subject, comprehensively observes these divided portions, and performs the examination while capturing images. However, Patent Document 1 has a problem that, for example, three-dimensional four-directional already imaged regions such as the greater curvature, lesser curvature, anterior wall, and posterior wall of the stomach cannot be displayed with good visibility.
[0007] The present invention aims to provide an image processing device, a control method for the image processing device, and an image processing program that enable efficient comprehensive imaging by showing captured and uncaptured areas in a format similar to the three-dimensional organ image that physicians are aware of during endoscopic examinations.
[0008] An image processing apparatus according to one aspect of the present invention includes a processor, which acquires an image of a subject, determines the anatomical parts of the subject shown in the image, and outputs imaging information indicating that the anatomical parts of the subject have been photographed, wherein the imaging information includes a first figure representing the subject, the outer perimeter of the first figure includes a first curved portion and a second curved portion, and a first part including the first curved portion, a second part including the second curved portion, a third part located between the first curved portion and the second curved portion, and a fourth part located between the first curved portion and the second curved portion are each associated with different anatomical parts.
[0009] A control method for an image processing apparatus according to one aspect of the present invention is a control method for an image processing apparatus that acquires an image of a subject, determines the anatomical parts of the subject shown in the image, and outputs imaging information indicating that the anatomical parts of the subject have been photographed, wherein the imaging information includes a first figure representing the subject, the outer perimeter of the first figure includes a first curved portion and a second curved portion, and a first part including the first curved portion, a second part including the second curved portion, a third part located between the first curved portion and the second curved portion, and a fourth part located between the first curved portion and the second curved portion are each associated with different anatomical parts.
[0010] An image processing program according to one aspect of the present invention is an image processing program that causes a computer to perform the following steps: acquire an image of a subject, determine the anatomical parts of the subject shown in the image, and output imaging information indicating that the anatomical parts of the subject have been photographed, wherein the imaging information includes a first figure representing the subject, the outer perimeter of the first figure includes a first curved portion and a second curved portion, and a first part including the first curved portion, a second part including the second curved portion, a third part located between the first and second curved portions, and a fourth part located between the first and second curved portions are each associated with different anatomical parts.
[0011] According to the present invention, by showing the captured and uncaptured areas in a format similar to the three-dimensional organ image that physicians are aware of during endoscopic examinations, comprehensive imaging can be performed efficiently.
[0012] This is a block diagram showing an endoscope system including an image processing device according to the first embodiment of the present invention. This is an explanatory diagram for illustrating the design of an organ model used in this embodiment. This is a diagram showing an example of organ classification. This is a flowchart for illustrating the operation of the first embodiment. This is an explanatory diagram showing model images G1 to G4 displayed on the display screen 31 of the display device 30. This is a flowchart showing the operation flow used in the second embodiment. This is an explanatory diagram showing an example of display. This is an explanatory diagram showing a modified example. This is an explanatory diagram showing a modified example. This is an explanatory diagram showing a modified example.
[0013] Embodiments of the present invention will be described in detail below with reference to the drawings.
[0014] (First Embodiment) Figure 1 is a block diagram showing an endoscope system including an image processing device according to the first embodiment of the present invention. Figure 2 is an explanatory diagram for illustrating the design of the organ model used in this embodiment. Figure 3 is a diagram showing an example of organ classification. This embodiment determines the regions of images that have been captured from the images obtained by the endoscope (hereinafter referred to as the captured regions) and outputs imaging information that enables the user to see the captured regions and regions that have not yet been captured by the endoscope (hereinafter referred to as the uncaptured regions). In this case, this embodiment employs an organ model design and display method that enables the captured regions and uncaptured regions of a subject to be shown three-dimensionally using a display device that performs a general two-dimensional display.
[0015] In the following explanation, the term "imaging" refers not only to the act of releasing the shutter to acquire an image of the area of interest, but also to the state in which the area of interest is within the field of view during endoscope operation, without necessarily releasing the shutter (in some cases, when it is within the field of view to the extent that there is no blur).
[0016] First, the design of the organ model in this embodiment will be explained with reference to Figures 2 and 3. Figure 2 shows an organ model in which the stomach is used as the subject, for example. As shown in Figure 3, the stomach is classified into anatomical parts (hereinafter also simply referred to as parts) such as the cardia, fundus, upper body, middle body, lower body, gastric angle, and antrum. Of these parts, the cardia, upper body, middle body, lower body, gastric angle, and antrum are further classified into three-dimensional detailed parts: the greater curvature, lesser curvature, anterior wall, and posterior wall. That is, the stomach is classified into three-dimensional anatomical parts: the greater curvature, a relatively long curved portion corresponding to the left side of the body, and the lesser curvature, a relatively short curved portion corresponding to the right side of the body, as well as the anterior wall, which corresponds to the front of the body, and the posterior wall, which corresponds to the back of the body.
[0017] Figure 2 shows an example of a stomach organ model in this embodiment, but this embodiment is not limited to the stomach and can be applied to various organ models of the human body. Furthermore, it can be applied not only to the human body but also to models of various objects that are photographed using an endoscope.
[0018] In Figure 2, organ model M1 is a schematic diagram having an external shape (organ shape ES) that mimics the shape of the stomach organ. Organ shape ES is divided into multiple parts by part division lines ABL. In the example of organ model M1 in Figure 2, the cardia AP1, fundus AP2, upper body AP3, middle body AP4, lower body AP5, gastric angle AP6, and antrum AP7 (hereinafter, these parts will be referred to as part AP when it is not necessary to distinguish between them) are divided by part division lines ABL. Note that the part division lines ABL dividing upper body AP3, middle body AP4, and lower body AP5 are shown as dashed lines.
[0019] In organ model M1, each organ AP is further divided into detailed areas that are three-dimensionally partitioned into the right, left, posterior, and anterior sides of the body. In this embodiment, the greater curvature, which is a detailed area, is indicated by a band-shaped region of a predetermined width (hereinafter referred to as the greater curvature indication area GCp) along the relatively long curved portion on the outer circumference of the organ shape ES. Similarly, the lesser curvature, which is a detailed area of each organ AP, is indicated by a band-shaped region of a predetermined width (hereinafter referred to as the lesser curvature indication area LCp) along the relatively short curved portion on the outer circumference of the organ shape ES.
[0020] The organ model M1 has a predetermined shaped mark (hereinafter referred to as the anterior wall indicator AWp) indicating the detailed part of the anterior wall and a predetermined shaped mark (hereinafter referred to as the posterior wall indicator BWp) indicating the detailed part of the posterior wall, within the region of each organ AP, specifically in the region between the greater curvature indicator GCp and the lesser curvature indicator LCp. In the example in Figure 2, the anterior wall indicator AWp and the posterior wall indicator BWp are represented by figures that partially overlap circular shapes, with the anterior wall indicator AWp displayed overlapping a part of the posterior wall indicator BWp. Therefore, the anterior wall indicator AWp is shown as a complete circle, while the posterior wall indicator BWp is shown with a part of the circle missing. Note that in Figure 2, the posterior wall indicator BWp is located closer to the lesser curvature indicator LCp than the anterior wall indicator AWp.
[0021] In other words, the organ model in this embodiment has a figure representing the organ shape ES, which is a simple connected figure, and its outer circumference has two first and second curved portions, the first portion (e.g., the greater curvature portion GCp) includes the first curved portion, the second portion (e.g., the lesser curvature portion LCp) includes the second curved portion, and a third portion (e.g., the posterior wall portion BWp) and a fourth portion (e.g., the anterior wall portion AWp) are provided in positions between the first and second curved portions, and these first portion (greater curvature portion GCp), second portion (lesser curvature portion LCp), third portion (posterior wall portion BWp), and fourth portion (anterior wall portion AWp) are each associated with different anatomical parts.
[0022] The third and fourth parts are contained within a closed surface enclosed by the first curved section, the second curved section, and the line connecting the first and second curved sections. Therefore, the sum of the areas of the third and fourth parts is smaller than the area of the closed surface.
[0023] Furthermore, the third and fourth parts are in contact with each other, and as mentioned above, by overlapping a part of the fourth part onto a part of the third part, the third part has a shape in which a part of the fourth part has been cut out. That is, visually, both the third and fourth parts are circular, and the third part appears to be partially covered by the fourth part. In other words, one can perceive two circles overlapping, with the lower circle being the third part and the upper circle being the fourth part, and the three-dimensional arrangement of the third and fourth parts is easily perceived. Due to this arrangement, the area of the third part is smaller than the area of the fourth part. Also, because it is a cut-out shape, the number of vertices of the third part is greater than the number of vertices of the fourth part. In addition, since the third and fourth parts can be said to be planar figures obtained by overlapping and offsetting identical shapes, the third and fourth parts have lines parallel to each other and have similar shapes in some parts.
[0024] In this embodiment, the greater curvature indicator portion GCp, the lesser curvature indicator portion LCp, the anterior wall indicator portion AWp, and the posterior wall indicator portion BWp are easily distinguishable by displaying them differently from other parts within the organ shape ES. For example, the greater curvature indicator portion GCp, the lesser curvature indicator portion LCp, the anterior wall indicator portion AWp, and the posterior wall indicator portion BWp may be shown with different gradations than other parts within the organ shape ES. Alternatively, the posterior wall indicator portion BWp and the anterior wall indicator portion AWp may be displayed with different gradations. For example, the anterior wall indicator portion AWp may be displayed brighter than the posterior wall indicator portion BWp.
[0025] For example, the display may be such that other parts within the organ shape ES, the greater curvature indicator GCp, the lesser curvature indicator LCp, the posterior wall indicator BWp, the anterior wall indicator AWp, and the region division line ABL are progressively brightened from a black level to a white level. In particular, by displaying the posterior wall indicator BWp (third part) as darker than the anterior wall indicator AWp (fourth part), it is made easier to recognize that the posterior wall corresponding to the posterior wall indicator BWp (third part) is located further back (posterior) than the anterior wall corresponding to the anterior wall indicator AWp (fourth part).
[0026] In this embodiment, when each detailed area is an unphotographed area, the marks corresponding to each detailed part, namely the greater curvature indicator portion GCp, the lesser curvature indicator portion LCp, the anterior wall indicator portion AWp, and the posterior wall indicator portion BWp (hereinafter, when it is not necessary to distinguish between these marks corresponding to detailed areas, they will be collectively referred to as the detailed indicator portion DAP), may be left uncolored. When an area is photographed, these detailed indicator portions DAP may be colored with a predetermined color. In the following description, an example in which unphotographed areas are left uncolored and photographed areas are colored will be described, but the display form may be different to allow for the identification of unphotographed and photographed areas, and patterns may be changed. In addition, for the fundus AP2 where no detailed parts exist, and for each part other than the stomach (esophagus, duodenum, etc.), the display form may also be different to allow for the identification of unphotographed and photographed areas.
[0027] Furthermore, in this embodiment, if all detailed areas within each area AP have been captured, the entire area AP demarcated by the area division line ABL may be colored with a predetermined color.
[0028] The endoscopic system 1 shown in Figure 1 is designed to display both captured and uncaptured areas of a subject observed by the endoscope 10. In this embodiment, an example of observing the stomach, a human organ, as the subject will be described. The endoscope 10 may have the function of observing in two observation modes: normal light observation and special light observation. The endoscopic system 1 receives endoscopic images (image data) directly or indirectly from the endoscope 10 to the video system center 20. The operator, as the user of the endoscopic system 1, performs an endoscopic examination of a predetermined tubular organ (also simply called a tubular organ or organ), such as the stomach, in a patient lying on a bed, for example. In the following description, an example of observing the stomach with an endoscope will be described, but it is applicable not only to the stomach but also to various tubular organs of the human body and various subjects other than the human body.
[0029] The endoscope 10 has an operating section (not shown) and a flexible insertion section. The endoscope 10 is, for example, an endoscope for upper gastrointestinal examination. Illumination light is supplied to the endoscope 10 from a light source device (not shown). This illumination light is emitted from the tip of the insertion section. This illumination light illuminates the inside of a predetermined tubular organ, which is the inside of the subject into which the tip of the insertion section is inserted.
[0030] The tip of the insertion section is provided with an objective optical system (not shown) and an image sensor. The image sensor's imaging surface is positioned at the imaging position of the objective optical system, and it converts the optical image of the subject formed on the imaging surface into an imaging signal. In this way, the image sensor images the inner wall of the stomach or other organ illuminated by the illumination light from the light source device. The objective optical system and the image sensor constitute an imaging unit (or imaging device) that images the inside of a tubular organ and outputs the image. The imaging signal obtained by the image sensor is supplied to the video system center 20.
[0031] The video system center 20 performs predetermined signal processing on the imaging signal from the endoscope 10 to obtain an endoscopic image. This endoscopic image is provided to the display device 30, and the endoscopic image becomes available for display on the display screen 31 of the display device 30. The display device 30 is a display device composed of, for example, an LCD (liquid crystal panel).
[0032] In this embodiment, an image processing device 40 is provided to distinguish between areas that have been photographed by the endoscope 10 and areas that have not yet been photographed by the endoscope 10 within a tubular organ. In Figure 1, the image processing device 40 is provided separately from the video system center 20, but it may also be provided within the video system center 20. Furthermore, some components of the image processing device 40 may be provided in the video system center 20. In addition, all or some components of the image processing device 40 may be located on the cloud.
[0033] The image processing device 40 includes an image acquisition unit 41, a shooting area determination unit 42, a shooting area display control unit 43, and a model storage unit 44. Each component within the image processing device 40 may be composed of a processor using a CPU (Central Processing Unit), FPGA (Field Programmable Gate Array), NPU (Neural Processing Unit), GPU (Graphics Processing Unit), etc., and may control each part by operating according to a program stored in a memory (not shown), or may implement some or all of the functions with hardware electronic circuits.
[0034] Endoscopic images generated by the video system center 20 are supplied to the image acquisition unit 41 of the image processing device 40. The image acquisition unit 41 takes the endoscopic images from the video system center 20 and provides them to the imaging site determination unit 42. The imaging site determination unit 42 determines the location within the organ that was photographed by the endoscope 10 based on the endoscopic images. For example, for the stomach, the imaging site determination unit 42 determines which of the above-mentioned areas was photographed: the cardia AP1, the fundus AP2, the upper body AP3, the middle body AP4, the lower body AP5, the gastric angle AP6, or the antrum AP7. Furthermore, for the cardia AP1, upper body AP3, middle body AP4, lower body AP5, gastric angle AP6, and antrum AP7, it also determines which of the detailed areas—the greater curvature, lesser curvature, anterior wall, or posterior wall—was photographed, and outputs the site determination result to the imaging site display control unit 43. The imaging area determination unit 42 is configured to perform area determination when the entire area AP or detailed area has been imaged. When the imaging area determination unit 42 obtains an area determination result indicating that an area has been determined, it means that the area has been imaged.
[0035] For example, the imaging area determination unit 42 may be configured with a site determination AI (artificial intelligence) having an inference model that determines the site included in the image captured by the endoscope 10 by performing inference processing on the input endoscopic image. Such an inference model can be constructed by obtaining a large amount of training data obtained by annotating each image of the endoscopic image obtained by the endoscope to indicate which site and / or detailed site each image belongs to, and then feeding this large amount of training data to the inference model for deep learning. Such a site determination AI infers the site included in the input endoscopic image and outputs site determination result information, such as the site name, as an inference result.
[0036] The model memory unit 44 stores information on organ models for various organs of the human body, for example. In this embodiment, the organ model stored in the model memory unit 44 is, for example, the organ model shown in Figure 2, in which the organ shape ES has first and second curved portions, and the first to fourth portions are each associated with different anatomical parts. In the example of organ model M1 in Figure 2, each part AP is partitioned by a part division line ABL, and some parts AP further have marks corresponding to the greater curvature, lesser curvature, anterior wall, and posterior wall details: greater curvature indicator part GCp, lesser curvature indicator part LCp, anterior wall indicator part AWp, and posterior wall indicator part BWp.
[0037] (Display Control) The imaging site display control unit 43 outputs imaging information indicating that an anatomical part has been imaged in the subject. The imaging information includes information for displaying a model such as the organ model M1 in Figure 2, and information for indicating whether each part, including part AP, and detailed parts are unimaged or have been imaged. That is, the imaging site display control unit 43 creates display data based on the model information read from the model storage unit 44 and outputs it to the display device 30. As a result, for example, the organ model M1 shown in Figure 2 is displayed on the display screen 31 of the display device 30. Based on the imaging site determination result from the imaging site determination unit 42, the imaging site display control unit 43 changes the display of marks indicating parts and detailed instruction parts DAP on the organ model M1. That is, if the imaging site determination result indicates that a predetermined part or detailed instruction part DAP has changed from an unimaged area to an imaged area, the imaging site display control unit 43 controls the display so that the display part of that part and the detailed instruction part DAP representing the detailed parts are displayed differently from the display of the unimaged area. For example, the imaging area display control unit 43 generates display data for coloring the marks (detailed indication area DAP) corresponding to the detailed areas of the already-imaging region and outputs it to the display device 30. Also, when the entire area has been imprinted, the imaging area display control unit 43 generates display data for coloring the entire area and outputs it to the display device 30.
[0038] In Figure 2, it was explained that for each part AP having a detailed indication area DAP, the change from an un-photographed area to a photographed area is indicated by a change in the display, such as coloring. Similarly, for the fundus AP2, which does not have a distinction for the detailed indication area DAP, the change from an un-photographed area to a photographed area is also indicated by a change in the display, such as coloring. Furthermore, for parts other than the stomach, such as the esophagus, esophagogastric junction, pylorus, duodenal bulb, and descending portion of the duodenum, the change from an un-photographed area to a photographed area is also indicated by a change in the display, such as coloring.
[0039] Furthermore, the imaging area display control unit 43 has a memory (not shown) that stores information in this memory indicating whether each detailed area and area AP is an area that has been imaged or not. If the imaging area display control unit 43 indicates, based on the information in the memory and the imaging area determination result, that both the anterior and posterior walls of a certain area AP are areas that have been imaged, it may apply a coloring process to the entire area AP, excluding the marked areas indicating the greater curvature indication portion GCp and the lesser curvature indication portion LCp. Also, if the imaging area display control unit 43 indicates, based on the information in the memory and the imaging area determination result, that all detailed areas within a area AP are areas that have been imaged, it may apply a coloring process to the entire area AP.
[0040] In the example shown, the imaging area display control unit 43 outputs the generated display data to the display device 30. However, the generated display data may also be output to the video system center 20, and the display control unit of the video system center 20 may simultaneously display the endoscopic image and the organ model M1 on the display screen 31 of the display device 30.
[0041] (Operation) Next, the operation of the embodiment configured as described above will be explained with reference to Figures 4 and 5. Figure 4 is a flowchart for explaining the operation of the first embodiment. Figure 5 is an explanatory diagram showing model images G1 to G4 displayed on the display screen 31 of the display device 30.
[0042] For example, during an endoscopic examination, the insertion part of the endoscope 10 is inserted into the body to observe the target area, such as the stomach. An image sensor is provided at the tip of the insertion part, and an endoscopic image is acquired by the image sensor. The video system center 20 outputs the endoscopic image acquired by the endoscope 10 to the image processing device 40.
[0043] The imaging area display control unit 43 of the image processing device 40 acquires a model image from the model storage unit 44 (S1). The imaging area display control unit 43 generates display data for the acquired model image and outputs it to the image processing device 30. In this way, the organ model M1 shown in Figure 2 is displayed on the display screen 31 of the display device 30 (S2).
[0044] The image acquisition unit 41 of the image processing device 40 acquires, from the video system center 20, an endoscopic image that is currently captured and output by the endoscope 10 (S3). The imaged region determination unit 42 receives the endoscopic image acquired by the image acquisition unit 41, and performs region determination for the region imaged by the endoscope 10 using, for example, a region determination AI (S4). Note that an image input to the region determination AI constituting the imaged region determination unit 42 may be an inappropriate image to be recorded, in which the mucous membrane is not clearly captured, for reasons such as being dark, having halation, insufficient air, or having mucus attached. In this case, the region determination AI outputs a determination result indicating that the image is inappropriate. Further, the region determination AI outputs a determination result of indeterminable when it cannot determine which region has been imaged. When a determination result indicating a region for the region or detailed region is obtained by the imaged region determination unit 42, the region or detailed region is regarded as an already-imaged region. The region determination result from the imaged region determination unit 42 is supplied to the imaged region display control unit 43.
[0045] The imaged region display control unit 43 determines whether or not a region determination result indicating that a region has become an already-imaged region has been input (S5). When a determination result of indeterminable or inappropriate is obtained from the region determination AI, that is, when a region determination result indicating already-imaged is not obtained by the imaged region determination unit 42 (NO in S5), the process proceeds to S5A to perform display indicating indeterminable or inappropriate, and then returns to S3. When a predetermined region or detailed region becomes an already-imaged region, and the region determination result for the region or detailed region is input to the imaged region display control unit 43, the imaged region display control unit 43 determines that the region or detailed region has been already imaged (YES in S5), and shifts the process to S6. The imaged region display control unit 43 reads out information related to the already-imaged region from the memory, and thereby determines whether or not the entire area of the region AP to which the relevant detailed region belongs has become an already-imaged region (S6).
[0046] When the imaging region display control unit 43 determines that any of the detailed regions included in the region to which the detailed region that has become an imaged region belongs has not been imaged (NO determination in S6), the imaging region display control unit 43 colors the detailed instruction portion DAP corresponding to the detailed region that has become an imaged region (S7), and returns the process to S3. In this way, the display of the organ model M1 on the display screen 31 of the display device 30 is updated.
[0047] An image G1 in FIG. 5A shows the organ model M1 in such a state, and indicates that the front wall instruction portion AWp, the back wall instruction portion BWp, and the greater curvature instruction portion GCp of the upper body AP3 are colored by the filling of the detailed instruction portion DAP. That is, it can be seen from the image G1 in FIG. 5A that only the esophagogastric junction, the front wall, the back wall and the greater curvature of the upper body AP3, and the pylorus have been imaged.
[0048] As described above, in the present embodiment, the two-dimensional display device 30 displays the unimaged regions and imaged regions of each region of the subject, for example, the stomach region and each detailed region, so that they can be recognized as three-dimensional regions that the doctor is aware of, and doctors and the like can reliably and easily determine the unimaged regions and imaged regions.
[0049] When the imaging region display control unit 43 uses the information of the imaged regions stored in the memory to determine that all the detailed regions included in the region to which the detailed region that has become an imaged region belongs have become imaged regions (YES determination in S6), the imaging region display control unit 43 colors the entire region AP including the detailed regions that have become imaged regions (S8), and shifts the process to S9.
[0050] An image G2 in FIG. 5B shows the organ model M1 in such a state, and indicates that the entire region showing the upper body AP3 is colored by filling. That is, from the image G2 in FIG. 5B, it can be seen that, of the entire region of the stomach, the cardia, the entire upper body AP3, that is, all of the front wall, the back wall, the greater curvature, and the lesser curvature, and the pylorus have been imaged.
[0051] In this embodiment, when all detailed areas of a particular part of the stomach have been photographed, the entire area of that part is colored, making it easy to see whether or not all areas of each part have been photographed.
[0052] In S9, the imaging area display control unit 43 determines whether all areas have been imaged. If the imaging area display control unit 43 determines that not all areas have been imaged, it returns to S3; if it determines that all areas have been imaged, it terminates the process. Even if all areas have been imaged, the imaging area determination unit 42 may continue to make determinations until it receives a signal indicating the end of the examination.
[0053] Thus, in this embodiment, a display device that performs a general two-dimensional display can show in three dimensions whether each part and detailed part within an organ has been photographed or not. This allows the user to easily and accurately determine which areas have been photographed and which have not.
[0054] (Second Embodiment) Figure 6 is a flowchart showing the operation flow adopted in the second embodiment. The hardware configuration of this embodiment is the same as in Figure 1. In the first embodiment, the entire region AP was colored when all detailed parts within the region AP had been captured. In contrast, in this embodiment, when both the anterior wall indication portion AWp and the posterior wall indication portion BWp within the region AP have been captured, the portion of the region AP excluding the greater curvature indication portion GCp and the lesser curvature indication portion LCp is colored.
[0055] The flowchart in Figure 6 differs from the flowchart in Figure 4 in that it adds the processes in steps S11 and S12. Specifically, in S6, if it is determined that all the detailed parts of the relevant area are not areas that have been photographed (NO in S6), the determination in S11 is made before proceeding to S7. In S11, the photographed area display control unit 43 determines whether both the front wall indicated part AWp and the rear wall indicated part BWp within the AP have been photographed. If neither of these detailed parts is determined to be an area that has been photographed, the photographed area display control unit 43 proceeds to S7 and colors the marks of the detailed parts that have been photographed.
[0056] In this embodiment, if the imaging area display control unit 43 determines in S11 that both the anterior wall indication portion AWp and the posterior wall indication portion BWp within the AP have been imaged, it proceeds to S12 to color the area of the AP excluding the greater curvature indication portion GCp and the lesser curvature indication portion LCp.
[0057] Figure 7 is an explanatory diagram showing an example of the display in this case. In image G3 of Figure 7A, the posterior wall indication portion BWp is colored (shown as filled in), indicating that the detailed area of the posterior wall of the upper body AP3 is an area that has been photographed. In this state, a photographic area determination result indicating that the detailed area of the anterior wall of the upper body AP3 has been photographed is supplied to the photographic area display control unit 43. In this case, in S11, the photographic area display control unit 43 determines that for the upper body AP3, the detailed areas of both the anterior and posterior walls are photographed, and in S12, it controls the area of the upper body AP3 excluding the greater curvature indication portion GCp and the lesser curvature indication portion LCp to be colored. Thus, the display screen 31 of the display device 30 is updated to the display shown in image G4 of Figure 7B.
[0058] In Figure 7B, a wide area including the front wall indication portion AWp and the rear wall indication portion BWp is colored (shown as a solid fill), making it easy to determine that the front and rear walls have already been photographed.
[0059] Thus, in this embodiment, the same effects as in the first embodiment can be obtained, and it is also possible to visually confirm that both the front and rear walls have been photographed.
[0060] (Modification) Figure 8 is an explanatory diagram showing a modification. Figure 8 shows three types of marks representing the front wall indicator portion and the rear wall indicator portion. In each of the above embodiments, as shown in Figure 8A, the front wall indicator portion AWp is represented by a circular mark, and the rear wall indicator portion BWp is represented by a circular mark with a portion missing, by overlapping the circular mark of the front wall indicator portion AWp onto a part of the circle. In Figure 8B, the front wall indicator portion AWp2 is represented by a square mark, and the rear wall indicator portion BWp2 is represented by a square mark with a portion missing, by overlapping the square mark of the front wall indicator portion AWp2 onto a part of the square. In addition, in Figure 8C, the front wall indicator portion AWp3 is represented by a triangular mark, and the rear wall indicator portion BWp3 is represented by a triangular mark with a portion missing, by overlapping the triangular mark of the front wall indicator portion AWp3 onto a part of the triangle.
[0061] In all of Figures 8A to 8C, a pair of identical shapes are arranged so that parts of them overlap. The overlapping, partially missing mark is used to indicate the back wall, and the unmissing mark is used to indicate the front wall, making it easier to visually distinguish between the back and front walls. Furthermore, by making the density of the mark indicating the back wall darker than that of the mark indicating the front wall, it is easier to visually recognize that the back wall is located further back than the front wall. Note that Figure 8 is just one example, and it is clear that various other shapes of marks can be used.
[0062] Furthermore, the examples in Figures 8D to 8F show cases where the size of the rear wall indicator portions BWp, BWp2, and BWp3 in Figures 8A to 8C has been reduced, and rear wall indicator portions BWps, BWp2s, and BWp3s have been adopted. That is, since the rear wall indicator portions BWps, BWp2s, and BWp3s are smaller in size than the front wall indicator portions AWp, AWp2, and AWp3, respectively, the rear wall and the front wall are made even easier to visually recognize.
[0063] While Figure 8 shows an example of combining figures of the same shape, you may also use combinations of shapes that are not similar but are quite close (combinations of N-sided polygons and M-sided polygons, where N=M), such as a rectangle and a square.
[0064] (Modified Examples) Figures 9 and 10 are explanatory diagrams showing modified examples. In each of the above embodiments, whether the detailed areas of the anterior and posterior walls of a certain region AP are unphotographed or already photographed is indicated by the uncolored or colored circular anterior wall indicator portion AWp and posterior wall indicator portion BWp. The anterior wall indicator portion AWp and posterior wall indicator portion BWp are shown as being positioned in a portion of the entire region AP, excluding the greater curvature indicator portion GCp and lesser curvature indicator portion LCp.
[0065] In contrast, the examples in Figures 9 and 10 divide the entire region AP, excluding the greater curvature indicator portion GCp and the lesser curvature indicator portion LCp, into two parts, with one part designated as the anterior wall indicator portion AWp and the other as the posterior wall indicator portion BWp. In the examples in Figures 9 and 10, the posterior wall indicator portion BWp is placed on the lesser curvature indicator portion LCp side, and the anterior wall indicator portion AWp is placed on the greater curvature indicator portion GCp side. Other configurations and operations, such as leaving unphotographed areas uncolored and coloring photographed areas, are the same as in the embodiments described above. Alternatively, the anterior wall indicator portion AWp may be placed on the lesser curvature indicator portion LCp side, and the posterior wall indicator portion BWp may be placed on the greater curvature indicator portion GCp side.
[0066] The examples in Figures 9 and 10 show different methods of displaying the anterior wall indication portion AWp and the posterior wall indication portion BWp, and the design and display methods of other organ models may be the same as those of the embodiments described above. For example, the example image G5 in Figure 9 shows that only the posterior wall indication portion BWp of the upper body AP3 is a captured area, and the other areas are not captured areas. Also, the example image G6 in Figure 10 shows that all detailed areas of the upper body AP3 and gastric angle AP6 are captured areas, only the posterior wall indication portion BWp of the middle body AP4 is not captured area, and only the posterior wall indication portion BWp of the lower body AP5 is captured area.
[0067] In the examples shown in Figures 9 and 10, if only one of the anterior or posterior walls of each AP has been imaged, the other unimaged area will be indicated by the letters AWpC (A in Figure 10) and BWpC (P in Figure 10) within the display areas of the anterior wall indicator AWp and the posterior wall indicator BWp, respectively, to show that it is an unimaged area and which of the anterior or posterior wall is the unimaged area. In the examples shown in Figures 9 and 10, the letter AWpC (A) indicates that the anterior wall of the lower body AP5 is an unimaged area, and the letter BWpC (P) indicates that the posterior wall of the midbody AP4 is an unimaged area.
[0068] Thus, the same effects as those of the above embodiments can be obtained in this modified form as well. Furthermore, in each part, if only one of the anterior wall or posterior wall has been imaged, characters clearly indicating the other unimaged area are displayed within the areas of the anterior wall indicator AWp and the posterior wall indicator BWp, allowing the user to easily identify the unimaged area.
[0069] The other configurations and effects are the same as in Figure 1.
[0070] The present invention is not limited to the embodiments described above, and the components can be modified and implemented in practice without departing from the spirit of the invention. Furthermore, various inventions can be formed by appropriately combining the multiple components disclosed in the embodiments described above. For example, some components of all the components shown in the embodiments may be deleted. Moreover, components from different embodiments may be appropriately combined.
[0071] Furthermore, many of the controls and functions described here, primarily those illustrated in the flowchart, can be configured by program, and these controls and functions can be realized by a computer reading and executing the program. This program can be recorded or stored, in whole or in part, as a computer program product on portable media such as flexible disks, CD-ROMs, or non-volatile memory, or on storage media such as hard disks or volatile memory. It can be distributed or provided at the time of product shipment or via portable media or communication lines. Users can easily implement the image processing apparatus, control method for the image processing apparatus, and image processing program of this embodiment by downloading and installing the program on their computer via a communication network, or by installing it from a recording medium to their computer.
[0072] This application is filed on the basis of priority claims to Provisional Application 63 / 788,929 filed with the United States Patent and Trademark Office on 15 April 2025 and Design Application 30 / 026,773 filed with the United States Patent and Trademark Office on 6 October 2025, and the above disclosures are incorporated herein by reference in the specification and claims.
Claims
1. The image processing device includes a processor, which acquires an image of a subject, determines the anatomical parts of the subject shown in the image, and outputs imaging information indicating that the anatomical parts of the subject have been photographed, wherein the imaging information includes a first figure representing the subject, the outer perimeter of the first figure includes a first curved portion and a second curved portion, and the first part including the first curved portion, the second part including the second curved portion, the third part located between the first curved portion and the second curved portion, and the fourth part located between the first curved portion and the second curved portion are each associated with different anatomical parts.
2. The image processing apparatus according to claim 1, wherein the first figure is a simply connected figure.
3. The image processing apparatus according to claim 1, wherein the first figure is shaped like a stomach.
4. The image processing apparatus according to claim 1, wherein the third portion and the fourth portion are housed in a closed surface enclosed by the first curved portion, the second curved portion, and the line connecting the first curved portion and the second curved portion.
5. The image processing apparatus according to claim 4, wherein the sum of the area of the third portion and the area of the fourth portion is smaller than the area of the closed surface.
6. The image processing apparatus according to claim 5, wherein the area of the third portion is smaller than the area of the fourth portion.
7. The image processing apparatus according to claim 5, wherein the third and fourth parts have similar shapes in part, and the sides constituting the third part are shorter than the corresponding sides of the fourth part.
8. The image processing apparatus according to claim 5, wherein the third part and the fourth part are in contact with each other.
9. The image processing apparatus according to claim 5, wherein the number of vertices in the third portion is greater than the number of vertices in the fourth portion.
10. The image processing apparatus according to claim 9, wherein the third part and the fourth part have lines parallel to each other.
11. The first figure is shaped like a stomach, the first part represents the greater curvature of the first anatomical region, the second part represents the lesser curvature of the first anatomical region, the third part represents the posterior wall of the first anatomical region, the fourth part represents the anterior wall of the first anatomical region, the third part and the fourth part are housed in a closed surface enclosed by the first curved part, the second curved part, and the line connecting the first curved part and the second curved part, and have a smaller area than the closed surface and are in contact with each other, the distance from the third part to the second part is shorter than that from the fourth part and the area is smaller than that of the fourth part, the image processing apparatus according to claim 1.
12. The image processing apparatus according to claim 1, wherein the image is obtained by an endoscope imaging the subject.
13. The image processing apparatus according to claim 12, wherein the imaging information includes information on whether the anatomical region is an area that has been imaged by the endoscope or an unimaged area that has not been imaged by the endoscope.
14. The image processing apparatus according to claim 12, wherein the processor displays the first to fourth parts in a different grayscale than the other parts of the first figure.
15. The image processing apparatus according to claim 12, wherein the processor displays the third portion darker than the fourth portion.
16. The image processing apparatus according to claim 13, wherein when the anatomical region becomes an area that has been photographed by the endoscope, the processor displays the corresponding portion of the first to fourth portions in a manner different from the unphotographed area.
17. The third and fourth portions are housed in a closed surface enclosed by the first curved portion, the second curved portion, and the line connecting the first and second curved portions, and the processor displays the entire closed surface in a manner different from the unphotographed area when all of the first to fourth portions within the anatomical region are areas that have been photographed. The image processing apparatus according to claim 13.
18. The third and fourth portions are housed in a closed surface enclosed by the first curved portion, the second curved portion, and the line connecting the first and second curved portions, and the processor displays the area of the entire closed surface excluding the first and second portions in a manner different from the unphotographed area when both the third and fourth portions within the anatomical region are areas that have been photographed.
19. A control method for an image processing device, comprising: acquiring an image of a subject; determining the anatomical parts of the subject shown in the image; and outputting imaging information indicating that the anatomical parts of the subject have been photographed, wherein the imaging information includes a first figure representing the subject, the outer perimeter of the first figure includes a first curved portion and a second curved portion, and a first part including the first curved portion, a second part including the second curved portion, a third part located between the first and second curved portions, and a fourth part located between the first and second curved portions are each associated with different anatomical parts.
20. An image processing program for causing a computer to perform the following steps: acquire an image of a subject, determine the anatomical parts of the subject shown in the image, and output imaging information indicating that the anatomical parts of the subject have been photographed, wherein the imaging information includes a first figure representing the subject, the outer perimeter of the first figure includes a first curved portion and a second curved portion, and a first part including the first curved portion, a second part including the second curved portion, a third part located between the first and second curved portions, and a fourth part located between the first and second curved portions are each associated with different anatomical parts.