Measurement device, measurement method, and program

Abstract

This measurement device has a control unit. The control unit acquires three-dimensional data including three-dimensional coordinates of three or more points on a subject, acquires edge information indicating whether or not each of the three or more points is an edge point on the edge of the subject, displays a three-dimensional image or a two-dimensional image on a display, sets a measurement reference indicating a reference position of measurement, sets a measurement region on the basis of a point designated on the three-dimensional image or the two-dimensional image of the subject by a pointing device, sets a point included in the measurement region as a measurement point, and generates distance information indicating the distance between the measurement reference and the measurement point. The edge information indicates that the measurement point is the edge point.

Description

Measuring device, measuring method, and program

[0001] The present invention relates to a measuring device, a measuring method, and a program. This application claims priority based on US Provisional Patent Application No. 63 / 720,364 filed on November 14, 2024, and International Patent Application PCT / JP2025 / 005312 filed on February 18, 2025, the contents of which are incorporated herein by reference.

[0002] Industrial endoscope devices are used for inspections (endoscopic inspections) of abnormalities and corrosion inside boilers, pipes, aircraft engines, heat exchangers, etc. Also, measuring endoscope devices having a measuring function for measuring the size of a specific region of a subject are used.

[0003] When a user designates one or more measurement points on a subject, the measuring endoscope device executes measurement processing based on the measurement points. For example, the measuring endoscope device calculates the distance between two measurement points, the distance between a measurement point and a reference plane, and the area of a region surrounded by three or more measurement points. Points on the edge of the subject are useful in measurement and are used as measurement points in various types of measurement. For example, the device disclosed in Patent Document 1 measures the area of a defective part of a subject.

[0004] Japanese Patent Application Laid-Open No. 2004-033367

[0005] The device disclosed in Patent Document 1 requires information on two or more points manually input by the user in order to obtain a line approximating the edge of the defective part. Since the user needs to manually input information on two or more points, it is laborious for the user.

[0006] An object of the present invention is to provide a measuring device, a measuring method, and a program that can reduce the labor of the user required for measurement and can execute measurement efficiently.

[0007] According to an aspect of the present invention, the measuring device includes a control unit. The control unit acquires three-dimensional data and edge information. The three-dimensional data includes the three-dimensional coordinates of three or more points on the subject calculated based on a two-dimensional image of the subject acquired by an endoscope. The edge information indicates whether each of the three or more points is an edge point on an edge of the subject. The control unit displays the three-dimensional image or the two-dimensional image on a display. The three-dimensional image is an image of a three-dimensional shape including points having the three-dimensional coordinates. The control unit sets a measurement reference indicating a reference position for measurement based on at least one of the three or more points, and sets a measurement area based on a point specified on the three-dimensional image or the two-dimensional image by a pointing device. The measurement area has a preset shape and is independent of the measurement reference. The control unit sets points included in the measurement area as measurement points. The edge information indicates that the measurement points are edge points. The control unit generates distance information indicating the distance between the measurement reference and the measurement points.

[0008] According to an aspect of the present invention, the control unit may calculate a line that approximates the edge of the subject based on the point specified on the three-dimensional image or the two-dimensional image and the edge point indicated by the edge information, and may set the measurement area based on the line.

[0009] According to an aspect of the present invention, the control unit may identify an edge including the edge point indicated by the edge information based on the three-dimensional data, or it may set one of the one or more edges of the subject as the measurement area based on the point specified on the three-dimensional image or the two-dimensional image.

[0010] According to an aspect of the present invention, the control unit may detect one or more feature regions on the subject based on the three-dimensional data or the two-dimensional image. The three-dimensional shape may have common features in each of the one or more feature regions. The control unit may set at least one of the one or more feature regions as the measurement region.

[0011] According to an aspect of the present invention, the control unit may set a new measurement area based on the measurement point and a point other than the measurement point.

[0012] According to an aspect of the present invention, the control unit may identify an edge containing the edge point indicated by the edge information based on the three-dimensional data, or it may set the new measurement area based on a point included in the same edge as the edge containing the measurement point.

[0013] According to an aspect of the present invention, the control unit may set two or more measurement points included in the measurement area, set a removal area including a part of the measurement area, or remove the measurement points included in the removal area from the two or more measurement points.

[0014] According to an aspect of the present invention, the control unit may generate distance information indicating the distance between the measurement reference and the remaining measurement points obtained by removing the measurement points included in the removal area from the two or more measurement points.

[0015] According to an aspect of the present invention, the control unit may set two or more measurement points included in the measurement area, and may generate first distance information indicating the distance between the measurement reference and each of the two or more measurement points. After the first distance information is generated, the control unit may set a removal area including a part of the measurement area. The control unit may remove the measurement points included in the removal area from the two or more measurement points. The control unit may generate second distance information indicating the distance between the measurement reference and the measurement points remaining after removing the measurement points included in the removal area from the two or more measurement points.

[0016] According to an aspect of the present invention, the control unit may generate the edge information based on the three-dimensional data.

[0017] According to an aspect of the present invention, the control unit may set the measurement reference which is one of a surface, a line, and a point.

[0018] According to an aspect of the present invention, the control unit may set the measurement reference before the measurement area is set.

[0019] According to an aspect of the present invention, the control unit may set the measurement area before the measurement standard is set.

[0020] According to an aspect of the present invention, the control unit may set the measurement area, which is a figure having a predetermined size.

[0021] According to an aspect of the present invention, the control unit may display the measurement area on the display.

[0022] According to an aspect of the present invention, the control unit may make the display state of a point on the three-dimensional image corresponding to the measurement point different from the display state of a point on the three-dimensional image corresponding to a point other than the measurement point.

[0023] According to an aspect of the present invention, the control unit may calculate the size of a figure that approximates the edge of the subject based on two or more measurement points, including the measurement point.

[0024] According to an aspect of the present invention, the control unit may generate distance information indicating the distance between the measurement reference and each of two or more measurement points including the measurement point.

[0025] According to an aspect of the present invention, the control unit may display the distance information on the display.

[0026] According to an aspect of the present invention, the measurement method includes the following: Acquiring three-dimensional data including the three-dimensional coordinates of three or more points on a subject calculated based on a two-dimensional image of the subject acquired by an endoscope. Acquiring edge information indicating whether each of the three or more points is an edge point on an edge of the subject. Displaying the three-dimensional image or the two-dimensional image on a display, the three-dimensional image being an image of a three-dimensional shape including points having the three-dimensional coordinates. Setting a measurement reference indicating a reference position for measurement based on at least one of the three or more points. Setting a measurement area based on a point specified on the three-dimensional image or the two-dimensional image by a pointing device, the measurement area having a preset shape and being independent of the measurement reference. Setting a point included in the measurement area as a measurement point, and the edge information indicating that the measurement point is an edge point. Generating distance information indicating the distance between the measurement reference and the measurement point.

[0027] According to an aspect of the present invention, the program causes the computer to perform the following processes: Obtain three-dimensional data including the three-dimensional coordinates of three or more points on a subject calculated based on a two-dimensional image of the subject obtained by an endoscope. Obtain edge information indicating whether each of the three or more points is an edge point on an edge of the subject. Display the three-dimensional image or the two-dimensional image on a display, and the three-dimensional image is an image of a three-dimensional shape including the points having the three-dimensional coordinates. Set a measurement reference indicating a reference position for measurement based on at least one of the three or more points. Set a measurement area based on a point specified on the three-dimensional image or the two-dimensional image by a pointing device, and the measurement area has a preset shape and is independent of the measurement reference. Set the points included in the measurement area as measurement points, and the edge information indicates that the measurement points are the edge points. Generate distance information indicating the distance between the measurement reference and the measurement points.

[0028] According to each of the above embodiments, the measuring device, measuring method, and program can reduce the effort required by the user for measurement and enable efficient measurement.

[0029] This is a perspective view showing the overall configuration of an endoscope device according to the first embodiment of the present invention. This is a block diagram showing an example of the internal configuration of an endoscope device according to the first embodiment of the present invention. This is a flowchart showing an example of the measurement process procedure in the first embodiment of the present invention. This is a diagram showing an example of the three-dimensional (3D) shape of a subject in the first embodiment of the present invention. This is a diagram showing an example of the 3D shape of a subject in the first embodiment of the present invention. This is a diagram showing an example of the measurement area in the first embodiment of the present invention. This is a diagram showing an example of the measurement point cloud in the first embodiment of the present invention. This is a diagram showing an example of a 3D image in the first embodiment of the present invention. This is a diagram showing an example of a 3D image in the first embodiment of the present invention. This is a diagram showing an example of a 3D image in the first embodiment of the present invention. This is a diagram showing an example of a straight line used to set the measurement area in the first modified example of the first embodiment of the present invention. This is a diagram showing an example of the measurement point cloud in the first modified example of the first embodiment of the present invention. This is a flowchart showing an example of the measurement process procedure in the second modified example of the first embodiment of the present invention. This is a diagram showing an example of the edge point vector in the second modified example of the first embodiment of the present invention. This is a diagram showing an example of the measurement point cloud in the second modified example of the first embodiment of the present invention. This is a flowchart showing an example of the measurement process procedure in the third modified example of the first embodiment of the present invention. This is a diagram showing an example of the measurement point cloud in the third modified example of the first embodiment of the present invention. This is a diagram showing an example of the measurement point cloud in the fourth modified example of the first embodiment of the present invention. This figure shows an example of a line segment used to set the measurement area in the fourth modified example of the first embodiment of the present invention. This flowchart shows an example of the measurement process procedure in the fifth modified example of the first embodiment of the present invention. This flowchart shows an example of the measurement point update process procedure in the fifth modified example of the first embodiment of the present invention. This figure shows an example of a measurement point group in the fifth modified example of the first embodiment of the present invention. This figure shows an example of a measurement point group in the fifth modified example of the first embodiment of the present invention. This flowchart shows an example of the measurement process procedure in the sixth modified example of the first embodiment of the present invention. This flowchart shows an example of the measurement point update process procedure in the sixth modified example of the first embodiment of the present invention. This flowchart shows an example of the measurement process procedure in the second embodiment of the present invention.This figure shows an example of a measurement area and a measurement point cloud in a second embodiment of the present invention. This figure shows an example of a reference point and distance information in a second embodiment of the present invention. This figure shows an example of a reference line in a third embodiment of the present invention. This figure shows an example of a measurement area and a measurement point cloud in a third embodiment of the present invention. This figure shows an example of distance information in a third embodiment of the present invention. This figure shows an example of a reference line, measurement area, and measurement point cloud in a modified example of the third embodiment of the present invention. This figure shows an example of distance information in a modified example of the third embodiment of the present invention. This flowchart shows an example of a measurement processing procedure in a fourth embodiment of the present invention. This figure shows an example of the 3D shape of a subject in a fourth embodiment of the present invention. This figure shows an example of a two-dimensional (2D) image of a subject in a fourth embodiment of the present invention. This flowchart shows an example of a measurement processing procedure in a fifth embodiment of the present invention. This figure shows an example of a first reference line and a second reference line in a fifth embodiment of the present invention. This figure shows an example of the 3D shape of a subject in a fifth embodiment of the present invention. This flowchart shows an example of a measurement processing procedure in a first modified example of the fifth embodiment of the present invention. This figure shows an example of the 3D shape of a subject in a first modified example of the fifth embodiment of the present invention. This flowchart shows an example of a measurement processing procedure in a second modified example of the fifth embodiment of the present invention. This figure shows an example of the 3D shape of the subject in the second modified example of the fifth embodiment of the present invention. This figure shows an example of the 3D shape of the subject in the third modified example of the fifth embodiment of the present invention. This flowchart shows an example of the measurement process procedure in the fourth modified example of the fifth embodiment of the present invention. This figure shows an example of the 3D shape of the subject in the fourth modified example of the fifth embodiment of the present invention. This figure shows an example of a 2D image of the subject in the fourth modified example of the fifth embodiment of the present invention. This flowchart shows an example of the measurement process procedure in the fifth modified example of the fifth embodiment of the present invention. This figure shows an example of the 3D shape of the subject in the fifth modified example of the fifth embodiment of the present invention. This flowchart shows an example of the measurement process procedure in the sixth embodiment of the present invention. This figure shows an example of the 3D shape of the subject in the sixth embodiment of the present invention.

[0030] Embodiments of the present invention will be described with reference to the drawings. In the following, an endoscope will be described as an example of a measuring device.

[0031] (First Embodiment) The configuration of the endoscope device 1 according to the first embodiment of the present invention will be explained using Figures 1 and 2. Figure 1 shows the external appearance of the endoscope device 1. Figure 2 shows an example of the internal configuration of the endoscope device 1.

[0032] The endoscope device 1 shown in Figure 1 comprises an insertion unit 2, a main unit 3, an operating unit 4, and a display unit 5. The endoscope device 1 images a subject and generates images. The subject is an industrial product. The user can perform various observations of different subjects by changing the optical adapter attached to the tip 20 of the insertion unit 2, selecting a built-in image processing program, and adding an image processing program.

[0033] The insertion section 2 is inserted into the body of the subject. The insertion section 2 is a long, slender tube that is flexible from the tip 20 to the base. The insertion section 2 images the subject and outputs the imaging signal to the main unit 3. An optical adapter is attached to the tip 20 of the insertion section 2. The main unit 3 is a control device equipped with a storage section for housing the insertion section 2. The operation unit 4 receives user input for the endoscope device 1. The display unit 5 has a display screen and displays images of the subject acquired by the insertion section 2, operation menus, etc. on the display screen.

[0034] The operation unit 4 is the user interface. The display unit 5 is a monitor (display) such as an LCD (Liquid Crystal Display). The display unit 5 may also be a touch panel. In that case, the operation unit 4 and the display unit 5 are integrated.

[0035] The main unit 3 shown in Figure 2 includes an endoscope unit 8, a CCU (Camera Control Unit) 9, and a control device 10.

[0036] The endoscope unit 8 includes a light source device and a bending device (not shown). The light source device supplies illumination light necessary for observation to the tip 20. The bending device bends the bending mechanism built into the insertion section 2.

[0037] The lens 21 and image sensor 28 are built into the tip 20 of the insertion section 2. The lens 21 is an observation optical system. The lens 21 captures the optical image of the subject formed by the optical adapter. The image sensor 28 is an image sensor. The image sensor 28 converts the optical image of the subject into an optical signal and generates an imaging signal. The lens 21 and image sensor 28 constitute a camera.

[0038] The CCU 9 drives the image sensor 28. The imaging signal output from the image sensor 28 is input to the CCU 9. The CCU 9 performs preprocessing on the imaging signal acquired by the image sensor 28, including amplification and noise reduction. The CCU 9 converts the preprocessed imaging signal into a video signal such as an NTSC signal.

[0039] The control device 10 includes a video signal processing circuit 12, a ROM (Read-Only Memory) 13, a RAM (Random Access Memory) 14, a card interface 15, an external device interface 16, a control interface 17, and a control unit 18.

[0040] The video signal processing circuit 12 performs predetermined video processing on the video signal output from the CCU 9. For example, the video signal processing circuit 12 performs video processing related to improving visibility. For example, this video processing includes color reproduction, gradation correction, noise suppression, and edge enhancement. For example, the video signal processing circuit 12 combines the video signal output from the CCU 9 with the graphic image signal generated by the control unit 18. The graphic image signal includes images of the operation screen, etc. The video signal processing circuit 12 outputs the combined video signal to the display unit 5.

[0041] ROM 13 is a non-volatile recording medium on which a program for the control unit 18 to control the operation of the endoscope device 1 is recorded. RAM 14 is a volatile recording medium that temporarily stores information used by the control unit 18 for controlling the endoscope device 1. The control unit 18 controls the operation of the endoscope device 1 based on the program recorded in ROM 13.

[0042] The control unit 18 controls each part of the endoscope device 1. The control unit 18 may be composed of at least one of a processor and a logic circuit. For example, the processor may be at least one of a CPU (Central Processing Unit), a DSP (Digital Signal Processor), and a GPU (Graphics Processing Unit). For example, the logic circuit may be at least one of an ASIC (Application Specific Integrated Circuit) and an FPGA (Field-Programmable Gate Array). The control unit 18 may include one or more processors. The control unit 18 may include one or more logic circuits.

[0043] The computer of the endoscope device 1 may read a program and execute the read program. The program includes instructions that define the operation of the control unit 18. That is, the functions of the control unit 18 may be realized by software.

[0044] The above program may be provided by a "computer-readable recording medium" such as a flash memory. The program may be transmitted to the endoscope device 1 from a computer that holds the program via a transmission medium or by a transmission wave in the transmission medium. The "transmission medium" for transmitting the program is a medium having a function of transmitting information. Media having a function of transmitting information include networks (communication networks) such as the Internet and communication lines (communication lines) such as telephone lines. The above program may realize a part of the above functions. Furthermore, the above program may be a differential file (differential program). The above functions may be realized by a combination of a program already recorded in the computer and the differential program.

[0045] Further, the control unit 18 controls the processing executed by the video signal processing circuit 12. The CCU 9 outputs a video signal. The video signal includes color data of each pixel of the image generated by the imaging device 28. The control unit 18 causes the video signal processing circuit 12 to output the video signal output from the CCU 9 to the display unit 5. The video signal processing circuit 12 outputs the video signal to the display unit 5. The display unit 5 displays an image based on the video signal output from the video signal processing circuit 12. Thereby, the control unit 18 displays the image generated by the imaging device 28 on the display unit 5.

[0046] The control unit 18 displays various types of information on the display unit 5. That is, the control unit 18 displays various types of information on the image.

[0047] For example, the control unit 18 generates a graphic image signal of various types of information. The control unit 18 outputs the generated graphic image signal to the video signal processing circuit 12. The video signal processing circuit 12 synthesizes the video signal output from the CCU 9 and the graphic image signal output from the control unit 18. Thereby, various types of information are superimposed on the image. The video signal processing circuit 12 outputs the synthesized video signal to the display unit 5. The display unit 5 displays an image on which various types of information are superimposed.

[0048] The memory card 42 is connected to the card interface 15. The memory card 42 is a removable recording medium for the endoscope device 1. The card interface 15 takes in control processing information, image information, etc. stored in the memory card 42 into the endoscope device 1. Also, the card interface 15 records control processing information, image information, etc. generated by the endoscope device 1 on the memory card 42.

[0049] An external device such as a USB device is connected to the external device interface 16. For example, a personal computer (PC) 41 is connected to the external device interface 16. The external device interface 16 transmits information to the PC 41 and receives information from the PC 41. Thereby, the PC 41 can display information. Also, the user can perform an operation related to the control of the endoscope device 1 by inputting an instruction to the PC 41.

[0050] The control interface 17 communicates with the operation unit 4, the endoscope unit 8, and the CCU 9 for operation control. The control interface 17 notifies the control unit 18 of information input by the user to the operation unit 4. The control interface 17 outputs control signals to the endoscope unit 8 for controlling the light source device and the bending device. The control interface 17 outputs control signals to the CCU 9 for controlling the image sensor 28.

[0051] The program executed by the control unit 18 may be recorded on a computer-readable recording medium. A computer other than the endoscope device 1 may read and execute the program recorded on this recording medium. For example, a PC 41 may read and execute the program. The PC 41 may control the endoscope device 1 by transmitting control information to the endoscope device 1 in accordance with the program. Alternatively, the PC 41 may acquire video signals from the endoscope device 1 and process the acquired video signals.

[0052] As described above, the endoscope device 1 has an image sensor 28 and a control unit 18. The image sensor 28 images the subject and generates an imaging signal. The imaging signal includes an image of the subject. Therefore, the image sensor 28 acquires an image of the subject generated by imaging the subject. This image is a two-dimensional (2D) image. The image acquired by the image sensor 28 is input to the control unit 18 via the video signal processing circuit 12.

[0053] The insertion section 2 constitutes an imaging device (camera). The image sensor 28 may be located in the main body section 3, or the optical fiber may be located in the insertion section 2. Light incident on the lens 21 may reach the image sensor 28 through the optical fiber. A borescope may also be used as a camera.

[0054] In the first embodiment, the object to be measured is a turbine blade or the like, and has two or more regions (objects). The two or more regions are not necessarily connected to each other, and gaps may exist between them. In the measurement process described below, the measurement reference is set in a first region of the object to be measured, and the measurement area is set in a second region of the object to be measured. In addition, in the measurement process, the distance between the measurement reference and the measurement point set in the measurement area is calculated.

[0055] Figure 3 shows an example of the measurement process performed by the endoscope device 1. The operation of the endoscope device 1 will be explained with reference to Figure 3.

[0056] In the following example, a stereo optical adapter with two fields of view is attached to the tip 20. The stereo optical adapter has a first optical system and a second optical system corresponding to the two fields of view. The first and second optical systems form two optical images of the subject on the image sensor 28. The image sensor 28 generates a stereo image corresponding to the first and second optical images. The stereo image includes a pair of two images (a first image and a second image). That is, the stereo image includes an image of the subject as seen from a first viewpoint and an image of the subject as seen from a second viewpoint.

[0057] The control unit 18 acquires the stereo image (2D image) output from the video signal processing circuit 12 (step S100). If a 2D image acquired in a previously executed measurement process is recorded on the memory card 42 or the like, the control unit 18 may acquire that 2D image from the memory card 42 or the like.

[0058] After step S100, the control unit 18 calculates the three-dimensional (3D) coordinates of three or more points on the subject using one or more 2D images and generates 3D data including those 3D coordinates (step S101).

[0059] The 3D coordinates contained in 3D data are defined in a 3D space corresponding to real space. Hereafter, points with 3D coordinates contained in 3D data are referred to as points in 3D data. The positions on the 2D image used to generate the 3D data and each point contained in the 3D data are related to each other.

[0060] If 3D data generated in a previously performed measurement process is recorded on a memory card 42 or the like, the control unit 18 may acquire that 3D data from the memory card 42 or the like.

[0061] After step S101, the control unit 18 determines whether each point included in the 3D data is an edge point. An edge point is a point on the edge of the subject. The control unit 18 generates edge information indicating whether each point included in the 3D data is an edge point (step S102).

[0062] For example, edge information includes a flag indicating whether each point in the 3D data is an edge point. Each point in the 3D data is associated with edge information. The 3D data may include edge information.

[0063] Step S102 will now be explained in detail. Figure 4 shows an example of the 3D shape of the subject. The subject SB10 shown in Figure 4 has two edges. The black area shown in Figure 4 is the background. The control unit 18 compares the number of points surrounding each point of the subject SB10 with a threshold. If the number is less than or equal to the threshold, the control unit 18 determines that the point of the subject SB10 is on an edge. If the number is greater than the threshold, the control unit 18 determines that the point of the subject SB10 is not on an edge. The control unit 18 generates edge information for each point of the subject SB10.

[0064] In Figure 4, there are two points around point P10. For example, when the threshold is 4, the control unit 18 determines that point P10 is on the edge. On the other hand, in Figure 4, there are seven points around point P11. The control unit 18 determines that point P11 is not on the edge.

[0065] The control unit 18 may detect edges by applying the Canny method to the 3D data. The control unit 18 may detect edges by applying a filtering process, such as a Sobel filter, to the 3D data. The control unit 18 may detect edges by applying OrganizedEdgeFromRGBNormal, which is included in the publicly available PCL (Point Cloud Library), to the 3D data.

[0066] All points included in the 3D data may be associated with edge information. In this case, the edge information indicates whether each point on the subject is an edge point or a non-edge point. Only one or more points on an edge on the subject may be associated with edge information. In this case, the edge information indicates that one or more of those points are edge points.

[0067] If edge information generated in a previously executed measurement process is recorded on a memory card 42 or the like, the control unit 18 may acquire that edge information from the memory card 42 or the like.

[0068] After step S102, the control unit 18 generates a graphic image signal of 3D data. The control unit 18 outputs this graphic image signal to the video signal processing circuit 12. The same processing as described above is performed, and the display unit 5 displays a 3D image of the subject. As a result, the control unit 18 displays the 3D image on the display unit 5 (step S103). The 3D image is an image of the 3D shape of the subject.

[0069] Figure 5 shows an example of the 3D shape of the subject. The subject shown in Figure 5 includes a first region R10 and a second region R11. The second region R11 includes four edges.

[0070] After the 3D image is displayed, the user specifies a point in a first region of the subject on the 3D image by operating the control unit 4. The 3D coordinates of the point specified by the user are included in the 3D data. If the display unit 5 is configured as a touch panel, the user specifies a point in the first region by touching the screen of the display unit 5. The control unit 4 or the touch panel is used as a pointing device. The control unit 18 receives point information indicating the point specified by the user. The control unit 18 sets a measurement reference based on the point indicated by the point information (step S104).

[0071] The measurement reference indicates a reference position for calculating the 3D distance described later. The measurement reference is one of a plane, a line, or a point. In the first embodiment, an example is described in which the measurement reference is a reference plane. The measurement reference is set in a space having three or more points included in the 3D data. The measurement reference includes one or more points in a first region of the subject.

[0072] Step S104 will now be described in detail. The control unit 18 identifies three or more points in the vicinity of the point indicated by the point information. Specifically, the control unit 18 identifies three or more points whose distance from the point indicated by the point information is less than or equal to a predetermined distance. In other words, the control unit 18 identifies three or more points within a sphere with a predetermined radius, centered on the point indicated by the point information. These three or more points lie on the first region, and the 3D coordinates of each of these three or more points are included in the 3D data.

[0073] The control unit 18 calculates a reference plane by performing a planar fitting using the 3D coordinates of three or more points. The reference plane approximates the surface of the subject in the first region. The control unit 18 may also calculate the reference plane using the 3D coordinates of three or more points, including the points indicated by the point information. The control unit 18 sets the calculated reference plane as the measurement reference.

[0074] For example, the reference plane is a flat plane. The reference plane may also be a curved surface, and the control unit 18 may calculate the reference plane by performing a curved surface fitting using the 3D coordinates of three or more points on it.

[0075] The control unit 18 may display the 2D image used to generate the 3D data on the display unit 5. The user may specify a point in a first region of the subject in the 2D image by operating the operation unit 4 or the touch panel. As described above, the position on the 2D image and each point included in the 3D data are related to each other. The control unit 18 may identify the point in the 3D data that corresponds to the point specified by the user. The 3D coordinates of the identified point are included in the 3D data. The control unit 18 may calculate a reference plane by using the 3D coordinates of the identified point.

[0076] The control unit 18 may display the reference plane calculated by the above method on the 3D image. The control unit 18 may display the reference plane on the 3D image in any color or pattern. As a result, the control unit 18 may make the display state of points on the 3D image that correspond to points included in the reference plane different from the display state of points on the 3D image that correspond to points not included in the reference plane.

[0077] After step S104, the control unit 18 selects the type of measurement area to be set in the second area of ​​the subject. The measurement area is defined by a shape such as a sphere, cube, or rectangular prism and extends in three dimensions. For example, the control unit 18 selects the type of measurement area according to setting information pre-recorded on a memory card 42 or the like. Alternatively, the control unit 18 selects the type of measurement area according to information entered by the user through the operation unit 4 or touch panel. The measurement area has a shape corresponding to its type. In the first embodiment, an example where the measurement area is a sphere is described. The control unit 18 further selects the diameter of the sphere according to the setting information or information entered by the user (step S105).

[0078] If setting information indicating the size of the measurement area, such as the diameter of a sphere, is recorded on the memory card 42 or the like, the control unit 18 may change the setting information according to the information entered by the user through the operation unit 4 or touch panel.

[0079] When step S105 is performed, the position of the measurement area has not yet been set. The user specifies a point in a first area of ​​the subject, and then specifies a point in a second area of ​​the subject on the 3D image by operating the operation unit 4 or the touch panel. The 3D coordinates of the points specified by the user are included in the 3D data. The control unit 18 receives point information indicating the points specified by the user. Based on the points indicated by the point information, the control unit 18 sets a measurement area independent of the measurement reference (step S106). The measurement area is set in a space having three or more points included in the 3D data.

[0080] Step S106 will now be described in detail. The control unit 18 sets a sphere as the measurement area. The center of the sphere is the point indicated by the point information, and the sphere has the diameter selected in step S105.

[0081] Figure 6 shows an example of a measurement area. The 3D shape shown in Figure 6 is the same as the 3D shape shown in Figure 5, and includes a first area R10 and a second area R11. When the user specifies a point in the second area R11, the control unit 18 sets a spherical measurement area MR10 centered on that point.

[0082] The control unit 18 may display the 2D image used to generate the 3D data on the display unit 5. The user may specify a point in a second region of the subject in the 2D image by operating the operation unit 4 or the touch panel. As described above, the position on the 2D image and each point included in the 3D data are related to each other. The control unit 18 may identify the point in the 3D data that corresponds to the point specified by the user. The 3D coordinates of the identified point are included in the 3D data. The control unit 18 may set the measurement area by using the 3D coordinates of the identified point.

[0083] The control unit 18 may display the measurement area set by the method described above on the 3D image. The control unit 18 may also display the measurement area on the 3D image in any color or pattern. In this way, the control unit 18 may make the display state of points on the 3D image that correspond to points included in the measurement area different from the display state of points on the 3D image that correspond to points not included in the measurement area.

[0084] After step S106, the control unit 18 sets one or more points included in the measurement area as measurement points. The edge information associated with each measurement point indicates that the measurement point is an edge point. Specifically, the control unit 18 identifies one or more edge points in the measurement area based on the edge information and sets the identified one or more edge points as measurement points (step S107).

[0085] In the following example, the control unit 18 sets two or more edge points included in the measurement area as measurement points. Hereafter, a group of two or more measurement points will be referred to as a measurement point group. In the following description, two or more measurement points may be replaced with a single measurement point.

[0086] Figure 7 shows an example of a measured point cloud. The 3D shape shown in Figure 7 is the same as the 3D shape shown in Figure 5, and includes the first region R10 and the second region R11. After the measurement region MR11 shown in Figure 7 is set, the control unit 18 sets the measured point cloud MP10 included in the measurement region MR11.

[0087] The control unit 18 may display the measurement point cloud set by the method described above on the 3D image. The control unit 18 may also display the measurement point cloud on the 3D image in any color or pattern. In this way, the control unit 18 may make the display state of the points on the 3D image that correspond to the measurement points included in the measurement point cloud different from the display state of the points on the 3D image that correspond to points other than the measurement points.

[0088] After step S107, the control unit 18 calculates the 3D distance between the measurement reference and each measurement point included in the measurement point cloud. In other words, the control unit 18 calculates the 3D distance between the reference plane and each measurement point. The control unit 18 generates distance information indicating the calculated 3D distance (step S108). The control unit 18 may record the generated distance information on a memory card 42 or the like.

[0089] The control unit 18 may perform statistical processing using distance information of two or more measurement points included in the measurement point cloud. For example, the control unit 18 may calculate statistical values ​​of the 3D distance between those two or more measurement points. These statistical values ​​may be the maximum value, minimum value, mean value, or standard deviation. The control unit 18 may generate distance information that shows these statistical values.

[0090] The control unit 18 may calculate the length of a line formed by connecting two or more measurement points. This line approximates the edge of the object being examined.

[0091] After step S108, the control unit 18 displays the distance information generated in step S108 on the display unit 5. For example, the control unit 18 displays the distance information on a 3D image (step S109).

[0092] Figure 8 shows an example of a 3D image displayed on the display unit 5 in step S109. The control unit 18 displays the 3D image IMG10 shown in Figure 8 on the display unit 5. The control unit 18 also displays distance information DI10 and distance information DI11 on the 3D image IMG10. Distance information DI10 indicates the minimum value of the 3D distance between two or more measurement points. Distance information DI11 indicates the maximum value of the 3D distance between two or more measurement points.

[0093] After step S109, the control unit 18 determines whether or not to perform the measurement again. For example, the control unit 18 makes this determination according to the information entered by the user through the operation unit 4 or touch panel (step S110).

[0094] If the control unit 18 determines in step S110 that the measurement should not be performed again, the measurement process shown in Figure 3 ends. If the control unit 18 determines in step S110 that the measurement should be performed again, step S105 is executed. The type of measurement area selected in the second step S105 may be the same as or different from the type of measurement area selected in the first step S105. If the type of measurement area is a sphere, the diameter of the sphere selected in the second step S105 may be the same as or different from the diameter of the sphere selected in the first step S105.

[0095] Figure 9 shows a first example of a 3D image displayed on the display unit 5 in step S109 when the measurement is performed two or more times. The control unit 18 displays the 3D image IMG11 shown in Figure 9 on the display unit 5.

[0096] The control unit 18 executes steps S105 to S109 four times and displays four sets of distance information on the 3D image IMG 11. The distance information shown in Figure 9 indicates the maximum and minimum values ​​of the 3D distance for two or more measurement points.

[0097] Figure 10 shows a second example of the 3D image displayed on the display unit 5 in step S109 when the measurement is performed two or more times. The control unit 18 displays the 3D image IMG12 shown in Figure 10 on the display unit 5.

[0098] In step S108, the control unit 18 performs the following processing. The control unit 18 calculates statistical values ​​using all distance information generated in measurements performed two or more times. These statistical values ​​are the maximum and minimum values, and the control unit 18 displays the maximum value MAX10 and minimum value MIN10 shown in Figure 10 on the display unit 5. The control unit 18 also displays points PX10 and PN10 on the display unit 5. Point PX10 corresponds to the point where the maximum value MAX10 was measured. Point PN10 corresponds to the point where the minimum value MIN10 was measured.

[0099] The control unit 18 may display the 2D image used to generate the 3D data on the display unit 5. The control unit 18 may also display at least one of the reference plane, measurement area, measurement point cloud, and distance information on the 2D image.

[0100] The order in which each step is executed in the measurement process is not limited to the order shown in Figure 3. For example, step S103 may be executed between steps S101 and S102. Step S104 may be executed between steps S105 and S108. In other words, the measurement reference may be set after the measurement area or measurement point has been set.

[0101] In the measurement process shown in Figure 3, the control unit 18 automatically sets the measurement points based on edge information. This reduces the effort required from the user and increases the efficiency of the measurement.

[0102] To obtain the statistical distance information shown in Figure 9 or Figure 10, it is necessary to set up a large number of measurement points. In the measurement process shown in Figure 3, the effort required of the user to set up a large number of measurement points is reduced, enabling efficient measurement.

[0103] A monocular optical adapter with a single field of view may be attached to the tip 20. A stereo optical adapter forms two optical images of the subject, while a monocular optical adapter forms one optical image of the subject. The image sensor 28 generates an image corresponding to the optical image formed by the monocular optical adapter. The image sensor 28 performs imaging from two or more different viewpoints and generates two or more images. The control unit 18 may calculate the 3D coordinates of three or more points on the subject by using the two or more images generated by the image sensor 28, and may generate 3D data including those 3D coordinates.

[0104] The main body 3 of the endoscope device 1 may include two or more separate units. These two or more units may communicate with each other, such as by transmitting images. These two or more units may also communicate wirelessly. Alternatively, these two or more units may be connected to each other by cables, and communication may be performed via these cables.

[0105] Each of the two or more units may have one or more control units. Therefore, the endoscope device 1 may have two or more control units. Some or all of the two or more control units may perform measurement processing by cooperating with each other. Therefore, the functions of the control unit 18 may be distributed among two or more control units.

[0106] Two or more control units may execute the processing sequentially. For example, the first control unit may execute a part of the measurement processing and output the processing result to the second control unit. The second control unit may then execute the rest of the measurement processing based on the processing result. Alternatively, two or more control units may execute the processing simultaneously. For example, the first control unit may generate edge information in step S102, and at the same time, the second control unit may display the 3D image on the display unit in step S103.

[0107] An endoscope system including an endoscope device 1 and external equipment may be used. For example, the external equipment is a PC 41. The external equipment may also be a cloud server on a network. The control unit 18 of the endoscope device 1 and one or more control units of the external equipment may cooperate with each other to perform the measurement process. Alternatively, one or more control units of the external equipment may perform the entire measurement process.

[0108] Each embodiment of the present invention's measuring device (endoscopic device 1) includes a control unit 18. The control unit 18 acquires 3D data including the 3D coordinates of three or more points on a subject calculated based on a 2D image of the subject acquired by the endoscope (insertion unit 2). The control unit 18 acquires edge information indicating whether each of the three or more points is an edge point on an edge of the subject. The control unit 18 displays a 3D image or a 2D image on a display unit 5 (display). The 3D image is an image of a 3D shape including points having 3D coordinates included in the 3D data. The control unit 18 sets a measurement reference indicating a reference position for measurement based on at least one of the three or more points. The control unit 18 sets a measurement area based on a point specified on the 3D image or 2D image by a pointing device. The measurement area has a preset shape and is independent of the measurement reference. The control unit 18 sets points included in the measurement area as measurement points. The edge information indicates that a measurement point is an edge point. The control unit 18 generates distance information indicating the distance between the measurement reference and the measurement point.

[0109] Each embodiment of the present invention comprises seven steps. In the first step (step S101), the control unit 18 acquires 3D data. In the second step (step S102), the control unit 18 acquires edge information. In the third step (step S103), the control unit 18 displays a 3D image or a 2D image on the display unit 5. In the fourth step (step S104), the control unit 18 sets a measurement reference. In the fifth step (step S106), the control unit 18 sets a measurement area. In the sixth step (step S107), the control unit 18 sets points included in the measurement area as measurement points. In the seventh step (step S108), the control unit 18 generates distance information.

[0110] Each aspect of the present invention provides a program that causes a computer to perform the first to seventh steps described above.

[0111] Each aspect of the present invention may include the following modifications. The control unit 18 generates edge information based on at least one of three or more points included in the 3D data.

[0112] Each aspect of the present invention may include the following modifications. The control unit 18 sets a measurement reference which is one of a surface, a line, and a point.

[0113] Each aspect of the present invention may include the following modifications. Before the measurement area is set, the control unit 18 sets the measurement reference.

[0114] Each aspect of the present invention may include the following modifications. The control unit 18 sets a measurement area which is a figure of a predetermined size.

[0115] Each aspect of the present invention may include the following modifications. The control unit 18 displays the measurement area on the display unit 5.

[0116] Each aspect of the present invention may include the following modifications. The control unit 18 makes the display state of the points on the 3D image corresponding to the measurement points different from the display state of the points on the 3D image corresponding to points other than the measurement points.

[0117] Each aspect of the present invention may include the following modifications. The control unit 18 generates distance information indicating the distance between a measurement reference and each of two or more measurement points.

[0118] Each aspect of the present invention may include the following modifications. The control unit 18 displays distance information on the display unit 5.

[0119] In the first embodiment, the control unit 18 automatically sets the edge points included in the measurement area as measurement points. Therefore, the endoscope device 1 can reduce the effort required by the user for measurement and can perform measurements efficiently.

[0120] (First Modification of the First Embodiment) A first modification of the first embodiment of the present invention will now be described. The measurement area in the first modification of the first embodiment is different from the measurement area in the first embodiment.

[0121] The control unit 18 executes the measurement process shown in Figure 3. In step S105, the control unit 18 selects a linear region as the type of measurement region. A linear region is a region whose distance from a reference straight line is less than or equal to a predetermined distance. In other words, a linear region is a cylindrical region. Also in step S105, the control unit 18 selects a specific value for that predetermined distance.

[0122] In step S106, the control unit 18 performs the following processing. The control unit 18 receives point information indicating a point on a 3D image specified by the user. The control unit 18 identifies two or more points in the vicinity of the point indicated by the point information. Specifically, the control unit 18 identifies two or more points whose distance from the point indicated by the point information is less than or equal to a predetermined distance. In other words, the control unit 18 identifies two or more points within a sphere with a predetermined radius, centered on the point indicated by the point information. These two or more points lie on a second region of the subject, and the 3D coordinates of each of these two or more points are included in the 3D data. In addition, the edge information associated with each of these two or more points indicates that each point is an edge point.

[0123] The control unit 18 calculates a straight line that approximates the edge of the subject by using the 3D coordinates of two or more points. If the point indicated by the point information is an edge point, the control unit 18 may calculate the straight line by using the 3D coordinates of two or more edge points including that edge point. The control unit 18 sets the area in which the distance from the above straight line is less than or equal to the distance selected in step S105 as the measurement area.

[0124] The control unit 18 may identify a point in the 3D data that corresponds to a point on the 2D image specified by the user. The control unit 18 may also identify two or more points in the vicinity of the identified point and perform the same processing as described above based on those two or more points.

[0125] Figure 11 shows an example of a straight line used to define the measurement area. The 3D shape shown in Figure 11 is the same as the 3D shape shown in Figure 5 and includes a first area R10 and a second area R11. When the user specifies a point P12 in the second area R11, the control unit 18 defines a straight line L10 based on two or more edge points in the vicinity of point P12. The control unit 18 defines the area where the distance from the straight line L10 is less than or equal to the distance selected in step S105 as the measurement area.

[0126] Figure 12 shows an example of a measured point cloud. The 3D shape shown in Figure 12 is the same as the 3D shape shown in Figure 5, and includes a first region R10 and a second region R11. After the measurement region is set using the straight line L10 shown in Figure 11, the control unit 18 sets the measured point cloud MP11 included in the measurement region.

[0127] The line approximating the edge of the subject is not limited to a straight line. The control unit 18 may calculate a curve that approximates the edge.

[0128] Each aspect of the present invention may include the following modifications. The control unit 18 calculates a line that approximates the edge of the subject based on a point specified on a 3D or 2D image of the subject and the edge point indicated by the edge information. The control unit 18 sets a measurement area based on the calculated line.

[0129] (Second Modification of the First Embodiment) A second modification of the first embodiment of the present invention will now be described. The measurement area in the second modification of the first embodiment is different from the measurement area in the first embodiment.

[0130] Figure 13 shows an example of the measurement process performed by the endoscope device 1. The operation of the endoscope device 1 will be explained with reference to Figure 13. Processes that are the same as those shown in Figure 3 will not be explained.

[0131] After step S102, the control unit 18 identifies an edge containing the edge point indicated by the edge information based on the 3D data (step S120). After step S120, step S103 is performed.

[0132] Step S120 will now be explained in detail. The control unit 18 selects the 3D coordinates of the edge point indicated by the edge information from the 3D coordinates included in the 3D data. The control unit 18 calculates the vector of each edge point by using the 3D coordinates of two adjacent edge points.

[0133] For example, when the first edge point and the second edge point are adjacent, and the second edge point and the third edge point are adjacent, the control unit 18 calculates a vector relating to the first edge point and the second edge point, and also calculates a vector relating to the second edge point and the third edge point. The calculated vector is associated with the edge point corresponding to the start and end point of the vector.

[0134] Figure 14 shows an example of a vector calculated in step S120. The 3D shape shown in Figure 14 is the same as the 3D shape shown in Figure 5, and includes a first region R10 and a second region R11. The control unit 18 calculates the vectors indicated by the arrows. To avoid complexity in the figure, the magnitude of the vectors indicated by each arrow in Figure 14 is exaggerated compared to their actual magnitudes. Also, Figure 14 shows only a portion of the many vectors.

[0135] The control unit 18 determines which edge each edge point belongs to based on the similarity of the vector directions. The details of this process are described below.

[0136] The control unit 18 selects a first edge point and a vector associated with the first edge point. If that vector is associated with both the first and second edge points, the control unit 18 selects a vector associated with the second edge point and a third edge point different from the first edge point. The control unit 18 determines the similarity of the directions of the two selected vectors.

[0137] When the directions of the two vectors are highly similar, the control unit 18 determines that the edge points associated with each of the two vectors are included in the same edge. In other words, the control unit 18 determines that the first edge point, the second edge point, and the third edge point are included in the same edge. When the directions of the two vectors are less similar, the control unit 18 determines that the first and second edge points are included in the first edge, and the second and third edge points are included in a second edge that is different from the first edge.

[0138] The control unit 18 selects a vector associated with the third edge point and a fourth edge point that is different from the second edge point, and repeats the same process as described above.

[0139] The vectors included in the vector group VG10 shown in Figure 14 have almost the same direction. Therefore, the control unit 18 determines that the edge points associated with the vectors included in the vector group VG10 belong to the same edge. Similarly, the control unit 18 determines that the edge points associated with the vectors included in the vector group VG11 shown in Figure 14 belong to the same edge.

[0140] The direction of the vectors included in vector group VG11 is different from the direction of the vectors included in vector group VG10. Therefore, the control unit 18 determines that the edge points associated with the vectors included in vector group VG10 are included in the first edge, and that the edge points associated with the vectors included in vector group VG11 are included in a second edge that is different from the first edge.

[0141] The control unit 18 adds information indicating the edge containing each edge point to the edge information. Based on the information added to the edge information, the control unit 18 can determine which edge each edge point belongs to.

[0142] In step S105, the control unit 18 selects an edge as the type of measurement area.

[0143] In step S106, the control unit 18 performs the following processing: The control unit 18 receives point information indicating a point on a 3D image specified by the user. The control unit 18 identifies edge points in the vicinity of the point indicated by the point information. Specifically, the control unit 18 identifies edge points whose distance from the point indicated by the point information is less than or equal to a predetermined distance. In other words, the control unit 18 identifies edge points within a sphere with a predetermined radius centered on the point indicated by the point information. Based on the edge information, the control unit 18 identifies the edge containing the identified edge points. The control unit 18 sets the identified edge as the measurement area.

[0144] The control unit 18 may identify a point in the 3D data that corresponds to a point on the 2D image specified by the user. The control unit 18 may also identify an edge point near the identified point and perform the same processing as described above based on that edge point.

[0145] In step S107, the control unit 18 sets two or more edge points included in the measurement area as measurement points. In other words, the control unit 18 sets two or more edge points included in the edge set as the measurement area as measurement points.

[0146] Figure 15 shows an example of a measurement point cloud. The 3D shape shown in Figure 15 is the same as the 3D shape shown in Figure 5, and includes a first region R10 and a second region R11. When the user specifies a point P13 in the second region R11, the control unit 18 identifies the edge point closest to point P13. The control unit 18 sets the edge containing the identified edge point as the measurement region and sets the measurement point cloud MP12 included in the measurement region.

[0147] Step S120 may be performed between steps S103 and S106.

[0148] Each aspect of the present invention may include the following modifications. The control unit 18 identifies an edge containing an edge point indicated by edge information based on 3D data. The control unit 18 sets one of one or more edges of the subject as a measurement area based on a point specified on the 3D image or 2D image of the subject.

[0149] (Third Modification of the First Embodiment) A third modification of the first embodiment of the present invention will now be described. The measurement area in the third modification of the first embodiment is different from the measurement area in the first embodiment.

[0150] Figure 16 shows an example of the measurement process performed by the endoscope device 1. The operation of the endoscope device 1 will be explained with reference to Figure 16. Processes that are the same as those shown in Figure 3 will not be explained.

[0151] After step S102, the control unit 18 performs segmentation. In segmentation, the control unit 18 detects one or more segments (feature regions) on the subject by using 3D data. The control unit 18 may also detect one or more segments by using one or more 2D images used to generate the 3D data. The one or more 2D images are one or more stereo images acquired using a stereo optical adapter, or two or more images acquired using a monocular optical adapter (step S130). After step S130, step S103 is performed.

[0152] Step S130 will now be described in detail. The control unit 18 extracts the features of the 3D shape of the object using the 3D data. The control unit 18 assigns each point corresponding to the 3D coordinates in the 3D data to one of two or more feature regions according to the extracted features. The 3D shape of the object has common features within one feature region. The features of the 3D shape of the object differ from each other between two or more different feature regions.

[0153] Only one point may be assigned to one feature region. If two or more points are assigned to one feature region, those two or more points satisfy common conditions that represent the 3D shape features of the subject. The conditions that a point in the first feature region satisfies are different from the conditions that a point in a second feature region, which is different from the first feature region, satisfies.

[0154] Segmentation is a simple method for classifying each point corresponding to 3D data. For example, the control unit 18 can use Euclidean Cluster Extraction for segmentation. This is a function included in PCL.

[0155] The control unit 18 uses this function to determine that points whose distance from each point in the 3D data is less than or equal to a predetermined distance are neighboring points. A point and its neighboring points are located on the same object. For example, if the subject has a first object and a second object that are separated from each other, each point in the 3D data is classified as either a point on the first object or a point on the second object. Each of the first and second objects is a feature region (segment). In this case, two or more points within a single feature region are characterized as being located on the same object.

[0156] The control unit 18 may use the Watershed algorithm or Deep Learning, etc., for segmentation. The control unit 18 may calculate normals perpendicular to the surface of the subject based on 3D data, and may detect edges or steps of the subject as feature regions based on changes in the direction of the normals. For example, the control unit 18 may detect a first feature region consisting of edges or steps, and a second feature region consisting of parts other than those edges or steps.

[0157] The control unit 18 may detect edges of the subject by performing image processing on the 2D image of the subject. The control unit 18 may detect feature regions corresponding to the edges on the 2D image from the 3D shape of the subject shown in the 3D data. The control unit 18 may detect feature regions based on the brightness or color of the 2D image. The control unit 18 may perform matching processing on the stereo image of the subject and detect feature regions based on the correlation values ​​obtained in the matching processing.

[0158] In step S105, the control unit 18 selects a feature area as the type of measurement area.

[0159] In step S106, the control unit 18 performs the following processing: The control unit 18 receives point information indicating a point on a 3D image specified by the user. The control unit 18 selects a feature region containing the point indicated by the point information. The control unit 18 sets the selected feature region as the measurement region. The control unit 18 may also identify a point in the 3D data corresponding to a point on a 2D image specified by the user. The control unit 18 may set the feature region containing the identified point as the measurement region. The control unit 18 may set two or more feature regions as measurement regions.

[0160] In step S107, the control unit 18 sets two or more edge points included in the measurement area as measurement points. In other words, the control unit 18 sets two or more edge points included in the feature area set as the measurement area as measurement points.

[0161] Figure 17 shows an example of a measurement point cloud. The 3D shape shown in Figure 17 is the same as the 3D shape shown in Figure 5, and includes a first region R10 and a second region R11. In step S130, the control unit 18 detects each of the first region R10 and the second region R11 as feature regions. When the user specifies a point in the second region R11, the control unit 18 sets the entire second region R11 as a measurement region and sets the measurement point cloud MP13 included in the measurement region. The measurement point cloud MP13 includes edge points on the four edges of the second region R11.

[0162] Step S130 may be performed between steps S103 and S106.

[0163] Each aspect of the present invention may include the following modifications. The control unit 18 detects one or more feature regions on a subject based on 3D data or a 2D image of the subject. The 3D shape of the subject has common features in each of the one or more feature regions. The control unit 18 sets at least one of the one or more feature regions as a measurement region.

[0164] (Fourth Modification of the First Embodiment) A fourth modification of the first embodiment of the present invention will now be described. In the fourth modification of the first embodiment, the measurement area is set again.

[0165] The control unit 18 executes the measurement process shown in Figure 13. A first example of the measurement process will be described.

[0166] After the control unit 18 determines in step S110 that it will perform the measurement again, the control unit 18 performs the following processing in step S106.

[0167] The user specifies a point in the second region of the subject by operating the control unit 4 or the touch panel. For example, the user specifies an edge point that is not included in the measurement point cloud. The 3D coordinates of the point specified by the user are included in the 3D data, and the edge information associated with that point indicates that it is an edge point. The control unit 18 receives point information indicating the point specified by the user.

[0168] The control unit 18 sets a new measurement area based on the points indicated by the point information. At least a portion of this new measurement area may differ from the measurement area set in the previously executed step S106. The control unit 18 may identify points in the 3D data that correspond to points on the 2D image specified by the user. The control unit 18 may set a new measurement area based on the identified points.

[0169] Figure 18 shows an example of a measured point cloud. The 3D shape shown in Figure 18 is the same as the 3D shape shown in Figure 5, and includes a first region R10 and a second region R11. In step S107, which is executed for the first time, the control unit 18 sets the measured point cloud MP10. In step S105, which is executed for the second time, the control unit 18 selects edge as the type of measurement region.

[0170] In step S106, which is executed for the second time, the control unit 18 sets the same edge as the edge that includes two or more edge points included in the measurement point cloud MP10 as the measurement region. At this time, the edge information identified in step S120 is used. The edge set as the measurement region includes two or more edge points included in the measurement point cloud MP10 and one or more edge points not included in the measurement point cloud MP10.

[0171] The user specifies point P14 on an edge containing the measured point cloud MP10. The edge information associated with point P14 indicates that point P14 is an edge point. Point P14 is not included in the measured point cloud MP10. The user may also specify a non-edge point in the vicinity of the edge.

[0172] The control unit 18 identifies two or more edge points between point P14 and point P15 based on edge information. Point P15 is an edge point at the end of the measurement point cloud MP10. Specifically, the control unit 18 identifies two or more edge points that are included in the same edge as the edge containing two or more measurement points included in the measurement point cloud MP10, and that are between point P14 and point P15, and sets these two or more edge points as measurement points. In this way, the control unit 18 sets the measurement point cloud MP14.

[0173] The control unit 18 may use two or more measurement points included in the measurement point cloud MP14 in step S108. Alternatively, the control unit 18 may use two or more measurement points included in the measurement point cloud MP10 and two or more measurement points included in the measurement point cloud MP14 in step S108.

[0174] The user may specify edge points that are not included in the measurement point cloud and edge points that are included in the measurement point cloud. Edge points included in the measurement point cloud may be edge points at the edges of the measurement point cloud or edge points other than those at the edges of the measurement point cloud. For example, the user may click on a location on the 3D image corresponding to an edge point included in the measurement point cloud, and may also click on a location on the 3D image corresponding to an edge point that is not included in the measurement point cloud. Edge points included in the measurement point cloud do not need to be edge points at the edges of the measurement point cloud.

[0175] Alternatively, the user may perform a drag operation from a position on the 3D image corresponding to an edge point included in the measurement point cloud to a position on the 3D image corresponding to an edge point not included in the measurement point cloud. The control unit 18 may identify two or more edge points between two edge points specified by the user based on the edge information, and may set those two or more edge points as measurement points.

[0176] A second example of the measurement process will be described. Figure 19 shows an example of a line segment used to define a new measurement region. The 3D shape shown in Figure 19 is the same as the 3D shape shown in Figure 5 and includes a first region R10 and a second region R11. In step S107, which is executed for the first time, the control unit 18 defines the measurement point cloud MP10. In step S105, which is executed for the second time, the control unit 18 selects a line region as the type of measurement region.

[0177] The user specifies point P16 on an edge containing the measured point cloud MP10. The edge information associated with point P16 indicates that point P16 is an edge point. Point P16 is not included in the measured point cloud MP10. The user may also specify a non-edge point in the vicinity of the edge.

[0178] In step S106, which is executed for the second time, the control unit 18 sets a line segment LS10 connecting point P15 and point P16. Point P15 is an edge point at the end of the measurement point group MP10. The control unit 18 sets a region where the distance from line segment LS10 is less than or equal to a predetermined distance as the measurement region. In step S107, which is executed for the second time, the control unit 18 sets two or more measurement points included in the measurement region.

[0179] Each aspect of the present invention may include the following modifications. The control unit 18 sets a new measurement area based on the set measurement points and points other than the set measurement points.

[0180] Each aspect of the present invention may include the following modifications. The control unit 18 identifies an edge containing an edge point indicated by edge information based on 3D data. The control unit 18 sets a new measurement area based on a point included in the same edge as the edge containing the set measurement point.

[0181] (Fifth Modification of the First Embodiment) A fifth modification of the first embodiment of the present invention will now be described. In the fifth modification of the first embodiment, after the measurement area and the measurement point cloud are set, measurement points included in a part of the measurement area are removed from the measurement point cloud. Thereafter, distance information is generated.

[0182] Figure 20 shows an example of the measurement process performed by the endoscope device 1. The operation of the endoscope device 1 will be explained with reference to Figure 20. Processes that are the same as those shown in Figure 3 will not be explained.

[0183] After step S107, the control unit 18 performs a measurement point update process (step S111). After step S111, step S108 is executed.

[0184] Figure 21 shows an example of the procedure for updating the measurement point in step S111. Referring to Figure 21, the operation of the endoscope device 1 in the measurement point update process will be explained.

[0185] The control unit 18 determines whether or not an update of the measurement point has been instructed. For example, the control unit 18 makes this determination according to the information entered by the user through the operation unit 4 or touch panel (step S1110).

[0186] For example, the display unit 5 displays the measurement area set in step S106 or the measurement point cloud set in step S107 on the 3D image displayed in step S103. The user checks the measurement area or measurement point cloud displayed on the 3D image and determines whether it is necessary to update the measurement points.

[0187] The user inputs information indicating the decision result by operating the operation unit 4 or the touch panel. The control unit 18 receives the information input by the user. Based on the information input by the user, the control unit 18 makes a decision in step S1110. If the information input by the user indicates that the measurement point needs to be updated, the control unit 18 determines in step S1110 that an update of the measurement point has been instructed. If the information input by the user indicates that an update of the measurement point does not need to be updated, the control unit 18 determines in step S1110 that an update of the measurement point has not been instructed.

[0188] If the control unit 18 determines in step S1110 that no instruction has been given to update the measurement points, the measurement point update process shown in Figure 21 ends. If the control unit 18 determines in step S1110 that an instruction has been given to update the measurement points, the control unit 18 sets a removal area (step S1111). The removal area includes the area to be removed from the measurement area.

[0189] Step S1111 will be explained in detail. For example, the user specifies a point in the measurement area by operating the operation unit 4 or the touch panel. The 3D coordinates of the point specified by the user are included in the 3D data. The control unit 18 receives point information indicating the point specified by the user. In step S1111, the control unit 18 sets the removal area based on the point indicated by the point information.

[0190] For example, the control unit 18 sets the area of ​​a sphere. The center of the sphere is the point indicated by the point information, and the sphere has a predetermined diameter. The user may specify the diameter of the sphere by operating the operation unit 4 or the touch panel. The control unit 18 sets the overlapping area between the sphere and the measurement area as a removal area. The removal area includes a part of the measurement area. In other words, the removal area includes a part of the measurement point cloud.

[0191] After step S1111, the control unit 18 identifies one or more measurement points included in the removal area set in step S1111. The control unit 18 removes the identified one or more measurement points from the measurement point group. As a result, the control unit 18 updates the measurement point group (step S1112). When step S1112 is executed, the measurement point update process shown in Figure 21 is completed.

[0192] Figures 22 and 23 show examples of measured point clouds. The 3D shapes shown in Figures 22 and 23 are the same as the 3D shapes shown in Figure 5, and include a first region R10 and a second region R11. In step S107, the control unit 18 sets the measured point cloud MP15.

[0193] For example, when a user specifies a point in the second region R11, the control unit 18 sets a spherical region R12 centered on that point (Figure 22). Region R12 includes a portion of the measurement point cloud MP15. The control unit 18 removes one or more measurement points included in region R12 from the measurement point cloud MP15. The measurement point cloud MP15 is updated to the measurement point cloud MP16 (Figure 23). The user can specify a region to be removed from the initially set measurement region.

[0194] The control unit 18 may display the 2D image used to generate the 3D data on the display unit 5. The user may specify a point in a second region of the subject in the 2D image by operating the operation unit 4 or the touch panel. As described above, the position on the 2D image and each point included in the 3D data are associated with each other. The control unit 18 may identify the point in the 3D data that corresponds to the point specified by the user. The 3D coordinates of the identified point are included in the 3D data. The control unit 18 may set a removal region by using the 3D coordinates of the identified point.

[0195] The control unit 18 may set two or more removal regions. The control unit 18 may remove one or more measurement points included in each of the two or more removal regions from the measurement point group.

[0196] The control unit 18 may identify overlapping regions where the measurement region and the removal region overlap. The control unit 18 may update the measurement region by removing the overlapping regions from the measurement region. The control unit 18 may set one or more points included in the updated measurement region as measurement points.

[0197] Each aspect of the present invention may include the following modifications. The control unit 18 sets two or more measurement points included in the measurement area and sets a removal area that includes a part of the measurement area. The control unit 18 removes the measurement points included in the removal area from the two or more measurement points.

[0198] Each aspect of the present invention may include the following modifications. The control unit 18 generates a measurement reference and distance information indicating the distance between the measurement point remaining after removing two or more measurement points included in the removal area.

[0199] (Sixth Modification of the First Embodiment) A sixth modification of the first embodiment of the present invention will now be described. In the sixth modification of the first embodiment, after distance information is generated, measurement points included in a part of the measurement area are removed from the measurement point group, similar to the fifth modification of the first embodiment. Thereafter, distance information is generated again.

[0200] Figure 24 shows an example of the measurement process performed by the endoscope device 1. The operation of the endoscope device 1 will be explained with reference to Figure 24. Processes that are the same as those shown in Figure 3 will not be explained.

[0201] After step S109, the control unit 18 performs a measurement point update process (step S112). After step S112, step S110 is executed.

[0202] Figure 25 shows an example of the procedure for updating the measurement point in step S112. Referring to Figure 25, the operation of the endoscope device 1 in the measurement point update process will be explained.

[0203] Steps S1120 to S1122 are the same as steps S1110 to S1112 shown in Figure 21. Therefore, steps S1120 to S1122 will not be explained.

[0204] After step S1122, the control unit 18 calculates the 3D distance between the measurement reference and each measurement point included in the measurement point cloud. In other words, the control unit 18 calculates the 3D distance between the reference plane and each measurement point. The control unit 18 generates distance information indicating the calculated 3D distance (step S1123).

[0205] After step S1123, the control unit 18 displays the distance information generated in step S1123 on the display unit 5. For example, the control unit 18 displays the distance information on a 3D image (step S1124). When step S1124 is executed, the measurement point update process shown in Figure 25 is completed.

[0206] In step S108, distance information (first distance information) is generated, and in step S109, the distance information is displayed. After the measurement point cloud is updated in step S1122, distance information (second distance information) is generated in step S1123, and in step S1124, the distance information is displayed.

[0207] Each aspect of the present invention may include the following modifications. The control unit 18 sets two or more measurement points included in the measurement area. The control unit 18 generates first distance information indicating the distance between a measurement reference and each of the two or more measurement points. After the first distance information is generated, the control unit 18 sets a removal area that includes a part of the measurement area. The control unit 18 removes the measurement points included in the removal area from the two or more measurement points. The control unit 18 generates second distance information indicating the distance between the measurement reference and the measurement points remaining after removing the measurement points included in the removal area from the two or more measurement points.

[0208] (Second Embodiment) A second embodiment of the present invention will now be described. In the second embodiment, a reference point is set as the measurement reference.

[0209] Figure 26 shows an example of the measurement process performed by the endoscope device 1. The operation of the endoscope device 1 will be explained with reference to Figure 26. Processes that are the same as those shown in Figure 3 will not be explained.

[0210] After step S103, step S105 is executed. After step S105, step S106a is executed. Step S106a is the same as step S106 shown in Figure 3, in which the control unit 18 sets the measurement area to the first area of ​​the subject instead of the second area of ​​the subject. After step S106a, step S107 is executed.

[0211] After step S107, step S104a is performed. Step S104a is the same as step S104 shown in Figure 3, and the control unit 18 sets the measurement reference to the second region instead of the first region. In the following example, the measurement reference is a reference point on the second region. After step S104a, step S108 is performed.

[0212] Step S104a will now be described in detail. The control unit 18 receives point information indicating a point on a 3D image specified by the user. The control unit 18 sets the point indicated by the point information as a reference point. The control unit 18 may also set a point in the 3D data that corresponds to a point on a 2D image specified by the user.

[0213] Figure 27 shows an example of a measurement region and a measurement point cloud. The 3D shape shown in Figure 27 includes a first region R20 and a second region R21. When the user specifies a point P20 in the first region R20, the control unit 18 sets a spherical measurement region MR20 centered on point P20. After the measurement region MR20 is set, the control unit 18 sets the measurement point cloud MP20 included in the measurement region MR20.

[0214] Figure 28 shows an example of reference point and distance information. The 3D shape shown in Figure 28 is the same as the 3D shape shown in Figure 27 and includes a first region R20 and a second region R21. When the user specifies a point P21 in the second region R21, the control unit 18 sets point P21 as the reference point. The control unit 18 may also detect a characteristic point in the second region R21 and set that point as the reference point.

[0215] The measurement point cloud MP20 shown in Figure 28 is the same as the measurement point cloud MP20 shown in Figure 27. The control unit 18 calculates the 3D distance between point P21 and each of two or more measurement points included in the measurement point cloud MP20. The control unit 18 identifies the maximum and minimum values ​​of the 3D distance between those two or more measurement points. For example, the maximum value corresponds to the length of the line segment LS20 shown in Figure 28, and the minimum value corresponds to the length of the line segment LS21 shown in Figure 28. The control unit 18 generates distance information including the maximum and minimum values.

[0216] Each aspect of the present invention may include the following modifications. Before the measurement reference is set, the control unit 18 sets the measurement area.

[0217] In the second embodiment, the endoscope device 1, similar to the first embodiment, can reduce the effort required by the user for measurement and can perform measurements efficiently.

[0218] (Third Embodiment) A third embodiment of the present invention will now be described. In the third embodiment, a reference line is set as a measurement reference. The reference line is a straight line, a line segment, a polyline, or a curve.

[0219] The control unit 18 executes the measurement process shown in Figure 3. In the following example, the first and second regions of the subject are the same.

[0220] In step S104, the control unit 18 receives point information indicating two points on a 3D image specified by the user. The 3D coordinates of each of these two points are included in the 3D data. The control unit 18 may automatically detect these two points. In step S104, the control unit 18 calculates a straight line containing these two points and sets this straight line as a reference line. The control unit 18 may also identify two points in the 3D data that correspond to two points on a 2D image specified by the user, and may perform the same processing as described above based on these two points.

[0221] Figure 29 shows an example of a reference line. The 3D shape shown in Figure 29 includes region R30. A recess or defect is formed in the specimen. When the user specifies points P30 and P31 in region R30, the control unit 18 sets a reference line RL30 that includes points P30 and P31.

[0222] In step S106, the control unit 18 receives point information indicating a point on a 3D image specified by the user. In step S106, the control unit 18 sets a measurement area based on the point indicated by the point information. In step S107, the control unit 18 sets two or more edge points included in the measurement area as measurement points. The control unit 18 may also identify a point in the 3D data corresponding to a point on a 2D image specified by the user, and may set a measurement area based on the identified point.

[0223] Figure 30 shows an example of a measurement region and a measurement point cloud. The 3D shape shown in Figure 30 includes region R30, similar to the 3D shape shown in Figure 29. The reference line RL30 shown in Figure 30 is the same as the reference line RL30 shown in Figure 29. When the user specifies point P32 in region R30, the control unit 18 sets a spherical measurement region MR30 centered on point P32. After the measurement region MR30 is set, the control unit 18 sets the measurement point cloud MP30 included in the measurement region MR30.

[0224] In step S108, the control unit 18 calculates the 3D distance between the reference line and each measurement point included in the measurement point cloud. In step S108, the control unit 18 generates distance information indicating the calculated 3D distance.

[0225] Figure 31 shows an example of distance information. The 3D shape shown in Figure 31 includes region R30, similar to the 3D shape shown in Figure 29. The reference line RL30 shown in Figure 31 is the same as the reference line RL30 shown in Figure 29. The measured point cloud MP30 shown in Figure 31 is the same as the measured point cloud MP30 shown in Figure 30.

[0226] The control unit 18 calculates the 3D distance between the reference line RL30 and each of two or more measurement points included in the measurement point cloud MP30. The control unit 18 identifies the maximum value Dmax and minimum value Dmin of the 3D distance between those two or more measurement points. The control unit 18 generates distance information including the maximum value Dmax and minimum value Dmin.

[0227] The control unit 18 may perform the measurement process shown in Figure 26.

[0228] In the third embodiment, the endoscope device 1, similar to the first embodiment, can reduce the effort required by the user for measurement and can perform measurements efficiently.

[0229] (Modification of the third embodiment) A modification of the third embodiment of the present invention will now be described. In the modification of the third embodiment, the size of the figure approximating the edge of the subject is calculated.

[0230] Figure 32 shows an example of a reference line, a measurement area, and a measurement point cloud. The 3D shape shown in Figure 32 includes area R31. A defect is formed in the specimen. When the user specifies points P33 and P34 in area R31, the control unit 18 sets a reference line RL31 that includes points P33 and P34.

[0231] When the user specifies point P35 in region R31, the control unit 18 sets a spherical measurement region MR31 centered on point P35. After the measurement region MR31 is set, the control unit 18 sets the measurement point group MP31 included in the measurement region MR31.

[0232] Figure 33 shows an example of distance information. The 3D shape shown in Figure 33 is the same as the 3D shape shown in Figure 32 and includes region R31. Point P33 shown in Figure 33 is the same as point P33 shown in Figure 32, and point P34 shown in Figure 33 is the same as point P34 shown in Figure 32. The reference line RL31 shown in Figure 33 is the same as reference line RL31 shown in Figure 32. Point P35 shown in Figure 33 is the same as point P35 shown in Figure 32. The measured point cloud MP31 shown in Figure 33 is the same as the measured point cloud MP31 shown in Figure 32.

[0233] The control unit 18 calculates the 3D distance between the reference line RL31 and each of two or more measurement points included in the measurement point cloud MP31. The control unit 18 identifies the maximum value Dmax of the 3D distance between the two or more measurement points. The control unit 18 generates distance information including the maximum value Dmax. The control unit 18 may also identify the minimum value of the 3D distance between the two or more measurement points, or generate distance information including the minimum value.

[0234] The control unit 18 calculates the 3D distance W between point P33 and point P34 and generates distance information indicating the 3D distance W. The control unit 18 may also display this distance information on the display unit 5.

[0235] The control unit 18 calculates the radius r of a circle C10 that approximates the edge of the missing portion by using the 3D coordinates of two or more measurement points included in the measurement point cloud. The control unit 18 may also display the circle C10 and the radius r on the display unit 5.

[0236] Each aspect of the present invention may include the following modifications. The control unit 18 calculates the size of a figure that approximates the edge of the object based on two or more measurement points. For example, the size corresponds to the radius r shown in Figure 33.

[0237] (Fourth Embodiment) A fourth embodiment of the present invention will now be described. In the fourth embodiment, clearance measurement using a reference line will be described. If a part of the 3D shape of the subject shown in the 3D data is missing, the size of the clearance between two objects may not be measured correctly. In the fourth embodiment, the endoscope device 1 measures the accurate size of the clearance by using a reference line and a 2D image of the subject.

[0238] Figure 34 shows an example of the measurement process performed by the endoscope device 1. The operation of the endoscope device 1 will be explained with reference to Figure 34. Processes that are the same as those shown in Figure 3 will not be explained.

[0239] In step S103, after the 3D image of the subject is displayed on the display unit 5, the control unit 18 displays the 2D image used to generate the 3D data on the display unit 5 (step S140). The control unit 18 may also display the 2D image on the display unit 5 before the 3D image is displayed on the display unit 5.

[0240] The user specifies a point in a first region of the subject on the 3D image by operating the control unit 4 or the touch panel. The 3D coordinates of the point specified by the user are included in the 3D data. The control unit 18 receives point information indicating the point specified by the user. The control unit 18 sets the point indicated by the point information as the reference point (step S141).

[0241] The user may specify a point in a first region of the subject in the 2D image by operating the control unit 4 or the touch panel. As described above, the position on the 2D image and each point included in the 3D data are related to each other. The control unit 18 may identify the point in the 3D data that corresponds to the point specified by the user. The 3D coordinates of the identified point are included in the 3D data. The control unit 18 may set the identified point as a reference point.

[0242] After step S141, the control unit 18 sets the reference surface to the first region using a method similar to the method for setting the reference surface in the first embodiment (step S104b).

[0243] The user specifies two points in a second region of the subject by operating the control unit 4 or the touch panel. The 3D coordinates of each of the two points specified by the user are included in the 3D data. The control unit 18 receives point information indicating the two points specified by the user. The control unit 18 sets the two points indicated by the point information as reference points (step S142).

[0244] The user may specify two points in a second region of the subject in the 2D image by operating the control unit 4 or the touch panel. The control unit 18 may identify two points in the 3D data that correspond to the two points specified by the user. The 3D coordinates of the two identified points are included in the 3D data. The control unit 18 may set the two identified points as reference points.

[0245] After step S142, the control unit 18 sets a reference line that includes the two reference points set in step S142 (step S143).

[0246] After step S143, the control unit 18 sets the measurement point on the reference line. For example, the control unit 18 may set one of the two reference points set in step S142 as the measurement point. Alternatively, the control unit 18 may set the midpoint of the two reference points set in step S142 as the measurement point. The control unit 18 displays the measurement point on the 2D image (step S144).

[0247] The user corrects the position of the measurement point in the 2D image by operating the control unit 4 or the touch panel. The control unit 18 identifies the point in the 3D data that corresponds to the measurement point on the 2D image specified by the user. The 3D coordinates of the identified point are included in the 3D data. The control unit 18 sets the identified point as the measurement point (step S145).

[0248] After step S145, the control unit 18 calculates the 3D distance between the reference plane and the measurement point. The control unit 18 generates distance information indicating the calculated 3D distance (step S108b). The calculated 3D distance corresponds to the size of the clearance between the first region and the second region.

[0249] Figure 35 shows an example of the 3D shape of the subject. The 3D shape shown in Figure 35 includes a first region R40 and a second region R41. When the user specifies a reference point RP40 in the first region R40, the control unit 18 sets a reference plane RS40 based on the reference point RP40. When the user specifies reference points RP41 and RP42 in the second region R41, the control unit 18 sets a reference line RL40 that includes reference points RP41 and RP42.

[0250] The 3D shape is missing in region R42 as shown in Figure 35. As will be described later, the user corrects the position of the measurement point in the 2D image. The measurement point on the reference line RL40 is changed to measurement point MP40. Although the 3D shape at measurement point MP40 is missing, measurement point MP40 is on the original edge.

[0251] Figure 36 shows an example of a 2D image of the subject. The 2D image SI40 shown in Figure 36 is used to generate 3D data corresponding to the 3D shape shown in Figure 35. The 2D image SI40 is one of two images included in the stereo image. The reference point RP43 shown in Figure 36 corresponds to the reference point RP40 shown in Figure 35, the reference point RP44 shown in Figure 36 corresponds to the reference point RP41 shown in Figure 35, and the reference point RP45 shown in Figure 36 corresponds to the reference point RP42 shown in Figure 35. The reference line RL41 shown in Figure 36 corresponds to the reference line RL40 shown in Figure 35.

[0252] The user corrects the position of the measurement point on the reference line RL41. The measurement point MP41 shown in Figure 36 is the measurement point corrected by the user and corresponds to the measurement point MP40 shown in Figure 35. The measurement point MP41 is on the reference line RL40 and is on the edge of the second region in the 2D image SI40.

[0253] When a measurement point in the 2D image SI40 is changed to a measurement point MP41, the control unit 18 calculates the 2D distance between the old measurement point and the measurement point MP41 in the 2D image SI40. The control unit 18 calculates the measurement point MP40 corresponding to the measurement point MP41 by moving the old measurement point in the second region R41 shown in Figure 35 along the reference line RL40 by the calculated 2D distance.

[0254] As shown in Figure 35, 3D data is missing in region R42, which includes the location where the measurement point should be set, making it difficult to set the correct measurement point in the 3D shape. On the other hand, in the 2D image SI40 shown in Figure 36, the user can confirm the location where the measurement point should be set. Therefore, the user can change the position of the measurement point set on the reference line RL41 to the correct position.

[0255] The control unit 18 may calculate a plane PL40 that includes the measurement point MP40 and is parallel to the reference plane RS40. When the 3D image of the 3D shape shown in Figure 35 is displayed on the display unit 5, the control unit 18 may display the reference plane RS40 and the plane PL40 on the 3D image. The user can easily grasp the 3D distance between the reference plane RS40 and the measurement point MP40.

[0256] The control unit 18 may display edges E40 corresponding to the edges of the 3D shape shown in Figure 35 on the 2D image SI40. By referring to the edges E40, the user can determine the discrepancy between the edges in the 3D shape and the edges in the 2D image SI40.

[0257] In the fourth embodiment, even if 3D data is missing at the position where the measurement point should be set, the endoscope device 1 can accurately measure the size of the clearance.

[0258] (Fifth Embodiment) A fifth embodiment of the present invention will now be described. A defect is formed in the subject in the fifth embodiment. The defect corresponds to a region that is missing in the subject. The 3D coordinates of points in that region are not included in the 3D data.

[0259] The specimen in the fifth embodiment has a curved edge of the defect and a straight first edge and a second edge connected to the edge of the defect. The first edge and the second edge are not parallel to each other. Often, the edge of the defect has a number of recesses and protrusions. Often, defects in turbine blades occur at the corners of the turbine blade.

[0260] As mentioned above, in the conventional technology, users are required to manually input information for two or more points, which is time-consuming for the user. The fifth embodiment aims to provide a measuring device that can reduce the user's effort required for measurement and can perform measurements efficiently.

[0261] Furthermore, in the conventional technology described above, accurate information about the edges of the defective area cannot be obtained. The fifth embodiment aims to provide a measuring device that can accurately measure the size of the defective area.

[0262] Figure 37 shows an example of the measurement process performed by the endoscope device 1. The operation of the endoscope device 1 will be explained with reference to Figure 37. Processes that are the same as those shown in Figure 3 will not be explained.

[0263] In step S103, after the 3D image of the subject is displayed on the display unit 5, the user specifies two points on the first edge of the subject on the 3D image by operating the operation unit 4 or the touch panel. The 3D coordinates of each of the two points specified by the user are included in the 3D data. The edge information associated with each of the two points indicates that each point is an edge point. At least one of the two points may be a point in the vicinity of the first edge. The control unit 18 receives point information indicating the two points specified by the user. The control unit 18 sets the two points indicated by the point information (step S150).

[0264] When the point indicated by the point information is a point near the first edge, the control unit 18 may set that point, or it may set an edge point near the point indicated by the point information based on the edge information.

[0265] The user may specify two points on the subject in the 2D image by operating the control unit 4 or the touch panel. As described above, the positions in the 2D image and each point included in the 3D data are related to each other. The control unit 18 may identify two points in the 3D data that correspond to the two points specified by the user. The 3D coordinates of each of the two identified points are included in the 3D data. The control unit 18 may set the two identified points.

[0266] After step S150, the control unit 18 sets a first reference line that includes the two points set in step S150 (step S151). The first reference line is set in a space that has three or more points included in the 3D data. For example, the first reference line is a straight line. The first reference line may also be a line segment, a polyline, or a curve.

[0267] After step S151, the control unit 18 sets a first missing endpoint on the first edge based on the first reference line and edge information (step S152). The first missing endpoint is the start or end point of the curved edge of the missing portion.

[0268] Step S152 will now be described in detail. For example, the edge information indicates that each of the three or more points is an edge point. The control unit 18 identifies a first missing endpoint that is included in the three or more points and lies on the first edge of the missing portion. Specifically, the control unit 18 identifies the edge point that is located at or above a predetermined distance from the first reference line and is closest to the first reference line as the first missing endpoint. The control unit 18 may also identify the point on the first reference line closest to that edge point as the first missing endpoint.

[0269] The user specifies two points on the first edge of the subject, and then specifies two points on the second edge of the subject by operating the control unit 4 or touch panel. The 3D coordinates of each of the two points specified by the user are included in the 3D data. The edge information associated with each of the two points indicates that each point is an edge point. At least one of the two points may be a neighboring point of the second edge. The control unit 18 receives point information indicating the two points specified by the user. The control unit 18 sets the two points indicated by the point information (step S153).

[0270] When the point indicated by the point information is a point near the second edge, the control unit 18 may set that point, or it may set an edge point near the point indicated by the point information based on the edge information.

[0271] The user may specify two points on the subject in the 2D image by operating the control unit 4 or the touch panel. The control unit 18 may identify two points in the 3D data that correspond to the two points specified by the user. The 3D coordinates of each of the two identified points are included in the 3D data. The control unit 18 may set the two identified points.

[0272] After step S153, the control unit 18 sets a second reference line that includes the two points set in step S153 (step S154). The second reference line is set in a space that has three or more points included in the 3D data. For example, the second reference line is a straight line. The second reference line may be a line segment, a polyline, or a curve, etc.

[0273] After step S154, the control unit 18 sets a second missing endpoint on the second edge based on the second reference line and edge information (step S155). The second missing endpoint is the start or end point of the curved edge of the missing portion and is different from the first missing endpoint.

[0274] Step S155 will now be described in detail. For example, the edge information indicates that each of the three or more points is an edge point. The control unit 18 identifies a second missing endpoint that is included in the three or more points and lies on the second edge of the missing portion. Specifically, the control unit 18 identifies the edge point that is located at or above a predetermined distance from the second reference line and is closest to the second reference line as the second missing endpoint. The control unit 18 may also identify the point on the second reference line closest to that edge point as the second missing endpoint.

[0275] After step S155, the control unit 18 calculates the intersection point of the first reference line and the second reference line (step S156).

[0276] Step S156 will now be described in detail. Figure 38 shows an example of a first reference line and a second reference line. The first reference line RL50 and the second reference line RL51 shown in Figure 38 do not intersect each other. The control unit 18 calculates the 3D coordinates of point P52, which is midway between point P50 on the first reference line RL50 and point P51 on the second reference line RL51. Point P50 is the point on the first reference line RL50 closest to the second reference line RL51. Point P51 is the point on the second reference line RL51 closest to the first reference line RL50. Point P52 corresponds to the intersection of the first reference line RL50 and the second reference line RL51. The 3D coordinates of point P52 are not included in the 3D data.

[0277] After step S156, the control unit 18 sets a reference plane that includes the first missing endpoint, the second missing endpoint, and the intersection point (step S157). The reference plane is set in a space that has three or more points included in the 3D data.

[0278] After step S157, the control unit 18 calculates the size of the defect based on the first reference line, the second reference line, the first defect endpoint, the second defect endpoint, the intersection point, and the reference plane (step S158). When step S158 is executed, the measurement process shown in Figure 37 is completed.

[0279] Step S158 will now be described in detail. The control unit 18 projects the first reference line, the second reference line, and the intersection point onto the reference plane. Based on the first missing endpoint, the second missing endpoint, and edge information, the control unit 18 identifies one or more edge points of the missing portion. For example, the control unit 18 identifies one or more points between the first missing endpoint and the second missing endpoint that are indicated by the edge information. These one or more points are the edge points of the missing portion. The 3D coordinates of these one or more points are included in the 3D data. The control unit 18 projects the one or more edge points of the missing portion onto the reference plane. The first reference line, etc., is projected onto the reference plane in a direction perpendicular to the reference plane.

[0280] The control unit 18 calculates the size of the defect region enclosed by a first reference line projected onto the reference plane, a second reference line projected onto the reference plane, an intersection point projected onto the reference plane, and one or more edge points of the defect projected onto the reference plane. For example, the control unit 18 calculates the length or area of ​​the defect region. The length of the defect region is the 3D distance between the first defect endpoint and the intersection point, or the 3D distance between the second defect endpoint and the intersection point.

[0281] Figure 39 shows an example of the 3D shape of the subject. A defect DP50 is formed in the subject shown in Figure 39. When the user specifies points P53 and P54 on the first edge of the subject, the control unit 18 sets a first reference line RL52 that includes points P53 and P54. The control unit 18 sets a first defect endpoint DE50.

[0282] When the user specifies points P55 and P56 on the second edge of the subject, the control unit 18 sets a second reference line RL53 that includes points P55 and P56. The control unit 18 sets a second missing endpoint DE51. The control unit 18 sets the intersection point IN50 of the first reference line RL52 and the second reference line RL53.

[0283] The control unit 18 sets a reference plane based on the first missing endpoint DE 50, the second missing endpoint DE 51, and the intersection IN 50. The control unit 18 identifies the endpoint EP 50 of the edge of the missing portion DP 50. In Figure 39, the endpoint EP 50 is drawn by a line representing a collection of multiple points.

[0284] The control unit 18 projects the first reference line RL52, the second reference line RL53, the intersection IN50, and the endpoint EP50 onto the reference plane. The control unit 18 calculates the area of ​​the missing region enclosed by these lines. The control unit 18 also calculates the 3D distance L50 between the first missing endpoint DE50 and the intersection IN50 projected onto the reference plane, and calculates the 3D distance L51 between the second missing endpoint DE51 and the intersection IN50 projected onto the reference plane.

[0285] The control unit 18 may display the first reference line, the second reference line, and the reference plane on the 3D image.

[0286] After step S158 is performed, the control unit 18 may display the size of the missing portion calculated in step S157 on the display unit 5.

[0287] The user may move the first or second missing endpoint. For example, the user may specify a new position for the first or second missing endpoint. The control unit 18 may change the position of the first or second missing endpoint to its new position.

[0288] In the measurement process shown in Figure 37, the control unit 18 automatically sets the endpoints of the missing portion based on edge information. This reduces the effort required from the user and improves the efficiency of the measurement.

[0289] Even when the shape of the defect is complex, the control unit 18 can accurately detect the edges of the subject based on 3D data, allowing for accurate measurement of the size of the defect. Compared to edge extraction using 2D images, using 3D data allows for accurate edge extraction in most cases.

[0290] The control unit 18 projects the first reference line, the second reference line, and the intersection point onto the reference plane. The control unit 18 also calculates the area of ​​the missing region enclosed by these projected lines onto the reference plane. This improves the accuracy of the measurement of the area of ​​the missing region.

[0291] The control unit 18 measures the 3D distance between one of the first reference line and the second reference line and the intersection point of the two reference lines. If the 3D distance is greater than or equal to a predetermined distance, the control unit 18 may display information on the display unit 5 indicating that the accuracy of the area of ​​the missing portion may be poor.

[0292] The control unit 18 may calculate the angle between the first reference line and the second reference line. If the difference between the calculated angle and the design value of that angle is large, the control unit 18 may display information on the display unit 5 indicating that the accuracy of the area of ​​the missing portion may be poor.

[0293] If the thickness of the sample is known, the control unit 18 may obtain a value indicating that thickness from a memory card 42 or the like. The control unit 18 may also calculate the volume of the missing portion by multiplying the thickness by the area of ​​the missing portion.

[0294] As mentioned above, the user may move the first or second defective endpoint. If the position of the first or second defective endpoint is deviated from the ideal position, the control unit 18 can obtain accurate measurement results by changing the position of the first or second defective endpoint.

[0295] Each embodiment of the present invention's measuring device (endoscopic device 1) includes a control unit 18. The control unit 18 acquires 3D data including the 3D coordinates of three or more points on a subject calculated based on a 2D image of the subject acquired by the endoscope (insertion unit 2). The control unit 18 acquires edge information indicating whether each of the three or more points is an edge point on an edge of the subject. The control unit 18 displays a 3D image or a 2D image on a display unit 5 (display). The 3D image is an image of a 3D shape including points having 3D coordinates included in the 3D data. The control unit 18 sets one or two reference lines indicating a reference position for measurement based on at least one of the three or more points. In the above example, the control unit 18 sets two reference lines. The two reference lines are not parallel to each other. The edge information indicates that one or more points included in one or each of the two reference lines are edge points. The control unit 18 calculates the size of the missing portion based on one or two reference lines and one or more points. The edge information indicates that each of the one or more points is an edge point. The missing area corresponds to the region that is missing in the subject.

[0296] Each embodiment of the present invention comprises five steps. In the first step (step S101), the control unit 18 acquires 3D data. In the second step (step S102), the control unit 18 acquires edge information. In the third step (step S103), the control unit 18 displays the 3D image on the display unit 5. In the fourth step (steps S151 and S154), the control unit 18 sets one or two reference lines. In the fifth step (step S158), the control unit 18 calculates the size of the missing portion.

[0297] A program according to each aspect of the present invention causes a computer to execute the first to fifth steps described above.

[0298] Each aspect of the present invention may include the following modifications. The control unit 18 sets each of two reference lines based on two or more points. Edge information indicates that each of the two or more points is an edge point. The control unit 18 sets two missing endpoints on the edge of the missing portion. Edge information indicates that the two missing endpoints are edge points. The control unit 18 calculates the size of the missing portion based on the two reference lines and the two missing endpoints.

[0299] Each aspect of the present invention may include the following modifications. The two missing endpoints are the endpoints of the edge of the missing portion.

[0300] Each aspect of the present invention may include the following modifications. The control unit 18 sets two reference lines and calculates the intersection of the two reference lines. The control unit 18 sets a reference plane based on the intersection and the two missing endpoints. The edge information indicates that each of the two missing endpoints is an edge point. The control unit 18 calculates the area, which is the size of the missing portion identified based on the intersection, the two missing endpoints, and the reference plane.

[0301] Each aspect of the present invention may include the following modifications. The control unit 18 calculates a first point on the reference plane by projecting the intersection of two reference lines onto the reference plane. The control unit 18 calculates a second point and a third point on the reference plane by projecting two missing endpoints onto the reference plane. The control unit 18 calculates the size of the missing portion by calculating the area of ​​the region enclosed by the first point, the second point, and the third point.

[0302] Each aspect of the present invention may include the following modifications. The control unit 18 calculates the size of the defect by calculating the distance between the intersection of two reference lines and the edge point (defect endpoint) indicated by the edge information.

[0303] Each aspect of the present invention may include the following modifications. The control unit 18 calculates the area of ​​the defect and obtains a value indicating the thickness of the defect. The control unit 18 calculates the size of the defect by multiplying this value by the area of ​​the defect.

[0304] In the fifth embodiment, the control unit 18 automatically sets the endpoints of the defective portion. Therefore, the endoscope device 1 can reduce the effort required by the user for measurement and perform measurements efficiently. Furthermore, the endoscope device 1 can accurately measure the size of the defective portion by generating edge information based on 3D data.

[0305] (First Modification of the Fifth Embodiment) A first modification of the fifth embodiment of the present invention will now be described. In the fifth embodiment described above, the user specifies two points on the first edge of the subject and two points on the second edge of the subject, and the control unit 18 sets two reference lines. On the other hand, in the first modification of the fifth embodiment, the user specifies one point on the first edge of the subject and one point on the second edge of the subject, and the control unit 18 sets two reference lines.

[0306] Figure 40 shows an example of the measurement process performed by the endoscope device 1. The operation of the endoscope device 1 will be explained with reference to Figure 40. Processes that are the same as those shown in Figure 37 will not be explained.

[0307] In step S103, after the 3D image of the subject is displayed on the display unit 5, the user specifies a point on the first edge of the subject on the 3D image by operating the operation unit 4 or the touch panel. The 3D coordinates of the point specified by the user are included in the 3D data. The edge information associated with that point indicates that it is an edge point. That point may be a point in the vicinity of the first edge. The control unit 18 receives point information indicating the point specified by the user and sets the point indicated by the point information (step S150a).

[0308] When the point indicated by the point information is a point near the first edge, the control unit 18 may set that point, or it may set a nearby edge point based on the edge information. The control unit 18 may also set a point in the 3D data that corresponds to a point on the 2D image specified by the user.

[0309] After step S150a, the control unit 18 identifies two or more points in the vicinity of the point set in step S150a. Specifically, the control unit 18 identifies two or more points whose distance from the point set in step S150a is less than or equal to a predetermined distance. In other words, the control unit 18 identifies two or more points within a sphere with a predetermined radius centered on the point set in step S150a. The edge information associated with each of these two or more points indicates that each edge point is an edge point. The control unit 18 sets a first reference line based on these two or more points. The first reference line is a straight line that approximates the first edge. For example, the control unit 18 sets the first reference line by applying RANSAC to these two or more points (step S151a). After step S151a, step S152 is executed.

[0310] The control unit 18 may set a first reference line by applying RANSAC to the point set in step S150a and one or more points whose distance from the point set in step S150a is less than or equal to a predetermined distance.

[0311] The user specifies one point on the first edge of the subject, and then specifies one point on the second edge of the subject by operating the control unit 4 or touch panel. The 3D coordinates of the points specified by the user are included in the 3D data. The edge information associated with that point indicates that it is an edge point. That point may be a neighboring point of the second edge. The control unit 18 receives point information indicating the point specified by the user. The control unit 18 sets the point indicated by the point information (step S153a).

[0312] When the point indicated by the point information is a point near the second edge, the control unit 18 may set that point, or it may set a nearby edge point based on the edge information. The control unit 18 may set a point in the 3D data that corresponds to a point on the 2D image specified by the user. Step S153a may be performed before step S152 is performed.

[0313] After step S153a, the control unit 18 identifies two or more points in the vicinity of the point set in step S153a. Specifically, the control unit 18 identifies two or more points whose distance from the point set in step S153a is less than or equal to a predetermined distance. In other words, the control unit 18 identifies two or more points within a sphere with a predetermined radius centered on the point set in step S153a. The edge information associated with each of these two or more points indicates that each edge point is an edge point. The control unit 18 sets a second reference line based on these two or more points. The second reference line is a straight line that approximates the second edge. For example, the control unit 18 sets the second reference line by applying RANSAC to these two or more points (step S154a). After step S154a, step S155 is performed.

[0314] The control unit 18 may set a second reference line by applying RANSAC to the point set in step S153a and one or more points whose distance from the point set in step S153a is less than or equal to a predetermined distance.

[0315] Figure 41 shows an example of the 3D shape of the subject. A defect DP50a is formed in the subject shown in Figure 41. When the user specifies a point P57 on the first edge of the subject, the control unit 18 sets a first reference line RL52a based on two or more edge points in the region R50 surrounding point P57. The control unit 18 sets a first defect endpoint DE50a.

[0316] When the user specifies a point P58 on the second edge of the subject, the control unit 18 sets a second reference line RL53a based on two or more edge points in the region R51 surrounding point P58. The control unit 18 sets a second missing endpoint DE51a. The control unit 18 sets the intersection point IN50a of the first reference line RL52a and the second reference line RL53a.

[0317] The control unit 18 sets a reference plane based on the first missing endpoint DE 50a, the second missing endpoint DE 51a, and the intersection IN 50a. The control unit 18 identifies the endpoint EP 50a of the edge of the missing portion DP 51a. In Figure 41, the endpoint EP 50a is drawn by a line representing a collection of many points.

[0318] The control unit 18 projects the first reference line RL52a, the second reference line RL53a, the intersection IN50a, and the endpoint EP50a onto the reference plane. The control unit 18 calculates the area of ​​the missing region enclosed by these lines. The control unit 18 also calculates the 3D distance L50a between the first missing endpoint DE50a and the intersection IN50a projected onto the reference plane, and calculates the 3D distance L51a between the second missing endpoint DE51a and the intersection IN50a projected onto the reference plane.

[0319] Each aspect of the present invention may include the following modifications. The control unit 18 sets each of two reference lines based on two or more points around a point corresponding to a point specified on a 3D or 2D image by a pointing device.

[0320] In the first modified example of the fifth embodiment, the number of points specified by the user is reduced compared to the fifth embodiment. As a result, the endoscope device 1 can reduce the effort required by the user for measurement and perform measurements efficiently.

[0321] (Second Modification of the Fifth Embodiment) A second modification of the fifth embodiment of the present invention will now be described. In the fifth embodiment described above, the user specifies two points on the first edge of the subject and two points on the second edge of the subject, and the control unit 18 sets two reference lines. On the other hand, in the second modification of the fifth embodiment, the user specifies two points on the first edge of the subject and one missing endpoint on the second edge of the subject, and the control unit 18 sets two reference lines.

[0322] Figure 42 shows an example of the measurement process performed by the endoscope device 1. The operation of the endoscope device 1 will be explained with reference to Figure 42. Processes that are the same as those shown in Figure 37 will not be explained.

[0323] In step S103, after the 3D image of the subject is displayed on the display unit 5, the user specifies an endpoint of the edge of the missing portion of the subject on the 3D image by operating the operation unit 4 or the touch panel. At this time, the user specifies an endpoint near the first edge of the subject. The 3D coordinates of the endpoint specified by the user are included in the 3D data. The edge information associated with that endpoint indicates that it is an edge point. That endpoint may be a point near the edge of the missing portion. The control unit 18 receives point information indicating the endpoint specified by the user and sets the endpoint indicated by the point information as the first missing endpoint (step S152b).

[0324] When the endpoint indicated by the point information is a point near the edge of the missing portion, the control unit 18 may set that endpoint as the first missing endpoint, or it may set a nearby edge point as the first missing endpoint based on the edge information. The control unit 18 may also set a first missing endpoint corresponding to a point on the 2D image specified by the user.

[0325] The user specifies the endpoint of the edge of the missing portion of the subject, and then specifies a point on the first edge of the subject by operating the operation unit 4 or touch panel. The 3D coordinates of the point specified by the user are included in the 3D data. The edge information associated with that point indicates that it is an edge point. That point may be a neighboring point of the first edge. The control unit 18 receives point information indicating the point specified by the user and sets the point indicated by the point information (step S150b). After step S150b, step S151 is executed.

[0326] When the point indicated by the point information is a point in the vicinity of the first edge, the control unit 18 may set that point, or it may set an edge point in the vicinity of the point indicated by the point information based on the edge information. The control unit 18 may also set a point in the 3D data that corresponds to a point on the 2D image specified by the user. Step S150b may be performed before step S152b is performed.

[0327] The user specifies a point on the first edge of the subject, and then specifies an endpoint of the edge of the missing portion of the subject by operating the operation unit 4 or touch panel. At this time, the user specifies an endpoint near the second edge of the subject. The 3D coordinates of the endpoint specified by the user are included in the 3D data. The edge information associated with that endpoint indicates that it is an edge point. That endpoint may be a point near the edge of the missing portion. The control unit 18 receives point information indicating the endpoint specified by the user and sets the endpoint indicated by the point information as the second missing endpoint (step S155b).

[0328] When the endpoint indicated by the point information is a point near the edge of the missing portion, the control unit 18 may set that endpoint as the second missing endpoint, or it may set a nearby edge point as the second missing endpoint based on the edge information. The control unit 18 may also set a second missing endpoint corresponding to a point on the 2D image specified by the user.

[0329] After step S155b, the control unit 18 sets a second reference line as a straight line that includes the second missing endpoint and is perpendicular to the first reference line (step S154b). The second reference line is perpendicular to the first reference line. After step S154b, step S156 is performed.

[0330] Figure 43 shows an example of the 3D shape of the subject. A defect DP50b is formed in the subject shown in Figure 43. When the user specifies a point P53b on the first edge of the subject and a first defect endpoint DE50b, the control unit 18 sets a first reference line RL52b that includes point P53b and the first defect endpoint DE50b.

[0331] When the user specifies a second missing endpoint DE51b on the second edge of the subject, the control unit 18 sets a second reference line RL53b that includes the second missing endpoint DE51b and is orthogonal to the first reference line RL52b. The control unit 18 sets the intersection point IN50b of the first reference line RL52b and the second reference line RL53b.

[0332] The control unit 18 sets a reference plane based on the first missing endpoint DE 50b, the second missing endpoint DE 51b, and the intersection IN 50b. The control unit 18 identifies the endpoint EP 50b of the edge of the missing portion DP 50b. In Figure 43, the endpoint EP 50b is drawn by a line representing a collection of multiple points.

[0333] The control unit 18 projects the first reference line RL52b, the second reference line RL53b, the intersection IN50b, and the endpoint EP50b onto the reference plane. The control unit 18 calculates the area of ​​the missing region enclosed by these lines. The control unit 18 also calculates the 3D distance L50b between the first missing endpoint DE50b and the intersection IN50b projected onto the reference plane, and calculates the 3D distance L51b between the second missing endpoint DE51b and the intersection IN50b projected onto the reference plane.

[0334] The control unit 18 may display a first reference line and a second reference line on the 3D image. The user can check the degree of agreement between the second reference line and the second edge. If the second reference line does not agree well with the second edge, the user may move the intersection point of the first reference line and the second reference line. For example, the user may specify a new position for the intersection point. The control unit 18 may change the position of the intersection point to the new position. The control unit 18 may set a second reference line that includes the intersection point and the second missing endpoint.

[0335] Each aspect of the present invention may include the following modifications. The control unit 18 sets a first reference line, which is one of two reference lines, based on two points. Edge information indicates that the two points are edge points. The control unit 18 sets a second reference line that intersects the first reference line, based on one point. Edge information indicates that the one point is an edge point. The second reference line is the other of the two reference lines. The control unit 18 calculates the size of the defect based on the two reference lines and the one point.

[0336] Each aspect of the present invention may include the following modifications. One of the two points mentioned above is the endpoint of the edge of the defect.

[0337] Each aspect of the present invention may include the following modifications: The second reference line is perpendicular to the first reference line.

[0338] In the second modification of the fifth embodiment, the number of points specified by the user is reduced compared to the fifth embodiment. As a result, the endoscope device 1 can reduce the effort required by the user for measurement and perform measurements efficiently.

[0339] Since the user specifies the first and second defect endpoints, the endoscope device 1 can set the first and second defect endpoints to the correct positions. Therefore, the endoscope device 1 can accurately measure the size of the defect.

[0340] (Third Modification of the Fifth Embodiment) A third modification of the fifth embodiment of the present invention will now be described. In the fifth embodiment described above, the control unit 18 sets a reference plane based on two missing endpoints and the intersection of two reference lines. On the other hand, in the third modification of the fifth embodiment, the control unit 18 sets a reference plane based on points around the two missing endpoints.

[0341] The endoscope device 1 performs the measurement process shown in Figure 37. In step S157, the control unit 18 performs the following process. The control unit 18 identifies three or more points around the first missing endpoint set in step S152 and the second missing endpoint set in step S155. Specifically, the control unit 18 sets up a sphere whose diameter is the line segment connecting the first missing endpoint and the second missing endpoint. The control unit 18 identifies three or more points within that sphere. The 3D coordinates of these three or more points are included in the 3D data.

[0342] The control unit 18 sets a reference plane by performing a planar fitting or surface fitting using the 3D coordinates of the three or more points. The control unit 18 may set a reference plane based on three or more points, including at least one of the first and second missing endpoints and one or more points within the sphere.

[0343] Figure 44 shows an example of the 3D shape of the subject. When the user specifies a first missing endpoint DE50c on a first edge of the subject and a second missing endpoint DE51c on a second edge of the subject, the control unit 18 sets up a sphere SP50 based on the first missing endpoint DE50c and the second missing endpoint DE51c. The control unit 18 sets up a reference plane based on three or more points within the sphere SP50.

[0344] The endoscope device 1 may perform the measurement process shown in Figure 40, or it may set a reference plane by performing the above process in step S157 shown in Figure 40. The endoscope device 1 may perform the measurement process shown in Figure 42, or it may set a reference plane by performing the above process in step S157 shown in Figure 42.

[0345] Each aspect of the present invention may include the following modifications. The control unit 18 sets a reference plane based on a sphere whose diameter is the line segment connecting the two missing endpoints.

[0346] In a third modification of the fifth embodiment, the control unit 18 sets the reference plane in a manner different from that of the fifth embodiment. The control unit 18 can select a method for setting the reference plane based on various factors such as the condition of the subject and the time constraints required for calculation.

[0347] (Fourth Modification of the Fifth Embodiment) A fourth modification of the fifth embodiment of the present invention will now be described. In the fifth embodiment described above, the control unit 18 generates edge information by using 3D data. If the 3D data contains noise, the position of some points included in the 3D data will differ from the correct position. Therefore, accurate edge information may not be generated. In the fourth modification of the fifth embodiment, the control unit 18 updates the edge information by using the 2D image used to generate the 3D data.

[0348] Figure 45 shows an example of the measurement process performed by the endoscope device 1. The operation of the endoscope device 1 will be explained with reference to Figure 45. Processes that are the same as those shown in Figure 37 will not be explained.

[0349] After step S102, the control unit 18 updates the edge information using the 2D image used to generate the 3D data in step S101 (step S160). After step S160, step S103 is executed.

[0350] Step S160 will be described in detail with reference to Figures 46 and 47. Figure 46 shows an example of the 3D shape of the subject. Figure 47 shows an example of a 2D image of the subject.

[0351] The edge E50 shown in Figure 46 is a set of points that were determined to be edge points in step S102. Because the 3D data contains noise, edge E50 differs from the actual edge E51. Therefore, the area A50 of the missing portion calculated in step S158 contains errors.

[0352] The edge E52 shown in Figure 47 corresponds to an actual edge of the subject. The control unit 18 detects edge E52 by performing edge detection processing using a first-order differential filter or a second-order differential filter. As mentioned above, the position on the 2D image and each point included in the 3D data are associated with each other. The control unit 18 calculates the 3D coordinates of each of one or more points of edge E51 that correspond to edge E52. If the edge information associated with each point of edge E51 indicates a non-edge point, the control unit 18 updates the edge information so that the edge information indicates an edge point.

[0353] Step S160 may be performed after step S103 has been executed.

[0354] Each aspect of the present invention may include the following modifications. The control unit 18 detects edges of a subject based on a 2D image of the subject. The control unit 18 identifies one or more points that are included in three or more points in the 3D data and that correspond to edges in the 2D image. The control unit 18 updates the edge information. The updated edge information indicates that each of the one or more points that correspond to edges in the 2D image is an edge point.

[0355] In the fourth modification of the fifth embodiment, the control unit 18 detects the edges of the subject based on a 2D image of the subject and updates the edge information associated with the points on the edges and the corresponding points in the 3D data. As a result, the endoscope device 1 can acquire accurate edge information.

[0356] (Fifth Modification of the Fifth Embodiment) A fifth modification of the fifth embodiment of the present invention will now be described. In the fifth embodiment described above, the user specifies two points on the first edge of the subject and two points on the second edge of the subject, and the control unit 18 sets two reference lines. On the other hand, in the fifth modification of the fifth embodiment, the user does not need to specify points, and the two reference lines are set automatically.

[0357] Figure 48 shows an example of the measurement process performed by the endoscope device 1. The operation of the endoscope device 1 will be explained with reference to Figure 48. Processes that are the same as those shown in Figure 37 will not be explained.

[0358] After step S103, the control unit 18 sets a first reference line (step S151d). After step S151d, step S152 is performed.

[0359] Step S151d will now be described in detail. For example, the control unit 18 detects a straight line that approximates a first edge of the subject by performing a Hough transform using the 3D data. The control unit 18 sets that straight line as a first reference line. The control unit 18 may also detect a straight line that approximates a first edge in the 2D image by performing a Hough transform using the 2D image used to generate the 3D data. The control unit 18 may set the first reference line based on two or more points on that straight line and two or more points in the 3D data that correspond to them.

[0360] After step S152, the control unit 18 sets a second reference line (step S154d). In step S154d, the control unit 18 performs the same processing as in step S151d. After step S154d, step S155 is executed.

[0361] Figure 49 shows an example of the 3D shape of the subject. The control unit 18 sets a first reference line RL52d that approximates the first edge of the subject. The control unit 18 also sets a second reference line RL53d that approximates the second edge of the subject.

[0362] Steps S151d and S154d may be combined into a single step. The control unit 18 may detect two or more straight lines. The control unit 18 may set one of the longest first straight line and the second longest second straight line as the first reference line, or set the other of the first and second straight lines as the second reference line.

[0363] Each aspect of the present invention may include the following modifications. The control unit 18 sets each of two reference lines that approximate the edge of the subject based on the edge points indicated by the edge information. The control unit 18 sets two missing endpoints on the edge of the missing portion. The edge information indicates that these two missing endpoints are edge points. The control unit 18 calculates the size of the missing portion based on the two reference lines and their two missing endpoints.

[0364] Each aspect of the present invention may include the following modifications. The two missing endpoints are the endpoints of the edge of the missing portion.

[0365] In the fifth modification of the fifth embodiment, the control unit 18 automatically sets two reference lines. In the fifth modification of the fifth embodiment, the number of points specified by the user is reduced compared to the fifth embodiment and the like. As a result, the endoscope device 1 can reduce the effort required by the user for measurement and can perform measurements efficiently.

[0366] (Sixth Embodiment) A sixth embodiment of the present invention will now be described. In the fifth embodiment described above, the control unit 18 sets two reference lines. On the other hand, in the sixth embodiment, the control unit 18 sets one reference line.

[0367] Figure 50 shows an example of the measurement process performed by the endoscope device 1. The operation of the endoscope device 1 will be explained with reference to Figure 50. Processes that are the same as those shown in Figure 37 will not be explained.

[0368] In step S151, the control unit 18 sets a first reference line, and in step S152, it sets a first missing endpoint. After step S152, step S155 is executed. The measurement process shown in Figure 50 does not include steps S153 and S154 shown in Figure 37. In other words, the control unit 18 does not set a second reference line. In step S155, the control unit 18 sets a second missing endpoint.

[0369] After step S155, the control unit 18 sets the missing vertex based on the first reference line set in step S151 and the second missing endpoint set in step S155. For example, the control unit 18 calculates a straight line that includes the second missing endpoint and is perpendicular to the first reference line, and calculates the intersection point between the first reference line and that straight line. The control unit 18 sets that intersection point as the missing vertex (step S170). The missing vertex corresponds to the vertex of the corner of the subject. Because that corner is missing, the 3D coordinates of the missing vertex are not included in the 3D data.

[0370] After step S170, the control unit 18 calculates the size of the defect based on the first defect endpoint, the second defect endpoint, and the defect vertex (step S158e). When step S158e is executed, the measurement process shown in Figure 50 is completed.

[0371] Step S158e will now be described in detail. The control unit 18 calculates the length of the missing region. The length of the missing region is the 3D distance between the first missing endpoint and the missing vertex, or the 3D distance between the second missing endpoint and the missing vertex. The control unit 18 may also calculate the approximate area of ​​the missing region based on the first missing endpoint, the second missing endpoint, and the missing vertex.

[0372] Figure 51 shows an example of the 3D shape of the subject. A defect DP50e is formed in the subject shown in Figure 51. When the user specifies points P53e and P54e on the first edge of the subject, the control unit 18 sets a first reference line RL52e that includes points P53e and P54e. The control unit 18 sets a first defect endpoint DE50e.

[0373] When the user specifies a second missing endpoint DE51e on the second edge of the subject, the control unit 18 calculates a straight line SL50 that includes the second missing endpoint DE51e and is perpendicular to the first reference line RL52e. The control unit 18 calculates the intersection point of the first reference line RL52e and the straight line SL50 and sets that intersection point as the missing vertex DE52. The control unit 18 calculates the 3D distance L50e between the first missing endpoint DE50e and the missing vertex DE52, and calculates the 3D distance L51e between the second missing endpoint DE51e and the missing vertex DE52.

[0374] In most cases, the user is aware of the angles of the corners of the subject. The control unit 18 may set the straight line such that the angle between the first reference line and the second line containing the missing endpoint matches a preset angle.

[0375] Each aspect of the present invention may include the following modifications. The control unit 18 calculates the size of the defect by calculating the distance between a point on a reference line and an edge point (second defect endpoint) indicated by edge information.

[0376] In the sixth embodiment, the number of points specified by the user is reduced compared to the fifth embodiment. Therefore, the endoscope device 1 can reduce the effort required by the user for measurement and perform measurements efficiently.

[0377] Since the user specifies the second defect endpoint, the endoscope device 1 can set the second defect endpoint to the correct position. Therefore, the endoscope device 1 can accurately measure the size of the defect.

[0378] While preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments and their variations. Additions, omissions, substitutions, and other modifications are possible without departing from the spirit of the invention. Furthermore, the present invention is not limited by the foregoing description, but only by the scope of the appended claims.

[0379] According to each embodiment of the present invention, the measuring device, measuring method, and program can reduce the effort required by the user for measurement and can perform measurements efficiently.

[0380] 1 Endoscope device 2 Insertion section 3 Main unit 4 Operation section 5 Display section 8 Endoscope unit 9 CCU 10 Control unit 12 Video signal processing circuit 13 ROM 14 RAM 15 Card interface 16 External device interface 17 Control interface 18 Control unit 20 Tip 21 Lens 28 Image sensor

Claims

1. A measuring device comprising a control unit, the control unit acquires three-dimensional data including the three-dimensional coordinates of three or more points on a subject calculated based on a two-dimensional image of the subject acquired by an endoscope, acquires edge information indicating whether each of the three or more points is an edge point on an edge of the subject, displays a three-dimensional image or the two-dimensional image on a display, the three-dimensional image is an image of a three-dimensional shape including points having the three-dimensional coordinates, sets a measurement reference indicating a reference position for measurement based on at least one of the three or more points, sets a measurement area based on a point specified on the three-dimensional image or the two-dimensional image by a pointing device, the measurement area has a preset shape and is independent of the measurement reference, sets points included in the measurement area as measurement points, the edge information indicates that the measurement points are edge points, and generates distance information indicating the distance between the measurement reference and the measurement points.

2. The measuring device according to claim 1, wherein the control unit calculates a line that approximates the edge of the subject based on the point specified on the three-dimensional image or the two-dimensional image and the edge point indicated by the edge information, and sets the measurement area based on the line.

3. The measuring device according to claim 1, wherein the control unit identifies an edge including the edge point indicated by the edge information based on the three-dimensional data, and sets one of the one or more edges of the subject as the measurement area based on the point specified on the three-dimensional image or the two-dimensional image.

4. The measuring device according to claim 1, wherein the control unit detects one or more feature regions on the subject based on the three-dimensional data or the two-dimensional image, the three-dimensional shape has common features in each of the one or more feature regions, and at least one of the one or more feature regions is set as the measurement region.

5. The measuring device according to claim 1, wherein the control unit sets a new measurement area based on the measurement point and a point other than the measurement point.

6. The measuring device according to claim 5, wherein the control unit identifies an edge containing the edge point indicated by the edge information based on the three-dimensional data, and sets the new measurement area based on a point included in the same edge as the edge containing the measurement point.

7. The measuring device according to claim 1, wherein the control unit sets two or more measurement points included in the measurement area, sets a removal area including a part of the measurement area, and removes the measurement points included in the removal area from the two or more measurement points.

8. The measuring device according to claim 7, wherein the control unit generates distance information indicating the distance between the measurement reference and the remaining measurement points by removing the measurement points included in the removal area from the two or more measurement points.

9. The measuring device according to claim 1, wherein the control unit sets two or more measurement points included in the measurement area, generates first distance information indicating the distance between the measurement reference and each of the two or more measurement points, sets a removal area including a part of the measurement area after the first distance information has been generated, removes the measurement points included in the removal area from the two or more measurement points, and generates second distance information indicating the distance between the measurement reference and the measurement points remaining after removing the measurement points included in the removal area from the two or more measurement points.

10. The measuring device according to claim 1, wherein the control unit generates the edge information based on the three-dimensional data.

11. The measuring device according to claim 1, wherein the control unit sets the measurement reference which is one of a surface, a line, and a point.

12. The measuring device according to claim 1, wherein the control unit sets the measurement reference before the measurement area is set.

13. The measuring device according to claim 1, wherein the control unit sets the measurement area before the measurement standard is set.

14. The measuring device according to claim 1, wherein the control unit sets the measurement area which is a figure having a predetermined size.

15. The measuring device according to claim 1, wherein the control unit displays the measurement area on the display.

16. The measuring device according to claim 1, wherein the control unit causes the display state of a point on the three-dimensional image corresponding to the measurement point to be different from the display state of a point on the three-dimensional image corresponding to a point other than the measurement point.

17. The measuring device according to claim 1, wherein the control unit calculates the size of a figure that approximates the edge of the object based on two or more measurement points including the measurement point.

18. The measuring device according to claim 1, wherein the control unit generates distance information indicating the distance between the measurement reference and each of two or more measurement points including the measurement point.

19. The measuring device according to claim 1, wherein the control unit displays the distance information on the display.

20. A measurement method comprising: acquiring three-dimensional data including the three-dimensional coordinates of three or more points on a subject calculated based on a two-dimensional image of the subject acquired by an endoscope; acquiring edge information indicating whether each of the three or more points is an edge point on an edge of the subject; displaying the three-dimensional image or the two-dimensional image on a display, wherein the three-dimensional image is an image of a three-dimensional shape including points having the three-dimensional coordinates; setting a measurement reference indicating a reference position for measurement based on at least one of the three or more points; setting a measurement area based on a point specified on the three-dimensional image or the two-dimensional image by a pointing device, wherein the measurement area has a preset shape and is independent of the measurement reference; setting points included in the measurement area as measurement points; the edge information indicating that the measurement points are edge points; and generating distance information indicating the distance between the measurement reference and the measurement points.

21. A program for causing a computer to perform the following steps: acquiring three-dimensional data including the three-dimensional coordinates of three or more points on a subject calculated based on a two-dimensional image of the subject acquired by an endoscope; acquiring edge information indicating whether each of the three or more points is an edge point on an edge of the subject; displaying a three-dimensional image or the two-dimensional image on a display, wherein the three-dimensional image is an image of a three-dimensional shape including points having the three-dimensional coordinates; setting a measurement reference indicating a reference position for measurement based on at least one of the three or more points; setting a measurement area based on a point specified on the three-dimensional image or the two-dimensional image by a pointing device, wherein the measurement area has a preset shape and is independent of the measurement reference; setting a point included in the measurement area as a measurement point, wherein the edge information indicates that the measurement point is an edge point; and generating distance information indicating the distance between the measurement reference and the measurement point.

Images