Medical endoscope-based image processing system, method and device

Abstract

A medical endoscope-based image processing system, method and device. The medical endoscope-based image processing system comprises: a single-channel endoscope, wherein a single imaging channel of the single-channel endoscope acquires a two-dimensional image; and a control device, wherein the control device is electrically connected to the single-channel endoscope, and the control device is configured to: in response to the two-dimensional image sent by the single-channel endoscope, extract an image part of a set region in the two-dimensional image; and perform three-dimensional reconstruction on the image part to obtain a fused image.

Description

Image processing systems, methods, and devices based on medical endoscopes Technical Field

[0001] This disclosure relates to the technical field of medical devices, and more specifically, to an image based on a medical endoscope. Background Technology

[0002] With the rapid development of medical equipment, doctors can improve both efficiency and accuracy in diagnosing and treating patients using these devices. Currently, to obtain three-dimensional images of a patient's internal organs, multiple two-dimensional images from different perspectives can be obtained using configured electronic endoscopes or dual-path optical endoscopes, and a three-dimensional image of the patient's internal organs can be constructed based on these two-dimensional images. However, this dual-path imaging method is often limited by manufacturing processes and mechanical dimensions, making it difficult to apply to some small types of endoscopes. Summary of the Invention

[0003] One objective of this disclosure is to provide a new technical solution for image processing based on medical endoscopes.

[0004] According to a first aspect of this disclosure, an image processing system based on a medical endoscope is provided, the system comprising:

[0005] A single-channel endoscope, wherein a single imaging channel of the single-channel endoscope acquires two-dimensional images;

[0006] A control device electrically connected to the single-channel endoscope is configured to: extract an image portion of a set region from the two-dimensional image in response to the two-dimensional image sent by the single-channel endoscope; and perform three-dimensional reconstruction on the image portion to obtain a fused image.

[0007] Optionally, the diameter of the first channel corresponding to the single imaging channel of the single-channel endoscope is less than or equal to 10 mm.

[0008] Optionally, the diameter of the first channel is the diameter of the front end of the single imaging channel.

[0009] Optionally, the system further includes a display device electrically connected to the control device, the display device being used to display the fused image in response to the fused image sent by the control device.

[0010] According to a second aspect of this disclosure, an image processing method based on a medical endoscope is also provided. The method is applied to the image processing system based on the medical endoscope, and the executing entity of the method is a control device within the image processing system based on the medical endoscope. The method includes:

[0011] In response to the two-dimensional image transmitted by a single-channel endoscope, an image portion of a defined region in the two-dimensional image is extracted;

[0012] A 3D reconstruction is performed on the image portion to obtain a fused image.

[0013] Optionally, the step of extracting a portion of a predetermined region from the two-dimensional image in response to the two-dimensional image transmitted by a single-channel endoscope includes:

[0014] In response to the two-dimensional image transmitted by the single-channel endoscope, the current image format of the two-dimensional image is converted into a to-be-processed image of a set image format;

[0015] The image portion of a designated region in the two-dimensional image is extracted using a preset ROI module.

[0016] Optionally, the step of performing three-dimensional reconstruction on the image portion to obtain the fused image includes:

[0017] The image portion is reconstructed in three dimensions using a pre-set multi-task deep learning model to obtain a reconstructed image.

[0018] The reconstructed image is registered and fused with the image portion to synthesize a fused image with depth information.

[0019] Optionally, the multi-task deep learning model includes sequentially connected generation nodes, fusion nodes, and rendering nodes;

[0020] The generation node is used to generate a first projected image from a predetermined viewpoint based on the image portion; the fusion node is used to fuse the first projected image into the three-dimensional point cloud data to generate a point cloud image; and the rendering node is used to render a second projected image corresponding to a specific viewpoint from the point cloud image to generate a depth image.

[0021] According to a third aspect of this disclosure, an image processing apparatus based on a medical endoscope is also provided, the apparatus comprising:

[0022] A response module is used to extract an image portion of a set region from the two-dimensional image in response to the two-dimensional image sent by a single-channel endoscope.

[0023] The module is used to perform 3D reconstruction of the image portion to obtain a fused image.

[0024] According to a third aspect of this disclosure, an image processing apparatus based on a medical endoscope is also provided, including a memory and a processor, the memory being used to store a computer program; the processor being used to execute the computer program to implement the method according to a second aspect of this disclosure.

[0025] According to a fourth aspect of this disclosure, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the method according to a second aspect of this disclosure.

[0026] According to a fifth aspect of this disclosure, a computer program product is also provided, comprising a computer program that, when executed by a processor, implements the method described according to a second aspect of this disclosure.

[0027] One beneficial effect of this disclosure is that the image processing system based on medical endoscopes provided by the present invention can acquire two-dimensional images through a single imaging channel of a single endoscope, and extract a portion of the image from a designated region in the two-dimensional image sent by the single endoscope through a control device, and perform three-dimensional reconstruction on the image portion to obtain a fused image. In other words, by using the control device to process a single two-dimensional image to construct a three-dimensional image, it is possible to effectively construct a three-dimensional image of the patient's interior in small-scale endoscope applications while ensuring the quality of the three-dimensional image, thus broadening the application scenarios of image processing systems based on medical endoscopes.

[0028] Other features and advantages of the embodiments of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings.

[0029] Attached Figure Description

[0030] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments of the present disclosure and, together with their description, serve to explain the principles of the embodiments of the present disclosure.

[0031] Figure 1 is a schematic diagram of the composition structure of an image processing system based on a medical endoscope according to one embodiment;

[0032] Figure 2 is a flowchart illustrating an image processing method based on a medical endoscope according to one embodiment;

[0033] Figure 3 is a structural topology diagram of a multi-task deep learning model according to an embodiment;

[0034] Figure 4 is a block diagram of an image processing device based on a medical endoscope according to an embodiment;

[0035] Figure 5 is a schematic diagram of the hardware structure of an image processing device based on a medical endoscope according to one embodiment. Detailed Implementation

[0036] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the invention.

[0037] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the invention or its application or use.

[0038] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0039] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0040] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0041] <System Implementation Example>

[0042] Figure 1 is a schematic diagram of the composition of an image processing system based on a medical endoscope according to one embodiment. As shown in Figure 1, the image processing system based on a medical endoscope may include a single-channel endoscope and a control device, and is applied in the context of medical endoscopes.

[0043] The endoscope acquires two-dimensional images through a single imaging channel;

[0044] The control device is electrically connected to a single-channel endoscope and is configured to: extract a portion of a set area from the two-dimensional image sent by the single-channel endoscope in response to the two-dimensional image; and perform three-dimensional reconstruction on the portion of the image to obtain a fused image.

[0045] In other words, this image processing system based on medical endoscopes can acquire two-dimensional images through a single imaging channel of a single endoscope, and through a control device, extract a portion of the image from a designated region in the two-dimensional image sent by the single endoscope, and then perform three-dimensional reconstruction on this portion to obtain a fused image. In other words, by using the control device to process a single two-dimensional image to construct a three-dimensional image, it is possible to effectively construct a three-dimensional image of the patient's interior in small-scale endoscope applications while ensuring the quality of the three-dimensional image, thus broadening the application scenarios of image processing systems based on medical endoscopes.

[0046] In some embodiments, in order to enable image acquisition of a two-dimensional image from a single imaging channel of a single-channel endoscope, the diameter of the first channel corresponding to the single imaging channel of the single-channel endoscope is less than or equal to 10 mm.

[0047] In some examples, the diameter of the first channel corresponding to the single imaging channel of a single-channel endoscope can be between 2.7mm and 10mm.

[0048] In some embodiments, to accommodate more endoscopes, the diameter of the first channel is the diameter of the front end of the single imaging channel.

[0049] In this embodiment, the front end of the single imaging channel can be the end that first extends into the patient's body.

[0050] In some embodiments, to facilitate doctors' viewing of the fused images, the system further includes a display device electrically connected to the control device, the display device being used to display the fused image in response to the fused image sent by the control device.

[0051] In the embodiments of this disclosure, the control device memory stores a computer program that controls the control device processor to operate according to the image processing method based on a medical endoscope according to any embodiment. Those skilled in the art can design the computer program based on the scheme of the embodiments of this disclosure. How the computer program controls the processor to operate is well known in the art and will not be described in detail here.

[0052] <Method Implementation>

[0053] Figure 2 is a schematic flowchart of an image processing method based on a medical endoscope according to one embodiment. The implementing entity is, for example, the control device shown in Figure 1.

[0054] As shown in Figure 2, the image processing method based on medical endoscope in this embodiment may include the following steps S210 and S220:

[0055] Step S210: In response to the two-dimensional image sent by the single-channel endoscope, extract the image portion of a set region from the two-dimensional image.

[0056] In some embodiments, in order to improve the computational efficiency of the control device and save the computing resources of the control device, step S210 may include the following steps S2101 and S2102:

[0057] Step S2101: In response to the two-dimensional image sent by the single-channel endoscope, the current image format of the two-dimensional image is converted into a processing image of the set image format.

[0058] In this embodiment, after a single-channel two-dimensional image is input via a single-channel endoscope, the control device is compatible with image inputs of various formats and interfaces, and preprocesses the image into a recognizable image format, that is, converts the current image format of the two-dimensional image into a set image format to be processed.

[0059] Step S2102: Extract the image portion of the set region in the two-dimensional image through the preset ROI module.

[0060] In this embodiment, the control device pre-sets a Region of Interest (ROI) module, which can identify a defined region of interest in an existing image. For small endoscopes, the display of a two-dimensional image cannot completely fill the entire screen. For the black border areas that are not displayed, the ROI module identifies the image portion that needs to be used for three-dimensional reconstruction, thereby saving the computing power of the control device and providing a better display effect.

[0061] In some examples, to further improve the image quality of the subsequently fused images, the method further includes the following step S2103 after step S2102:

[0062] Step S2103: Perform image enhancement processing on the image portion to obtain the enhanced image portion.

[0063] Step S220: Perform three-dimensional reconstruction on the image portion to obtain a fused image.

[0064] In some embodiments, step S220 may include the following steps S2201 and S2202:

[0065] Step S2201: The image is reconstructed in three dimensions using a pre-set multi-task deep learning model to obtain the reconstructed image.

[0066] In this embodiment, the multi-task deep learning model includes a generation node, a fusion node, and a rendering node connected in sequence. The generation node is used to generate a first projection image from a predetermined viewpoint based on the image portion. The fusion node is used to fuse the first projection image into the 3D point cloud data to generate a point cloud image. The rendering node is used to render a second projection image corresponding to a specific viewpoint from the point cloud image to generate a depth image.

[0067] In some examples, as shown in Figure 3, in the generation node, 2D structure generation occurs: the generator takes a single 2D image (i.e., the image portion) and uses a preset image generation algorithm to generate a 2D projection (i.e., the first projected image) from a predetermined viewpoint. In the fusion node, point cloud fusion occurs: the 2D projection is fused into the 3D point cloud data to generate a point cloud map (i.e., a point cloud image). In the fusion node, pseudo-rendering occurs: the fused point cloud map is rendered with different 2D projections from a specific viewpoint to generate corresponding depth images.

[0068] In some examples, as shown in Figure 3, the multi-task deep learning model also includes an optimization node that can compare the available 3D images during model training to further optimize the image generation algorithm.

[0069] Step S2202 involves registering and fusing the reconstructed image with the image portion to synthesize a fused image with depth information.

[0070] In this embodiment, the fused image can be a three-dimensional RGB image.

[0071] <Equipment Example 1>

[0072] Figure 4 is a schematic block diagram of an image processing apparatus based on a medical endoscope according to one embodiment. As shown in Figure 4, the image processing apparatus 400 based on a medical endoscope may include:

[0073] The response module 410 is used to extract a portion of the image from a set area in response to a two-dimensional image sent by a single-channel endoscope.

[0074] Module 420 is obtained, which is used to perform three-dimensional reconstruction of the image portion to obtain a fused image.

[0075] Optionally, the response module 410 is also configured to, in response to a two-dimensional image sent by a single-channel endoscope, convert the current image format of the two-dimensional image into a to-be-processed image of a set image format; and extract the image portion of a set region in the two-dimensional image through a preset ROI module.

[0076] Optionally, module 420 is also used to perform three-dimensional reconstruction of the image portion using a pre-set multi-task deep learning model to obtain a reconstructed image; and to register and fuse the reconstructed image with the image portion to synthesize a fused image with depth information.

[0077] The image processing device 400 based on the medical endoscope can be the control device 10 in Figure 1.

[0078] <Equipment Example 2>

[0079] Figure 5 is a schematic diagram of the hardware structure of an image processing device based on a medical endoscope according to another embodiment.

[0080] As shown in FIG5, the medical endoscope-based image processing device 500 includes a processor 510 and a memory 520. The memory 520 is used to store an executable computer program, and the processor 510 is used to execute the method as described in any of the above method embodiments under the control of the computer program.

[0081] The image processing device 500 based on the medical endoscope can be the control device 10 in Figure 1.

[0082] Each module of the image processing device 400 based on the medical endoscope described above can be implemented by the processor 510 in this embodiment executing the computer program stored in the memory 520, or it can be implemented by other structures, which are not limited here.

[0083] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0084] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0085] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0086] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of the invention.

[0087] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0088] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0089] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0090] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It will be known to those skilled in the art that implementation in hardware, implementation in software, and implementation using a combination of software and hardware are equivalent.

[0091] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the invention is defined by the appended claims.

[0092]

Claims

1. An image processing system based on a medical endoscope, characterized in that, The system includes: A single-channel endoscope, wherein a single imaging channel of the single-channel endoscope acquires two-dimensional images; A control device electrically connected to the single-channel endoscope is configured to: extract an image portion of a set region from the two-dimensional image in response to the two-dimensional image sent by the single-channel endoscope; and perform three-dimensional reconstruction on the image portion to obtain a fused image.

2. The image processing system based on a medical endoscope according to claim 1, characterized in that, The diameter of the first channel corresponding to the single imaging channel of the single-channel endoscope is less than or equal to 10 mm.

3. The image processing system based on a medical endoscope according to claim 1, characterized in that, The diameter of the first channel is the diameter of the front end of the single imaging channel.

4. The image processing system based on a medical endoscope according to claim 1, characterized in that, The system also includes a display device electrically connected to the control device, the display device being used to display the fused image in response to the fused image sent by the control device.

5. An image processing method based on a medical endoscope, characterized in that, The method is applied to the image processing system based on a medical endoscope, and the execution subject of the method is the control device in the image processing system based on a medical endoscope. The method includes: In response to the two-dimensional image transmitted by a single-channel endoscope, an image portion of a defined region in the two-dimensional image is extracted; A 3D reconstruction is performed on the image portion to obtain a fused image.

6. The image processing method based on a medical endoscope according to claim 1, characterized in that, The step of extracting a portion of a predetermined region from the two-dimensional image in response to a single-channel endoscope-transmitted image includes: In response to the two-dimensional image transmitted by the single-channel endoscope, the current image format of the two-dimensional image is converted into a to-be-processed image of a set image format; The image portion of a designated region in the two-dimensional image is extracted using a preset ROI module.

7. The image processing method based on a medical endoscope according to claim 1, characterized in that, The process of performing three-dimensional reconstruction on a portion of the image to obtain a fused image includes: The image portion is reconstructed in three dimensions using a pre-set multi-task deep learning model to obtain a reconstructed image. The reconstructed image is registered and fused with the image portion to synthesize a fused image with depth information.

8. The image processing method based on a medical endoscope according to claim 7, characterized in that, The multi-task deep learning model includes sequentially connected generation nodes, fusion nodes, and rendering nodes; The generation node is used to generate a first projected image from a predetermined viewpoint based on the image portion; the fusion node is used to fuse the first projected image into the three-dimensional point cloud data to generate a point cloud image; and the rendering node is used to render a second projected image corresponding to a specific viewpoint from the point cloud image to generate a depth image.

9. An image processing device based on a medical endoscope, characterized in that, The device includes: A response module is used to extract an image portion of a set region from the two-dimensional image in response to the two-dimensional image sent by a single-channel endoscope. The module is used to perform 3D reconstruction of the image portion to obtain a fused image.

10. An image processing device based on a medical endoscope, characterized in that, It includes a memory and a processor, the memory being used to store a computer program, and the processor being used, under the control of the computer program, to execute the image processing method based on a medical endoscope according to any one of claims 5 to 8.

Images