Medical image processing device and operation method thereof

Abstract

This operation method of a medical image processing device comprises the steps of: acquiring, from a first external device, two-dimensional first images including a patient's vascular structure and pelvic structure by tomographically imaging the inside of the patient's pelvic cavity; generating three-dimensional modeling data by performing three-dimensional modeling on the first images; when a medical staff performs labeling to record a vessel name for a target vessel of the vascular structure in the three-dimensional modeling data, performing labeling in real time so that the vessel name is recorded for the target vessel in the first images; and registering the three-dimensional modeling data with a second image which includes the patient's pelvic cavity structure and pelvic curvature and is acquired from a second external device by three-dimensional scanning of the patient, generating an augmented reality-based composite image in which a medical image is output on the patient's body in the second image, and transmitting the first images and the three-dimensional modeling data to an output device that outputs the composite image.

Description

Medical image processing device and method of operation thereof

[0001] The present invention relates to a medical image processing device and a method of operating the same. More specifically, the present invention relates to a medical image processing device used in an augmented reality-based medical image processing system and a method of operating the same.

[0002] There is a high level of interest in the medical industry in improving technologies for visualizing the inside of a patient during medical procedures. For example, various imaging technologies, such as computed tomography (CT) scans and magnetic resonance imaging (MRI), can be utilized to enable medical professionals to visualize organs, bones, and other tissues inside a patient's body.

[0003] <Prior Art Literature>

[0004] <Patent Literature>

[0005] (Patent Document 0001) Korean Published Patent No. 10-2022-0112246

[0006] (Patent Document 0002) Korean Published Patent No. 10-2024-0068176

[0007] One objective of the present invention is to provide a medical image processing device used in an augmented reality-based medical image processing system.

[0008] Another objective of the present invention is to provide a method of operating the medical image processing device.

[0009] However, the objectives of the present invention are not limited to such objectives, and may be expanded in various ways without departing from the spirit and scope of the present invention.

[0010] To achieve the aforementioned objective of the present invention, a method of operation of a medical image processing device according to one embodiment of the present invention may include the steps of: obtaining two-dimensional first images including the patient's vascular structure, pelvic structure, and skin structure by tomographically scanning the inside of the patient's pelvic cavity from a first external device; performing three-dimensional modeling on the first images to generate three-dimensional modeling data; labeling the target blood vessel in the first image in real time so that the blood vessel name is recorded on the target blood vessel in the three-dimensional modeling data when a medical professional labels the target blood vessel of the vascular structure in the three-dimensional modeling data; and aligning the three-dimensional modeling data with a second image including the patient's pelvic cavity structure and pelvic curvature obtained by three-dimensionally scanning the patient from a second external device, generating an augmented reality-based synthetic image that outputs a medical image on the patient's body in the second image, and transmitting the first images and the three-dimensional modeling data to an output device that outputs the synthetic image.

[0011] To achieve the aforementioned objective of the present invention, one embodiment of the present invention may provide a computer-readable recording medium on which a computer program for performing a method of operating a medical image processing device using augmented reality is recorded.

[0012] To achieve the aforementioned objective of the present invention, a medical image processing device according to one embodiment of the present invention may include: a data receiving unit that receives two-dimensional first images including a patient's vascular structure, pelvic structure, and skin structure obtained by tomographically scanning the inside of the patient's pelvis from a first external device; an image processing unit that performs three-dimensional modeling on the first images to generate three-dimensional modeling data; an output unit that outputs the first images and the three-dimensional modeling data; a labeling unit that labels the target blood vessel in the first image in real time so that the blood vessel name is recorded on the target blood vessel in the vascular structure when a medical professional labels the target blood vessel in the three-dimensional modeling data; and a data transmission unit that aligns the three-dimensional modeling data with a second image including the patient's pelvic structure and pelvic curvature obtained by three-dimensionally scanning the patient from a second external device, generates an augmented reality-based synthetic image that outputs a medical image on the patient's body in the second image, and transmits the first images and the three-dimensional modeling data to an output device that outputs the synthetic image.

[0013] In embodiments of the present invention, by performing 3D modeling on a 2D image including a patient's vascular structure, pelvic structure, and skin structure, and labeling the 3D modeling data generated by performing 3D modeling so that the names of the target blood vessels corresponding to each other are recorded in real time, medical staff can identify the exact location of the target blood vessels in the vascular structure. Accordingly, the surgical time of medical staff can be reduced, and the fatigue of medical staff and patients due to the procedure or surgery can be reduced.

[0014] However, the effects of the present invention are not limited to the above effects, and may be extended in various ways without departing from the spirit and scope of the present invention.

[0015] FIG. 1 is a block diagram showing a medical image processing system according to one embodiment of the present invention.

[0016] Figure 2 is a drawing showing an example of a screen displayed in the medical image processing device of Figure 1.

[0017] Figure 3 is a drawing showing another example of a screen displayed in the medical image processing device of Figure 1.

[0018] Figure 4 is a drawing showing a screen displayed on the output device of Figure 1.

[0019] FIG. 5 is a flowchart for explaining the operation method of a medical image processing device according to one embodiment of the present invention.

[0020] FIG. 6 is a flowchart illustrating a method of operation of a medical image processing device according to an embodiment of the present invention.

[0021] With respect to the embodiments of the present invention disclosed in the text, specific structural or functional descriptions are provided merely for the purpose of explaining the embodiments of the present invention, and the embodiments of the present invention may be implemented in various forms and should not be interpreted as being limited to the embodiments described in the text.

[0022] The present invention is capable of various modifications and may take various forms, and specific embodiments are illustrated in the drawings and described in detail in the text. However, this is not intended to limit the invention to the specific disclosed forms, and it should be understood that the invention includes all modifications, equivalents, and substitutions that fall within the spirit and scope of the invention.

[0023] Terms such as first, second, etc. may be used to describe various components, but said components should not be limited by said terms. said terms may be used for the purpose of distinguishing one component from another component. For example, without departing from the scope of the present invention, the first component may be named the second component, and similarly, the second component may be named the first component.

[0024] When it is stated that one component is "connected" or "connected" to another component, it should be understood that while it may be directly connected or connected to that other component, there may also be other components in between. Conversely, when it is stated that one component is "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between. Other expressions describing the relationships between components, such as "between" and "exactly between," or "adjacent to" and "directly adjacent to," should be interpreted in the same way.

[0025] The terms used in this application are used merely to describe specific embodiments and are not intended to limit the invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, terms such as "comprising" or "having" are intended to specify the existence of the described features, numbers, steps, actions, components, parts, or combinations thereof, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0026] Additionally, terms such as "~part" mentioned in the text refer to a unit that processes at least one function or operation, and this may be implemented in hardware, software, or a combination of hardware and software.

[0027] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art to which the present invention pertains. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in this application.

[0028] Hereinafter, embodiments of the present invention will be described in more detail with reference to the attached drawings. Identical or similar reference numerals are used for identical components in the drawings.

[0029] In this specification, "medical image processing device" and "output device" include all various devices capable of performing computational processing and providing results to a user. For example, the medical image processing device and output device according to the present invention may each include a computer, a server device, and a portable terminal, or may take the form of any one of them.

[0030] For example, the above computer may include a notebook, desktop, laptop, tablet PC, slate PC, etc. equipped with a web browser.

[0031] For example, the above server device is a server that processes information by communicating with an external device, and may include an application server, a computing server, a database server, a file server, a game server, a mail server, a proxy server, a web server, etc.

[0032] For example, the above-mentioned portable terminal is a wireless communication device that ensures portability and mobility, and may include all kinds of handheld-based wireless communication devices such as PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (WCode Division Multiple Access), WiBro (Wireless Broadband Internet) terminals, smartphones, etc., and wearable devices such as watches, rings, bracelets, anklets, necklaces, glasses, contact lenses, or head-mounted devices (HMDs).

[0033] Augmented Reality (AR) is a technology that overlays three-dimensional virtual images onto real-world images or backgrounds to display them as a single image. This augmented reality technology is being applied in various fields, such as games, health, and map services, through various smart devices, and recently, augmented reality technology is being utilized in the medical field.

[0034] FIG. 1 is a block diagram illustrating a medical image processing system according to embodiments of the present invention. FIG. 2 is a diagram illustrating one example of a screen displayed in the medical image processing device of FIG. 1. FIG. 3 is a diagram illustrating another example of a screen displayed in the medical image processing device of FIG. 1. FIG. 4 is a diagram illustrating a screen displayed in the output device of FIG. 1.

[0035] Referring to FIGS. 1, 2, and 3, a medical image processing system (1) according to embodiments of the present invention may include a first external device (100), a second external device (200), a medical image processing device (300), and an output device (400).

[0036] The medical image processing system (1) can acquire medical information before a patient's surgery or procedure, extract feature data necessary for the surgery or procedure from the acquired medical information, and process it into data necessary for the surgery or procedure.

[0037] For example, the first external device (100) can process the internal cross-section of the human body into an image by reconstructing the results of X-ray or ultrasound of the patient's body using a computer through Computed Tomography (CT). Optionally, the first external device (100) may also capture an image of the human body through Magnetic Resonance Imaging (MRI) or Positron Emission Tomography (PET). In this specification, the first external device (100) capturing an image of the human body through Computed Tomography is described as an example.

[0038] For example, the second external device (200) can acquire three-dimensional image information of an object (e.g., a patient) using a depth camera and / or a stereo camera. Specifically, the second external device (200) can acquire three-dimensional image information by three-dimensionally scanning the object.

[0039] In one embodiment, the medical image processing system (1) can perform marker-less medical image processing through data analysis of the medical image processing device (300) and the output device (400) without the physical installation of artificial markers for extracting medical images. Accordingly, the medical image processing system (1) can suppress the occurrence of human error in all processes of maintaining the relative position between the patient's body and the marker, or where markers are not required to be installed.

[0040] In one embodiment, the medical image processing system (1) can output an augmented reality-based medical image and project it onto the lower body of the patient. Accordingly, the medical image processing system (1) can visually display the location and depth of vascular structures (e.g., female reproductive organs), bone tissue structures (e.g., pelvic structures), and skin structures (e.g., buttock structures) located within the patient's pelvic cavity (e.g., the part where female reproductive organs, where gynecological cancer may occur, are located), and medical personnel can perform an accurate, safe, and rapid surgery or procedure using the medical image processing system (1).

[0041] The medical image processing device (300) may include an input unit (310), a data receiving unit (320), an image processing unit (330), a labeling unit (340), a data transmission unit (350), an output unit (360), and a database (370).

[0042] The input unit (310) may be an input means for receiving information from a user (e.g., medical staff). For example, the input unit (310) may include an input means such as a keyboard, mouse, touch panel, etc. However, embodiments of the present invention are not necessarily limited thereto.

[0043] The data receiving unit (320) can receive medical information images of the inside of the patient's pelvis from an external device. Specifically, the data receiving unit (320) can receive medical information images of the inside of the patient's pelvis from the external device via wired or wireless communication. In one embodiment, the data receiving unit (320) can obtain a plurality of first images including the patient's vascular structure, bone tissue structure, and skin structure by tomographically scanning the inside of the patient's lower body from the first external device (100). For example, the first images may be the two-dimensional images (SS1, SS2, SS3) of FIGS. 2 and FIGS. 3.

[0044] The image processing unit (330) can perform image processing configured to process data constituting an image according to a predefined operation. Specifically, the image processing unit (330) can generate 3D modeling data by performing 3D modeling on 2D images (SS1, SS2, SS3) obtained through the first external device (100).

[0045] In one embodiment, the image processing unit (330) can perform patient region segmentation in slice units of the medical image. The entire segmented image can be used as volume data for a mesh. In one embodiment, some of the medical image may be data that has been 3D modeled in the form of a mesh. For example, the 3D modeled data may be 3D mesh information including 3D vertex information, connection information thereof, and surface information generated therefrom.

[0046] In one embodiment, the image processing unit (330) can perform segmentation of a medical image (e.g., a vascular structure) using U-Net series deep learning. The algorithm of the U-Net series deep learning can expand from a narrow range to a wide range in a contracting path, extract context information from image pixels, switch from the contracting path to an expanding path in a bottleneck, and then combine the context information with the pixel location information in the expanding path to distinguish which object each pixel belongs to.

[0047] In one embodiment, the image processing unit (330) can generate a blood vessel mesh by utilizing a marching cube technique. The mesh may be 3D modeling data. When generating the mesh, the image processing unit (330) can apply a Gaussian filter to the 3D surface to smooth the volume data. As a result, the same effect as applying smoothing to the blood vessel mesh generated through the marching cube can be achieved.

[0048] In one embodiment, the image processing unit (330) can process the first-1 region (A11) and the first-2 region (A12) of the blood vessel structure in the two-dimensional images (SS1, SS2, SS3) as in FIGS. 2 and 3 so that each has a specific color. For example, the first-1 region (A11) and the first-2 region (A12) may be displayed in different colors. That is, the first-1 region (A11) may have a first color, and the first-2 region (A12) may have a second color different from the first color. In this case, the second-1 region (A21) of the blood vessel structure corresponding to the first-1 region (A11) in the three-dimensional modeling data may have the first color, and the second-2 region (A22) of the blood vessel structure corresponding to the first-2 region (A12) may have the second color.

[0049] Likewise, the image processing unit (330) can process the skin structure and the bone tissue structure in the two-dimensional images (SS1, SS2, SS3) to have different colors that are different from the first and second colors, respectively, and in this case, the skin structure and the bone tissue structure corresponding to this in the three-dimensional modeling data may have different colors.

[0050] The output unit (360) may be an output means for displaying (or outputting) information (e.g., medical images) processed by the medical image processing device (300). For example, the output unit (360) may include output means such as a display device, a speaker, etc. In one embodiment, the output unit (360) may display execution screen information of an application program running on the medical image processing device (300) (e.g., first images (e.g., two-dimensional images) including the vascular structure, pelvic structure, and skin structure of the patient obtained by tomographically scanning the inside of the patient's pelvis, and three-dimensional modeling data generated by performing three-dimensional modeling on the first images), or UI (User Interface) and GUI (Graphic User Interface) information according to such execution screen information.

[0051] The output unit (360) may selectively display (or output) the patient's vascular structure, bone tissue structure, and skin structure in the three-dimensional modeling data. For example, the output unit (360) may not display (or output) at least one of the patient's vascular structure, bone tissue structure, and skin structure in the three-dimensional modeling data. In one embodiment, the output unit (360) may display (or output) only the patient's vascular structure as shown in FIGS. 2 and FIGS. 3. However, embodiments of the present invention are not necessarily limited thereto.

[0052] In one embodiment, the medical staff can rotate and move the three-dimensional modeling data displayed through the output unit (360). For example, the medical staff can use a mouse to rotate and move the three-dimensional modeling data displayed through the output unit (360). By rotating and moving the three-dimensional modeling data, the medical staff can view the three-dimensional modeling data from various directions.

[0053] In one embodiment, the rendering quality of the 3D modeling data may be limited while the 3D modeling data is rotated and moved. If the rendering quality of the 3D modeling data is limited while the 3D modeling data is rotated and moved, there may be no interruption of the 3D modeling data while the 3D modeling data is rotated and moved. When the rotation and movement of the 3D modeling data are finished, the 3D modeling data rendered with the original quality may be displayed.

[0054] In one embodiment, the labeling unit (340) can label the target blood vessel of the blood vessel structure in real time so that the blood vessel name is recorded on the target blood vessel in the 3D modeling data when the medical staff labels the target blood vessel of the blood vessel structure in the 3D modeling data.

[0055] Specifically, as illustrated in FIG. 2, the output unit (360) can output a recording window (e.g., first to fifth recording windows (RW1, RW2, RW3, RW4, RW5)) when the medical staff clicks the target blood vessel in the three-dimensional modeling data, and a recording window (RW2') can be output to the corresponding target blood vessel in the first two-dimensional images (SS1, SS2, SS3) in real time.

[0056] Next, the labeling unit (340) can label the blood vessel name of the target blood vessel corresponding to the blood vessel name in the two-dimensional first images (SS1, SS2, SS3) in real time when the medical staff records the blood vessel name of the target blood vessel in the record window.

[0057] After the blood vessel names are labeled to be recorded at the corresponding locations of the target blood vessels in the above 3D modeling data (e.g., TS of FIG. 2 and FIG. 3) and the 2D first images (SS1, SS2, SS3), the output unit (360) can display (or output) a blood vessel list (BVL) in which the names of the target blood vessels of the patient are recorded (see FIG. 2).

[0058] However, the embodiments of the present invention are not necessarily limited thereto, and the labeling unit (340) can label the target blood vessel of the blood vessel structure corresponding to the blood vessel name in the 3D modeling data in real time when the medical staff labels the target blood vessel of the blood vessel structure in the 2D first images (SS1, SS2, SS3) so that the blood vessel name is recorded thereon.

[0059] For example, the target blood vessel may include at least one of the uterine artery, superior vesical artery, aorta, common iliac artery, internal iliac artery, external iliac artery, and renal vein located inside the patient's pelvis.

[0060] As a result, the blood vessel name is labeled to be recorded at the corresponding location of the target blood vessel in the three-dimensional modeling data (e.g., TS in FIGS. 2 and 3) and the two-dimensional first images (SS1, SS2, SS3), thereby allowing the medical staff to identify the exact location of the target blood vessel in the blood vessel structure. Accordingly, the surgical time of the medical staff can be reduced, and the fatigue of the medical staff and the patient due to the procedure or surgery can be reduced.

[0061] As described above, in the first images (SS1, SS2, SS3) of the two-dimensional structure, the first-1 region (A11) and the first-2 region (A12) of the blood vessel structure are each processed to have the first color (e.g., red) and the second color (e.g., blue), and in this case, the second-1 region (A21) and the second-2 region (A22) of the blood vessel structure corresponding to the first-1 region (A11) and the first-2 region (A12) in the three-dimensional modeling data (e.g., ST) may each have the first color and the second color.

[0062] In the first two-dimensional images (SS1, SS2, SS3), a specific region of the blood vessel structure may not be processed with the first color or the second color, and in this case, the part corresponding to the specific region in the three-dimensional modeling data (e.g., ST) may not be displayed. Alternatively, in the first two-dimensional images (SS1, SS2, SS3), a specific region other than the blood vessel structure (e.g., the skin structure or the bone tissue structure) may be processed with the first color or the second color, and in this case, the part corresponding to the specific region in the three-dimensional modeling data (e.g., ST) may be displayed with the first color or the second color.

[0063] In one embodiment, when the medical staff labels at least some of the colors of the first-1 region (A11) and the first-2 region (A12) of the blood vessel structure in the first two-dimensional images (SS1, SS2, SS3) to be modified (i.e., deleted and / or added), the labeling unit (340) can label at least some of the corresponding second-1 region (A21) and the second-2 region (A22) of the blood vessel structure in the three-dimensional modeling data (e.g., ST) to be modified (i.e. deleted and / or added) and displayed in real time. In another embodiment, when a specific area other than the blood vessel structure in the first two-dimensional images (SS1, SS2, SS3) is processed with the first color or the second color, if the medical staff labels the first color or the second color of the specific area in the two-dimensional images (SS1, SS2, SS3) to be deleted, the first color or the second color of the part corresponding to the specific area in the three-dimensional modeling data (e.g., ST) can be labeled to be deleted in real time.

[0064] In one embodiment, a 3D coordinate value for a specific location of the blood vessel structure can be obtained from 3D modeling data (ST), and a 2D coordinate value of the blood vessel structure corresponding to the 3D coordinate value can be obtained from 2D first images (SS1, SS2, SS3). Through this, the medical staff can compare the 3D coordinate value and the 2D coordinate value to verify whether labeling has been performed at corresponding locations in the 3D modeling data (ST) and the 2D first images (SS1, SS2, SS3).

[0065] The data transmission unit (350) can transmit the first images obtained through the first external device (100) and the three-dimensional modeling data generated through the image processing unit (330) to the output device (400). Specifically, the data transmission unit (350) can transmit the first images obtained through the first external device (100) and the three-dimensional modeling data generated through the image processing unit (330) to the output device (400) via wired or wireless communication.

[0066] The data receiving unit (310), image processing unit (330), labeling unit (340), and data transmission unit (350) may each be implemented as an algorithm for controlling operations or as a program that reproduces the algorithm. In this case, each program may be implemented on a computer as a memory that stores each program and at least one processor that performs the aforementioned operations using the data stored in the memory. For example, the memory and the processor may be implemented as separate chips. Optionally, the memory and the processor may be implemented as a single chip.

[0067] The above processor may be configured to control the overall organic operation of various functional units of the medical image processing device (300), such as processing one or more commands required for the control of the medical image processing device (300), performing calculations according to the commands, and making judgments according to program logic. The above processor may provide or process appropriate information or functions to the user by processing signals, data, information, etc. that are input or output through the input unit (350), data receiving unit (310), image processing unit (330), labeling unit (340), data transmission unit (350), or output unit (360), or by running an application program stored in the database (370). The processed data may be stored in memory or used to build the database (370), or transmitted externally through the input unit (350), data receiving unit (310), image processing unit (330), labeling unit (340), data transmission unit (350), or output unit (360).

[0068] Such a processor may be implemented as a general-purpose processor, a dedicated processor, or an application processor. For example, the processor may be implemented as a computational processor (e.g., CPU (Central Processing Unit), GPU (Graphic Processing Unit), AP (Application Processor), etc.) including dedicated logic circuits (e.g., FPGA (Field Programmable Gate Array), ASICs (Application Specific Integrated Circuits), etc.). However, the embodiments of the present invention are not necessarily limited thereto. For example, the processor may be implemented as a DSP (Digital Signal Processor) capable of converting analog signals into digital signals for high-speed processing, an MCU (Micro Controller Unit), or an NPU (Neural Processing Unit) specialized in processing artificial neural networks.

[0069] In one embodiment, the medical image processing device (300) may include a desktop, a laptop, etc. However, the embodiments of the present invention are not necessarily limited thereto.

[0070] A database (370) may refer to a collection of data that is integrated and managed for the purpose of being shared and used by storing various information. The database (370) may store data temporarily or semi-permanently. For example, the database (370) may store an operating system (OS) for operating at least one device, data for hosting a website, or data regarding an application. Additionally, the database (370) may store modules in the form of computer code. The database (370) may be managed through middleware that is separate from the application.

[0071] The database (370) may include a relational database, a key-value database, an object database, a document database, a memory database, etc.

[0072] The database (370) can store data supporting various functions of the medical image processing device (300) and programs for the operation of the control unit, and can store input / output data (e.g., music files, still images, videos, etc.), and can store a number of applications running on the medical image processing device (300), data for the operation of the medical image processing device (300), and commands. At least some of these applications can be downloaded from an external server via wireless communication.

[0073] The database (370) may be implemented as a memory that is separated from the medical image processing device (300) or connected via wired or wireless connection. In this case, the memory may include at least one type of storage medium among flash memory type, hard disk type, SSD type (Solid State Disk type), SSD type (Solid Disk Drive type), multimedia card micro type, card type memory (e.g., SD or XD memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, and optical disk.

[0074] The output device (400) can receive a second image (i.e., a real-time spatial image) including the pelvic structure and pelvic curvature of the patient obtained by three-dimensional scanning of the patient from the second external device (200), and the two-dimensional first images and the three-dimensional modeling data from the medical image processing device (300).

[0075] After receiving the first images, the second image, and the three-dimensional modeling data, the output device (400) can align the three-dimensional modeling data to the second image based on information about the reference skin. In one embodiment, the output device (400) can perform alignment using a first alignment algorithm and a second alignment algorithm. For example, the output device (400) can calculate the approximate location of a first feature point corresponding to the reference skin from the three-dimensional modeling data and a second feature point corresponding to the reference skin from the second image based on the first alignment algorithm. Here, the approximate location may mean a position and a direction.

[0076] For example, the output device (400) can calculate the approximate location of the first feature point and the second feature point as the first matching algorithm. In one embodiment, the output device (400) can, as the first matching algorithm, extract a first bounding box from the 3D modeling data and extract a second bounding box from the second image, and then match the center point of the first bounding box (i.e., the first feature point) with the center point of the second bounding box (i.e., the second feature point). In the present invention, the accuracy of the matching can be increased by additionally using the second matching algorithm together with the first matching algorithm.

[0077] The output device (400) can calculate the approximate positions of the first feature point and the second feature point, and then calculate the precise positions of the first feature point and the second feature point based on the second matching algorithm. For example, the output device (400) can calculate the precise positions of the first feature point and the second feature point using the ICP (Iterative Closest Point) algorithm as the second matching algorithm, starting from the aforementioned approximate positions. Here, the precise position may refer to movement and rotation.

[0078] That is, the output device (400) can detect the position and direction as an approximate position through the first matching algorithm and detect movement and rotation as a precise position through the second matching algorithm (e.g., the ICP algorithm) so that the distance between the points becomes closer.

[0079] In another embodiment, the output device (400) may perform alignment using only the second alignment algorithm. For example, the output device (400) may set an initial value by performing pre-alignment at a plurality of pre-set points. Here, the pre-alignment may be performed through the ICP algorithm. For example, the output device (400) may perform pre-alignment at the center of the upper surface of a scene bounding box corresponding to a real-time image and at the center of a second bounding box. Subsequently, the output device (400) may perform alignment by applying the ICP algorithm from the initial value to calculate the precise location of the first feature point and the second feature point.

[0080] The output device (400) can generate an augmented reality-based synthetic image that outputs a medical image on the patient's body in the second image after aligning the 3D modeling data (e.g., the blood vessel structure (402) of FIG. 4) with the second image (i.e., a real-time spatial image).

[0081] As illustrated in FIG. 4, the output device (400) can display (or output) a two-dimensional CT plane (401) as medical information on the patient's body in the second image. For example, if the medical staff selects a specific first image among the first images, the output device (400) can place the specific first image on the patient's body in the second image at a location corresponding to the location of the specific first image (i.e., inside the patient's torso).

[0082] Optionally, the output device (400) may stack the first images to generate three-dimensional data and generate an anatomical plane image in a desired direction from the three-dimensional data to represent it.

[0083] As illustrated in FIG. 4, the output device (400) can display (or output) the augmented reality-based composite image. Additionally, the output device (400) can continuously display (or output) the augmented reality-based composite image by tracking spatial information based on the depth and direction of the second external device (200).

[0084] For example, the output device (400) may display (or output) the augmented reality-based synthetic image including the blood vessel structure (402). Optionally, the output device (400) may additionally display (or output) three-dimensional modeling data including at least one of a pelvic structure and a skin structure in addition to the blood vessel structure (402) in the synthetic image.

[0085] For example, the output device (400) may display (or output) a recording window in which the name of the blood vessel of the blood vessel structure is recorded in the three-dimensional modeling data of the synthetic image. However, embodiments of the present invention are not necessarily limited thereto.

[0086] For example, the output device (400) may include a tablet, smartphone, laptop, wearable device, etc. However, embodiments of the present invention are not necessarily limited thereto.

[0087] In one embodiment, the medical image processing device (300) and the output device (400) may each be composed of different hardware. However, the embodiments of the present invention are not necessarily limited thereto, and at least some parts of the medical image processing device (300) and the output device (400) may be composed of the same hardware.

[0088] In one embodiment, the output device (400) and the second external device (200) may be composed of the same hardware. However, embodiments of the present invention are not necessarily limited thereto, and the output device (400) and the second external device (200) may each be composed of different hardware.

[0089] FIG. 5 is a flowchart for explaining the operation method of a medical image processing device according to one embodiment of the present invention.

[0090] Referring to FIG. 5, in a method of operation of a medical image processing device according to an embodiment of the present invention, two-dimensional first images including the vascular structure, pelvic structure, and skin structure of the patient can be obtained (S110). Specifically, the two-dimensional first images can be obtained by tomographically scanning the inside of the patient's pelvis using a tomographic scanning device.

[0091] After obtaining the first images, three-dimensional modeling can be performed on the images to obtain three-dimensional modeling data (S120).

[0092] After acquiring the above 3D modeling data, if the medical staff labels the target blood vessel of the blood vessel structure in the above 3D modeling data so that the blood vessel name is recorded thereon, the blood vessel name can be recorded thereon in real time on the target blood vessel corresponding to the first image (S130).

[0093] After labeling the blood vessel name of the target blood vessel in the above 3D modeling data and the above 1 image so that it is recorded, the above 1 images and the above 3D modeling data can be transmitted to an output device (e.g., output device (400) of FIG. 1) (S140).

[0094] FIG. 6 is a flowchart illustrating a method of operation of a medical image processing device according to an embodiment of the present invention.

[0095] In the following, descriptions that overlap with the operation method of the medical image processing device described with reference to FIG. 5 are omitted or simplified.

[0096] Referring to FIG. 6, in a method of operation of a medical image processing device according to one embodiment of the present invention, two-dimensional first images including the vascular structure, pelvic structure, and skin structure of the patient can be obtained (S110).

[0097] After obtaining the first images, three-dimensional modeling of the images can be performed to obtain three-dimensional modeling data (S120). In the first images and the three-dimensional modeling data, the blood vessel structure may have a first color or a second color different from the first color.

[0098] After acquiring the above 3D modeling data, if the medical staff labels the target blood vessel of the blood vessel structure in the above 3D modeling data so that the blood vessel name is recorded thereon, the blood vessel name can be recorded thereon in real time on the target blood vessel corresponding to the first image (S130).

[0099] After acquiring the above 3D modeling data, if the medical staff labels the first specific area in the first image to be modified, the second specific area corresponding to the first specific area in the 3D modeling data can be labeled to be modified in real time (S135).

[0100] For example, if the medical staff labels the color of at least a portion of the blood vessel structure in the first image to be modified (i.e., deleted and / or added), the corresponding at least portion of the blood vessel structure in the three-dimensional modeling data may be labeled to be modified (i.e. deleted and / or added) and displayed in real time. Optionally, if the medical staff labels the first color or the second color of a specific area other than the blood vessel structure in the first image to be deleted, the first color or the second color of a portion corresponding to the specific area in the three-dimensional modeling data may be labeled to be deleted in real time.

[0101] After labeling the blood vessel name of the target blood vessel in the 3D modeling data and the first image to be recorded, and labeling specific regions of the blood vessel structures corresponding to each other in the 3D modeling data and the first image to be modified, the first images and the 3D modeling data can be transmitted to an output device (e.g., the output device (400) of FIG. 1) (S140).

[0102] Meanwhile, the disclosed embodiments may be implemented in the form of a recording medium that stores instructions executable by a computer. The instructions may be stored in the form of program code and, when executed by a processor, may generate a program module to perform the operation of the disclosed embodiments. The recording medium may be implemented as a computer-readable recording medium.

[0103] Computer-readable recording media include all types of recording media that store instructions that can be decoded by a computer. Examples include ROM (Read Only Memory), RAM (Random Access Memory), magnetic tape, magnetic disk, flash memory, optical data storage devices, etc.

[0104] Although embodiments of the present invention have been described above with reference to the drawings, the above description is illustrative and may be modified and changed by those skilled in the art without departing from the technical spirit of the present invention.

[0105] <Explanation of Symbols>

[0106] 1: Medical image processing system

[0107] 100: First external device

[0108] 200: Second external device

[0109] 300: Medical image processing device

[0110] 400: Output device

Claims

1. In a method of operating a medical image processing device, A step of obtaining two-dimensional first images including the patient's vascular structure, pelvic structure, and skin structure by tomographically imaging the inside of the patient's pelvis from a first external device; A step of generating 3D modeling data by performing 3D modeling on the first images; A step of labeling the target blood vessel of the blood vessel structure in the above 3D modeling data to record the blood vessel name, and labeling the target blood vessel in the first image in real time so that the blood vessel name is recorded thereon; and A method of operating a medical image processing device comprising the steps of: aligning the 3D modeling data with a second image including the pelvic internal structure and pelvic curvature of the patient obtained by 3D scanning the patient from a second external device; generating an augmented reality-based synthetic image that outputs a medical image on the patient's body in the second image; and transmitting the first images and the 3D modeling data to an output device that outputs the synthetic image.

2. In claim 1, the step of labeling the target blood vessel in the 3D modeling data so that the blood vessel name is recorded on the target blood vessel in real time in the first image is: The above medical staff includes the step of clicking the target blood vessel in the above 3D modeling data to output a record window, and A method of operation of a medical image processing device characterized by the fact that when the medical staff records the name of the target blood vessel in the record window, the name of the target blood vessel is recorded in the first image in real time.

3. A method of operation of a medical image processing device according to claim 1, characterized in that the target blood vessel comprises at least one of the patient's uterine artery, superior vesical artery, aorta, common iliac artery, internal iliac artery, external iliac artery, and renal vein.

4. In claim 1, after the step of generating the three-dimensional modeling data, A method of operation of a medical image processing device characterized by further including the step of rotating and moving the above-mentioned three-dimensional modeling data.

5. A method of operation of a medical image processing device according to claim 4, characterized in that, in the step of rotating and moving the three-dimensional modeling data, the rendering quality of the three-dimensional modeling data is limited while the three-dimensional modeling data is rotated and moved.

6. In claim 1, after the step of acquiring the first images, In the first image above, the first-1 region and the first-2 region of the blood vessel structure are each processed to have a first color and a second color different from the first color, and A method of operation of a medical image processing device characterized in that the 2-1 region and the 2-2 region of the blood vessel structure, corresponding respectively to the 1-1 region and the 1-2 region in the above 3D modeling data, each have the 1 color and the 2 color.

7. In claim 6, after the step of generating the three-dimensional modeling data, A method of operation of a medical image processing device characterized by further including the step of labeling at least a portion of the 1-1 region and the 1-2 region of the blood vessel structure in the 1st image to be modified, and labeling at least a portion of the 2-1 region and the 2-2 region of the blood vessel structure corresponding thereto in the 3D modeling data to be modified and displayed in real time.

8. In claim 6, after the step of generating the three-dimensional modeling data, A method of operation of a medical image processing device characterized by further including the step of, when a specific area other than the blood vessel structure in the first image is processed with the first color or the second color, labeling the first color or the second color of the specific area in the first image to be deleted, and labeling the first color or the second color of the corresponding part of the specific area in the 3D modeling data to be deleted in real time.

9. A computer-readable recording medium having a computer program recorded thereon for performing a method of operating a medical image processing device using augmented reality according to any one of claims 1 to 8.

10. A data receiving unit that receives two-dimensional first images including the patient's vascular structure, pelvic structure, and skin structure obtained by tomographically scanning the inside of the patient's pelvis from a first external device; An image processing unit that performs three-dimensional modeling on the first images to generate three-dimensional modeling data; An output unit that outputs the first images and the three-dimensional modeling data; A labeling unit that labels the target blood vessel in the first image in real time so that the blood vessel name is recorded when a medical professional labels the target blood vessel of the blood vessel structure in the above 3D modeling data; and A medical image processing device comprising: a data transmission unit that aligns the 3D modeling data with a second image including the pelvic internal structure and pelvic curvature of the patient obtained by 3D scanning the patient from a second external device; generates an augmented reality-based synthetic image that outputs a medical image on the patient's body in the second image; and transmits the first images and the 3D modeling data to an output device that outputs the synthetic image.

11. In Paragraph 10, The above output unit outputs a record window when the medical staff clicks the target blood vessel in the 3D modeling data, and A medical image processing device characterized by the above labeling unit labeling the blood vessel name of the target blood vessel in the first image in real time when the medical staff records the blood vessel name of the target blood vessel in the recording window.

12. In Paragraph 10, It further includes a processor that controls the operation of the data receiving unit, the image processing unit, the labeling unit, and the data transmission unit, and A medical image processing device characterized by the processor controlling the rendering quality of the three-dimensional modeling data to be limited while the three-dimensional modeling data is rotated and moved.

13. In claim 10, the image processing unit is, In the first image above, the 1-1 region and the 1-2 region of the blood vessel structure are processed to each have a first color and a second color different from the first color, and A medical image processing device characterized in that the 2-1 region and 2-2 region of the blood vessel structure, corresponding respectively to the 1-1 region and the 1-2 region in the above 3D modeling data, each have the 1 color and the 2 color.

14. In claim 13, if the medical staff labels at least a portion of the 1-1 region and the 1-2 region of the vascular structure in the 1 image to modify the color, A medical image processing device characterized by the above labeling unit labeling in real time so that at least a portion of the 2-1 region and the 2-2 region of the vascular structure corresponding thereto in the 3D modeling data is modified and displayed.

15. In claim 13, if a specific area other than the blood vessel structure in the first image is processed with the first color or the second color, if the medical staff labels the first color or the second color of the specific area in the first image to be deleted, A medical image processing device characterized by the above labeling unit labeling to delete the first color or the second color of a part corresponding to the specific area corresponding thereto in the three-dimensional modeling data in real time.

Images