Body surface three-dimensional point cloud target region extraction method and system

Abstract

A body surface three-dimensional point cloud target region extraction method and system. The method comprises the following steps: performing three-dimensional scanning and imaging on the body surface of a target patient provided with a windowed membrane to obtain three-dimensional point cloud data and two-dimensional images of the body surface of the target patient (S1); constructing a mapping relationship on the basis of the three-dimensional point cloud data and the two-dimensional images (S2); inputting each two-dimensional image into an image segmentation model to obtain an ROI recognition result for the outer contour of a windowed membrane region in the two-dimensional image (S3); and traversing three-dimensional points in the three-dimensional point cloud data, mapping a three-dimensional point to a two-dimensional point of a two-dimensional image on the basis of the mapping relationship, determining whether the two-dimensional point is within the ROI recognition result at a viewpoint corresponding to the two-dimensional image, and if yes, adding the three-dimensional point to a target region set, otherwise, determining the next three-dimensional point; and when the traversal has been completed, outputting the target region set as a target region extraction result (S4).

Description

A method and system for extracting target regions from three-dimensional point clouds on the body surface Technical Field

[0001] This invention relates to the field of computer vision technology, and more specifically, to a method and system for extracting target regions from three-dimensional point clouds on the body surface. Background Technology

[0002] Tumor radiotherapy is a local treatment method that uses radiation to treat tumors. During radiotherapy, it is crucial to ensure the accuracy of the radiation irradiation position to avoid damage to other tissues and organs. A low-temperature thermoplastic membrane is commonly used to fix the patient's position during radiotherapy, and windows are created in the membrane as needed clinically. To improve the accuracy of patient positioning, a surface optical image guidance system is used to perform a three-dimensional surface scan of the patient, obtaining a three-dimensional point cloud of the patient's surface. The windowed area of ​​the membrane is then registered on the surface to confirm the accuracy of the positioning and improve the safety of the treatment.

[0003] Current approaches propose acquiring the object's position in a 2D image and generating a 2D bounding box. This is followed by predicting the 3D bounding box in the 3D point cloud data, identifying the physical center of the 3D point cloud data, and finally performing perimeter detection and recognition on a virtual 3D model based on the physical center and the 3D bounding box to identify the membrane window region in the 3D point cloud of the patient's body surface. However, due to the lack of texture information in the acquired 3D point cloud of the patient's body surface, it is impossible to identify the corresponding region of the membrane window based on the membrane color. Furthermore, because the membrane adheres closely to the patient's body surface, it is impossible to identify the corresponding region of the membrane window based on the shape of the window in the 3D point cloud of the patient's body surface. In addition, during radiotherapy, the rotation of the treatment gantry can obstruct the scanning of the window region, making it difficult to accurately identify the target region of the membrane window in the 3D point cloud of the patient's body surface. Summary of the Invention

[0004] To overcome the shortcomings of the prior art in accurately identifying the target area of ​​the membrane opening in the three-dimensional point cloud of the patient's body surface, the present invention provides a method and system for extracting the target area of ​​the three-dimensional point cloud of the body surface.

[0005] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows:

[0006] A method for extracting target regions from three-dimensional point clouds on the body surface includes the following steps:

[0007] The surface of the target patient equipped with a fenestrated membrane was scanned and photographed in three dimensions to obtain three-dimensional point cloud data and two-dimensional images of the target patient's body surface.

[0008] A mapping relationship is constructed based on the three-dimensional point cloud data and the two-dimensional image;

[0009] The two-dimensional image is input into the image segmentation model to obtain the ROI recognition result of the outer contour of the diaphragm window region in the two-dimensional image;

[0010] Traverse the 3D points in the 3D point cloud data, and based on the mapping relationship, assign the 3D point P to... i Two-dimensional point p mapped to a one- or two-dimensional image i Above, and determine the two-dimensional point p i If the point P is located within the ROI recognition result of its corresponding 2D image, then the 3D point P is... i Add to the target region set; otherwise, evaluate the next 3D point.

[0011] After completing the traversal, the set of target regions is output as the target region extraction result.

[0012] Furthermore, this invention also proposes a system for extracting target regions from three-dimensional point clouds on the body surface, applying the method for extracting target regions from three-dimensional point clouds on the body surface proposed in this invention. The system includes:

[0013] The acquisition module is used to scan and photograph the surface of the target patient with the fenestrated membrane to obtain three-dimensional point cloud data and two-dimensional images of the target patient's body surface.

[0014] The mapping module is used to construct a mapping relationship between the three-dimensional point cloud data and the two-dimensional image;

[0015] The ROI recognition module is equipped with an image segmentation model, which is used to input the two-dimensional image into the image segmentation model and output the ROI recognition result of the outer contour of the membrane window region in the two-dimensional image;

[0016] The target region extraction module is used to traverse the 3D points in the 3D point cloud data and extract the 3D points P based on the mapping relationship. i Two-dimensional point p mapped to a one- or two-dimensional image i Above, and determine the two-dimensional point p i If the point P is located within the ROI recognition result of the viewpoint of the 2D image, then the 3D point P is... i Add the target region to the target region set; otherwise, judge the next 3D point. After completing the traversal, output the target region set as the target region extraction result.

[0017] Furthermore, the present invention also proposes an apparatus comprising a memory and a processor, wherein the memory stores computer-readable instructions, wherein when the computer-readable instructions are executed by the processor, the processor causes the processor to perform the steps of the method for extracting target regions of three-dimensional point clouds on the body surface as proposed in the present invention.

[0018] Furthermore, the present invention also proposes a storage medium storing computer-readable instructions, wherein when the computer-readable instructions are executed by a processor, the steps of the method for extracting target regions of three-dimensional point clouds on the body surface as proposed in the present invention are implemented.

[0019] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0020] This invention performs three-dimensional scanning and two-dimensional image capture on the body surface of a target patient with a fenestrated membrane. It uses a trained image segmentation model to accurately extract the ROI recognition results of the outer contour of the fenestrated area of ​​the membrane. Finally, by mapping the three-dimensional point cloud onto the two-dimensional image, it filters the three-dimensional point cloud located within the ROI target area, thus effectively and accurately extracting the target area of ​​the three-dimensional point cloud on the body surface.

[0021] This invention utilizes an image segmentation model to identify the Region of Interest (ROI) of the outer contour of the diaphragm window area. It can adapt to different patients, different treatment sites, different diaphragm window sizes and positions, and different ambient light conditions for 2D image ROI identification. It can accurately extract the ROI region of the diaphragm window, thereby effectively improving the accuracy of extracting the target region of the 3D point cloud on the patient's body surface. Attached Figure Description

[0022] Figure 1 is a flowchart of a method for extracting target regions from three-dimensional point clouds on the body surface according to an embodiment of the present invention.

[0023] Figure 2 is a flowchart illustrating the target area identification and judgment process according to an embodiment of the present invention.

[0024] Figure 3 is a schematic diagram of the ROI recognition result of a two-dimensional image windowed region according to an embodiment of the present invention.

[0025] Figure 4 is a diagram showing the target region extraction effect according to an embodiment of the present invention.

[0026] Figure 5 is a schematic diagram of the ROI recognition result of a windowed area in a two-dimensional image with occlusion according to an embodiment of the present invention.

[0027] Figure 6 is a diagram showing the target region extraction effect according to an embodiment of the present invention.

[0028] Figure 7 is an architecture diagram of a three-dimensional point cloud target region extraction system for the body surface according to an embodiment of the present invention. Detailed Implementation

[0029] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0030] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The singular forms “a,” “the,” and “the” used in this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0031] It should be understood that although the terms first, second, third, etc., may be used in this invention to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be referred to as second information without departing from the scope of this invention, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0032] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0033] Example 1

[0034] This embodiment proposes a method for extracting target regions from three-dimensional point clouds on the body surface. As shown in Figure 1, this is a flowchart of the method for extracting target regions from three-dimensional point clouds on the body surface in this embodiment.

[0035] The method for extracting the target region of a three-dimensional point cloud on the body surface proposed in this embodiment includes the following steps:

[0036] S1. Perform three-dimensional scanning and imaging on the target patient's body surface equipped with a fenestrated membrane to obtain three-dimensional point cloud data and two-dimensional images of the target patient's body surface.

[0037] S2. Construct a mapping relationship between the three-dimensional point cloud data and the two-dimensional image;

[0038] S3. Input the two-dimensional image into the image segmentation model to obtain the ROI recognition result of the outer contour of the diaphragm window region in the two-dimensional image;

[0039] S4. Traverse the three-dimensional points in the three-dimensional point cloud data, and assign the three-dimensional point P based on the mapping relationship. iTwo-dimensional point p mapped to a one- or two-dimensional image i Above, and determine the two-dimensional point p i If the point P is located within the ROI recognition result of the viewpoint of the 2D image, then the 3D point P is... i Add the target region to the target region set; otherwise, judge the next 3D point. After completing the traversal, output the target region set as the target region extraction result.

[0040] In this embodiment, the target patient's body surface with a fenestrated membrane is scanned and photographed in three dimensions. The ROI recognition result of the outer contour of the fenestrated membrane area is accurately extracted by a trained image segmentation model. Finally, it is determined whether the three-dimensional points in the three-dimensional point cloud data are located in the ROI area of ​​any two-dimensional image. Based on the mapping relationship between the two-dimensional image and the three-dimensional point cloud, the target area in the three-dimensional point cloud is determined, which is the fenestrated membrane area of ​​the three-dimensional point cloud of the patient's body surface.

[0041] For example, in step S1, before performing three-dimensional scanning and imaging on the target patient, the target patient undergoes a fixed window membrane fabrication in a molding room. Window membranes are made in areas with high correlation to target area movement, and the target patient is positioned and fixed in the treatment room using the membrane.

[0042] In an optional embodiment, step S1 involves performing a three-dimensional scan and imaging of the target patient's body surface equipped with a fenestrated membrane, including the following steps:

[0043] The target patient's body surface is scanned using a scanning device to obtain three-dimensional point cloud data and two-dimensional images from different perspectives. At least two scanning devices are used and are uniformly calibrated. The scanning devices are set at the same height and adjacent scanning devices are set at equal intervals.

[0044] Furthermore, in an optional embodiment, the scanning device includes a three-dimensional scanning device and a two-dimensional scanning device, or a binocular structured light camera.

[0045] When the scanning device uses a binocular structured light camera, the binocular structured light camera is used to acquire two-dimensional images for three-dimensional measurement and reconstruction, thereby obtaining three-dimensional point cloud data of the target patient's body surface.

[0046] When the scanning device uses both a 3D scanning device and a 2D scanning device, the 3D scanning device acquires 3D point cloud data of the target patient's body surface, while the 2D scanning device acquires 2D images of the target patient's body surface from different perspectives. The two are then correlated through calibration.

[0047] Specifically, the number and / or location of the aforementioned scanning devices are set according to the actual needs of the treatment room space and / or the location, shape, and size of the diaphragm window configured on the target patient, and the present invention does not impose any limitations on this.

[0048] For example, a three-dimensional scanning device and three two-dimensional imaging devices are configured in the treatment room.

[0049] For example, when the number of scanning devices installed is greater than or equal to two, the distance between each scanning device and the fenestrated membrane of the target patient is set to be equal, and the adjacent scanning devices are set to be spaced at equal intervals.

[0050] For example, the scanning device is mounted at a first angle of less than or equal to 30° with the view downward.

[0051] For example, the scanning device is mounted at a downward 30° angle.

[0052] Specifically, by uniformly calibrating 3D scanning equipment and obtaining its intrinsic parameters and coordinate system transformation parameters, the mapping relationship between 3D point cloud data and 2D images can be further constructed. By setting the installation positions and viewing angles of multiple 2D imaging devices and uniformly calibrating them, the mapping accuracy between 3D points and 2D pixels can be improved.

[0053] For example, during the calibration process of a 3D scanning device, a calibration plate can be optionally placed within the scanning area, ensuring it is parallel to the lens of the 3D scanning device. The camera distance and angle are then adjusted to ensure the calibration plate completely covers the camera's field of view. The 3D scanning device is then used to scan the calibration plate. Optionally, images from multiple perspectives can be acquired by moving, rotating, or tilting the 3D scanning device. After all necessary images have been acquired and saved, image processing software is used to extract the markers on the calibration plate from the images and calculate camera parameters to obtain the extrinsic parameter matrix, intrinsic parameter matrix, and distortion coefficients, which are then saved.

[0054] For example, in the calibration process of a two-dimensional imaging device, Zhang's calibration method can be optionally adopted. Specifically, by placing a series of parallel calibration plates on a plane, the camera's intrinsic and extrinsic parameter matrices are solved and saved using the least squares method based on the image coordinates and physical coordinates of the calibration plates at different positions and attitudes.

[0055] In step S1, a three-dimensional scan is performed on the target patient's body surface equipped with a fenestrated membrane to obtain three-dimensional point cloud data of the target patient's body surface, which optionally includes the spatial normal vectors of the three-dimensional points.

[0056] In an optional instance, step S2, constructing a mapping relationship between the 3D point cloud data and the 2D image, includes the following steps:

[0057] The 3D point cloud coordinates P in the world coordinate system w =(X w ,Y w Z w Convert to camera coordinate system P c =(X c ,Y c Z c The expression for the world coordinate system to camera coordinate system transformation is:

[0058] Among them, M extrinsic =(R 3×3 ,t 3×1 R is the extrinsic parameter matrix. 3×3 ,t 3×1 The external parameters obtained through the calibration of the scanning equipment are denoted as the three-dimensional rotation matrix and the three-dimensional translation matrix, respectively.

[0059] The 3D point cloud coordinates P in the camera coordinate system c =(X c ,Y c Z c The conversion from camera coordinates to pixel coordinates p = (u, v) is expressed as follows:

[0060] in, Let f be the intrinsic parameter matrix. x and f y , respectively, are the focal lengths of the camera in the x and y coordinate directions, representing the equivalent value of the focal length from the camera's optical center to the image plane in the pixel coordinate system, with units of pixels; u0 and v0 are the x and y coordinates of the intersection point of the camera's optical axis and the image plane in the image coordinate system, representing the projection point of the camera's optical axis on the image, usually close to the geometric center of the image, with units of pixels.

[0061] Based on the world coordinate system-camera coordinate system transformation formula and the camera coordinate system-pixel coordinate system transformation formula, construct a 3D point cloud P. w =(X w ,Y w Z w The mapping relationship between (u, v) and the two-dimensional point p = (u, v).

[0062] Specifically, based on the mapping relationship, a three-dimensional point P is determined. i =(X i ,Y i Z i In the process of determining whether a point P lies within the ROI identification result of any two-dimensional image, the three-dimensional point P is calculated using the mapping formula. i The mapped 2D pixel pi = (u, v), then determine the two-dimensional pixel point p. i =Whether (u,v) is located within the ROI region.

[0063] Alternatively, the two-dimensional pixel p can be determined by whether the product of the normal vector and the unit vector of the three-dimensional point is less than 0. i = (u,v) whether it is located at the viewpoint of the corresponding two-dimensional image.

[0064] In an optional embodiment, in step S3, the image segmentation model used for identifying the outer contour ROI of the diaphragm window region is pre-trained, including the following steps:

[0065] Two-dimensional images of different patients undergoing treatment at different sites were collected, and the outer contour of the fenestrated area of ​​the membrane in the two-dimensional images was labeled to obtain training data.

[0066] The training data is augmented to obtain a training dataset;

[0067] The training dataset is input into the image segmentation model for training, resulting in a trained image segmentation model used to extract the ROI recognition results of the outer contour of the diaphragm window region from the input two-dimensional image.

[0068] For example, the acquired two-dimensional images include two-dimensional images of different patients, different treatment sites, different diaphragm window sizes and positions, and different ambient light conditions.

[0069] For example, the labelme annotation tool is used to annotate the outer contour of the diaphragm window region in a two-dimensional image.

[0070] For example, data augmentation operations on training data include rotation, cropping, deformation, etc.

[0071] For example, during the training of an image segmentation model, an adaptive learning rate adjustment strategy can be optionally adopted, which can dynamically adjust the learning rate according to the loss value during training, helping to maintain the stability and convergence of the model during training.

[0072] Alternatively, the image segmentation model may be a YOLOv5 network.

[0073] For example, the image segmentation model includes a backbone network, a neck network, a head network, a post-processing network, and a loss function. The backbone network extracts useful features from the input image; the neck network integrates feature maps from different levels; the head network predicts the target category and bounding box location; the post-processing network processes the detection results, removing redundant detection boxes to obtain the final detection result; and the loss function optimizes the model's parameters during training, guiding the model's update direction by calculating the difference between predicted and true values.

[0074] For example, during the training of an image segmentation model, the CIoU (Complete Intersection over Union) loss function can be optionally used to measure the degree of matching of the target boxes, which can more accurately reflect the matching of the target boxes and thus improve the detection accuracy.

[0075] In this embodiment, after labeling the outer contour of the windowed region of the membrane in the training dataset, the image segmentation model is input for learning. Even in two-dimensional images, due to differences in the position and shape of the membrane, excessively bright or dark ambient light, noise such as occlusion, the image segmentation network can still effectively extract the outer contour detection box of the windowed region and output it as the ROI recognition result.

[0076] For example, the image segmentation model in this invention may also optionally use neural networks such as U-Net, R-FCN, and Mask R-CNN, which can effectively extract ROIs. This invention does not limit the scope of the model.

[0077] Compared with other neural networks, the YOLOv5 network is chosen as the image segmentation model because it has advantages in many aspects, such as high efficiency, accuracy, ease of use, real-time performance and versatility in ROI extraction.

[0078] In an optional embodiment, step S4, determining whether a 3D point lies within the ROI recognition result of any 2D image, includes the following steps:

[0079] (1) For the i-th three-dimensional point P i =(X i ,Y i Z i Based on the mapping relationship, the three-dimensional point P is... i Mapped to the m-th two-dimensional image A m Two-dimensional point p m,i And determine the two-dimensional point p m,i Is it in the m-th two-dimensional image A? m Within the ROI identification results:

[0080] If so, proceed to step (2);

[0081] Otherwise, for the next two-dimensional image A m+1 Execute step (1) until all two-dimensional images have been judged, then proceed to the next three-dimensional point P. i+1 Execute step (1) until all three-dimensional points have been judged;

[0082] (2) The three-dimensional point P i Convert to camera coordinate system P i,c =(X i,c ,Y i,c Z i,c ), calculate the 3D point P in the camera coordinate system i normal vector With unit vector Is the product less than 0?

[0083] If so, then the three-dimensional point P i,c Add to the target region set, and for the next 3D point P i+1 Execute step (1) until all three-dimensional points have been judged;

[0084] Otherwise, for the next two-dimensional image A m+1 Execute step (1) until all two-dimensional images have been judged, then proceed to the next three-dimensional point P. i+1 Execute step (1) until all three-dimensional points have been judged;

[0085] Where i = 1, 2, ..., N, N is the total number of three-dimensional points; m = 1, 2, ..., M, M is the total number of two-dimensional images.

[0086] In this embodiment, each two-dimensional image represents an image captured by a scanning device from a specific viewpoint. By evaluating each two-dimensional image individually, it is possible to determine whether the current three-dimensional point falls within the ROI of the corresponding device viewpoint from different perspectives. If it is determined that the current three-dimensional point falls within the ROI of the current two-dimensional image, the ROI of the current three-dimensional point will not be determined again in the next image, thus avoiding duplicate determinations.

[0087] Among them, the three-dimensional point P in the camera coordinate system i normal vector According to P i,c =(X i,c ,Y i,c Z i,c )get.

[0088] Figure 2 shows the flowchart for target area identification and judgment.

[0089] In this embodiment, the determination condition in step (1) is that the three-dimensional point is located within the ROI recognition result of the two-dimensional image. Specifically, the two-dimensional point mapped from the three-dimensional point to the two-dimensional image is inside the closed outer contour ROI.

[0090] For example, in determining a two-dimensional point p m,i Is it in the m-th two-dimensional image A? m When identifying the ROI in the results, the following steps are included:

[0091] Step 1: Traverse the adjacent two-dimensional points in the ROI recognition results to form line segments, and finally form a set of line segments with multiple ordered index IDs;

[0092] Step 2: Based on the mapped two-dimensional point coordinates p m,i (x m,i ,y m,i ), calculate the line Y = y in the two-dimensional image. m,i Does the line Y intersect with any line segment in the set of line segments? Specifically, when there exists a line segment with endpoints p1(x1,y1) and p2(x2,y2) such that y≥y1 and y≤y2, or y≥y2 and y≤y1, it represents the line Y = y m,i If there is an intersection point with a line segment in the set of line segments, calculate the x-axis coordinate of the intersection point X = x1 + (y - y1) / (y2 - y1) * (x2 - x1) to obtain the coordinates of the intersection point;

[0093] Step 3: For each two-dimensional point obtained after mapping, calculate the set of intersection points, sort the x-coordinates of the intersection point set from smallest to largest, and record the sorted index number ID;

[0094] Step 4: In the sorted set of intersection points, find the first one that is not less than x. m,i Find the index number id of the intersection point and determine the two-dimensional point p. m,i (x m,i ,y m,i Whether the point p is located inside a closed outer contour ROI; where, let the index difference of the first intersection point after sorting be id-0, and if id%2=1, then determine whether the two-dimensional point p is inside the closed outer contour ROI; m,i (x m,i ,y m,i It is located inside the closed outer contour ROI.

[0095] Based on the above determination rules, the two-dimensional point p can be determined. m,i Is it in the m-th two-dimensional image A? m Within the ROI identification results.

[0096] In this embodiment, the determination condition in step (2) is the three-dimensional point P. iNormal vector and unit vector The product of is less than 0, meaning that the product of and is less than 0 for the three-dimensional point P. i If the angle between the normal vector and the unit vector of the optical axis of the 3D scanning device is greater than 90°, it indicates that the 3D point P... i Or, towards the scanning end of the 3D scanning device, or the 3D point P. i It is located within the scanning range of the 3D scanning equipment.

[0097] Specifically, a point obtained by scanning equipment should be a point scanned by the equipment when the angle between the unit vector of the optical axis of the scanning equipment and the normal vector of the three-dimensional point cloud is greater than 90° from the perspective of the scanning equipment, that is, when the product of the normal vector of the three-dimensional point cloud and the unit vector of the optical axis is less than 0. Otherwise, it means that the three-dimensional point belongs to the three-dimensional point behind the perspective of the scanning equipment and does not match the current perspective.

[0098] This embodiment uses dual-judgment to extract the target region of the three-dimensional point cloud on the body surface, which can effectively improve the accuracy of the target region extraction.

[0099] Specifically, after the traversal is completed, the target region set is marked in the three-dimensional point cloud data, and the marked three-dimensional point cloud data is output. The marked region is the target region extraction result.

[0100] Specifically, this embodiment also verifies the method for extracting the target region of the three-dimensional point cloud on the body surface described in this invention through experiments.

[0101] Figure 3 shows a schematic diagram of the ROI recognition result of a two-dimensional image windowed region according to an embodiment of the present invention, wherein parts (a), (b), and (c) in Figure 3 are two-dimensional images with ROI recognition results at different angles.

[0102] Figure 4 shows the target region extraction effect according to an embodiment of the present invention, wherein parts (a) and (b) in Figure 4 are comparison images before and after target region extraction.

[0103] As shown in Figure 5, this is a schematic diagram of the ROI recognition result of a windowed two-dimensional image with occlusion according to an embodiment of the present invention. Parts (a), (b), and (c) in Figure 5 are two-dimensional images of the ROI recognition result with windowed regions at different angles, and part (c) contains occlusion indication.

[0104] Figure 6 is a diagram showing the target region extraction effect according to an embodiment of the present invention.

[0105] The results of target region extraction shown in the figure demonstrate that the proposed method for extracting target regions from three-dimensional point clouds on the body surface can still accurately identify the target regions of the membrane opening in the three-dimensional point cloud of the patient's body surface even in the presence of occlusion. Specifically, by mapping and matching the ROI region of the two-dimensional image with the three-dimensional point cloud data, it is determined whether the three-dimensional point is located within the ROI recognition result of any two-dimensional image and falls within the scanning range of the corresponding three-dimensional scanning device. If so, the three-dimensional point is taken as the three-dimensional point within the corresponding target region of the membrane opening. Finally, the set of target regions is output as the target region extraction result, thus completing the extraction of the target region from the three-dimensional point cloud on the body surface.

[0106] Example 2

[0107] This embodiment proposes a system for extracting target regions from three-dimensional point clouds on the body surface, applying the method for extracting target regions from three-dimensional point clouds on the body surface proposed in Embodiment 1. Figure 7 shows the architecture diagram of the system for extracting target regions from three-dimensional point clouds on the body surface in this embodiment.

[0108] The three-dimensional point cloud target region extraction system for body surface proposed in this embodiment includes:

[0109] The acquisition module is used to scan and photograph the surface of the target patient with the fenestrated membrane to obtain three-dimensional point cloud data and two-dimensional images of the target patient's body surface.

[0110] The mapping module is used to construct a mapping relationship between the three-dimensional point cloud data and the two-dimensional image;

[0111] The ROI recognition module is equipped with an image segmentation model, which is used to input the two-dimensional image into the image segmentation model and output the ROI recognition result of the outer contour of the membrane window region in the two-dimensional image;

[0112] The target region extraction module is used to traverse the three-dimensional points in the three-dimensional point cloud data and, based on the mapping relationship, determine whether the three-dimensional point is located within the ROI recognition result of the viewpoint of the two-dimensional image. If so, the three-dimensional point is added to the target region set; otherwise, the next three-dimensional point is judged. After the traversal is completed, the target region set is output as the target region extraction result.

[0113] In one optional embodiment, the acquisition module is used to scan the body surface of the target patient to obtain three-dimensional point cloud data and two-dimensional images from different perspectives of the target patient's body surface.

[0114] The acquisition module may optionally include at least two scanning devices, and the scanning devices are uniformly calibrated.

[0115] Alternatively, the scanning devices are arranged at the same height, and adjacent scanning devices are arranged at equal intervals.

[0116] Furthermore, in an optional embodiment, the scanning device includes a three-dimensional scanning device and a two-dimensional scanning device, or a binocular structured light camera.

[0117] Furthermore, in an optional embodiment, when the mapping module constructs a mapping relationship based on the 3D point cloud data and the 2D image, it performs the following steps:

[0118] The 3D point cloud coordinates P in the world coordinate system w =(X w ,Y w Z w Convert to camera coordinate system P c =(X c ,Y c Z c The expression for the world coordinate system to camera coordinate system transformation is:

[0119] Among them, M extrinsic =(R 3×3 ,t 3×1 R is the extrinsic parameter matrix. 3×3 ,t 3×1 These are the extrinsic parameters obtained through the calibration of the scanning equipment;

[0120] The 3D point cloud coordinates P in the camera coordinate system c =(X c ,Y c Z c The conversion from camera coordinates to pixel coordinates p = (u, v) is expressed as follows:

[0121] in, Let f be the intrinsic parameter matrix. x and f y , respectively, are the focal lengths of the camera in the x and y coordinate directions, and u0 and v0 are the x and y coordinates of the intersection of the camera optical axis and the image plane in the image coordinate system;

[0122] Based on the world coordinate system-camera coordinate system transformation formula and the camera coordinate system-pixel coordinate system transformation formula, construct a 3D point cloud P. w =(X w ,Y w Z w The mapping relationship between (u, v) and the two-dimensional point p = (u, v).

[0123] Furthermore, in an optional embodiment, the acquisition module is also used to acquire two-dimensional images of different patients undergoing treatment at different sites, and to annotate the outer contour of the diaphragm window area in the two-dimensional images to obtain training data.

[0124] Furthermore, the training data is used to pre-train the image segmentation model mounted in the ROI recognition module. The training process includes:

[0125] The training data is augmented to obtain a training dataset;

[0126] The training dataset is input into the image segmentation model for training, resulting in a trained image segmentation model used to extract the ROI recognition results of the outer contour of the diaphragm window region from the input two-dimensional image.

[0127] Furthermore, in an optional embodiment, the image segmentation model mounted in the ROI recognition module includes the YOLOv5 network model.

[0128] Furthermore, in an optional embodiment, when determining whether a 3D point is located within the ROI recognition result of any 2D image, the target region extraction module performs the following steps:

[0129] (1) For the i-th three-dimensional point P i =(X i ,Y i Z i Based on the mapping relationship, determine whether it exists in the m-th two-dimensional image A. m Within the ROI identification results:

[0130] If so, proceed to step (2);

[0131] Otherwise, for the next two-dimensional image A m+1 Execute step (1) until all two-dimensional images have been judged, then proceed to the next three-dimensional point P. i+1 Perform step (1);

[0132] (2) Convert the i-th 3D point to the camera coordinate system P i,c =(X i,c ,Y i,c Z i,c ), calculate the three-dimensional point P i,c normal vector With unit vector Is the product less than 0?

[0133] If so, then the three-dimensional point P i,c Add to the target region set, and for the next 3D point P i+1 Execute step (1) until all three-dimensional points have been judged;

[0134] Otherwise, for the next two-dimensional image A m+1 Execute step (1) until all two-dimensional images have been judged, then proceed to the next three-dimensional point P.i+1 Execute step (1) until all three-dimensional points have been judged;

[0135] Where i = 1, 2, ..., N, N is the total number of three-dimensional points; m = 1, 2, ..., M, M is the total number of two-dimensional images.

[0136] It is understood that the system in this embodiment corresponds to the method in Embodiment 1 above, and other options in Embodiment 1 above are also applicable to this embodiment, so they will not be described again here.

[0137] Example 3

[0138] This embodiment proposes a computer device, including a memory and a processor. The memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor performs the steps of the method for extracting the target region of a three-dimensional point cloud on the body surface as proposed in Embodiment 1.

[0139] Example 4

[0140] This embodiment proposes a storage medium storing computer-readable instructions, wherein when the computer-readable instructions are executed by a processor, they implement the steps of the method for extracting the target region of the three-dimensional point cloud on the body surface proposed in Embodiment 1.

[0141] By way of example, the storage medium includes, but is not limited to, USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks or optical disks, and other media capable of storing program code.

[0142] For example, the instructions, programs, code sets, or instruction sets may be implemented using conventional programming languages.

[0143] For example, the processor includes, but is not limited to, smartphones, personal computers, servers, network devices, etc., for executing all or part of the steps of the three-dimensional point cloud target region extraction method for body surface described in Embodiment 1.

[0144] The various embodiments in this invention are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The device embodiments described above are merely exemplary. The modules described as separate components may or may not be physically separate. When implementing the present invention, the functions of each module can be implemented in one or more software and / or hardware. Alternatively, some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0145] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A method for extracting a target region of a three-dimensional point cloud on a body surface, characterized by Includes the following steps: The surface of the target patient equipped with a fenestrated membrane was scanned and photographed in three dimensions to obtain three-dimensional point cloud data and two-dimensional images of the target patient's body surface. A mapping relationship is constructed based on the three-dimensional point cloud data and the two-dimensional image; The two-dimensional image is input into the image segmentation model to obtain the ROI recognition result of the outer contour of the diaphragm window region in the two-dimensional image; traversing three-dimensional points in the three-dimensional point cloud data, mapping a three-dimensional point P i to a two-dimensional point p i on a two-dimensional image based on the mapping relationship, and determining whether the two-dimensional point p i is located within an ROI identification result of a view angle at which the two-dimensional image is located, and if so, adding the three-dimensional point P i to a target region set, and otherwise determining a next three-dimensional point; After completing the traversal, the set of target regions is output as the target region extraction result.

2. The method of claim 1, wherein, The determination of whether it is located within the ROI recognition result of any two-dimensional image includes the following steps: (1) For the i-th three-dimensional point P i =(X i ,Y i Z i Based on the mapping relationship, the three-dimensional point P is... i Mapped to the m-th two-dimensional image A m Two-dimensional point p m,i And determine the two-dimensional point p m,i Is it in the m-th two-dimensional image A? m Within the ROI identification results: If so, proceed to step (2); Otherwise, the next two-dimensional image A m+1 Step (1) is performed until all two-dimensional images are judged, and the next three-dimensional point P i+1 Step (1) is performed until all three-dimensional points are judged. (2) convert the three-dimensional point P i to the camera coordinate system P i,c = (X i,c , Y i,c , Z i,c ), calculate the normal vector i of the three-dimensional point P with the unit vector Is the product less than 0? If yes, the three-dimensional point P i adding the target region set, and judging the next three-dimensional point P i+1 performing step (1) until all three-dimensional points are judged Otherwise, the next two-dimensional image A m+1 Step (1) is performed until all two-dimensional images are judged, and the next three-dimensional point P i+1 Step (1) is performed until all three-dimensional points are judged. Where i = 1, 2, ..., N, N is the total number of three-dimensional points; m = 1, 2, ..., M, M is the total number of two-dimensional images. 3.The method of claim 1, wherein, The method further includes the following steps: Two-dimensional images of different patients undergoing treatment at different sites were collected, and the outer contour of the fenestrated area of ​​the membrane in the two-dimensional images was labeled to obtain training data. The training data is augmented to obtain a training dataset; The training dataset is input into the image segmentation model for training, resulting in a trained image segmentation model used to extract the ROI recognition results of the outer contour of the diaphragm window region from the input two-dimensional image.

4. The method of claim 3, wherein, The image segmentation model includes the YOLOv5 network model.

5. The method of claim 1, wherein, The process of constructing a mapping relationship between the 3D point cloud data and the 2D image includes the following steps: The 3D point cloud coordinates P in the world coordinate system w =(X w ,Y w Z w Convert to camera coordinate system coordinates P c =(X c ,Y c Z c The expression for the world coordinate system to camera coordinate system transformation is: wherein M extrinsic = (R 3×3 ,t 3×1 ) is an extrinsic matrix, R 3×3 ,t 3×1 is an extrinsic parameter obtained by calibrating the scanning device; The three-dimensional point cloud coordinates P of the camera coordinate system are converted into pixel coordinate system coordinates p = (u, v), and the expression of the camera coordinate system-pixel coordinate conversion formula is: c = (X c ,Y c ,Z c ) wherein, Let f be the intrinsic parameter matrix. x and f y , respectively, are the focal lengths of the camera in the x and y coordinate directions, and u0 and v0 are the x and y coordinates of the intersection of the camera optical axis and the image plane in the image coordinate system; According to the world coordinate system-camera coordinate system conversion formula and the camera coordinate system-pixel coordinate conversion formula, a mapping relationship formula of a three-dimensional point cloud P and a two-dimensional point p=(u, v) is constructed. w = (X w , Y w , Z w ) 6. The method according to any one of claims 1 to 5, wherein, The three-dimensional scanning and imaging of the target patient's body surface equipped with a fenestrated membrane includes the following steps: The target patient's body surface was scanned using a scanning device to obtain three-dimensional point cloud data and two-dimensional images from different perspectives. The scanning devices are at least two in number and are uniformly calibrated; the scanning devices are set at the same height and adjacent scanning devices are set at equal intervals.

7. The method of claim 6, wherein, The scanning equipment includes a three-dimensional scanning device and a two-dimensional scanning device, or a binocular structured light camera.

8. A system for extracting target regions from three-dimensional point clouds on the body surface, characterized in that, The acquisition module is used to scan and photograph the surface of the target patient with the fenestrated membrane to obtain three-dimensional point cloud data and two-dimensional images of the target patient's body surface. The mapping module is used to construct a mapping relationship between the three-dimensional point cloud data and the two-dimensional image; The ROI recognition module is equipped with an image segmentation model, which is used to input the two-dimensional image into the image segmentation model and output the ROI recognition result of the outer contour of the membrane window region in the two-dimensional image; The target region extraction module is used to traverse the 3D points in the 3D point cloud data and extract the 3D points P based on the mapping relationship. i Two-dimensional point p mapped to a one- or two-dimensional image i Above, and determine the two-dimensional point p. i If the point P is located within the ROI recognition result of the viewpoint of the 2D image, then the 3D point P is... i Add to the target region set; otherwise, evaluate the next 3D point. After completing the traversal, the set of target regions is output as the target region extraction result. 9.The body surface three-dimensional point cloud target region extraction system according to claim 8, characterized in that, The target region extraction module executes the following steps when judging whether the two-dimensional point p i is located within the ROI recognition result of the view angle in which the two-dimensional image is located. (1) For the i-th three-dimensional point P i =(X i ,Y i Z i Based on the mapping relationship, the three-dimensional point P is... i Mapped to the m-th two-dimensional image A m Two-dimensional point p m,i And determine the two-dimensional point p m,i Is it in the m-th two-dimensional image A? m Within the ROI identification results: If so, proceed to step (2); Otherwise, the next two-dimensional image A m+1 Step (1) is performed until all two-dimensional images are judged, and the next three-dimensional point P i+1 Step (1) is performed until all three-dimensional points are judged. (2) convert the three-dimensional point P i to the camera coordinate system P i,c = (X i,c , Y i,c , Z i,c ), calculate the normal vector i of the three-dimensional point P with the unit vector Is the product less than 0? If yes, the three-dimensional point P i adding the target region set, and judging the next three-dimensional point P i+1 performing step (1) until all three-dimensional points are judged Otherwise, the next two-dimensional image A m+1 Step (1) is performed until all two-dimensional images are judged, and the next three-dimensional point P i+1 Step (1) is performed until all three-dimensional points are judged. Where i = 1, 2, ..., N, N is the total number of three-dimensional points; m = 1, 2, ..., M, M is the total number of two-dimensional images. 10.The body surface three-dimensional point cloud target region extraction system according to claim 8, characterized in that, The acquisition module is also used to acquire two-dimensional images of different patients undergoing treatment at different sites, and to annotate the outer contour of the diaphragm window area in the two-dimensional images to obtain training data. The training data is used to train the image segmentation model; Its training process includes: The training data is augmented to obtain a training dataset; The training dataset is input into the image segmentation model for training, resulting in a trained image segmentation model used to extract the ROI recognition results of the outer contour of the diaphragm window region from the input two-dimensional image.

11. The body surface three-dimensional point cloud target region extraction system according to claim 10, characterized in that, The image segmentation model includes the YOLOv5 network model. 12.The body surface three-dimensional point cloud target region extraction system according to claim 8, characterized in that, When constructing a mapping relationship between the 3D point cloud data and the 2D image, the mapping module performs the following steps: The three-dimensional point cloud coordinates P of the world coordinate system w = (X w , Y w , Z w ) are converted into camera coordinate system coordinates P c = (X c , Y c , Z c ), and the expression of the world coordinate system-camera coordinate system conversion formula is: wherein M extrinsic = (R 3×3 ,t 3×1 ) is an extrinsic matrix, R 3×3 ,t 3×1 is an extrinsic parameter obtained by calibrating the scanning device; The three-dimensional point cloud coordinates P of the camera coordinate system are converted into pixel coordinate system coordinates p = (u, v), and the expression of the camera coordinate system-pixel coordinate conversion formula is: c = (X c ,Y c ,Z c ) wherein, Let f be the intrinsic parameter matrix. x and f y , respectively, are the focal lengths of the camera in the x and y coordinate directions, and u0 and v0 are the x and y coordinates of the intersection of the camera optical axis and the image plane in the image coordinate system; According to the world coordinate system-camera coordinate system conversion formula and the camera coordinate system-pixel coordinate conversion formula, a mapping relationship formula of a three-dimensional point cloud P and a two-dimensional point p=(u, v) is constructed. w = (X w , Y w , Z w ) and a two-dimensional point p=(u, v).

13. The body surface three-dimensional point cloud target region extraction system according to any one of claims 8 to 12, characterized by, When the acquisition module scans and photographs the body surface of a target patient equipped with a fenestrated membrane, it performs the following steps: The target patient's body surface was scanned to obtain three-dimensional point cloud data and two-dimensional images from different perspectives. The acquisition module includes at least two scanning devices, which are uniformly calibrated; the scanning devices are set at the same height, and adjacent scanning devices are set at equal intervals.

14. The body surface three-dimensional point cloud target region extraction system according to claim 13, characterized in that, The scanning equipment includes a three-dimensional scanning device and a two-dimensional scanning device, or a binocular structured light camera.

15. An apparatus comprising a memory and a processor, said memory having computer readable instructions stored therein, wherein, When the computer-readable instructions are executed by the processor, the processor performs the steps of the method for extracting target regions of three-dimensional point clouds on the body surface as described in any one of claims 1 to 7.

16. A storage medium having stored thereon computer readable instructions, characterized in that, When the computer-readable instructions are executed by a processor, they implement the steps of the method for extracting the target region of a three-dimensional point cloud on the body surface as described in any one of claims 1 to 7.

Images