Three-dimensional reconstruction method and system
By combining distortion correction and depth completion network models, the problem of missing depth maps was solved, the accuracy and effect of 3D reconstruction were improved, and the accuracy and integrity of the 3D model were ensured.
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- HUNAN HAPPLY SUNSHINE INTERACTIVE ENTERTAINMENT MEDIA CO LTD
- Filing Date
- 2025-04-08
- Publication Date
- 2026-06-04
AI Technical Summary
In existing 3D reconstruction technologies, depth maps often suffer from large-area gaps, affecting the accuracy and effectiveness of dense reconstruction.
Distortion correction is performed by obtaining distortion coefficients and camera intrinsic parameters. Sparse point cloud and camera extrinsic parameters are calculated. After removing outliers, depth completion is performed using a depth completion network model to construct the final 3D model.
It improves the accuracy of dense reconstruction and the effect of 3D reconstruction, ensuring the accuracy and stability of 3D reconstruction results.
Smart Images
Figure CN2025087738_04062026_PF_FP_ABST
Abstract
Description
A three-dimensional reconstruction method and system
[0001] This invention claims priority to Chinese Patent Application No. 202411733636.7, filed on November 29, 2024, entitled "A Three-Dimensional Reconstruction Method and System", the entire contents of which are incorporated herein by reference. Technical Field
[0002] This invention relates to the field of image processing technology, and specifically to a three-dimensional reconstruction method and system. Background Technology
[0003] 3D reconstruction technology is used to recover the structure of a 3D scene from a 2D image. 3D reconstruction technology has broad application prospects in game development, film production and cultural relic protection.
[0004] Current 3D reconstruction techniques involve calculating a depth map from the sparse point cloud obtained through sparse reconstruction and the RGB image, and then using the depth map for dense reconstruction to obtain a dense point cloud. However, in the aforementioned 3D reconstruction process, the depth map often suffers from quality issues such as large-area missing areas, which affects the accuracy of dense reconstruction and results in poor 3D reconstruction performance. Summary of the Invention
[0005] In view of this, embodiments of the present invention provide a three-dimensional reconstruction method and system to solve the problems of poor three-dimensional reconstruction effect in existing three-dimensional reconstruction technologies.
[0006] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:
[0007] The first aspect of this invention discloses a three-dimensional reconstruction method, the method comprising:
[0008] Using the first set of images of the object to be reconstructed acquired by the camera, the distortion coefficients and camera intrinsic parameters of the camera are obtained;
[0009] The distortion of the first images in the first image set is corrected using the distortion coefficients to obtain the second image set;
[0010] Calculate the sparse point cloud, and calculate the camera extrinsic parameters and the first depth map of the second image contained in the second image set;
[0011] Outlier removal is performed on the first depth map;
[0012] The first depth map, after removing outliers, is input into a pre-trained depth completion network model for depth completion, resulting in an optimized second depth map.
[0013] Based on the depth information of the second depth map, the camera intrinsic parameters, the camera extrinsic parameters, and the sparse point cloud, the final 3D model of the object to be reconstructed is constructed.
[0014] Preferably, calculating the sparse point cloud and calculating the camera extrinsic parameters and the first depth map of the second image contained in the second image set includes:
[0015] The second image set contains the second images, which are then divided into multiple image groups.
[0016] By utilizing the similarity between feature points of each second image in the image group, multiple sets of successfully matched feature points are obtained;
[0017] Based on the camera intrinsic parameters, the second image set, the image group, and the multiple sets of successfully matched feature points, calculate the sparse point cloud and the camera extrinsic parameters of the second image;
[0018] The first depth map corresponding to the second image is calculated based on the camera intrinsic parameters, the camera extrinsic parameters, and the sparse point cloud.
[0019] Preferably, the final 3D model of the object to be reconstructed is constructed based on the depth information of the second depth map, the camera intrinsic parameters, the camera extrinsic parameters, and the sparse point cloud, including:
[0020] Calculate the dense point cloud based on the depth information of the second depth map, the camera intrinsic parameters, the camera extrinsic parameters, and the sparse point cloud;
[0021] The dense point cloud is meshed to obtain an initial 3D model of the object to be reconstructed;
[0022] By utilizing the mapping relationship between points on the second image and points in space, the RGB pixels of the second image are mapped to the texture space to obtain the texture map of the initial three-dimensional model;
[0023] The texture map is mapped onto the initial 3D model to obtain the final 3D model.
[0024] Preferably, outlier removal of the first depth map includes:
[0025] Identify valid points in the first depth map;
[0026] The valid points that meet the preset anomaly conditions are identified as outliers;
[0027] Remove the outliers from the first depth map.
[0028] Preferably, using a first set of images of the object to be reconstructed acquired by a camera, the distortion coefficients and camera intrinsic parameters of the camera are obtained, including:
[0029] Read the image metadata of the first image contained in the first image set of the object to be reconstructed, which was acquired by the camera;
[0030] Obtain the distortion coefficients and camera intrinsics of the camera from the image metadata.
[0031] Preferably, the deep completion network model includes at least: an encoder composed of a deep residual network, a decoder composed of multiple upsampling modules, a local network composed of multiple consecutive convolutional layers, and a refinement network.
[0032] A second aspect of this invention discloses a three-dimensional reconstruction system, the system comprising:
[0033] The acquisition unit is used to acquire distortion coefficients and camera intrinsic parameters of the camera using a first set of images of the object to be reconstructed acquired by the camera.
[0034] A correction unit is used to perform distortion correction on the first images contained in the first image set using the distortion coefficients to obtain a second image set.
[0035] The computing unit is used to compute sparse point clouds and compute camera extrinsic parameters and a first depth map of the second images contained in the second image set.
[0036] The removal unit is used to remove outliers from the first depth map;
[0037] The depth completion unit is used to input the first depth map (with outliers removed) into a pre-trained depth completion network model to perform depth completion, thereby obtaining an optimized second depth map.
[0038] The construction unit is used to construct the final 3D model of the object to be reconstructed based on the depth information of the second depth map, the camera intrinsic parameters, the camera extrinsic parameters, and the sparse point cloud.
[0039] Preferably, the computing unit includes:
[0040] The sub-unit is used to divide the second images contained in the second image set into multiple image groups;
[0041] The sub-unit is used to obtain multiple sets of successfully matched feature points by utilizing the similarity between feature points of each second image in the image group.
[0042] The first calculation subunit is used to calculate the sparse point cloud and the camera extrinsic parameters of the second image based on the camera intrinsic parameters, the second image set, the image group, and the multiple sets of successfully matched feature points;
[0043] The second calculation subunit is used to calculate the first depth map corresponding to the second image based on the camera intrinsic parameters, the camera extrinsic parameters, and the sparse point cloud.
[0044] Preferably, the building unit includes:
[0045] The calculation subunit is used to calculate the dense point cloud based on the depth information of the second depth map, the camera intrinsic parameters, the camera extrinsic parameters, and the sparse point cloud;
[0046] Meshable sub-units are used to mesh the dense point cloud to obtain an initial three-dimensional model of the object to be reconstructed;
[0047] The first mapping subunit is used to map the RGB pixels of the second image to the texture space by utilizing the mapping relationship between points on the second image and points in space, so as to obtain the texture map of the initial three-dimensional model.
[0048] The second mapping subunit is used to map the texture map onto the initial 3D model to obtain the final 3D model.
[0049] Preferably, the removal unit includes:
[0050] An identification subunit is used to identify valid points in the first depth map;
[0051] A sub-unit is defined to identify the valid points that meet the preset abnormal conditions as outliers;
[0052] The removal sub-unit is used to remove the outliers in the first depth map.
[0053] Based on the above embodiments of the present invention, a 3D reconstruction method and system are provided. The method comprises: using a first image set of the object to be reconstructed acquired by a camera, obtaining distortion coefficients and camera intrinsic parameters; performing distortion correction on the first images contained in the first image set using the distortion coefficients to obtain a second image set; calculating a sparse point cloud, and calculating the camera extrinsic parameters and a first depth map of the second images contained in the second image set; removing outliers from the first depth map; inputting the first depth map with outliers removed into a depth completion network model for depth completion to obtain an optimized second depth map; and constructing the final 3D model of the object to be reconstructed based on the depth information of the second depth map, camera intrinsic parameters, camera extrinsic parameters, and sparse point cloud. In this scheme, after calculating the first depth map, outliers in the first depth map are first removed, and then a depth completion network model is used to perform depth completion on the first depth map with outliers removed to obtain a more complete second depth map. The second depth map is then used to construct the final 3D model of the object to be reconstructed, thereby improving the accuracy of dense reconstruction and the effect of 3D reconstruction. Attached Figure Description
[0054] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0055] Figure 1 is a flowchart of a three-dimensional reconstruction method provided in an embodiment of the present invention;
[0056] Figure 2 is a structural example diagram of the deep completion network model provided in an embodiment of the present invention;
[0057] Figure 3 is a flowchart of the calculation of sparse point cloud and first depth map provided in an embodiment of the present invention;
[0058] Figure 4 is a flowchart of constructing the final three-dimensional model provided in an embodiment of the present invention;
[0059] Figure 5 is a structural block diagram of a three-dimensional reconstruction system provided in an embodiment of the present invention. Detailed Implementation
[0060] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0061] In this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0062] 3D reconstruction technology is used to recover the structure of a 3D scene from a 2D image. 3D reconstruction technology has broad application prospects in game development, film production, virtual reality, augmented reality, cultural relic protection and medical imaging.
[0063] Current 3D reconstruction techniques involve calculating a depth map from the sparse point cloud obtained through sparse reconstruction and the RGB image, and then using the depth map for dense reconstruction to obtain a dense point cloud. However, in the aforementioned 3D reconstruction process, the depth map often suffers from quality issues such as large-area missing areas, which affects the accuracy of dense reconstruction and results in poor 3D reconstruction performance.
[0064] To address the aforementioned issues, this invention provides a three-dimensional reconstruction method and system. After calculating a first depth map, outliers in the first depth map are removed. Then, a depth completion network model is used to perform depth completion on the first depth map after outlier removal to obtain a more complete and higher-quality second depth map. The second depth map is then used to construct the final three-dimensional model of the object to be reconstructed, thereby improving the accuracy of dense reconstruction and the effect of three-dimensional reconstruction.
[0065] In practical applications, this solution can be used for 3D reconstruction in areas such as cultural relic protection, cultural heritage research, and industrial design.
[0066] Referring to Figure 1, a flowchart of a three-dimensional reconstruction method provided by an embodiment of the present invention is shown. The three-dimensional reconstruction method includes:
[0067] Step S101: Using the first set of images of the object to be reconstructed acquired by the camera, obtain the distortion coefficients and the camera's intrinsic parameters.
[0068] It should be noted that image metadata includes, but is not limited to: shooting date, camera model, shooting location, aperture value, shutter speed, resolution, file format, ISO, camera intrinsic parameters (focal length and principal point coordinates), distortion coefficients (radial distortion and tangential distortion), etc.
[0069] In the specific implementation step S101, the object to be reconstructed (such as a cultural relic) is placed at a designated location, and then the object to be reconstructed is photographed from multiple angles using a camera (such as a digital camera) to obtain multiple first images, which together form a first image set.
[0070] Read the image metadata of the first image in the first image set of the object to be reconstructed, which is acquired by the camera, and obtain the distortion coefficients and camera intrinsic parameters from the image metadata.
[0071] For example, place the object to be reconstructed in the acquisition box and take pictures of the object from three angles: top, eye level, and bottom. The tilt angle is between 30 and 45 degrees. Turn on the turntable and take 16-24 first pictures from each angle to obtain the first picture set.
[0072] Step S102: Perform distortion correction on the first images contained in the first image set using distortion coefficients to obtain the second image set.
[0073] In the specific implementation step S102, the obtained distortion coefficients are used to correct the distortion of the first image contained in the first image set to obtain the second image set. The second image set contains multiple second images, which are the first images after distortion correction.
[0074] Step S103: Calculate the sparse point cloud, and calculate the camera extrinsic parameters and the first depth map of the second image contained in the second image set.
[0075] In the specific implementation step S103, sparse reconstruction is performed using camera intrinsic parameters and the second image set to obtain sparse point cloud, camera extrinsic parameters of the second image, and the first depth map corresponding to the second image.
[0076] It should be noted that the camera extrinsic parameters of the second image specifically refer to the extrinsic parameters of the camera used to capture the second image.
[0077] Step S104: Remove outliers from the first depth map.
[0078] It should be noted that the first depth map often contains some outliers due to calculation errors. These outliers can interfere with the depth completion operation on the first depth map, so it is necessary to remove the outliers in the first depth map.
[0079] In the specific implementation step S104, valid points in the first depth map are identified, and valid points that meet the preset abnormal conditions are determined to be outliers, and outliers in the first depth map are removed.
[0080] The preset outlier condition is: the value exceeds the mean by 3 standard deviations; outliers are valid points whose values exceed the mean by 3 standard deviations.
[0081] Specifically, for all valid points in the first depth map, these valid point values are denoted as x1 to x... n If the value of one of the valid points is x i If (i is greater than or equal to 1 and less than or equal to n) satisfies the preset abnormality condition shown in formula (1), then the valid point is determined to be an outlier and needs to be removed.
[0082] In formula (1), The mean of all valid points is s, and the variance is s; the mean is calculated using formula (2). The variance s is calculated using formula (3).
[0083] Step S105: Input the first depth map with outliers removed into the pre-trained depth completion network model for depth completion to obtain the optimized second depth map.
[0084] In the specific implementation step S105, after removing outliers from the first depth map, the first depth map with outliers removed is input into the pre-trained depth completion network model for depth completion, resulting in an optimized second depth map (equivalent to the optimized first depth map).
[0085] In other words, a depth completion network model is pre-trained, and this model is used to perform depth completion on the first depth map after removing outliers, thereby obtaining an optimized second depth map.
[0086] In some embodiments, the deep completion network model includes at least: an encoder composed of a deep residual network, a decoder composed of multiple upsampling modules, a local network composed of multiple consecutive convolutional layers, and a refinement network.
[0087] More specifically, the structure of the deep completion network model is shown in Figure 2. The deep completion network model includes: concat, Encoder, Decoder, Local Network, Global Depth, Local Depth, fusion, Refined Network, and Refined Depth.
[0088] The Encoder is a deep residual network (such as ResNet34); the Decoder is a decoder consisting of 5 upsampling modules (for example only), each of which consists of convolution, pixel shuffle, and channel attention branches.
[0089] A Local Network consists of three (for example only) consecutive convolutional layers; a Refined Network uses a Cross-Stage Partial Network (CSPNet).
[0090] In addition, in Figure 2, RGB Image is the input color image, Sparse Depth is the input sparse depth map, concat is the channel stitching operation, Global Depth is the globally optimized depth map, Local Depth is the locally optimized depth map, fusion is the fusion operation, and Refined Depth is the refined depth map.
[0091] During the training of the deep completion network model, the loss function used consists of three parts: L1 loss (absolute loss), RMAE loss (root mean square absolute error loss), and gradient loss. In order to ensure the training stability of the deep completion network model, the L1 loss of the global depth, local depth, and re-refined depth are calculated simultaneously during training. After the deep completion network model is trained and stabilized, only the L1 loss of the refining depth is calculated to obtain the optimal result.
[0092] Assuming that in the true depth map B, all points greater than 0.001 are considered valid points, and these valid points constitute a point set P, which contains N valid points, then the calculation formulas for the L1 loss, RMAE loss and gradient loss of the predicted depth map A and the true depth map B are as shown in formulas (4) to (6).
[0093] In formulas (4)-(6), L1 loss (A,B) represents the L1 loss, RMAE loss (A,B) represents the RMAE loss, Grandient loss (A,B) represents the gradient loss, where i and j are the coordinates of the pixel. i,j To predict the value at point (i,j) on depth map A, B i,j Let (i,j) be the value of the point (i,j) on the true depth map B;
[0094] Conv represents convolution computation, and kx represents convolution computation that calculates the gradient along the x-axis. For example, if kx is specified as... ky represents the convolution that calculates the gradient along the y-axis. For example, if ky is specified as...
[0095] Based on the above formulas (4) to (6), the total training loss LOSS during the training of the deep completion network model is calculated by formula (7).
[0096] In formula (7), epoch is the current training round number, Epochs is the total number of training rounds, and Depth is the depth. Global For the global depth map, Depth Local This is a local depth map. Refined To refine the depth map (i.e., predict the depth map), Depth Truth This is a true depth map.
[0097] The above is an explanation of the deep completion network model.
[0098] Step S106: Based on the depth information of the second depth map, camera intrinsic parameters, camera extrinsic parameters, and sparse point cloud, construct the final 3D model of the object to be reconstructed.
[0099] In the specific implementation step S106, after obtaining the second depth map through the depth completion network model, the final three-dimensional model of the object to be reconstructed is constructed based on the depth information of the second depth map, camera intrinsic parameters, camera extrinsic parameters of the second image, and sparse point cloud.
[0100] In this embodiment of the invention, after calculating the first depth map, outliers in the first depth map are removed, and then the depth completion network model is used to perform depth completion on the first depth map after removing outliers to obtain a more complete second depth map. The second depth map is then used to construct the final three-dimensional model of the object to be reconstructed, thereby improving the accuracy of dense reconstruction and the effect of three-dimensional reconstruction.
[0101] Regarding the calculation of the sparse point cloud and the first depth map involved in step S103 of Figure 1 in the above embodiment of the present invention, referring to Figure 3, a flowchart of the calculation of the sparse point cloud and the first depth map provided by the embodiment of the present invention is shown, including the following steps:
[0102] Step S301: Divide the second images contained in the second image set into multiple image groups.
[0103] In the specific implementation step S301, feature extraction operators are used to extract the image features of the second images contained in the second image set. The feature extraction operators used are any one of the following: Scale-Invariant Feature Transform (SIFT), Domain-size pooling in local descriptors (DSPSIFT for short), and Accelerated KAZE algorithm; where KAZE can be called Kaze Features.
[0104] Based on the extracted image features of the second image, the second image is matched to divide the second image into multiple image groups, that is, the second image set is divided into multiple image groups;
[0105] Specifically, an image matching algorithm is used to match the second image, thereby dividing the second image into multiple image groups. The image matching algorithm used is any one of the following: Vocabulary Tree matching, Sequential matching, Exhaustive matching, and Frustum matching.
[0106] Step S302: Utilize the similarity between feature points of each second image in the image group to obtain multiple sets of successfully matched feature points.
[0107] In the specific implementation step S302, the similarity between feature points of each second image in the image group is calculated, and then multiple sets of successfully matched feature points are obtained according to the similarity between feature points. Among them, "successfully matched feature points" refers to feature points whose distance in the feature vector space is less than a threshold.
[0108] Specifically, the similarity between feature points of each second image in the image group is calculated by vector dot product, and then the RANSAC algorithm (Random Sample Consensus Algorithm) is used to filter out mismatched feature points, thereby obtaining multiple sets of successfully matched feature points.
[0109] Step S303: Based on the camera intrinsic parameters, the second image set, the image group, and multiple successfully matched feature points, calculate the camera extrinsic parameters of the sparse point cloud and the second image.
[0110] In the specific implementation step S303, based on the camera intrinsic parameters, the second image set, the image group, and multiple sets of successfully matched feature points, sparse reconstruction is performed using the SFM algorithm (or Global SFM algorithm) to calculate the sparse point cloud and the camera extrinsic parameters of the second image.
[0111] Among them, camera extrinsic parameters include, but are not limited to, camera pose (including translation and rotation parameters), SFM algorithm is a structure recovery algorithm from motion, and Global SFM algorithm is a global structure recovery algorithm from motion.
[0112] It should be noted that during sparse reconstruction using the SFM algorithm, camera intrinsic parameters are used to determine the position of feature points in 3D space. The SFM algorithm uses bundle adjustment to optimize the camera's extrinsic parameters (position and pose) and the position of 3D points. In this optimization process, camera intrinsic parameters are used as known constraints. Accurate camera intrinsic parameters can help the optimization algorithm converge to a better solution more quickly, thereby improving the efficiency and accuracy of the sparse reconstruction process.
[0113] Step S304: Calculate the first depth map corresponding to the second image based on the camera intrinsic parameters, camera extrinsic parameters, and sparse point cloud.
[0114] In the specific implementation step S304, based on the camera intrinsic parameters, camera extrinsic parameters, and sparse point cloud, a semi-global stereo matching algorithm (SGM algorithm) or a local stereo matching algorithm (SAD algorithm) is used to calculate the first depth map corresponding to the second image. A corresponding first depth map is calculated for each second image.
[0115] The above is an explanation of how to calculate sparse point clouds and first depth maps.
[0116] Regarding the construction of the final three-dimensional model of the object to be reconstructed involved in step S106 of Figure 1 in the above embodiment of the present invention, referring to Figure 4, a flowchart of the construction of the final three-dimensional model provided by the embodiment of the present invention is shown, including the following steps:
[0117] Step S401: Calculate the dense point cloud based on the depth information of the second depth map, camera intrinsic parameters, camera extrinsic parameters, and sparse point cloud.
[0118] It should be noted that the depth information in the second depth map is the distance between the pixel and the camera in physical space.
[0119] In the specific implementation step S401, dense reconstruction is performed based on the depth information of the second depth map, camera intrinsic parameters, camera extrinsic parameters and sparse point cloud to calculate dense point cloud.
[0120] Step S402: Mesh the dense point cloud to obtain the initial 3D model of the object to be reconstructed.
[0121] In the specific implementation step S402, the calculated dense point cloud is meshed to obtain the initial three-dimensional model (mesh model) of the object to be reconstructed.
[0122] Specifically, algorithms such as Poisson reconstruction, growth reconstruction, or greedy projection triangulation are used to mesh dense point clouds, thereby obtaining the initial 3D model of the object to be reconstructed.
[0123] Step S403: Using the mapping relationship between points on the second image and points in space, map the RGB pixels of the second image to the texture space to obtain the texture map of the initial 3D model.
[0124] It should be noted that the initial 3D model of the object to be reconstructed is a colorless 3D model. Therefore, it is necessary to combine the second image to generate a texture map to apply texture mapping to the initial 3D model.
[0125] In the specific implementation step S403, the mapping relationship between points on the second image and points in space is used to map the RGB pixels of the second image to the texture space, thereby obtaining the texture map of the initial three-dimensional model.
[0126] Step S404: Map the texture map onto the initial 3D model to obtain the final 3D model.
[0127] In the specific implementation step S404, after obtaining the texture map of the initial three-dimensional model, the texture map is mapped onto the initial three-dimensional model to color the initial three-dimensional model (equivalent to applying texture mapping to the initial three-dimensional model), thereby obtaining the final three-dimensional model of the object to be reconstructed (a three-dimensional model with color).
[0128] The above is an explanation of how to construct the final 3D model of the object to be reconstructed.
[0129] As can be seen from the above embodiments, this solution reduces the interference of outliers on depth completion and dense reconstruction by removing outliers from the depth map, performs depth completion on the depth map, thereby improving the integrity and accuracy of the depth map, and thus ensuring the accuracy and stability of the 3D reconstruction results. This makes the final 3D model obtained from the reconstruction more accurate and provides a high-quality 3D reconstruction solution for practical applications.
[0130] Corresponding to the three-dimensional reconstruction method provided in the above embodiments of the present invention, referring to FIG5, the present invention also provides a structural block diagram of a three-dimensional reconstruction system, which includes: an acquisition unit 100, a correction unit 200, a calculation unit 300, a removal unit 400, a depth completion unit 500, and a construction unit 600;
[0131] The acquisition unit 100 is used to acquire distortion coefficients and camera intrinsic parameters of the camera using the first image set of the object to be reconstructed acquired by the camera.
[0132] In a specific implementation, the acquisition unit 100 is specifically used to: read the image metadata of the first image contained in the first image set of the object to be reconstructed acquired by the camera; and obtain the distortion coefficients and camera intrinsic parameters from the image metadata.
[0133] The correction unit 200 is used to correct the distortion of the first image contained in the first image set by means of the distortion coefficient, so as to obtain the second image set.
[0134] The computing unit 300 is used to compute sparse point clouds and to compute camera extrinsic parameters and a first depth map of the second images contained in the second image set.
[0135] The removal unit 400 is used to remove outliers from the first depth map.
[0136] The depth completion unit 500 is used to input the first depth map with outliers removed into the pre-trained depth completion network model for depth completion, so as to obtain an optimized second depth map.
[0137] In some embodiments, the deep completion network model includes at least: an encoder composed of a deep residual network, a decoder composed of multiple upsampling modules, a local network composed of multiple consecutive convolutional layers, and a refinement network.
[0138] The construction unit 600 is used to construct the final 3D model of the object to be reconstructed based on the depth information of the second depth map, camera intrinsic parameters, camera extrinsic parameters, and sparse point cloud.
[0139] In this embodiment of the invention, after calculating the first depth map, outliers in the first depth map are removed, and then the depth completion network model is used to perform depth completion on the first depth map after removing outliers to obtain a more complete second depth map. The second depth map is then used to construct the final three-dimensional model of the object to be reconstructed, thereby improving the accuracy of dense reconstruction and the effect of three-dimensional reconstruction.
[0140] Preferably, as shown in Figure 5, the calculation unit 300 includes a division subunit, an acquisition subunit, a first calculation subunit, and a second calculation subunit. The execution principle of each subunit is as follows:
[0141] The sub-unit division is used to divide the second images contained in the second image set into multiple image groups.
[0142] The sub-unit is used to obtain multiple sets of successfully matched feature points by utilizing the similarity between feature points of each second image in the image group.
[0143] The first computational subunit is used to calculate the sparse point cloud and the camera extrinsic parameters of the second image based on the camera intrinsic parameters, the second image set, the image group, and multiple sets of successfully matched feature points.
[0144] The second computational subunit is used to calculate the first depth map corresponding to the second image based on the camera intrinsic parameters, camera extrinsic parameters, and sparse point cloud.
[0145] Preferably, as shown in Figure 5, the construction unit 600 includes a computation sub-unit, a meshing sub-unit, a first mapping sub-unit, and a second mapping sub-unit. The execution principle of each sub-unit is as follows:
[0146] The computational subunit is used to compute dense point clouds based on the depth information of the second depth map, camera intrinsic parameters, camera extrinsic parameters, and sparse point clouds.
[0147] Meshable sub-units are used to mesh dense point clouds to obtain an initial 3D model of the object to be reconstructed.
[0148] The first mapping subunit is used to map the RGB pixels of the second image to the texture space by utilizing the mapping relationship between points on the second image and points in space, so as to obtain the texture map of the initial three-dimensional model.
[0149] The second mapping subunit is used to map the texture map onto the initial 3D model to obtain the final 3D model.
[0150] Preferably, referring to the content shown in Figure 5, the removal unit 400 includes an identification sub-unit, a determination sub-unit, and a removal sub-unit. The execution principle of each sub-unit is as follows:
[0151] The identification sub-unit is used to identify valid points in the first depth map.
[0152] The sub-unit is determined to identify valid points that meet preset abnormal conditions as outliers.
[0153] Remove sub-cells to remove outliers from the first depth map.
[0154] In summary, the embodiments of the present invention provide a three-dimensional reconstruction method and system. After calculating the first depth map, outliers in the first depth map are removed. Then, a depth completion network model is used to perform depth completion on the first depth map after removing outliers to obtain a more complete second depth map. The second depth map is then used to construct the final three-dimensional model of the object to be reconstructed, thereby improving the accuracy of dense reconstruction and the effect of three-dimensional reconstruction.
[0155] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.
[0156] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.
[0157] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A three-dimensional reconstruction method, characterized by, The method includes: Using the first set of images of the object to be reconstructed acquired by the camera, the distortion coefficients and camera intrinsic parameters of the camera are obtained; The distortion of the first images in the first image set is corrected using the distortion coefficients to obtain the second image set; Calculate the sparse point cloud, and calculate the camera extrinsic parameters and the first depth map of the second image contained in the second image set; Outlier removal is performed on the first depth map; The first depth map, after removing outliers, is input into a pre-trained depth completion network model for depth completion, resulting in an optimized second depth map. Based on the depth information of the second depth map, the camera intrinsic parameters, the camera extrinsic parameters, and the sparse point cloud, the final 3D model of the object to be reconstructed is constructed.
2. The method of claim 1, wherein, Calculating the sparse point cloud, and calculating the camera extrinsic parameters and the first depth map of the second image contained in the second image set, including: The second image set contains the second images, which are then divided into multiple image groups. By utilizing the similarity between feature points of each second image in the image group, multiple sets of successfully matched feature points are obtained; Based on the camera intrinsic parameters, the second image set, the image group, and the multiple sets of successfully matched feature points, calculate the sparse point cloud and the camera extrinsic parameters of the second image; The first depth map corresponding to the second image is calculated based on the camera intrinsic parameters, the camera extrinsic parameters, and the sparse point cloud.
3. The method of claim 1, wherein, Based on the depth information of the second depth map, the camera intrinsic parameters, the camera extrinsic parameters, and the sparse point cloud, the final 3D model of the object to be reconstructed is constructed, including: Calculate the dense point cloud based on the depth information of the second depth map, the camera intrinsic parameters, the camera extrinsic parameters, and the sparse point cloud; The dense point cloud is meshed to obtain an initial 3D model of the object to be reconstructed; By utilizing the mapping relationship between points on the second image and points in space, the RGB pixels of the second image are mapped to the texture space to obtain the texture map of the initial three-dimensional model; The texture map is mapped onto the initial 3D model to obtain the final 3D model.
4. The method of claim 1, wherein, Outlier removal from the first depth map includes: Identify valid points in the first depth map; The valid points that meet the preset anomaly conditions are identified as outliers; Remove the outliers from the first depth map.
5. The method of claim 1, wherein, Using a first set of images of the object to be reconstructed acquired by a camera, distortion coefficients and camera intrinsic parameters of the camera are obtained, including: Read the image metadata of the first image contained in the first image set of the object to be reconstructed, which was acquired by the camera; Obtain the distortion coefficients and camera intrinsics of the camera from the image metadata.
6. The method according to any one of claims 1 to 5, characterized in that, The deep completion network model includes at least: an encoder composed of a deep residual network, a decoder composed of multiple upsampling modules, a local network composed of multiple consecutive convolutional layers, and a refinement network.
7. A three-dimensional reconstruction system, characterized by The system includes: The acquisition unit is used to acquire distortion coefficients and camera intrinsic parameters of the camera using a first set of images of the object to be reconstructed acquired by the camera. A correction unit is used to perform distortion correction on the first images contained in the first image set using the distortion coefficients to obtain a second image set. The computing unit is used to compute sparse point clouds and compute camera extrinsic parameters and a first depth map of the second images contained in the second image set. The removal unit is used to remove outliers from the first depth map; The depth completion unit is used to input the first depth map (with outliers removed) into a pre-trained depth completion network model to perform depth completion, thereby obtaining an optimized second depth map. The construction unit is used to construct the final 3D model of the object to be reconstructed based on the depth information of the second depth map, the camera intrinsic parameters, the camera extrinsic parameters, and the sparse point cloud.
8. The system of claim 7, wherein, The computing unit includes: The sub-unit is used to divide the second images contained in the second image set into multiple image groups; The sub-unit is used to obtain multiple sets of successfully matched feature points by utilizing the similarity between feature points of each second image in the image group. The first calculation subunit is used to calculate the sparse point cloud and the camera extrinsic parameters of the second image based on the camera intrinsic parameters, the second image set, the image group, and the multiple sets of successfully matched feature points; The second calculation subunit is used to calculate the first depth map corresponding to the second image based on the camera intrinsic parameters, the camera extrinsic parameters, and the sparse point cloud.
9. The system according to claim 8, characterized in that, The building unit includes: The calculation subunit is used to calculate the dense point cloud based on the depth information of the second depth map, the camera intrinsic parameters, the camera extrinsic parameters, and the sparse point cloud; Meshable sub-units are used to mesh the dense point cloud to obtain an initial three-dimensional model of the object to be reconstructed; The first mapping subunit is used to map the RGB pixels of the second image to the texture space by utilizing the mapping relationship between points on the second image and points in space, so as to obtain the texture map of the initial three-dimensional model. The second mapping subunit is used to map the texture map onto the initial 3D model to obtain the final 3D model.
10. The system according to claim 7, characterized in that, The removal unit includes: An identification subunit is used to identify valid points in the first depth map; A sub-unit is defined to identify the valid points that meet the preset abnormal conditions as outliers; The removal sub-unit is used to remove the outliers in the first depth map.