Three-dimensional modeling system and method using online multi-view stereo

Abstract

The present invention relates to a three-dimensional modeling technique, and more specifically to a three-dimensional modeling system and method using online multi-view stereo, in which a high-precision depth map is estimated at a high resolution, and accurate mapping utilizing strong geometric priors is implemented so as to reconstruct and render a high-quality real-time three-dimensional model. According to one embodiment of the present invention, a high-precision depth map and confidence map are generated through multi-view stereo and a depth fusion network, thereby enhancing the accuracy and quality of a 3D model.

Description

3D modeling system and method using online multi-view stereo

[0001] The present invention relates to 3D modeling technology, and more specifically, to a 3D modeling system and method using online multi-view stereo that estimates a high-precision depth map at high resolution, realizes accurate mapping using powerful geometric prior information, and reconstructs and renders a high-quality real-time 3D model.

[0002] Precise 3D models are essential in various fields, including digital twins, augmented and virtual reality, industrial design, architectural visualization, and robotic airports. Generating these 3D models requires 3D reconstruction technology, and one of the most widely used methods is Multi-View Stereo (MVS). MVS generates highly precise 3D models by identifying detailed correspondences between images captured from multiple viewpoints. Recently, MVS has been combined with Neural Rendering techniques and is also being utilized in Novel View Synthesis. Neural Rendering can realistically reproduce complex scenes through deep learning technology, and in particular, the 3D Gaussian Splatting (3DGS) technique enables real-time and high-quality rendering. 3DGS utilizes a Gaussian-based particle system to provide excellent detail and real-time rendering performance, even in complex scenes.

[0003] However, existing multi-view stereo algorithms are designed to operate on a linear basis and typically require a significant amount of time to process 3D models in batches. This is a major factor limiting the use of multi-view stereo in fields requiring real-time processing, such as robotics or real-time graphics applications. For example, autonomous vehicles or real-time AR / VR systems require capturing scenes in real time and generating 3D models immediately, but existing multi-view stereo methods are not suitable for meeting these requirements. To address this problem, the Dense SLAM (Simultaneous Localization and Mapping) method is used. Dense SLAM aims for 3D reconstruction in online environments and estimates depth maps by applying multi-view stereo to a continuous sequence of images within a local time window. This allows for the simultaneous estimation of camera positions in real time and the generation of 3D models of the surrounding environment. Recent research has adopted the map representation method of Dense SLAM for techniques such as Neural Rediance Fields (NeRF) or 3DGS, enabling more effective real-time 3D modeling, rendering, and viewpoint synthesis.

[0004] However, existing methods focus primarily on estimating coarse 3D scenes for faster computation, which limits their ability to achieve precise 3D reconstruction. To ensure real-time computational performance, most methods utilize downsampled images or lightweight networks, resulting in a significant degradation of the quality of the generated 3D models. Furthermore, depth information estimated based on images can be inaccurate due to factors such as motion blur, occlusion, and areas lacking texture; this leads to reduced reliability of the depth data and causes noisy reconstructions. Therefore, a new approach is required to estimate high-resolution yet accurate depth maps and realize high-quality mapping by utilizing robust geometric prior information.

[0005] The present invention provides a 3D modeling system and method using online multi-view stereo that can generate and render high-quality 3D models in real time by utilizing image information from a camera.

[0006] The technical problems that the present invention aims to solve are not limited to those mentioned above, and other unmentioned technical problems will be clearly understood by those skilled in the art to which the present invention belongs from the description below.

[0007] According to one aspect of the present invention, a three-dimensional modeling system using online multi-view stereo is provided.

[0008] A 3D modeling system using online multi-view stereo according to one embodiment of the present invention includes a first execution module for estimating a depth map and a confidence map and a second execution module for generating a 3D model, and the first execution module may include a camera estimation unit that tracks the camera pose of an input keyframe, a depth estimation unit that performs depth estimation based on multi-view stereo (MVS), and a filtering unit that refines depth values ​​and filters noise.

[0009] According to another aspect of the present invention, a three-dimensional modeling method using online multi-view stereo and a computer program for executing the same are provided.

[0010] A three-dimensional modeling method using online multi-view stereo according to one embodiment of the present invention and a computer program for executing the same may include, in the front end, a step of estimating the camera pose of an input keyframe, a step of calculating a depth map and a confidence map based on the keyframe, a step of storing the depth map and the confidence map in a keyframe buffer, and a step of refining the depth map and filtering out noise, and in the back end, a step of performing adaptive density control to dynamically adjust the Gaussian point density, a step of sequentially integrating the depth map and the confidence map to create a three-dimensional model, and a step of iteratively optimizing the Gaussian parameters to optimize the three-dimensional model.

[0011] According to one embodiment of the present invention, a high-precision depth map and a reliability map are generated through multi-view stereo and a depth fusion network to improve the accuracy and quality of a 3D model.

[0012] According to one embodiment of the present invention, real-time processing is possible by introducing a structure that processes the front-end and back-end in parallel.

[0013] In addition, according to one embodiment of the present invention, reliable 3D reconstruction is supported even in situations where dynamic scenes must be processed in real time, and it operates efficiently even in large-scale scenes.

[0014] The effects of the present invention are not limited to the effects described above, and should be understood to include all effects that can be inferred from the composition of the invention described in the description or claims of the present invention.

[0015] FIGS. 1 to 3 are drawings illustrating a three-dimensional modeling system using online multi-view stereo according to an embodiment of the present invention.

[0016] FIG. 4 is a diagram illustrating a 3D modeling method using online multi-view stereo according to an embodiment of the present invention.

[0017] FIGS. 5 to 7 are examples of experiments performed by a 3D modeling system using online multi-view stereo according to an embodiment of the present invention.

[0018] FIG. 8 is a diagram illustrating a computing device implementing a three-dimensional modeling system using online multi-view stereo according to an embodiment of the present invention.

[0019] The present invention is susceptible to various modifications and may have various embodiments. Specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, and it should be understood that it includes all modifications, equivalents, and substitutions that fall within the spirit and scope of the invention. In describing the present invention, detailed descriptions of related prior art are omitted if it is determined that such detailed descriptions would unnecessarily obscure the essence of the invention. Furthermore, singular expressions used in this specification and claims should generally be interpreted as meaning "one or more" unless otherwise stated.

[0020] In this specification, the term "module" includes a unit composed of hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be a component formed integrally, or a minimum unit or part thereof that performs one or more functions. For example, a module may be composed of an Application-Specific Integrated Circuit (ASIC).

[0021] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In describing with reference to the accompanying drawings, identical or corresponding components are given the same reference numerals, and redundant descriptions thereof will be omitted.

[0022]

[0023] FIGS. 1 and 2 are drawings for explaining a three-dimensional modeling system using online multi-view stereo according to an embodiment of the present invention.

[0024] Referring to FIG. 1, a 3D modeling system (10) using online multi-view stereo can perform real-time 3D rendering by configuring the first execution module (100) and the second execution module (200) in parallel as independent threads.

[0025] The first execution module (100) is a module executed in the frontend and is executed in parallel as an independent thread from the second execution module (200).

[0026] The first execution module (100) is a module executed in the frontend that tracks camera poses using SLAM and estimates depth maps through multi-view stereo (MVS). More specifically, the first execution module (100) tracks the position of the camera in the input sequence of images using image-based SLAM. When a new keyframe is detected, the first execution module (100) performs multi-view stereo (MVS)-based depth estimation based on the frame and stores the generated depth map and confidence map in the keyframe buffer.

[0027] Referring to FIG. 2, the first execution module (100) includes a camera estimation unit (110), a depth estimation unit (120), and a filtering unit (130).

[0028] The camera estimation unit (110) estimates the camera pose of an input keyframe using SLAM. For example, the camera estimation unit (110) predicts high-density pixel correspondence between adjacent frames based on DROID_SLAM utilizing an optical flow network, and fine-tunes the camera pose value by continuously performing high-density bundle adjustment between keyframes.

[0029] When a new keyframe is detected, the depth estimation unit (120) performs multi-view stereo (MVS) based depth estimation based on the keyframe. More specifically, the depth estimation unit (130) performs a given keyframe F k Keyframe F based on k Use as a reference frame, and its neighboring keyframes ( Using ) as the source frame, the depth map (D k ) and confidence map(C k ) estimates. The depth estimation unit (130) estimates the source frame from a given keyframe (F k Neighboring consecutive keyframes before and after ( Decide as ).

[0030] The depth estimation unit (120) estimates depth using MVS-Former, a deep learning-based multi-view stereo (MVS) method. MVSFormer can generate a sophisticated depth map by using hierarchical vision transformers to extract multi-scale visual features at each layer.

[0031] The depth estimation unit (120) uses a Cascade Cost-Volume-based multi-view stereo (MVS) method. The depth estimation unit (120) significantly reduces GPU memory usage and execution time by accumulating Cost Volume step by step through a cascade structure, thereby enabling real-time high-resolution depth map generation and providing performance suitable for online 3D reconstruction systems.

[0032]

[0033] FIG. 3 is a diagram briefly illustrating a process for refining a depth map and filtering out outliers in a 3D modeling system using online multi-view stereo according to an embodiment of the present invention.

[0034] FIGS. 3(b) and FIGS. 3(c) are depth maps and confidence maps using multi-view stereo, FIGS. 3(d) and FIGS. 3(e) are depth maps and confidence maps refined using V-Fuse, FIG. 3(f) is a depth map with noise (outliers) filtered by verifying the geometric consistency of depth, and FIG. 3(g) is a point cloud generated by transforming depth values.

[0035] The filtering unit (130) can solve the problem of the accuracy of the depth map being lower than that of offline multi-view stereo (MVS) and the problem of a lot of noise occurring due to the characteristics of real-time multi-view stereo (MVS).

[0036] The filtering unit (130) sequentially uses a depth fusion network based on the estimated depth and reliability information to refine the depth values ​​and filter out noise (outlier values).

[0037] The filtering unit (130) is a reference keyframe F k Depth map of (D k ) and confidence map(C k ), and adjacent keyframes( ) depth map( )) and confidence map( Based on )) and integrating this, F k Refined depth map for ( ) and confidence map( ...is generated. To explain in detail, the filtering unit (130) calculates the mean and standard deviation based on the input depth map (Fig. 3(b)) and confidence map (Fig. 3(c)) to set the depth hypothesis range for each pixel, and then analyzes the visibility constraints to construct a visibility constraint volume. The filtering unit (130) normalizes the constructed visibility constraint volume through a 3D-CNN (Convolutional Neural Network) and finally generates a refined depth map (Fig. 3(d)) through a regression process.

[0038] The filtering unit (130) is a refined depth map ( In ), the confidence map( Filter out low-confidence depth values ​​by applying a threshold value to ).

[0039] The filtering unit (130) is a reference keyframe (F k ) depth map(D k ) and adjacent keyframes( ) depth map ( The geometric consistency of ) is examined to further remove depth values ​​with low consistency. To explain in detail, the filtering unit (130) each adjacent keyframe ( For ), depth map(D k ) adjacent keyframes ( It is adjusted to match the view of ). The filtering unit (130) adjusts the adjusted depth map (D k ) and adjacent keyframes( ) depth map( The filtering unit (130) calculates the difference in relative depth values ​​between the views. If the difference in relative depth values ​​is smaller than a defined threshold, the depth value is considered geometrically consistent, and depth values ​​that do not show consistency in at least three views are filtered out.

[0040] Referring again to FIG. 1, the second execution module (200) sequentially integrates the generated depth map and confidence map to progressively generate and optimize a 3D Gaussian Splatting (3DGS) model.

[0041] The second execution module (200) identifies unexplored areas of the depth map during the integration process, converts the depth values ​​of the areas into a point cloud, initializes them into new Gaussian points, and adds them to the 3DGS model.

[0042] The second execution module (200) performs the task of continuously optimizing Gaussian parameters in parallel with the frontend first execution module (100) to gradually update the model.

[0043] Referring to FIG. 2, the second execution module (200) includes a density control unit (210) and an optimization unit (220).

[0044] The density control unit (210) optimizes the distribution of Gaussian points through adaptive density control. More specifically, the density control unit (210) dynamically adjusts the point density according to the level of detail required in various areas of the scene, thereby allocating more Gaussian points to high-frequency detail areas and fewer points to low-frequency areas, thereby efficiently utilizing computational resources. Originally, 3D Gaussian splatting (3DGS) is inefficient for real-time mapping and rendering, especially in large-scale scenes, because a large number of points are required to capture high-frequency detail information. However, the second execution module (200) uses a generalized exponential splatting (GES) method, which is an extension of the generalized exponential function (GEF), to solve this problem.

[0045] The density control unit (210) performs more adaptive and efficient rendering by processing shape parameters that control the sharpness of the splat into trainable elements in various frequency bands. In this way, the second execution module (200) can effectively process high-frequency components that caused difficulties for the original 3DGS approach. For example, the second execution module (200) can produce a more natural and realistic image by allowing closer Gaussian points to have more influence on the final color and farther Gaussian points to blend into the background.

[0046]

[0047] The optimization unit (220) continuously optimizes the Gaussian parameters. To be more specific, the optimization unit (220) iteratively updates the Gaussian parameters of the 3DGS map using Differentiable Rendering and Gradient-based Optimization.

[0048] The optimization unit (220) performs optimization in each iteration process by using the current keyframe and previously input keyframes together. Since the calculation time is limited, the second execution module (200) randomly selects some keyframes rather than using all input keyframes. The optimization unit (220) can perform optimization through iteration whenever a new frame is input. For half of the total iterations, the optimization unit (220) uses frames selected within a specific time window based on the current frame, and for the remaining half of the iterations, it uses frames selected from the entire set of keyframes. In this way, the second execution module (200) mitigates the 'Forgetting Problem,' where the influence of initial input frames gradually decreases over time.

[0049]

[0050] FIG. 4 is a diagram illustrating a three-dimensional modeling method using online multi-view stereo according to an embodiment of the present invention. Each process described below is a process performed by each functional part constituting the three-dimensional modeling system using online multi-view stereo in each step; however, for the concise and clear explanation of the present invention, the subject of each step will be collectively referred to as the three-dimensional modeling system using online multi-view stereo.

[0051] Referring to FIG. 4, steps S410 to S440 are performed in the front end, and steps S450 to S470 are performed in the back end. The 3D modeling system (10) using online multi-view stereo can perform real-time 3D rendering by configuring the front end and back end in parallel as independent threads.

[0052] In step S410, the 3D modeling system (10) using online multi-view stereo estimates the camera pose of the input keyframe. For example, the 3D modeling system (10) using online multi-view stereo predicts high-density pixel correspondence between adjacent frames based on DROID_SLAM and continuously performs high-density bundle adjustment between keyframes to fine-tune the camera pose value and precisely estimate the camera pose.

[0053] In step S420, the 3D modeling system (10) using online multi-view stereo estimates a depth map and a confidence map based on the input keyframe. The 3D modeling system (10) using online multi-view stereo estimates the input keyframe F k Keyframe F based on k Use as the reference frame, and use the source frame to create the depth map (D k ) and confidence map(C k ) estimates. At this time, the 3D modeling system (10) using online multi-view stereo estimates the source frame as a given keyframe (F k Neighboring consecutive keyframes before and after ( It is determined as ). A 3D modeling system (10) using online multi-view stereo can perform depth estimation by utilizing MVS-Former, which is a deep learning-based multi-view stereo (MVS) method. A 3D modeling system (10) using online multi-view stereo can use a Cascade Cost-Volume-based multi-view stereo (MVS) method.

[0054] In step S430, the 3D modeling system (10) using online multi-view stereo stores the estimated depth map and confidence map in a keyframe buffer.

[0055] In step S440, the 3D modeling system (10) using online multi-view stereo refines (details) the depth map and filters out noise to correct the depth map and the reliability map. The 3D modeling system (10) using online multi-view stereo sequentially refines the depth values ​​and filters out noise (outlier values) using a Depth Fusion Network based on depth and reliability information. For example, the 3D modeling system (10) using online multi-view stereo reference keyframe F k Depth map of (D k ) and confidence map(C k ), and adjacent keyframes( ) depth map(D nbr ) and confidence map(C nbr Based on ), integrating this to F k Corrected depth map for ( ) and confidence map( Creates ).

[0056] A 3D modeling system (10) using online multi-view stereo performs the task of continuously optimizing Gaussian parameters in parallel with a front-end first execution module (100) to gradually update the model.

[0057] In step S450, the 3D modeling system (10) using online multi-view stereo performs adaptive density control that dynamically adjusts the Gaussian point density according to the level of detail required in various areas of the scene. For example, the 3D modeling system (10) using online multi-view stereo can efficiently utilize computational resources by allocating more Gaussian points to high-frequency detail areas and fewer points to low-frequency areas.

[0058] In step S460, the 3D modeling system (10) using online multi-view stereo sequentially integrates the estimated depth map and confidence map to generate a 3DGS (Gaussian Splatting) model.

[0059] In step S470, the 3D modeling system (10) using online multi-view stereo continuously optimizes Gaussian parameters. The 3D modeling system (10) using online multi-view stereo iteratively updates the Gaussian parameters of the 3DGS model using Differentiable Rendering and Gradient-based Optimization. In each iteration, the 3D modeling system (10) performs optimization of the generated 3DGS model by using the current keyframe and the previously input keyframes together.

[0060]

[0061] FIG. 5 is an example drawing of a rendering result of a 3D modeling system using online multi-view stereo according to an embodiment of the present invention.

[0062] Referring to FIG. 5, a 3D modeling system (10) using online multi-view stereo can perform precise 3D rendering using precise depth information.

[0063] Figure 5(a) is a 3DGS (Gaussian Splatting) model generated based on inaccurate depth information (right box) of the existing Dense SLAM method (Photo-SLAM), and Figure 5(b) is a rendering result.

[0064] FIG. 5(a) is a conventional 3D reconstruction model that focuses on generating a precise model based on multiple fixed viewpoints in an offline environment, but it has difficulties with real-time processing and cannot respond quickly to dynamic changes in large-scale scenes. In particular, the quality of the 3D model in conventional 3D reconstruction systems degrades as shown in FIG. 5(b) due to noise generated during the real-time camera pose tracking and depth map generation processes.

[0065] On the other hand, FIG. 5(c) is a 3DGS (Gaussian Splatting) model generated based on precise depth information (right box) according to an embodiment of the present invention, and FIG. 5(d) is a rendering result.

[0066] The present invention improves the quality of a 3D model by generating a high-precision depth map and a confidence map through Multi-View Stereo (MVS) and a Depth Fusion Network, operates efficiently even in large-scale scenes, and increases the accuracy of the 3D model by effectively filtering noise through a depth map refinement process.

[0067]

[0068] FIGS. 6 and 7 are experimental examples to illustrate the performance of a 3D modeling system using online multi-view stereo according to an embodiment of the present invention.

[0069] Referring to FIGS. 6 and FIGS. 7, it can be seen that the rendering results ( FIGS. 5(b) and FIGS. 6(c)) of the 3D modeling system (10) using online multi-view stereo are the most accurate in both indoor and outdoor environments.

[0070] A 3D modeling system (10) using online multi-view stereo can achieve higher PSNR (Peak Signal to Noise Ratio) and SSIM (Structural Similarity Index Measure) performance than conventional SLAM methods, thereby obtaining more accurate rendering results.

[0071] [Table 1] shows the rendering accuracy in the Replica RGB dataset, and [Table 2] shows the rendering accuracy in the TUM-RGBD dataset. According to [Table 1] and [Table 2], the 3D rendering system (10) using online multi-view stereo records the highest performance in average PSNR and LPIPS (a measure of perceptual image quality), and also shows good results in average SSIM.

[0072]

[0073]

[0074]

[0075] FIG. 8 illustrates a computing device that implements a three-dimensional modeling system using online multi-view stereo according to an embodiment of the present invention.

[0076] An embodiment of the present invention described by FIGS. 1 to 7 can be implemented as a computing device (900) operated by at least one processor.

[0077] The computing device (900) may include a processor (910), memory (920), storage (930), a communication interface (940), a system interconnect (950), and a display (960).

[0078] The processor (910) includes a CPU (Central Processing Unit), an MPU (Micro Processor Unit), an MCU (Micro Controller Unit), a GPU (Graphic Processing Unit), and an APU (Application Processing Unit).

[0079] The memory (920) interacts with the processor (910) to perform the function of storing data and enabling rapid access to necessary information so that the program can be executed efficiently. The memory (920) includes at least one of a register, a cache memory, a main memory, a read-only memory, a virtual memory, and a non-volatile memory.

[0080] Storage (930) serves to permanently store and manage data. Storage preserves data even after the computing system is turned off or rebooted, and is used to store operating systems, applications, user files, etc. Storage (930) includes at least one of a hard disk drive (HDD), a solid-state drive (SSD), an optical disc, network storage, and cloud storage.

[0081] The communication interface (940) provides a path for exchanging data between various devices inside and outside the computing system. The communication interface (940) can support at least one of the following communication methods: USB (Universal Serial Bus), PCIe (Peripheral Component Interconnect Express), SATA (Serial ATA), Ethernet, Wi-Fi, Thunderbolt, and HDMI (High-Definition Multimedia Interface).

[0082] The system interconnect (950) serves to exchange data and signals between various components within the computing system. The system interconnect (950) can support at least one of a bus, point-to-point, crossbar switch, or network-on-chip (NoC) method.

[0083] The display (960) is an output device of the computing system and performs the function of providing visual information to the user.

[0084] According to the above configuration, the program according to the embodiment of the present invention is executed based on instructions executed by the processor (910) and can be stored in memory (920) or storage (930).

[0085]

[0086] The method for re-recognizing a person using a thermal image according to the embodiment of the present invention described above may be implemented in the form of program instructions that can be executed through various computer components and recorded on a computer-readable recording medium. A computer-readable recording medium may include program instructions, data files, data structures, etc., either individually or in combination. The program instructions recorded on the computer-readable recording medium may be specially designed and configured for the embodiment of the present invention, or may be known and available to a person skilled in the art of computer software. A computer-readable recording medium includes hardware configured to store and execute program instructions, such as magnetic recording media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; ROM, RAM, and flash memory. Program instructions include machine code generated by a compiler and high-level language code that can be executed on a computer using an interpreter. Hardware may be configured to operate as one or more software modules to process the method according to the present invention, and vice versa.

[0087] A method for recognizing a person using a thermal image according to an embodiment of the present invention can be executed in the form of a program instruction on an electronic device. The electronic device includes portable communication devices such as smartphones or smartpads, computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, and home appliances.

[0088] A method for recognizing a person using a thermal image according to an embodiment of the present invention may be provided as included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable recording medium or online through an application store. In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created in a storage medium such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0089] Each component, such as a module or a program, according to an embodiment of the present invention may be composed of a single or multiple sub-components, and some of these sub-components may be omitted or additional sub-components may be included. Some components (modules or programs) may be integrated into a single entity and may perform the functions performed by each corresponding component prior to integration in the same or similar manner. Operations performed by a module, program, or other component according to an embodiment of the present invention may be executed sequentially, in parallel, iteratively, or heuristically, or at least some operations may be executed in a different order, omitted, or additional operations may be added.

[0090]

[0091] Although all components constituting an embodiment of the present invention have been described above as being combined into one or operating in combination, the present invention is not necessarily limited to such embodiments. That is, within the scope of the purpose of the present invention, all components may be selectively combined in one or more ways to operate.

[0092] Although operations are depicted in a specific order in the drawings, it should not be understood that the operations must necessarily be executed in the specific order depicted or in a sequential order, or that all depicted operations must be executed to obtain the desired result. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various configurations in the embodiments described above should not be understood as necessarily required, and it should be understood that the described program components and systems can generally be integrated together into a single software product or packaged into multiple software products.

[0093] The present invention has been described above with reference to its embodiments. Those skilled in the art will understand that the present invention may be implemented in modified forms without departing from the essential characteristics of the invention. Therefore, the disclosed embodiments should be considered in an illustrative rather than a restrictive sense. The scope of the invention is defined by the claims, not by the foregoing description, and all variations within the scope of the claims should be interpreted as being included in the invention.

[0094] The scope of the present invention is defined by the claims set forth below, and all modifications or variations derived from the meaning and scope of the claims and equivalent concepts thereof should be interpreted as being included within the scope of the present invention.

[0095] The modes for carrying out the invention are described together in the best mode for carrying out the invention above.

[0096] The present invention improves the accuracy and quality of 3D models by generating high-precision depth maps and reliability maps through multi-view stereo and a depth fusion network, and has industrial applicability in various fields such as digital twins, augmented reality and virtual reality, industrial design, architectural visualization, and robotic airports.

Claims

1. In a 3D modeling system using online multi-view stereo, A first execution module for estimating depth maps and confidence maps; and It includes a second execution module that performs the creation of a three-dimensional model, and The above-mentioned first execution module is A camera estimation unit that tracks the camera pose of an input keyframe; A depth estimation unit that performs multi-view stereo (MVS) based depth estimation; and A 3D modeling system using online multi-view stereo, comprising a filtering unit that refines the depth value and filters out noise.

2. In Paragraph 1, The above first execution module and the above second execution module are A 3D modeling system using online multi-view stereo that performs real-time 3D rendering by being configured in parallel with independent threads for the frontend and backend, respectively.

3. In Paragraph 1, The above second execution module is A 3D modeling system using online multi-view stereo that sequentially integrates the depth map and confidence map to progressively generate a 3D Gaussian splatting model.

4. In Paragraph 1, The above second execution module is A 3D modeling system using online multi-view stereo that performs the task of iteratively optimizing Gaussian parameters in parallel with the first execution module.

5. In a 3D modeling method using online multi-view stereo performed by a 3D modeling system using online multi-view stereo, In the front end Step of estimating the camera pose of an input keyframe; A step of calculating a depth map and a confidence map based on the above keyframes; The step of storing the depth map and confidence map in a keyframe buffer; and The above includes the step of refining the depth map and filtering out noise, In the backend A step of performing adaptive density control to dynamically adjust Gaussian point density; A step of generating a 3D model by sequentially integrating the depth map and confidence map; and A 3D modeling method using online multi-view stereo, comprising the step of optimizing the 3D model by iteratively optimizing Gaussian parameters.

6. In Paragraph 5 The above frontend and backend are A 3D modeling method using online multi-view stereo configured in parallel with independent threads to perform real-time 3D rendering.

7. A computer program recorded on a computer-readable recording medium for executing a three-dimensional modeling method using online multi-view stereo pursuant to paragraph 5.

Images