System for enhancing three-dimensional volumetric model using diffusion model

Abstract

A system for enhancing a three-dimensional volumetric model according to one embodiment of the present invention is a system for enhancing a three-dimensional volumetric model connected to a user terminal via a network, the system comprising: a model information unit that stores a three-dimensional volumetric model and three-dimensional volumetric model data generated by modeling object images captured by a plurality of cameras using a volumetric technique; a model reconstruction unit that selects optimal data from the three-dimensional volumetric model data corresponding to the three-dimensional volumetric model and reconstructs the three-dimensional volumetric model on the basis of the selected optimal data; a model conversion unit that converts the reconstructed three-dimensional volumetric model into a mesh format; and a model enhancement unit that processes the converted three-dimensional volumetric model using a diffusion model.

Description

3D Volume Model Improvement System Using Diffusion Model

[0001] The present invention relates to a three-dimensional volume model improvement system, and more specifically, to a three-dimensional volume model improvement system using a diffusion model that removes noise and restores a three-dimensional volume model generated through volumetric technology.

[0002] Volumetric technology is a technique for collecting and representing data in three-dimensional space. It involves capturing objects or scenes from various angles using cameras (e.g., RGB-D sensors) and generating three-dimensional images based on the resulting data. While it is difficult to represent depth information in general three-dimensional images or videos, volumetric technology can create a realistic three-dimensional environment. This technology has a particular strength in maximizing immersion by allowing viewers to observe the scene while moving freely from various angles.

[0003] However, 3D images generated using volumetric technology have difficulty reproducing details such as hair, textures, and the microstructure of the skin, and small features tend to be blurred or disappear. As a result, volumetric technology produces 3D images that result in low immersion in the user experience and is unsuitable for creating high-quality content.

[0004] The technical problem that the 3D volume model improvement system according to the technical concept of the present invention aims to solve is to provide a 3D volume model improvement system that can improve the quality of a 3D volume model using volumetric technology.

[0005] In addition, the technical problem that the 3D volume model improvement system according to the technical concept of the present invention aims to achieve is to provide a 3D volume model improvement system that enables the easy production of high-quality content using a 3D volume model.

[0006] The technical problems that the three-dimensional volume model improvement system according to the technical concept of the present invention aims to solve are not limited to those mentioned above, and other problems not mentioned will be clearly understood by a person skilled in the art from the description below.

[0007] A three-dimensional volume model improvement system according to one embodiment of the technical concept of the present invention may be characterized by comprising: a model information unit that stores a three-dimensional volume model and three-dimensional volume model data resulting from modeling object images by a plurality of cameras using volumetric technology; a model reconstruction unit that selects optimal data from the three-dimensional volume model data based on the three-dimensional volume model and reconstructs the three-dimensional volume model based on the selected optimal data; a model conversion unit that converts the reconstructed three-dimensional volume model into a mesh form; and a model improvement unit that processes the converted three-dimensional volume model using a diffusion model.

[0008] In addition, the 3D volume model forms a point cloud composed of multiple points, and the 3D volume model data may include point-specific coordinate values ​​of the 3D volume model, a camera identifier corresponding to the 3D volume model, and camera coordinate values.

[0009] In addition, the model reconstruction unit divides the coordinate system for the 3D volume model into multiple voxels, includes a 3D volume model corresponding to each voxel for each camera, selects the point of the 3D volume model closest to the camera in the voxel, and can reconstruct the 3D volume model based on the 3D volume model and 3D volume model data corresponding to the selected point.

[0010] In addition, the model conversion unit can generate a Voronoi diagram from the reconstructed 3D volume model and form a triangular mesh shape by connecting the central axes of the Voronoi diagram.

[0011] In addition, the model transformation unit can generate a Voronoi diagram by selecting a set of points of the reconstructed 3D volume model, calculating the nearest region based on each point in the selected set of points, and defining the boundaries between the regions.

[0012] In addition, the model transformation unit adjusts the 3D volume model in the form of a mesh, and the mesh shape can be adjusted into an acute triangle structure through a combination of Edge Split, Edge Collapse, Edge Flip, and Vertex Shift.

[0013] In addition, the model improvement unit obtains depth images of the 3D volume model in 6 directions according to the camera viewpoint from the converted 3D volume model and inputs them into a diffusion model, specifically a latent diffusion model, converts data according to the depth images of the 3D volume model in 6 directions from latent space into latent vectors, adds noise to the latent vectors in the forward propagation step, and estimates the amount of noise included in the latent vectors and removes the noise in the backpropagation step to restore them.

[0014] In addition, the model improvement unit can combine the restored depth images to generate a resulting 3D volumetric model and transmit it to a user terminal.

[0015] In addition, the user terminal is connected to the model information unit to select a three-dimensional volume model and transmits selection information corresponding to the selected three-dimensional volume model to the model reconstruction unit, and the model reconstruction unit can reconstruct a three-dimensional volume model corresponding to the received selection information.

[0016] Additionally, the 3D volume model improvement system further includes a model inspection unit that receives inspection request information based on the result 3D volume model from a user terminal, wherein the model inspection unit searches for the image most similar to the result 3D volume model, calculates a first degree of agreement which is the degree of agreement between the result 3D volume model and the searched most similar image, calculates a second degree of agreement which is the degree of agreement between the result 3D volume model and the original object image, and calculates a third degree of agreement which is the degree of agreement between the searched most similar image and the original object image; when the first degree of agreement, the second degree of agreement, and the third degree of agreement are all greater than or equal to a set threshold, the model inspection unit generates inspection information as 'Excellent' for the result 3D volume model and transmits it to the user terminal that transmitted the inspection request information, and when at least one of the first degree of agreement, the second degree of agreement, and the third degree of agreement is lower than a set threshold, the model inspection unit generates inspection information as 'Poor' for the result 3D volume model and transmits it to the user terminal that transmitted the inspection request information. It is possible.

[0017] A three-dimensional volume model improvement system according to embodiments based on the technical concept of the present invention has the following effects.

[0018] (1) Improve the quality of the three-dimensional volumetric model using volumetric technology.

[0019] (2) High-quality content can be easily produced using a 3D volumetric model.

[0020] However, the effects that can be achieved by a three-dimensional volume model processing system according to one embodiment of the present invention are not limited to those mentioned above, and other unmentioned effects will be clearly understood by a person skilled in the art from the description below.

[0021] A brief description of each drawing is provided to help to better understand the drawings cited in this specification.

[0022] FIG. 1 is a block diagram illustrating a three-dimensional volume model improvement system according to one embodiment of the present invention.

[0023] FIG. 2 is an implementation flowchart of a three-dimensional volume model improvement system according to one embodiment of the present invention.

[0024] FIG. 3 is a diagram exemplarily illustrating the data distribution per camera inside a voxel in a three-dimensional volume model improvement system according to one embodiment of the present invention.

[0025] FIG. 4 is a diagram illustrating a method for selecting overlapping points by a camera in a three-dimensional volume model improvement system according to an embodiment of the present invention.

[0026] FIG. 5 is a diagram illustrating a neighbor voxel search and display method in a three-dimensional volume model improvement system according to an embodiment of the present invention.

[0027] FIG. 6 is a diagram exemplarily illustrating a Voronoi diagram and a triangularly partitioned mesh structure generated in a three-dimensional volumetric model improvement system according to one embodiment of the present invention.

[0028] FIG. 7 is a diagram exemplarily illustrating the adjustment of a mesh structure generated in a three-dimensional volume model improvement system according to one embodiment of the present invention.

[0029] FIG. 8 is a diagram illustrating an algorithm using a latent diffusion model in a three-dimensional volume model improvement system according to one embodiment of the present invention.

[0030] FIG. 9 is a diagram illustrating the correction of a depth image using a latent diffusion model in a three-dimensional volume model improvement system according to one embodiment of the present invention.

[0031] FIG. 10 is a diagram illustrating the structure of a mesh surface according to a depth image at a specific viewpoint of a 3D volume model in a 3D volume model improvement system according to one embodiment of the present invention.

[0032] FIG. 11 is a diagram illustrating the texture area of ​​a mesh surface according to a depth image at a specific viewpoint of a 3D volume model in a 3D volume model improvement system according to one embodiment of the present invention.

[0033] The present invention is capable of various modifications and may have various embodiments, and specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, and it should be understood that the present invention includes all modifications, equivalents, and substitutions that fall within the spirit and scope of the invention.

[0034] In describing the present invention, if it is determined that a detailed description of related prior art may unnecessarily obscure the essence of the present invention, such detailed description is omitted. Additionally, numbers used in the description of this specification (e.g., 1st, 2nd, etc.) are merely identification symbols to distinguish one component from another.

[0035] In addition, when a component is described in this specification as being "connected" or "connected" to another component, it should be understood that the component may be directly connected to or directly connected to the other component, but unless otherwise specifically stated, it may also be connected or connected through another component in between.

[0036] In addition, components expressed as '~part' in this specification may consist of two or more components combined into a single component, or a single component may be divided into two or more components according to more detailed functions. Furthermore, each component described below may additionally perform some or all of the functions performed by other components in addition to the primary function it is responsible for, and it goes without saying that some of the primary functions performed by each component may be exclusively performed by other components.

[0037] Hereinafter, embodiments based on the technical concept of the present invention will be described in detail in turn.

[0038] FIG. 1 is a block diagram illustrating a three-dimensional volume model improvement system according to one embodiment of the present invention, and FIG. 2 is an implementation flowchart of a three-dimensional volume model improvement system according to one embodiment of the present invention.

[0039] As illustrated in FIGS. 1 and 2, a three-dimensional volumetric model improvement system (100) according to one embodiment of the present invention may be connected to a user terminal (10) via a network (1) and may include a model information unit (101), a model reconstruction unit (102), a model conversion unit (103), and a model improvement unit (104). Here, the user may refer to a person who creates and manages three-dimensional characters.

[0040] The model information unit (101), model reconstruction unit (102), model conversion unit (103), and model improvement unit (104) may include at least one of a processor, memory, and data transceiver.

[0041] The user can access the three-dimensional volume model improvement system (100) using a user terminal (10) and exchange signals with the three-dimensional volume model improvement system (100). The three-dimensional volume model improvement system (100) may display a web page through a web browser running on the user terminal (10) and allow access to the three-dimensional volume model improvement system (100) after logging in on the web page, or an application capable of accessing the three-dimensional volume model improvement system (100) may be installed and run on the user terminal (10).

[0042] The user terminal (10) can be implemented as a computer capable of connecting to a remote server or terminal via a network (1). Here, the computer may include, for example, a notebook, desktop, laptop, etc. equipped with a web browser. Additionally, the user terminal (10) can be implemented as a terminal device capable of connecting to a remote server or terminal via a network (1). The terminal device may include, for example, all types of handheld-based wireless communication devices such as PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), Wibro (Wireless Broadband Internet) terminals, smartphones, smartpads, tablet PCs, etc.

[0043] Here, the network (1) refers to a connection structure capable of exchanging information between each node, such as multiple terminals and servers. Examples of such networks include, but are not limited to, a 3GPP (3rd Generation Partnership Project) network, an LTE (Long Term Evolution) network, a 5G network, a WIMAX (World Interoperability for Microwave Access) network, the Internet, a LAN (Local Area Network), a Wireless LAN (Wireless Local Area Network), a WAN (Wide Area Network), a PAN (Personal Area Network), a Bluetooth network, a satellite broadcasting network, an analog broadcasting network, and a DMB (Digital Multimedia Broadcasting) network.

[0044] The model information unit (101) can store a three-dimensional volume model and three-dimensional volume model data (S101). Here, the three-dimensional volume model refers to an object image generated by capturing an object (e.g., a person, etc.) using multiple cameras (especially RGB-D cameras) and modeling it using volumetric technology, and can form a point cloud composed of multiple points. Additionally, the three-dimensional volume model data is data for the three-dimensional volume model and may include point-by-point coordinate values ​​of the three-dimensional volume model, a camera identifier corresponding to the three-dimensional volume model, camera coordinate values, etc.

[0045] Meanwhile, the user terminal (10) can collect object images and object data from a camera and transmit them to the model information unit (101), and the model information unit (101) can generate a three-dimensional volume model and three-dimensional volume model data based on the object images. Additionally, the three-dimensional volume model can be corrected and stored in the model information unit (101).

[0046] The model reconstruction unit (102) can select optimal data from the 3D volume model data according to the 3D volume model and reconstruct the 3D volume model based on the selected optimal data (S102).

[0047] In particular, the model reconstruction unit (102) can divide the coordinate system for the three-dimensional volume model into multiple voxels. Here, the voxels can be formed in the form of small three-dimensional grids. In addition, each voxel may contain a three-dimensional volume model corresponding to the voxel for each camera and can form a point cloud (see FIG. 3).

[0048] The model reconstruction unit (102) can select the point of the 3D volume model closest to the camera in the voxel as optimal data, and can reconstruct the 3D volume model based on the 3D volume model and 3D volume model data corresponding to the selected point which is the optimal data. Here, the point of the 3D volume model closest to the camera may refer to the point with the shortest distance from the camera calculated based on the camera coordinate values ​​and the point coordinate values ​​of the 3D volume model. In addition, the distance between the camera and the point may be calculated through the Euclidean distance between the camera and the point. This is because the depth image obtained from the camera exhibits more accurate characteristics the closer it is to the camera.

[0049] Figure 4(a) shows the selection of points during the process of reconstructing a three-dimensional volume model, and Figure 4(b) shows the reconstructed three-dimensional volume model.

[0050] As described above, the model reconstruction unit (102) can reconstruct a three-dimensional volume model by grouping all point clouds within each voxel space by camera and keeping only the points corresponding to the camera closest to the voxel location.

[0051] Meanwhile, when the distance between each camera and voxel is similar or the influence of the camera corresponding to the selected point is discontinuous between voxels, the model reconstruction unit (102) can search all voxels based on a voxel-based neighbor search method and perform clustering so that the influence of each camera selected in each voxel is continuously distributed. In addition, the voxel search can be performed by repeatedly searching the face with the smallest index based on the z-axis for the voxel resolution of the z-axis, and when the voxel face to be searched is determined as shown in FIG. 5(a), 8 voxels adjacent to the current position are selected as shown in FIG. 5(b), and the maximum frequency value of the camera selected in the 8 voxels is set as the camera of the current position voxel to give continuity to each voxel.

[0052] Additionally, the user terminal (10) can be connected to the model information unit (101) to select a three-dimensional volume model and can transmit selection information corresponding to the selected three-dimensional volume model to the model reconstruction unit (102). The model reconstruction unit (102) can reconstruct a three-dimensional volume model corresponding to the received selection information.

[0053] The model conversion unit (103) can convert the reconstructed three-dimensional volume model into a mesh form (S103). Here, the model conversion unit (103) can generate the reconstructed three-dimensional volume model into a Voronoi diagram (see FIG. 6(a)) and can generate a plurality of triangular mesh shapes by connecting the central axes of the Voronoi diagram (see FIG. 6(b)).

[0054] Additionally, the model conversion unit (103) can generate a Voronoi diagram by selecting a set of points of the reconstructed three-dimensional volume model, calculating the nearest region based on each point in the selected set of points, and defining the boundaries between the regions.

[0055] Additionally, the model transformation unit (103) can adjust the mesh-shaped 3D volume model. Here, the mesh shape can be adjusted into an acute triangle structure by a combination of edge split, edge collapse, edge flip, and vertex shift (see FIG. 7). As a result, the mesh-shaped 3D volume model can be adjusted to have a mesh shape with uniform triangle angles and a uniformly aligned surface.

[0056] The model improvement unit (104) can process the converted 3D volume model using a diffusion model (S104). Here, the diffusion model is a generative probabilistic model and may refer to a machine learning model that generates a high-quality converted 3D volume model through a process of progressively denoising the converted 3D volume model and then restoring it in reverse. In this embodiment, a latent diffusion model (LDM), which is an example of a diffusion model, may be applied. Samplers such as DDPM (Denoising Diffusion Probabilistic Model) and DDIM (Denoising Diffusion Implicit Model) can support the latent diffusion model and learn the diffusion model for high-resolution image synthesis by sampling a small amount of data in the initial noise removal step and aiming to find a more suitable latent space.

[0057] In particular, as illustrated in FIG. 8, the model improvement unit (104) can acquire depth images of the 3D volume model in six directions (front, left side, back, right side, top, and bottom) according to the camera viewpoint from the converted 3D volume model and input them into a diffusion model, specifically a latent diffusion model. The inference of the latent diffusion model can be performed by training a U-Net using the depth images of each camera and using the weights of the U-Net together with the Control-Net weights trained to condition the shape of the actual object, and can transfer features of the high-quality image by assigning them to a unique identifier to train the U-Net more efficiently. In addition, the control net enables the diffusion model to reflect various input conditions and copies the weights of the diffusion model into a trainable copy and a locked copy, respectively. The trainable copy of the model learns a dataset designed to perform specific tasks based on the form of the input conditions, while the locked copy functions to preserve pre-trained existing weights.

[0058] When a depth image of a 6-directional 3D volume model is input into a diffusion model in the model improvement unit (104), the data corresponding to the depth image of the 6-directional 3D volume model can be converted from a latent space into a latent vector through a Variational Autoencoder (VAE). Here, the latent vector can have a size of 1 / 8 of the total input resolution. In addition, the converted latent vector can be conditioned through a control net.

[0059] The latent diffusion model can restore the latent vector by adding noise to it during the forward propagation step using a sampler (e.g., DDPM, DDIM), and then estimating the amount of noise included in the latent vector by a U-Net at each time step during the backward propagation step and removing the noise. As a result, the depth image can be improved and acquired with enhanced quality.

[0060] Through a latent diffusion model, rays are generated toward an object by the depth value per pixel from the enhanced depth image acquired at each time point, and the vertex position of the corresponding mesh surface can be adjusted through the average length of each ray.

[0061] The model improvement unit (104) can generate a result 3D volume model based on the improved depth image and transmit it to a user terminal (10). As a result, the user can use the improved result 3D volume model.

[0062] FIG. 9 is a set of drawings illustrating depth images at different viewpoints of a 3D volume model using a latent diffusion model in a 3D volume model improvement system according to an embodiment of the present invention. FIG. 9(a) shows depth images at four viewpoints of a volume model synthesized through a latent diffusion model. FIG. 9(b) can be seen to show the noise in the depth information shown in FIG. 9(a) and the area that could not be synthesized into a mesh shape due to this, which is restored and filled in.

[0063] FIG. 10 is a diagram illustrating the structure of a mesh surface according to a depth image at a specific viewpoint of a 3D volume model in a 3D volume model improvement system according to one embodiment of the present invention.

[0064] Figure 10 shows the structure of the improved mesh surface through viewpoint depth images of the 3D volume model improved in Figure 9. Each legend is divided into three stages by angle to clearly distinguish between the noise and the improved surface. Figure 10(a) shows the image before the application of the latent diffusion model, and Figure 10(b) shows the result after the application of the latent diffusion model. In Figure 10(b), the result of the surface curvature caused by the noise seen in Figure 10(a) being relieved can be observed.

[0065] FIG. 11 is a diagram illustrating the texture area of ​​a mesh surface according to a depth image at a viewpoint of a 3D volume model in a 3D volume model improvement system according to an embodiment of the present invention. Compared to the image before improvement in FIG. 11(a), a texture of better quality can be observed in the image after improvement in FIG. 11(b).

[0066] The 3D volume model improvement system (100) according to the present embodiment can form an improved 3D volume model by reconstructing a 3D volume model generated through volumetric technology, converting it into a mesh form, and then processing it using a diffusion model, particularly a latent diffusion model. As a result, the 3D volume model improvement system (100) according to the present embodiment can improve the quality of the 3D volume model generated by volumetric technology, thereby enabling easy production of high-quality content using the 3D volume model.

[0067] Meanwhile, the three-dimensional volume model improvement system (100) according to the present embodiment may further include a model inspection unit (105).

[0068] The model inspection unit (105) can receive inspection request information from the user terminal (10). Here, the user terminal (10) can generate inspection request information based on the received result 3D volume model and transmit it to the model inspection unit (105). Additionally, the inspection request information may include an identifier of the result 3D volume model, indicating that an inspection of the result 3D volume model is requested.

[0069] Additionally, the model inspection unit (105) can obtain a result 3D volume model corresponding to the result verification information from the model improvement unit (104) and proceed to subsequent steps.

[0070] First, the model verification unit (105) can search for the image most similar to the result 3D volume model. Here, the model verification unit (105) stores images collected through a network using a web crawling method and searches for the image most similar to the result 3D volume model among the stored images.

[0071] For example, the model verification unit (105) can extract features of a three-dimensional volume model and individually extract features of multiple collected images, and can convert the extracted features of the three-dimensional volume model and each of the extracted features of the collected images into a common vector space. Here, the three-dimensional volume model and the collected images can be converted into vectors of the same dimension. The model verification unit (105) calculates the similarity between the three-dimensional volume model and the collected images using the vectors, determines the ranking of the collected images based on the calculated similarity, and can search for the top-ranked collected image as the most similar image.

[0072] Next, the model inspection unit (105) can calculate a first degree of agreement, which is the degree of agreement between the result 3D volume model and the most similar image found.

[0073] For example, the model verification unit (105) can align the coordinate system of the image most similar to the result 3D volume model, convert the result 3D volume model into a 2D projected image, or expand the image most similar to 3D. In this state, the model verification unit (105) can analyze the texture and brightness differences between the result 3D volume model and the image most similar, calculate the ratio of the overlapping area in the common area and the distance between the data, and then calculate a first degree of agreement based on this.

[0074] Next, the model inspection unit (105) can calculate a second degree of agreement, which is the degree of agreement between the result 3D volume model and the original object image. Here, the original object image may refer to an object image corresponding to the result 3D volume model, and may be generated per camera and consist of multiple images.

[0075] For example, the model inspection unit (105) can align the coordinate systems of the resulting 3D volume model and the original object image, convert the resulting 3D volume model into a 2D projected image, or expand the original object image into 3D. In this state, the model inspection unit (105) can analyze the texture and brightness differences between the resulting 3D volume model and the original object image, calculate the ratio of the overlapping area in the common area and the distance between the data, and then calculate a second degree of agreement based on this.

[0076] Next, the model inspection unit (105) calculates a third degree of agreement, which is the degree of agreement between the most similar image found and the original object image.

[0077] The model inspection unit (105) can align the coordinate system of the most similar image found and the original object image, analyze the texture and brightness differences between the most similar image found and the original object image, calculate the ratio of the overlapping area in the common area and the distance between the data, and then calculate the third degree of agreement based on this.

[0078] When the first degree of agreement, the second degree of agreement, and the third degree of agreement are all above the set threshold, the model inspection unit (105) can generate inspection information as ‘excellent’ for the transmitted result 3D volume model and transmit it to the user terminal (10) that transmitted the inspection request information. As a result, the user can perceive that the result 3D volume model has little to no distortion or is minimized based on the original object.

[0079] Meanwhile, when at least one of the first degree of agreement, the second degree of agreement, and the third degree of agreement is lower than a set threshold, the model inspection unit (105) can generate inspection information as ‘poor’ for the transmitted result 3D volume model and transmit it to the user terminal (10) that transmitted the inspection request information. As a result, the user can perceive that the result 3D volume model has distortion based on the original object and has not sufficiently represented the original object.

[0080] In addition, when the first degree of agreement is lower than the set threshold, the second degree of agreement and the third degree of agreement may not be calculated. In addition, when the first degree of agreement is greater than or equal to the set threshold and the first degree of agreement is lower than the set threshold, the third degree of agreement may not be calculated.

[0081] As described above, the model inspection unit (105) can objectively evaluate the quality of the resulting three-dimensional volume model, particularly the degree of distortion, based on the degree of agreement, and, if necessary, provide information necessary to modify or improve the resulting three-dimensional volume model, and can be used to improve the process for improving the three-dimensional volume model in the future.

[0082] The functional operations and embodiments relating to the subject matter described above in this specification may be implemented in digital electronic circuits, computer software, firmware, or hardware, or in a combination of one or more of these, including the structures disclosed in this specification and their structural equivalents.

[0083] Embodiments of the subject matter described herein may be implemented as one or more computer program products, that is, as one or more modules relating to computer program instructions encoded on a tangible program medium for execution by a data processing device or for controlling the operation thereof. The tangible program medium may be a radio signal or a computer-readable medium. A radio signal is an artificially generated signal, such as an electrical, optical, or electromagnetic signal generated by a machine, for example, to encode information for transmission to a suitable receiver device for execution by a computer. A computer-readable medium may be a machine-readable storage device, a machine-readable storage substrate, a memory device, a combination of materials affecting a machine-readable radio signal, or a combination of one or more of these.

[0084] A computer program (also known as a program, software, software application, script, or code) may be written in any form of a programming language, including compiled or interpreted languages, or a priori or procedural languages, and may be developed in any form, including a standalone program, modules, components, subroutines, or other units suitable for use in a computer environment.

[0085] A computer program does not necessarily correspond to a file in a file system. A program may be stored within a single file provided to the requested program, within multiple interacting files (e.g., a file storing one or more modules, subprograms, or parts of code), or within a part of a file containing other programs or data (e.g., one or more scripts stored within a markup language document).

[0086] Computer programs can be deployed to be executed on multiple computers or a single computer that are located at one site or distributed across multiple sites and interconnected by a communication network.

[0087] Additionally, the logical flow and structural block diagrams described herein describe corresponding actions and / or specific methods supported by corresponding functions and steps supported by the disclosed structural means, and can also be used to construct corresponding software structures and algorithms and their equivalents.

[0088] The processes and logic flows described in this specification can be performed by one or more programmable processors that execute one or more computer programs to perform functions by operating on input data and generating outputs.

[0089] Processors suitable for the execution of computer programs include, for example, both general-purpose and special-purpose microprocessors and any one or more processors of any type of digital computer. Generally, the processor will receive instructions and data from read-only memory or random access memory or both.

[0090] The core elements of a computer are one or more memory devices for storing instructions and data, and a processor for executing instructions. Additionally, a computer will generally be combined with or include one or more mass storage devices for storing data, such as magnetic, magneto-optical, or optical discs, to be operable to receive data from, transmit data to, or perform both of these operations. However, a computer does not need to have such devices.

[0091] The description provided herein presents the best mode of the invention and offers examples to explain the invention and to enable those skilled in the art to manufacture and use the invention. The specification thus written is not intended to limit the invention to the specific terms presented.

[0092] Accordingly, although the present invention has been described in detail with reference to the examples above, those skilled in the art may make modifications, changes, and variations to these examples without departing from the scope of the present invention. In short, it is stated that in order to achieve the intended effect of the present invention, it is not necessary to separately include all functional blocks shown in the drawings or to follow all sequences shown in the drawings exactly as shown; such matters may fall within the technical scope of the present invention as described in the claims even if they are not.

[0093] [Explanation of the symbol]

[0094] 00: 3D Volume Model Improvement System

[0095] 101: Model Information Department

[0096] 102: Model Reconstruction Section

[0097] 103: Model conversion section

[0098] 104: Model Improvement Section

[0099] 105: Model Inspection Department

Claims

1. In a three-dimensional volumetric model improvement system connected to a user terminal via a network, A model information unit that stores a 3D volumetric model and 3D volumetric model data resulting from modeling object images from multiple cameras using volumetric technology; A model reconstruction unit that selects optimal data from 3D volume model data according to a 3D volume model and reconstructs the 3D volume model based on the selected optimal data; A model conversion unit that converts a reconstructed 3D volumetric model into a mesh form; and A 3D volume model improvement system characterized by including a model improvement unit that processes a converted 3D volume model using a diffusion model.

2. In Paragraph 1, The 3D volumetric model forms a point cloud composed of multiple points, and A 3D volume model improvement system characterized by 3D volume model data including point-by-point coordinate values ​​of the 3D volume model, a camera identifier corresponding to the 3D volume model, and camera coordinate values.

3. In Paragraph 2, the model reconstruction unit, The coordinate system for the 3D volumetric model is divided into multiple voxels, and the 3D volumetric model corresponding to each voxel is included for each camera, and A 3D volume model improvement system characterized by selecting the point of a 3D volume model closest to a camera in a voxel, and reconstructing the 3D volume model based on the 3D volume model and 3D volume model data corresponding to the selected point.

4. In Paragraph 3, A 3D volume model improvement system characterized by a model conversion unit generating a Voronoi diagram from a reconstructed 3D volume model and forming a triangular mesh shape by connecting the central axes of the Voronoi diagram.

5. In Paragraph 4, A 3D volumetric model improvement system characterized by a model transformation unit selecting a set of points of a reconstructed 3D volumetric model, calculating the nearest region based on each point in the selected set of points, and defining boundaries between regions to generate a Voronoi diagram.

6. In Paragraph 4, A 3D volume model improvement system characterized in that the model conversion unit adjusts a 3D volume model in the form of a mesh, wherein the mesh form is adjusted into an acute triangle structure through a combination of division, merging, flipping, and vertex rearrangement.

7. In Paragraph 1, A 3D volume model improvement system characterized by a model improvement unit acquiring depth images of a 6-directional 3D volume model according to a camera viewpoint from a converted 3D volume model and inputting them into a diffusion model, particularly a latent diffusion model, converting data according to the depth images of the 6-directional 3D volume model into latent vectors in a latent space, adding noise to the latent vectors in a forward propagation step, and estimating the amount of noise included in the latent vectors and removing the noise to restore them in a backpropagation step.

8. In Paragraph 7, A 3D volume model improvement system characterized by a model improvement unit that combines restored depth images to generate a resulting 3D volume model and transmits it to a user terminal.

9. In Paragraph 1, The user terminal is connected to the model information unit to select a three-dimensional volumetric model and transmits selection information corresponding to the selected three-dimensional volumetric model to the model reconstruction unit, and A 3D volume model improvement system characterized by a model reconstruction unit that reconstructs a 3D volume model corresponding to transmitted selection information.

10. In paragraph 8, the three-dimensional volumetric model improvement system is, It further includes a model inspection unit that receives inspection request information based on the result 3D volume model from a user terminal, The model verification unit searches for the image most similar to the resulting 3D volumetric model, calculates a first degree of agreement which is the degree of agreement between the resulting 3D volumetric model and the searched most similar image, calculates a second degree of agreement which is the degree of agreement between the resulting 3D volumetric model and the original object image, and calculates a third degree of agreement which is the degree of agreement between the searched most similar image and the original object image, and When the first, second, and third degrees of agreement are all above the set threshold, the model inspection unit generates inspection information as 'Excellent' for the resulting 3D volumetric model and transmits it to the user terminal that transmitted the inspection request information, and A 3D volume model improvement system characterized by the fact that when at least one of the first degree of agreement, the second degree of agreement, and the third degree of agreement is lower than a set threshold, the model inspection unit generates inspection information as 'poor' for the resulting 3D volume model and transmits it to the user terminal that transmitted the inspection request information.

Images