Texturing method for generating three-dimensional virtual model, and computing device therefor

Abstract

A texturing method for generating a three-dimensional virtual model, according to one embodiment of the present disclosure, is performed in a computing device that generates a three-dimensional virtual model for a real space on the basis of a plurality of data sets respectively generated at a plurality of imaging points of the real space, and comprises the steps of: generating a three-dimensional mesh model for the real space by using the plurality of data sets; selecting a texturing image from the plurality of data sets on the basis of the presence of a non-target object, and texturing the three-dimensional mesh model by using the selected texturing image; and providing the textured three-dimensional virtual model to a user by means of a user interface.

Description

Texturing method for generating a 3D virtual model and computing device for the same

[0001] The present invention relates to a texturing method for generating a three-dimensional virtual model of real space and a computing device for the same.

[0002] Recently, virtual space implementation technologies are being developed that provide an online virtual space corresponding to the actual real world, enabling users to experience being in that space without physically visiting it. Various developments are underway for these real-world-based virtual technologies, which serve as tools for implementing digital twins.

[0003] In order to implement a virtual space based on real space, it is necessary to acquire images taken of the actual real space to be implemented, and based on this, generate a three-dimensional virtual image, that is, a 3D virtual model, to provide the virtual space.

[0004] These three-dimensional virtual models are generated based on data captured at various points in the real world. For example, to construct a three-dimensional virtual model, data can be collected from various points in the real world and a three-dimensional virtual model can be generated based on this.

[0005] However, when collecting data on such real-world spaces, there is a problem in that unwanted objects are reflected. For example, when collecting data on the interior of a factory facility, if people passing by are filmed, these people are reflected in the images, and the human figures are textured onto the 3D virtual model.

[0006] One technical aspect of the present application is intended to solve the aforementioned problem and to provide a texturing technology for generating a 3D virtual model that can prevent non-target objects, which are unintended subject objects, from being textured when generating a 3D virtual space corresponding to real space based on a shooting data set collected from multiple shooting points in real space.

[0007] One technical aspect of the present application is to provide a texturing technology for generating a 3D virtual model that can efficiently select a texturing image without non-target objects with a small amount of computation by identifying non-target objects for object identification of an image and setting color data for the object area of ​​the non-target objects by replacing padding values.

[0008] One technical aspect of the present application is to provide a texturing technology for generating a 3D virtual model that can efficiently select an optimal texturing image even when there are multiple non-target objects by setting padding values ​​for color data of the object area of ​​non-target objects.

[0009] However, the problems to be solved in this disclosure are not limited to those mentioned above, and may be expanded in various ways without departing from the spirit and scope of this disclosure.

[0010] One technical aspect of the present invention proposes a texturing method for generating a three-dimensional virtual model. The texturing method for generating a three-dimensional virtual model is a texturing method performed in a computing device that generates a three-dimensional virtual model of a real space based on a plurality of data sets generated at a plurality of shooting points in a real space, and comprises: a step of generating a three-dimensional mesh model of a real space using the plurality of data sets; a step of selecting a texturing image based on the presence of a non-target object in the plurality of data sets and performing texturing on the three-dimensional mesh model using the selected texturing image; and a step of providing the three-dimensional virtual model with completed texturing to a user through a user interface.

[0011] Another technical aspect of the present invention proposes a text-based personal information identification device. The text-based personal information identification device includes at least one processor; and a memory for storing instructions. When the instructions are executed individually or collectively by the at least one processor, the processor generates a three-dimensional mesh model of real space using a plurality of data sets, selects a texturing image based on the presence of a non-target object in the plurality of data sets, performs texturing on the three-dimensional mesh model using the selected texturing image, and provides a three-dimensional virtual model with completed texturing to a user through a user interface.

[0012] Another technical aspect of the present invention proposes a storage medium. The storage medium is a storage medium that stores computer-readable instructions, wherein the instructions, when executed by a computing device, perform the following operations: generating a three-dimensional mesh model of real space using a plurality of data sets; selecting a texturing image based on the presence of a non-target object in the plurality of data sets and performing texturing on the three-dimensional mesh model using the selected texturing image; and providing a three-dimensional virtual model with completed texturing to a user through a user interface.

[0013] The means for solving the above-mentioned problem do not enumerate all features of the present invention. Various means for solving the problem of the present invention may be understood in more detail by referring to specific embodiments in the following detailed description.

[0014] According to the present application, there is one or more of the following effects.

[0015] According to one embodiment of the present invention, when creating a three-dimensional virtual space corresponding to real space based on a shooting data set collected from multiple shooting points in real space, there is an effect of preventing non-target objects, which are unintended subject objects, from being textured in advance.

[0016] According to one embodiment of the present invention, by identifying non-target objects for object identification in an image and setting color data for the object area of ​​the non-target object by replacing it with a padding value, it is possible to efficiently select a texturing image without non-target objects with a small amount of computation.

[0017] According to one embodiment of the present invention, by setting the padding value of the color data for the object area of ​​a non-target object as a penalty, there is an effect of efficiently selecting an optimal texturing image even when there are multiple non-target objects.

[0018] The effects of the present application are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art from the description in the claims.

[0019] FIG. 1 is a drawing illustrating a three-dimensional virtual model providing system according to one embodiment disclosed in the present application.

[0020] FIG. 2 is a drawing for explaining each component of a three-dimensional virtual model providing system according to one embodiment disclosed in the present application.

[0021] FIG. 3 is a flowchart illustrating a texturing method for generating a three-dimensional virtual model according to one embodiment disclosed in the present application.

[0022] FIG. 4 is a flowchart illustrating a method for generating a three-dimensional mesh model according to one embodiment disclosed in the present application.

[0023] FIG. 5 is a flowchart illustrating a method for selecting and performing texturing on a texturing image according to one embodiment disclosed in the present application.

[0024] FIG. 6 is a flowchart illustrating a method for processing identification of non-target objects according to one embodiment disclosed in the present application.

[0025] FIG. 7 is a flowchart illustrating a method for selecting texturing images according to one embodiment disclosed in the present application.

[0026] FIG. 7 is a flowchart illustrating a method for selecting texturing images according to another embodiment disclosed in the present application.

[0027] FIGS. 9 and 10 are drawings for explaining an example of selecting a texturing image based on a non-target object according to one embodiment disclosed in the present application.

[0028] FIGS. 11 to 14 are drawings illustrating examples of texturing results according to one embodiment disclosed in the present application.

[0029] FIG. 15 is a drawing illustrating an exemplary structure and operating environment of a computing device according to one embodiment of the present invention.

[0030] Hereinafter, embodiments of the present disclosure are described in detail with reference to the drawings so that those skilled in the art can easily practice them. However, the present disclosure may be embodied in various different forms and is not limited to the embodiments described herein. In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and brevity.

[0031] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order).

[0032] The term “module” as used in the various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0033] Various embodiments of this document may be implemented as software (e.g., a program) comprising one or more instructions stored in a storage medium (e.g., memory) readable by a machine or device. For example, the processor of the machine or device may call at least one of the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by a machine may be provided in the form of a non-transitory storage medium. Here, "non-transitory" simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and this term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily in the storage medium.

[0034] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0035] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0036] In this disclosure, the term "processor" may refer to hardware capable of performing functions and operations according to each designation described herein, computer program code capable of performing specific functions and operations, or an electronic recording medium loaded with computer program code capable of performing specific functions and operations. According to the embodiments, the operation of the processor may be defined and / or interpreted as the operation of a digital signage content providing device, but is not limited thereto. The processor may refer to a functional and / or structural combination of hardware for carrying out the technical concept of this disclosure and / or software for driving said hardware.

[0037] FIG. 1 is a drawing illustrating a three-dimensional virtual model providing system according to one embodiment disclosed in the present application.

[0038] A three-dimensional virtual model providing system may include a scanning device (100), a computing device (300), and a user terminal (500).

[0039] The scan device (100) is an electronic device for generating a shooting data set at each shooting point in real space, and is a portable electronic device including a camera and a distance measuring sensor.

[0040] The scan device (100) is driven to scan at each of the multiple shooting points in real space, and can generate multiple data sets for the multiple shooting points in real space.

[0041] A plurality of data sets may include image and depth data acquired at each shooting point, and, depending on the embodiment, may further include indoor location data.

[0042] In one embodiment, the data set generated at each shooting point may be a 360-degree panoramic image acquired at the shooting point and 360-degree depth data acquired at the shooting point—e.g., a depth map.

[0043] The computing device (300) can construct a three-dimensional virtual model corresponding to the real space based on a plurality of data sets collected from the scanning device (100).

[0044] The computing device (300) can receive multiple data sets generated at various shooting points indoors from the scanning device (100). The computing device (300) can generate a 3D virtual model, which is a 3D virtual space corresponding to real space, by using the multiple data sets, namely images and depth data generated at various points indoors.

[0045] Here, the term "image" encompasses all images expressed in color and is not limited to images of a specific method of expression. Therefore, color images can be applied in various ways, such as RFG images expressed in RGB (Red Green Blue) as well as CMYK images expressed in CMYK (Cyan Magenta Yellow Key).

[0046] Depth data is a representation that encompasses data providing depth information about the subject space. For example, each pixel in the depth data may be distance information from the shooting point to each point in the subject space—a point in space corresponding to each pixel.

[0047] The computing device (300) can generate a three-dimensional mesh model of the real space based on a plurality of data sets for each of a plurality of shooting points in the real space received from the scanning device (100).

[0048] Here, the 3D mesh model is a 3D model that represents a spatial volume constituting real space as a set of polygons, and the 3D mesh model can be represented by multiple vertices, edges connecting two vertices, and faces divided by multiple edges.

[0049] The computing device (300) can perform texturing by mapping a portion of an image onto the surface of a three-dimensional mesh model to apply a texture similar to each region of real space, and by completing this texturing, can create a three-dimensional virtual model corresponding to real space.

[0050] The computing device (300) can provide the user terminal (500), etc., with the experience of a three-dimensional virtual space corresponding to real space by providing the three-dimensional virtual model created in this way.

[0051] The user terminal (500) is an electronic device that allows the user to experience a three-dimensional virtual model corresponding to real space. The user can use the user terminal (500) to connect to the computing device (300) and experience the three-dimensional virtual model provided by the computing device (300).

[0052] FIG. 2 is a drawing for explaining each component of a three-dimensional virtual model providing system according to one embodiment disclosed in the present application.

[0053] The computing device (300) may include a processing unit (310) and a system memory (320).

[0054] The processing unit (310) may include, as an example, at least one of a microprocessor, a central processing unit, a processor core, a multi-core processor, a multiprocessor, an ASIC (Application-Specific Integrated Circuit), or an FPGA (Field Programmable Gate Array), but is not limited thereto.

[0055] The computing device (300) may include a module for object detection and a module for performing texturing. These modules may be artificial intelligence models or software modules that operate individually, and may be implemented as part of a processing unit (310), but are not limited thereto, and a separate processor may be used for each module.

[0056] System memory (320) can store instructions (or programs) executable by the processing unit (310). System memory (320) may include volatile memory or non-volatile memory. Volatile memory may be implemented as dynamic random access memory (DRAM), static random access memory (SRAM), thyristor RAM (T-RAM), zero capacitor RAM (Z-RAM), or twin transistor RAM (TTRAM). Non-volatile memory can be implemented as EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory, MRAM (Magnetic RAM), Spin-Transfer Torque (STT)-MRAM, Conductive Bridging RAM (CBRAM), FeRAM (Ferroelectric RAM), PRAM (Phase change RAM), Resistive RAM (RRAM), Nanotube RRAM, Polymer RAM (PoRAM), Nano Floating Gate Memory (NFGM), holographic memory, Molecular Electronic Memory Device, or Insulator Resistance Change Memory.

[0057] The computing device (300) may be implemented as a server, but is not limited thereto.

[0058] The scan device (100) includes a camera for acquiring an image and a depth scanner for acquiring depth data.

[0059] A depth scanner may include a specific sensor for measuring distance, such as a LiDAR sensor, an infrared sensor, an ultrasonic sensor, etc. Alternatively, to acquire depth data, the depth scanner may include a stereocamera, a stereoscopic camera, a 3D depth camera (3D, depth camera), etc., which can measure distance information in place of a sensor.

[0060] The user terminal (500) is a device used by the user, and the user can be connected to the computing device (300) through the user terminal (500).

[0061] The user terminal (500) is an electronic device on which software such as an application runs, for example, a mobile phone, a smartphone, a laptop computer, a digital broadcasting terminal, a PDA (personal digital assistants), a PMP (portable multimedia player), a navigation device, a personal computer (PC), a tablet PC, an ultrabook, and a wearable device, for example, a smartwatch, a smart glass, a head-mounted display (HMD), a Virtual Reality (V) device, or an Augmented Reality (AR) device.

[0062] FIG. 3 is a flowchart illustrating a texturing method for generating a three-dimensional virtual model according to one embodiment disclosed in the present application.

[0063] The scan device (100) can generate multiple data sets by taking photos at multiple shooting points in real space (S301). That is, the scan device (100) can generate one data set for each shooting point in real space.

[0064] For example, the dataset may include a 360-degree panoramic image acquired at the shooting point and 360-degree depth data acquired at the shooting point. According to an embodiment, the dataset may further include location information data for the shooting point—e.g., Simultaneous Localization and Mapping (SLAM), etc.

[0065] The scan device (100) can transmit multiple data sets of real space to the computing device (300) (S302).

[0066] The computing device (300) receives a plurality of data sets from the scanning device (100) and can generate a 3D mesh model, which is a 3D spatial model corresponding to real space, using the plurality of data sets (S303).

[0067] For example, a computing device (300) may generate multiple point clouds in a three-dimensional space by placing multiple data sets corresponding to each of multiple shooting points in real space, and generate a three-dimensional mesh model based on these point clouds.

[0068] The computing device (300) can generate a 3D virtual model by performing texturing on the generated 3D mesh model (S304).

[0069] The computing device (300) can perform texturing by considering non-target objects. Here, a non-target object refers to an object that does not correspond to spatial information of such real space when configuring a virtualization of real space, and for example, a person object may be included.

[0070] That is, the computing device (300) can select a texturing image based on the presence of a non-target object in a plurality of data sets and perform texturing on a 3D mesh model using the selected texturing image (S304).

[0071] In this way, by performing texturing while considering non-target objects, non-target objects are not projected onto the textures expressed in the 3D virtual model, thereby improving the texturing quality of the 3D virtual model.

[0072] Various embodiments of texturing considering such non-target objects will be described in more detail below with reference to FIGS. 5 to 8.

[0073] The computing device (300) can provide a user experience interface to a user terminal (500) for a three-dimensional virtual model with completed texturing, thereby providing the user with an experience of a three-dimensional virtual space corresponding to real space (S305). Various VR (Virtual Reality) based viewers can be applied as this experience interface.

[0074] FIG. 4 is a flowchart illustrating a method for generating a three-dimensional mesh model according to one embodiment disclosed in the present application.

[0075] Referring to FIG. 4, the computing device (300) can prepare multiple data sets acquired from multiple shooting points in real space (S410).

[0076] The computing device (300) can generate multiple 3D point clouds by reflecting depth data captured at each of the multiple shooting points on a 3D spatial coordinate system (S420).

[0077] For example, a computing device (300) can generate a point cloud for each shooting point by individually setting three-dimensional spatial coordinates for each shooting point and reflecting a data set at the shooting point to the individual three-dimensional spatial coordinates.

[0078] For example, a computing device (300) can generate a single integrated point cloud from a plurality of three-dimensional point clouds based on the integration of three-dimensional spatial coordinate systems (S430).

[0079] Subsequently, the computing device (300) can generate an integrated point cloud based on integrated 3D spatial coordinates for real space by reflecting the point clouds of individual 3D spatial coordinates generated at each shooting point into a single integrated 3D spatial coordinate based on location information for each shooting point. For this integration, the computing device (300) can perform normalization of the point cloud, etc., and perform processing for the redundancy of the point cloud resulting from the integration of multiple coordinates.

[0080] The computing device (300) can generate a 3D mesh model based on an integrated 3D spatial coordinate-based point cloud as described above (S440).

[0081] For example, a computing device (300) can generate a three-dimensional mesh model by selecting multiple vertices based on a point cloud, forming edges between the multiple vertices, and then setting multiple faces based on the edges.

[0082] For example, the computing device (300) can generate a face based on three adjacent vertices, in which case the face may be a flat triangle set with three vertices. As another example, a square face may be set based on four vertices.

[0083] FIG. 5 is a flowchart illustrating a method for selecting and performing texturing on a texturing image according to one embodiment disclosed in the present application.

[0084] Referring to FIG. 5, the computing device (300) can select multiple images included in multiple data sets (S510). Since the 3D mesh model is a model in which there is no texture on the surface of the face, in order to perform texturing on such a face, an image related to the face must be selected. That is, since the source of texturing is an image, the computing device (300) can select an image from the data set for texturing.

[0085] The computing device (300) can identify objects in an image. That is, the computing device (300) can identify objects by generating pixel-unit object area information for multiple images (S520).

[0086] In one embodiment, the computing device (300) may use a deep learning model to identify objects in an image. For example, the computing device (300) may identify objects in an image using a semantic segmentation deep learning model that processes a segmentation task to determine whether each pixel in an image or video belongs to a specific object.

[0087] Such semantic segmentation deep learning models may include backbone, upsampling, and mask prediction processes. The backbone process is the process of extracting key features from input images using feature extractors such as VGG and ResNet. Since semantic segmentation deep learning models primarily use feature maps with reduced image size, a process of restoring the image to its original size through upsampling is required, and pixel-level prediction can be performed during this upsampling process. Subsequently, semantic segmentation deep learning models process the prediction of not only bounding boxes but also the masks of each object, thereby enabling the accurate segmentation of the object's shape.

[0088] Deep learning models for such semantic segmentation may include Fully Convolutional Networks (FCN), U-Net, and DeepLab models, but are not limited to these.

[0089] The computing device (300) can perform identification processing for non-target objects among the objects identified as above (S530). For example, the computing device (300) can select non-target objects among the identified objects and perform separate identification processing on the area of ​​the selected non-target objects. An example of such identification processing is described below with reference to FIG. 6.

[0090] The computing device (300) can perform texturing by selecting a texturing image for each face of a three-dimensional mesh model based on the identification processing described above (S540). Various examples of such identification processing are described below with reference to FIGS. 7 and 8.

[0091] In this way, the computing device (300) identifies non-target objects in an image and performs a separate identification process on the area of ​​the non-target objects, thereby quickly identifying non-target objects during the texturing process using the image and selecting an image that has no non-target objects or has the smallest non-target objects as a texturing image.

[0092] FIG. 6 is a flowchart illustrating a method for identifying a non-target object according to an embodiment disclosed in the present application. FIG. 6 illustrates an embodiment of the identification process for a non-target object, which is step S530 shown in FIG. 5.

[0093] Referring to FIG. 6, the computing device (300) can confirm the existence of non-target objects among the identified objects (S610). For example, the computing device (300) can identify a person object that is a non-target object among the various types of objects identified in an image.

[0094] The computing device (300) can perform padding processing to fill the object area of ​​a non-target object with a preset padding value (S620).

[0095] For example, the computing device (300) can set a padding value for each pixel of the object area of ​​a non-target object. That is, the padding value can be set by replacing the color value of each pixel of the image.

[0096] Here, the object area of ​​a non-target object refers to the area in an image where the non-target object is displayed; for example, if a person's upper body is present in the image, the entire area of ​​that person's upper body can be the object area.

[0097] In one embodiment, the padding value may be a pre-set penalty value. This penalty value may be used in the evaluation process for candidate images described below, for example, in a manner where a candidate image with a low penalty value is set as the texturing image.

[0098] In one embodiment, the padding value may be a pre-set null value. In this case, since the null value has no value, it may be represented as a hole that does not contain any texture even if used for texturing. In this case as well, a candidate image with no or few null values ​​may be used as a texturing image.

[0099] This padding process can be performed in various stages depending on the embodiment. For example, FIG. 5 describes an embodiment in which padding processing for non-target objects is performed for all of the multiple images, but it is not limited thereto. That is, various modifications can be made, such as selecting candidate images described below for each face and then performing padding processing for non-target objects on the candidate images.

[0100] FIG. 7 is a flowchart illustrating a method for selecting texturing images according to one embodiment disclosed in the present application.

[0101] FIG. 7 discloses an example of a process for performing texturing by selecting a texturing image for each face in a three-dimensional mesh model, as an embodiment of step S540 illustrated in FIG. 5.

[0102] Referring to FIG. 7, the computing device (300) can select a first face, which is one of the faces in a three-dimensional mesh model (S710).

[0103] The computing device (300) is based on the reference angle and reference distance for the first phase.

[0104] Multiple candidate images can be selected (S720).

[0105] For example, the computing device (300) may set a predetermined reference angle based on a normal vector for a first face and set a predetermined distance from the first face as a reference distance. The computing device (300) may calculate the angle and distance of each image for the first face based on the camera pose and position for each image. The computing device (300) may set images satisfying the reference angle and reference distance as a plurality of candidate images.

[0106] In the example above, an example of setting candidate images based on a reference angle and a reference distance was explained; however, this is for illustrative purposes only, and candidate images can also be set based on a reference angle or a reference distance.

[0107] In one embodiment, the computing device (300) can actively change the reference angle or reference distance so that the number of candidate images is greater than a certain number. That is, if the number of candidate images is set to at least 5, and there are fewer than 5 images satisfying the previously set reference angle and reference distance, the computing device (300) can actively change the reference angle or reference distance—set it to a larger value—so that the number of candidate images satisfying the reference angle and reference distance is at least 5.

[0108] The computing device (300) can determine whether padding values ​​exist for a plurality of candidate images based on whether padding values ​​exist and exclude candidate images that have padding values ​​(S730).

[0109] This step S730 is a processing step that excludes images in which non-target objects are displayed from the candidate images, thereby ensuring that no non-target objects exist in the finally selected texturing image.

[0110] The computing device (300) can select one image as a texturing image based on proximity to a reference angle and a reference distance for the remaining candidate images after such exclusion processing (S740).

[0111] Various methods can be applied to select one of these candidate images as the texturing image. For example, an image with the smallest reference angle value and an area displaying the corresponding face larger than a certain size can be selected as the final texturing image. As another example, an image with the largest image resolution for the face—e.g., area size—can be selected as the texturing image. In addition to these, various other methods for selecting the final texturing image can be applied.

[0112] Subsequently, the computing device (300) can perform texturing processing on the first face using the selected texturing image. For example, texturing can be performed by selecting an area corresponding to the first face from the selected texturing image and projecting it onto the first face. Since various methods can be applied to the texturing method for such a face, the description thereof is omitted here.

[0113] Figure 9 illustrates such an example. In Figure 9, Figure (a) and Figure (b) are exemplified as candidate images for the face of a blackboard object. Figure (a) is an image taken from the front of the blackboard, where the reference angle corresponds to the normal vector and has the highest score for the reference angle. Additionally, the resolution of the blackboard object is higher than in Figure (b), so the score for the reference distance is also higher. On the other hand, Figure (b) is an image taken from a slightly further distance from the left side of the blackboard, and has a worse score than Figure (a) when considering the reference angle and reference distance.

[0114] On the other hand, in Figure (a), the blackboard object 911 and the non-target object 913 are displayed together, but in Figure (b), only the blackboard object 921 is displayed. Therefore, in this case, the blackboard object face can be selected as the texturing image in Figure (b), where there is no non-target object in the blackboard object face.

[0115] If the face to be textured is a chair object, since the chair object 912 in figure (a) is superior to the chair object 922 in figure (b) in reference angle and / or reference distance, figure (a) can be selected as the texturing image for the chair object face.

[0116] FIG. 8 is a flowchart illustrating a method for selecting texturing images according to another embodiment disclosed in the present application.

[0117] FIG. 8 discloses another example of a process for performing texturing by selecting a texturing image for each face in a three-dimensional mesh model, as another embodiment of step S540 illustrated in FIG. 5.

[0118] Referring to FIG. 8, the computing device (300) selects a first face of a three-dimensional mesh model (S810) and can select a plurality of candidate images based on a reference angle and a reference distance for the first face (S820). The computing device (300) can select the candidate image with the smallest sum of padding values ​​among the plurality of candidate images as the texturing image (S830). The computing device (300) can texture the first face using the selected texturing image (S840).

[0119] The example illustrated in FIG. 8 relates to an embodiment for selecting candidate images based on the sum of padding values ​​of non-target objects, wherein the padding values ​​of non-target objects are used as a penalty, i.e., a deduction factor.

[0120] Figure 10 illustrates such an example. In Figure 10, Figures (a), (b), and (c) are exemplified as candidate images for the face of a blackboard object. Figure (a) is an image taken from the front of the blackboard, where the reference angle corresponds to the normal vector and has the highest score for the reference angle; it also has a higher resolution for the blackboard object and a higher score for the reference distance. On the other hand, Figure (b) is an image taken from a slightly further distance from the left side of the blackboard, and has a worse score than Figure (a) when considering the reference angle and reference distance. Figure (c) is an image taken from a slightly further distance from the left side of the blackboard, and has a worse score than Figures (a) and (b) when considering the reference angle and reference distance.

[0121] On the other hand, in Figure (a), blackboard object 1011 and non-target object 1013 are displayed together, and the area of ​​non-target object 1013 is obscured the largest. In Figure (b), the area of ​​non-target object 1022 that overlaps with blackboard object 1021 is the smallest. In Figure (c), the first non-target object 1032 does not overlap with blackboard object 1031, but the area taken by the second non-target object 1033 is the second largest.

[0122] Therefore, since the penalty increases by the size of the area corresponding to the non-target object, in this case, Figure (b) will be selected as the texturing image for the blackboard object face.

[0123] In the descriptions above, the chalkboard object was described as a single face for convenience of explanation and is not limited thereto. In reality, the chalkboard object may include multiple polygons, such as multiple triangular faces, and the processing of FIG. 7 or FIG. 8 will be performed individually for each face. That is, the presence of non-target objects or padding values ​​can be processed individually for each face.

[0124] FIGS. 11 to 14 are drawings illustrating examples of texturing results according to one embodiment disclosed in the present application.

[0125] In FIG. 11, as an example of a general texturing method before the application of the present invention, it can be seen that a human figure is textured on the exterior wall of a building. On the other hand, FIG. 12 shows an example of applying an embodiment of the present invention to the same data set, and it can be seen that no human object is textured on the exterior wall of a building, etc.

[0126] FIG. 13 also shows an example of a general texturing method before the application of the present invention, in which a human figure is textured on a part of a building wall. On the other hand, FIG. 14 shows an example of applying an embodiment of the present invention to the same data set, in which it can be seen that no human object is textured on the building exterior wall, etc.

[0127] FIG. 15 is a drawing illustrating an exemplary environment of a computing device according to one embodiment of the present invention.

[0128] FIG. 15 is intended to provide a general and simplified description of a suitable computing environment in which various embodiments of a computing device may be implemented. Referring to FIG. 14, a computing device (100) is illustrated.

[0129] The computing device (100) may include at least a processing unit (310) and a system memory (320).

[0130] The computing device may include multiple processing units that cooperate when executing a program. Depending on the exact configuration and type of the computing device, the system memory (301) may be volatile (e.g., RAM), non-volatile (e.g., ROM, flash memory, etc.), or a combination thereof. The system memory (320) includes a suitable operating system (330) for controlling the operation of the platform, which may be, for example, the WINDOWS operating system from Microsoft. The system memory (320) may include one or more software applications, such as program modules, applications, etc.

[0131] The computing device may include additional data storage devices (340), such as magnetic disks, optical disks, or tapes. These additional storage devices may be removable storage and / or fixed storage. Computer-readable storage media may include volatile and non-volatile, removable and fixed media implemented by any method or technique for storing information such as computer-readable instructions, data structures, program modules, or other data. System memory (320) and data storage devices (340) are merely examples of computer-readable storage media. Computer-readable storage media may include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory techniques, CD-ROM, DVD or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that stores desired information and can be accessed by the computing device (100).

[0132] Input devices (350) of a computing device may include, for example, a keyboard, a mouse, a pen, a voice input device, a touch input device, and comparable input devices. Output devices (360) may include, for example, a display, a speaker, a printer, and other types of output devices. Since these devices are widely known in the art, a detailed description is omitted.

[0133] The computing device may include a communication device (370) that allows the device to communicate with other devices through a network in a distributed computing environment, such as a wired / wireless network, a satellite link, a cellular link, a local area network, and a comparable mechanism. The communication device (370) is one example of a communication medium, and the communication medium may contain computer-readable instructions, data structures, program modules, or other data. For example, the communication medium includes, but is not limited to, wired media such as a wired network or direct wired connection, and wireless media such as acoustic, RF, infrared, and other wireless media.

[0134] Although the embodiments have been described above with reference to limited examples and drawings, those skilled in the art can make various modifications and variations from the description above. For example, suitable results can be achieved even if the described techniques are performed in a different order than described, and / or the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0135] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below.

[0136] Although specific embodiments have been described in the detailed description of this document, it will be obvious to those skilled in the art that various modifications are possible within the scope of this document.

[0137] [Sasa]

[0138] This invention was filed overseas with the support of the following research project supported by the government of the Republic of Korea.

[0139] Assignment 1

[0140] Assignment No.: 20023322

[0141] Ministry Name: Ministry of Trade, Industry and Energy

[0142] Specialized Research Management Agency: Korea Institute of Industrial Technology Evaluation and Management

[0143] Research Project Name: Excellent Corporate Research Institute Development Project (ATC+)

[0144] Research Project Title: Advanced Development of an Intelligent Digital Twin Service Platform Based on a Converged Visual Localization Algorithm Supporting Various Types of Collected Data

[0145] Organizer: 3i Co., Ltd.

[0146] Research Period: 2024.01.01~2024.12.31

[0147]

[0148] Assignment 2

[0149] Project ID: 2710006711

[0150] Assignment No.: 00229330

[0151] Ministry Name: Ministry of Science and ICT

[0152] Project Management (Specialized) Agency Name: Korea Institute of Information and Communications Technology Planning and Evaluation

[0153] Research Project Name: Development of Core Technologies for Immersive Content

[0154] Research Project Title: Streaming 3D Digital Media Service Technology

[0155] Project Executing Organization Name: Seoul National University of Science and Technology Industry-Academic Cooperation Foundation

[0156] Research Period: 2024.01.01~2024.12.31

[0157]

[0158] According to one embodiment of the present invention, when generating a three-dimensional virtual space corresponding to real space based on a shooting data set collected from multiple shooting points in real space, there is an effect of preventing non-target objects, which are unintended subject objects, from being textured in advance, thus having high potential for industrial use.

[0159] Furthermore, by identifying non-target objects during image object identification and setting the color data of the non-target object's object region by replacing it with padding values, it is possible to efficiently select texturing images free of non-target objects with minimal computation, making it highly promising for industrial application.

[0160] Furthermore, by setting padding values ​​for the color data of non-target objects' object regions, it is possible to efficiently select the optimal texturing image even when multiple non-target objects exist, making it highly suitable for industrial applications.

Claims

1. A texturing method performed on a computing device that generates a three-dimensional virtual model of a real space based on multiple data sets generated at each of multiple shooting points in a real space, A step of generating a 3D mesh model of real space using multiple data sets; A step of selecting texturing images based on the presence of non-target objects in the plurality of data sets, and performing texturing on the 3D mesh model using the selected texturing images; and A step comprising providing a 3D virtual model with completed texturing to a user through a user interface; Texturing method for generating a 3D virtual model.

2. In paragraph 1, the plurality of data sets are, 360-degree panoramic image acquired at the shooting point; and 360-degree depth data acquired at the above shooting point; comprising, Texturing method for generating a 3D virtual model.

3. In claim 1, the step of performing texturing on the three-dimensional mesh model using the texturing image is: A step of identifying objects by generating object region information in pixel units for a plurality of images included in the plurality of data sets above; A step of selecting non-target objects among identified objects and performing identification processing; and A step of selecting a texturing image for each face of a 3D mesh model based on the above identification processing; comprising Texturing method for generating a 3D virtual model.

4. In paragraph 3, the above identification processing is, Padding the color data of the object area of ​​the above-mentioned non-target object by replacing it with a preset padding value. Texturing method for generating a 3D virtual model.

5. In paragraph 4, the step of selecting a texturing image for each face of a 3D mesh model based on the identification processing is: A step of selecting a first face of the above 3D mesh model; A step of selecting a plurality of candidate images based on a reference angle and a reference distance for the first face; and A step of excluding candidate images having padding values ​​among the plurality of candidate images; comprising Texturing method for generating a 3D virtual model.

6. In paragraph 5, the step of selecting a texturing image for each face of a 3D mesh model based on the identification processing above is: For the remaining candidate images, the step of selecting one image as a texturing image based on proximity to a reference angle and a reference distance; further comprising Texturing method for generating a 3D virtual model.

7. In paragraph 4, the step of selecting a texturing image for each face of a 3D mesh model based on the identification processing above is: A step of selecting a first face of the above 3D mesh model; A step of selecting a plurality of candidate images based on a reference angle and a reference distance for the first face; and A step of selecting, among the plurality of candidate images, the candidate image having the smallest sum of padding values ​​as the texturing image; comprising Texturing method for generating a 3D virtual model.

8. In a text-based personal information identification device, At least one processor; and It includes memory for storing instructions, When the above instructions are executed individually or collectively by the at least one processor, the processor, Generate a 3D mesh model of real space using multiple datasets, and Texturing images are selected based on the presence of non-target objects in the plurality of data sets, and texturing is performed on the 3D mesh model using the selected texturing images. Providing a textured 3D virtual model to the user through a user interface, Computing device for generating a three-dimensional virtual model.

9. In paragraph 8, the plurality of data sets are, 360-degree panoramic image acquired at the shooting point; and 360-degree depth data acquired at the above shooting point; comprising Computing device for generating a three-dimensional virtual model.

10. In paragraph 8, the text-based personal information identification device is, In order to perform texturing on the 3D mesh model using the above texturing image, For a plurality of images included in the plurality of data sets above, object region information is generated in pixel units to identify objects, and Select non-target objects from among the identified objects and perform identification processing, Selecting texturing images for each face of a 3D mesh model based on the above identification processing, Computing device for generating a three-dimensional virtual model.

11. In Paragraph 10, the above identification processing is, Padding the color data of the object area of ​​the above-mentioned non-target object by replacing it with a preset padding value. Computing device for generating a three-dimensional virtual model.

12. In Paragraph 11, the text-based personal information identification device is, In order to select a texturing image for each face of a 3D mesh model based on the above identification processing, Select the first face of the above 3D mesh model, and Selecting a plurality of candidate images based on a reference angle and a reference distance for the first face, and Excluding candidate images having padding values ​​for the plurality of candidate images above, For the remaining candidate images, one image is selected as the texturing image based on proximity to the reference angle and reference distance. Computing device for generating a three-dimensional virtual model.

13. In Paragraph 11, the text-based personal information identification device is, In order to select a texturing image for each face of a 3D mesh model based on the above identification processing, Select the first face of the above 3D mesh model, and Selecting a plurality of candidate images based on a reference angle and a reference distance for the first face, and Among the above plurality of candidate images, selecting the candidate image with the smallest sum of padding values ​​as the texturing image. Computing device for generating a three-dimensional virtual model.

14. In a storage medium storing computer-readable instructions, When the above instructions are executed by a computing device, the computing device, An operation to generate a 3D mesh model of real space using multiple data sets; An operation of selecting texturing images based on the presence of non-target objects in the plurality of data sets and performing texturing on the 3D mesh model using the selected texturing images; and The operation of providing a textured 3D virtual model to a user through a user interface; which performs the operation of Computer-readable storage media.

Images