Method for three-dimensional registration of heterogeneous three-dimensional virtual models and computing device therefor

Abstract

A method for three-dimensional (3D) registration of three-dimensional virtual models, performed by a computing device, according to one embodiment of the present disclosure, comprises the steps of: generating a 3D interior digital model of an indoor space using a plurality of datasets collected in the indoor space; generating a 3D exterior digital model of an outdoor space using a plurality of image sets collected in the outdoor space; generating outline-based 3D volumetric models for the 3D interior digital model and the 3D exterior digital model; and performing registration between the 3D interior digital model and the 3D exterior digital model on the basis of external shape matching between a 3D interior volumetric model and a 3D exterior volumetric model.

Description

3D alignment method for heterogeneous 3D virtual models and computing device for the same

[0001] The present invention relates to a three-dimensional alignment method for heterogeneous three-dimensional virtual models and a computing device for the same.

[0002] Recently, virtual space implementation technologies are being developed that provide an online virtual space corresponding to the actual real world, enabling users to experience being in that space without physically visiting it. Various developments are underway for these real-world-based virtual technologies, which serve as tools for implementing digital twins.

[0003] Meanwhile, various methods for implementing digital models that constitute virtual space are being developed. For example, when constructing a digital model of a wide area using multiple images captured by a drone, a method can be used to create a 3D model by inversely calculating the camera positions and poses of the multiple images and then using common feature points. On the other hand, when using distance measuring sensors such as LiDAR for indoor spaces, a method can be used to create a 3D model by projecting the shooting point onto a 3D spatial coordinate system and then reflecting the distance data.

[0004] As such, different digital model construction technologies exist depending on the space being captured and the capturing device; consequently, there are limitations such as a lack of interoperability between digital models implemented in these differing ways and the requirement to implement separate, individual systems for each. Consequently, this poses an obstacle to the widespread adoption of digital twins.

[0005] Therefore, the need for technology to converge different heterogeneous digital models is growing.

[0006] One technical aspect of the present application aims to solve the aforementioned problem by providing a 3D alignment technology for 3D virtual models that can provide a converged digital twin, which provides heterogeneous digital models in a single converged environment, by expressing heterogeneous 3D models configured in different ways through a single interface.

[0007] One technical aspect of the present application is to provide a 3D alignment technology for 3D virtual models that enables the alignment of heterogeneous digital models efficiently and quickly with fewer resources by generating an outline-based volume model for heterogeneous 3D digital models configured in different ways and performing alignment.

[0008] One technical aspect of the present application is to provide a 3D alignment technology for 3D virtual models that enables accurate scaling with efficient and low resources by performing scaling based on an outline-based volume model for heterogeneous 3D digital models configured in different ways.

[0009] One technical aspect of the present application is to provide a 3D alignment technology for 3D virtual models that can express heterogeneous 3D digital models configured in different ways so as to be seamlessly switched between them within a single interface, and also provide a high level of user experience by matching and scaling between heterogeneous digital models, and display more complex digital models through a single user interface.

[0010] However, the problems to be solved in this disclosure are not limited to those mentioned above, and may be expanded in various ways without departing from the spirit and scope of this disclosure.

[0011] One technical aspect of the present invention proposes a three-dimensional alignment method for three-dimensional virtual models. The three-dimensional alignment method for the three-dimensional virtual models is performed on a computing device and includes the steps of: generating a three-dimensional interior digital model for an indoor space using a plurality of data sets collected in an indoor space; generating a three-dimensional exterior digital model for an outdoor space using a plurality of image sets collected in an outdoor space; generating an outline-based three-dimensional volume model for the three-dimensional interior digital model and the three-dimensional exterior digital model; and performing alignment between the three-dimensional interior digital model and the three-dimensional exterior digital model based on external alignment between the three-dimensional interior volume model and the three-dimensional exterior volume model.

[0012] Another technical aspect of the present invention proposes a computing device for performing three-dimensional alignment of heterogeneous three-dimensional virtual models. The computing device includes at least one processor and a memory for storing instructions. When the instructions are executed individually or collectively by the at least one processor, the processor generates a three-dimensional interior digital model for an indoor space using a plurality of data sets collected in an indoor space, generates a three-dimensional exterior digital model for an outdoor space using a plurality of image sets collected in an outdoor space, generates an outline-based three-dimensional volume model for the three-dimensional interior digital model and the three-dimensional exterior digital model, and performs alignment between the three-dimensional interior digital model and the three-dimensional exterior digital model based on the external alignment between the three-dimensional interior volume model and the three-dimensional exterior volume model.

[0013] Another technical aspect of the present invention proposes a storage medium. The storage medium is a storage medium that stores computer-readable instructions, wherein when the instructions are executed by a computing device, the computing device performs the following operations: generating a three-dimensional interior digital model for an indoor space using a plurality of data sets collected in an indoor space; generating a three-dimensional exterior digital model for an outdoor space using a plurality of image sets collected in an outdoor space; generating an outline-based three-dimensional volume model for the three-dimensional interior digital model and the three-dimensional exterior digital model; and performing alignment between the three-dimensional interior digital model and the three-dimensional exterior digital model based on external alignment between the three-dimensional interior volume model and the three-dimensional exterior volume model.

[0014] The means for solving the above-mentioned problem do not enumerate all features of the present invention. Various means for solving the problem of the present invention may be understood in more detail by referring to specific embodiments in the following detailed description.

[0015] According to the present application, there is one or more of the following effects.

[0016] According to one embodiment of the present invention, by expressing heterogeneous three-dimensional models configured in different ways through a single interface, there is an effect of providing a converged digital twin that provides heterogeneous digital models in a single converged environment.

[0017] According to one embodiment of the present invention, by generating an outline-based volume model for heterogeneous three-dimensional digital models configured in different ways and performing alignment, the heterogeneous digital models can be aligned efficiently and quickly with fewer resources.

[0018] According to one embodiment of the present invention, by performing scaling based on an outline-based volume model for heterogeneous three-dimensional digital models configured in different ways, it is possible to perform scaling accurately with efficient and low resources.

[0019] According to one embodiment of the present invention, the effect of being able to seamlessly switch between heterogeneous three-dimensional digital models configured in different ways within a single interface is provided. In addition, by matching the alignment and scaling between heterogeneous digital models, the effect of providing a high level of user experience and enabling more complex digital models to be displayed through a single user interface is provided.

[0020] The effects of the present application are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art from the description in the claims.

[0021] FIG. 1 is a drawing illustrating a three-dimensional virtual model fusion providing system according to one embodiment disclosed in the present application.

[0022] FIG. 2 is a drawing for explaining each component of a three-dimensional virtual model fusion providing system according to one embodiment disclosed in the present application.

[0023] FIG. 3 is a flowchart illustrating a method for providing fusion of three-dimensional virtual models according to one embodiment disclosed in the present application.

[0024] FIG. 4 is a flowchart illustrating a method for generating a three-dimensional exterior model according to one embodiment disclosed in the present application.

[0025] FIG. 5 is a flowchart illustrating a method for generating a three-dimensional interior model according to one embodiment disclosed in the present application.

[0026] FIG. 6 is a drawing illustrating an example of a three-dimensional exterior model according to one embodiment of the present invention, and FIG. 7 is a drawing illustrating an example of a three-dimensional interior model according to one embodiment of the present invention.

[0027] FIG. 8 is a flowchart illustrating an outline-based volume model generation method according to one embodiment disclosed in the present application.

[0028] FIG. 9 is a drawing illustrating an example of an outline-based volume model according to one embodiment of the present invention.

[0029] FIG. 10 is a flowchart illustrating a method of matching between heterogeneous digital models according to one embodiment disclosed in the present application.

[0030] FIGS. 11 and 12 are drawings illustrating an example of alignment between heterogeneous digital models according to one embodiment disclosed in the present application.

[0031] FIG. 13 is a flowchart illustrating a scaling method between heterogeneous digital models according to one embodiment disclosed in the present application.

[0032] FIGS. 14 to 16 are drawings illustrating examples of scaling between heterogeneous digital models according to one embodiment disclosed in the present application.

[0033] FIG. 17 is a flowchart illustrating a method for providing a seamless switching interface between heterogeneous digital models according to one embodiment disclosed in the present application.

[0034] FIG. 18 is a drawing illustrating an example of a seamless switching interface between heterogeneous digital models according to one embodiment disclosed in the present application.

[0035] FIG. 19 is a drawing illustrating an exemplary structure and operating environment of a computing device according to one embodiment of the present invention.

[0036] Hereinafter, embodiments of the present disclosure are described in detail with reference to the drawings so that those skilled in the art can easily practice them. However, the present disclosure may be embodied in various different forms and is not limited to the embodiments described herein. In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and brevity.

[0037] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. This

[0038] In the document, each of the phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish a corresponding component from another corresponding component and do not limit the components in any other aspect (e.g., importance or order).

[0039] The term “module” as used in the various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0040] Various embodiments of this document may be implemented as software (e.g., a program) comprising one or more instructions stored in a storage medium (e.g., memory) readable by a machine or device. For example, the processor of the machine or device may call at least one of the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by a machine may be provided in the form of a non-transitory storage medium. Here, "non-transitory" simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and this term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily in the storage medium.

[0041] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0042] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0043] In this disclosure, the term "processor" may refer to hardware capable of performing functions and operations according to each designation described herein, computer program code capable of performing specific functions and operations, or an electronic recording medium loaded with computer program code capable of performing specific functions and operations. According to the embodiments, the operation of the processor may be defined and / or interpreted as the operation of a digital signage content providing device, but is not limited thereto. The processor may refer to a functional and / or structural combination of hardware for carrying out the technical concept of this disclosure and / or software for driving said hardware.

[0044] The three-dimensional digital model described below encompasses models expressed in three dimensions in virtual space and is not limited to a specific construction method. Since a three-dimensional digital model refers to an external three-dimensional virtual model, it is obvious that various applications are possible, such as constructing a digital twin based on such a three-dimensional digital model.

[0045] FIG. 1 is a drawing illustrating a three-dimensional virtual model fusion providing system according to one embodiment disclosed in the present application.

[0046] A three-dimensional virtual model fusion providing system may include a scanning device (100), a computing device (300), and a user terminal (500).

[0047] The scan device (100) is an electronic device for generating a shooting data set at each shooting point in real space.

[0048] The scan device (100) may include a first scan device (101) for photographing an outdoor space to create a digital model of an outdoor space, and a second scan device (102) for photographing an indoor space to create a digital model of an indoor space.

[0049] For example, the first scan device (101) can collect a set of shooting data from multiple shooting points in the air above an outdoor space as a flying vehicle such as a drone. For example, such a set of shooting data may be a set of images taken from multiple shooting points, but is not limited thereto.

[0050] For example, the second scan device (102) may include a camera and a distance measuring sensor for collecting a shooting data set from multiple shooting points in an indoor space. For example, such a shooting data set may be a multiple data set captured from multiple shooting points. Each data set may include a 360-degree panoramic image, 360-degree depth data (e.g., Depthamp, etc.) and indoor location information (e.g., SLAM-based positioning data, etc.) for the shooting points in the indoor space, which are individually acquired at the respective shooting points.

[0051] The computing device (300) can construct a three-dimensional virtual model corresponding to the real space based on a plurality of data sets collected from the scanning device (100).

[0052] The computing device (300) can create a three-dimensional digital model of an outdoor space using a plurality of image sets provided from the first scan device (101), and the three-dimensional digital model of an outdoor space created in this way is described below as a 'three-dimensional exterior digital model' or 'exterior digital model'.

[0053] A three-dimensional exterior digital model has the characteristic of representing a wide area of ​​space and can be generated based on data with a relatively low amount of information (e.g., resolution). Figure 6 illustrates an example of such a three-dimensional exterior digital model.

[0054] Additionally, the computing device (300) can create a three-dimensional digital model of an indoor space using a plurality of data sets provided from the second scanning device (102), and the three-dimensional digital model of an indoor space created in this way is described below as a 'three-dimensional interior digital model' or 'interior digital model'.

[0055] A 3D interior digital model can be generated based on data that has a relatively high amount of information and features the ability to represent a relatively narrow space, particularly an indoor space. Figure 7 illustrates an example of such a 3D interior digital model.

[0056] Meanwhile, the 3D exterior digital model and the 3D interior digital model are constructed in different ways based on different shooting data. Therefore, these two models have low consistency with each other. That is, since the two models differ in orientation, size, etc., in order to simultaneously and fusedly represent the two models, the computing device (300) secures external uniformity of the two models through alignment and scaling of the heterogeneous 3D digital models constructed in these different ways, and also provides an interface that provides smooth switching between the two models.

[0057] The computing device (300) can provide a fused digital environment of heterogeneous 3D virtual models to a user terminal (500) based on the fusion processing of two heterogeneous 3D virtual models in this way.

[0058] FIG. 2 is a drawing for explaining each component of a three-dimensional virtual model fusion providing system according to one embodiment disclosed in the present application.

[0059] The computing device (300) may include a processing unit (310) and a system memory (320).

[0060] The processing unit (310) may include, as an example, at least one of a microprocessor, a central processing unit, a processor core, a multi-core processor, a multiprocessor, an ASIC (Application-Specific Integrated Circuit), or an FPGA (Field Programmable Gate Array), but is not limited thereto.

[0061] The computing device (300) includes modules that constitute different types of digital models, and may include an outline model and an integration interface for fusing these different types of digital models. These models or modules may be artificial intelligence models or software modules that operate individually and may be implemented as part of a processing unit (310), but are not limited thereto, and a separate processor may be used for each module.

[0062] System memory (320) can store instructions (or programs) executable by the processing unit (310). System memory (320) may include volatile memory or non-volatile memory. Volatile memory may be implemented as dynamic random access memory (DRAM), static random access memory (SRAM), thyristor RAM (T-RAM), zero capacitor RAM (Z-RAM), or twin transistor RAM (TTRAM). Non-volatile memory can be implemented as EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory, MRAM (Magnetic RAM), Spin-Transfer Torque (STT)-MRAM, Conductive Bridging RAM (CBRAM), FeRAM (Ferroelectric RAM), PRAM (Phase change RAM), Resistive RAM (RRAM), Nanotube RRAM, Polymer RAM (PoRAM), Nano Floating Gate Memory (NFGM), holographic memory, Molecular Electronic Memory Device, or Insulator Resistance Change Memory.

[0063] The computing device (300) may be implemented as a server, but is not limited thereto.

[0064] The first scan device (101) may include a camera for acquiring an image, and the second scan device (102) may include a camera for acquiring an image and a depth scanner for acquiring depth data. The depth scanner may include a predetermined sensor for measuring distance, such as a LiDAR sensor, an infrared sensor, an ultrasonic sensor, etc. Alternatively, the depth scanner may include a stereocamera, a stereoscopic camera, a 3D depth camera, etc., which can measure distance information in place of a sensor to acquire depth data.

[0065] The user terminal (500) is a device used by the user, and the user can be connected to the computing device (300) through the user terminal (500).

[0066] The user terminal (500) is an electronic device on which software such as an application runs, for example, a mobile phone, a smartphone, a laptop computer, a digital broadcasting terminal, a PDA (personal digital assistants), a PMP (portable multimedia player), a navigation device, a personal computer (PC), a tablet PC, an ultrabook, and a wearable device, for example, a smartwatch, a smart glass, a head-mounted display (HMD), a Virtual Reality (V) device, or an Augmented Reality (AR) device.

[0067] FIG. 3 is a flowchart illustrating a method for providing fusion of heterogeneous three-dimensional virtual models according to one embodiment disclosed in the present application.

[0068] The first scan device (101) can generate multiple image sets by taking photos at multiple shooting points in an outdoor space (S301), and can transmit multiple image sets of the outdoor space to a computing device (300) (S303). These shooting and transmission may not be performed simultaneously, and various methods of collection and transmission are possible, such as collecting multiple image sets from the air using a drone and then transmitting the collected multiple image sets to the computing device (300) all at once through a terminal such as a smartphone linked with the drone.

[0069] The second scan device (102) can generate multiple data sets by taking photos at multiple shooting points in an indoor space (S302). The second scan device (102) can transmit multiple data sets of the real space to a computing device (300) (S304).

[0070] The computing device (300) receives shooting data, i.e., a plurality of image sets and a plurality of data sets, from the first scanning device (101) and the second scanning device (102), and can use these to generate a three-dimensional virtual space model corresponding to an outdoor space / indoor space, respectively.

[0071] The computing device (300) can generate a three-dimensional interior digital model of an indoor space using a plurality of data sets provided from the second scanning device (102) (S305).

[0072] The computing device (300) can generate a three-dimensional exterior digital model of an outdoor space using a plurality of image sets provided from the first scan device (101) (S306).

[0073] The computing device (300) can perform preprocessing for the fusion of a three-dimensional exterior digital model and a three-dimensional interior digital model. This preprocessing can convert the digital model into a volume model to smoothly handle alignment or scaling between the two digital models.

[0074] The computing device (300) can generate an outline-based volume model for an interior digital model and an exterior digital model (S307). Hereinafter, a volume model generated from a three-dimensional interior digital model is referred to as an "interior volume model" or "three-dimensional interior volume model," and a volume model generated from a three-dimensional exterior digital model is referred to as an "exterior volume model" or "three-dimensional exterior volume model." These volume models can be easily understood by referring to FIGS. 8 and 9 and the following description.

[0075] The computing device (300) can perform processing for fusion between a three-dimensional exterior digital model and a three-dimensional interior digital model. This processing for fusion may include alignment to align the orientation between the two digital models and scaling to align the ratio between the two digital models.

[0076] For example, a computing device (300) can perform alignment between two digital models based on external alignment between an interior volume model and an exterior volume model (S308).

[0077] For example, the computing device (300) can perform scaling between two digital models based on an external size comparison between an interior volume model and an exterior volume model (S309).

[0078] Afterwards, the computing device (300) can implement a seamless switching interface between different heterogeneous digital models based on such alignment and scaling (S310).

[0079] Hereinafter, with reference to FIGS. 4 to 18, various specific embodiments for such a computing device (300) and the fusion of heterogeneous digital models performed thereon will be described.

[0080] FIG. 4 is a flowchart illustrating a method for generating a three-dimensional exterior model according to one embodiment disclosed in the present application.

[0081] The computing device (300) can prepare multiple sets of images acquired from multiple shooting points in an outdoor space (S410).

[0082] A computing device (300) can extract feature points for each image (S420) and estimate the camera pose for each image based on common feature point matching (S430). Here, feature points can be extracted using various algorithms such as SIFT (Scale-Invariant Feature Transform) or ORB (Oriented FAST and Rotated BRIEF) or using a deep learning model trained to extract feature points, and multiple feature points can be extracted even from a single image.

[0083] The computing device (300) can generate a point cloud by mapping feature points extracted from each image into a three-dimensional space based on the estimated camera pose (S440). The RANSAC algorithm can be used for this feature point matching, but is not limited thereto. For example, a point cloud can be generated by generating a sparse point cloud by reflecting feature points from multiple images into a single three-dimensional spatial coordinate system, and then generating a dense point cloud by extracting more points based on this sparse point cloud.

[0084] The computing device (300) can generate a three-dimensional mesh model based on the point cloud generated as above (S450).

[0085] Here, the 3D mesh model is a 3D model that represents a spatial volume constituting real space as a set of polygons, and the 3D mesh model can be represented by multiple vertices defined in 3D spatial coordinates, edges connecting two vertices, and faces partitioned by multiple edges. Here, the multiple vertices can be set based on a point cloud.

[0086] For example, the computing device (300) can generate a face based on three adjacent vertices, in which case the face may be a flat triangle set with three vertices. As another example, a square face may be set based on four vertices.

[0087] The computing device (300) can perform texturing by selecting a texturing image for each face of the three-dimensional mesh model (S460).

[0088] The texturing process is performed by selecting a texturing image for each face of the mesh model and then applying a texture to the face using the image region within the texturing image that corresponds to the area of ​​the face. Here, the texturing image may be selected by considering at least some of various factors, such as the angle, distance, and resolution relative to the face.

[0089] That is, the computing device (300) can perform texturing by mapping a portion of an image onto the surface of a three-dimensional mesh model to apply a texture similar to each area of ​​real space, and by completing this texturing, a three-dimensional exterior digital model corresponding to the real outdoor space can be constructed. FIG. 6 illustrates an example of such a three-dimensional exterior digital model, and it can be seen that a three-dimensional digital model for a considerably wide outdoor area is constructed.

[0090] FIG. 5 is a flowchart illustrating a method for generating a three-dimensional interior model according to one embodiment disclosed in the present application.

[0091] The computing device (300) can prepare multiple data sets acquired from multiple shooting points in an indoor space (S510).

[0092] For example, the computing device (300) can generate a 3D interior model, which is a 3D virtual model corresponding to real space, by using multiple data sets, namely images and depth data generated at various points in the room.

[0093] The image used here in the exterior model is a representation that encompasses all images expressed in color, and is not limited to images of a specific representation method. Therefore, color images can be applied in various ways, such as RFG images expressed in RGB (Red Green Blue) as well as CMYK images expressed in CMYK (Cyan Magenta Yellow Key). Depth data is a representation that encompasses data providing depth information regarding the subject space; for example, each pixel in the depth data may be distance information from the shooting point to each point in the subject space—a point in space corresponding to each pixel.

[0094] The computing device (300) can generate multiple 3D point clouds by reflecting depth data captured at each of the multiple shooting points on a 3D spatial coordinate system (S520).

[0095] The computing device (300) can generate a single integrated point cloud from a plurality of 3D point clouds based on the integration of 3D spatial coordinate systems (S530).

[0096] Above, an example was described of a method in which point clouds are constructed at each individual shooting location and then projected onto a common 3D coordinate system to form a single integrated point cloud, but this is not limited to this. Therefore, it is also possible to modify the method by first projecting each shooting point onto a common 3D coordinate system, generating individual point clouds at each projected 3D shooting point location, and then integrating them.

[0097] This integration requires a process of mapping each shooting point onto three-dimensional spatial coordinates, and for this purpose, coordinate information at each shooting point in an indoor space can be used.

[0098] The computing device (300) can generate a three-dimensional mesh model based on an integrated point cloud (S540), and generate a three-dimensional interior digital model by selecting a texturing image for each face of the three-dimensional mesh model and performing texturing (S550). Since this three-dimensional mesh model and texturing process correspond to what was described above with reference to FIG. 4, a redundant explanation is omitted here.

[0099] Through the above process, a three-dimensional digital interior model corresponding to a real indoor space can be constructed. Figure 7 illustrates an example of such a three-dimensional digital interior model, and it can be seen that a three-dimensional digital model is constructed for the indoor space with high precision and resolution.

[0100] FIG. 8 is a flowchart illustrating an outline-based volume model generation method according to one embodiment disclosed in the present application.

[0101] Here, a volume model refers to a model that simply represents the digital model based on its external shape, that is, its outline. As shown in the example in Fig. 9, a volume model 910 for a 3D interior digital model is illustrated. Such a volume model can be represented as polyhedra or combinations of polyhedra; for instance, if the subject is a building, it can be represented as a rectangular prism or a combination of rectangular prisms. The steps for generating a volume model are explained with reference to Fig. 8.

[0102] Referring to FIG. 8, the computing device (300) can extract a plurality of boundary line outlines for a three-dimensional digital model (S810).

[0103] For example, to extract an outline, the computing device (300) can detect flat surfaces such as ceilings, floors, and walls in a three-dimensional digital model, and extract boundary lines where these planes meet to set them as outlines. The computing device (300) may use algorithms such as RANSAC (Random Sample Consensus) for plane detection, but is not limited thereto. The computing device (300) can perform simplification processing when performing boundary extraction. For example, irregularities or complex boundary lines can be simplified into a straight line by aligning them with surrounding boundary lines. This is because, since the volume model is a model for estimating the overall volume, detailed shape parts are omitted, thereby making it easier to perform the alignment and scaling operations described later.

[0104] The computing device (300) can perform outline simplification processing on the extracted multiple boundary line outlines based on whether they are the outermost (S820).

[0105] If a boundary line exists on the interior side, for example, a boundary line for the interior of a room, it is not used when comparing the overall volume; therefore, if the boundary line outline is not the outermost line, it can be deleted to simplify the process.

[0106] The computing device (300) can set up a three-dimensional volume model represented by polyhedra and combinations of polyhedra based on a simplified processed outline (S830). FIG. 9 illustrates an example of a volume model after the entire process described above is completed, and as shown, it can be seen that the volume model is processed into a rectangular prism based on the outermost outline.

[0107] The computing device (300) can generate a volume model for each of the interior digital model and the exterior digital model.

[0108] That is, the computing device (300) can generate a three-dimensional interior volume model based on a three-dimensional interior digital model and generate a three-dimensional exterior volume model based on a three-dimensional exterior digital model.

[0109] By using these interior volume models and exterior volume models to calculate alignment or scaling between the two volume models, and by reflecting the calculated alignment or scaling results in their source models, the interior digital model and exterior digital model, it is possible to process alignment or scaling between the interior digital model and the exterior digital model more efficiently.

[0110] FIG. 10 is a flowchart illustrating a method of matching between heterogeneous digital models according to one embodiment disclosed in the present application.

[0111] The computing device (300) can attempt an external alignment between the interior volume model and the exterior volume model (S1010).

[0112] Here, since the volume models can be polygonal—e.g., rectangular prisms—the external alignment between the two volume models can be attempted quickly and efficiently using plane-based alignment.

[0113] If the alignment direction is specified as one—that is, if there is only one alignment direction (S1020, no)—the computing device (300) can align the 3D interior volume model and the 3D exterior volume model with the corresponding alignment direction—that is, the unique alignment direction—(S1031). This case applies when the two volume models are in the form of a set of rectangular prisms, etc. For example, if the two volume models are in the form of two rectangular prisms joined at a predetermined angle, such as '<', only one unique alignment is possible in this case.

[0114] The computing device (300) can align two digital models, namely a three-dimensional interior digital model and a three-dimensional exterior digital model, in correspondence with the mutual alignment of these two volume models.

[0115] Here, alignment refers to the positional alignment between two volume models, or more broadly, two digital models. From the result of this alignment, the positional association between the two models—e.g., representation through a common 3D coordinate system—can be established. For example, it can be used to determine the position between the two models when implementing them in a single virtual environment. As another example, it can be used to set the position and orientation when a transition occurs between the two digital models.

[0116] Meanwhile, there may be cases where there are two or more alignment directions for the two volume models (S1020, e.g.). In the example of FIG. 11, figure (a) illustrates an example where the interior digital model—red solid line—and the exterior digital model—blue dotted line—are viewed from a planar direction, and in this case, there are two cases where the interior is aligned by changing the vertical direction by 180 degrees.

[0117] Therefore, in this case, for more accurate matching, the computing device (300) can perform matching based on common object identification (S1030 to S1040).

[0118] That is, the computing device (300) can identify indoor and outdoor common objects in two digital models, and then project the identified indoor and outdoor common objects onto each of the two volume models (S1030).

[0119] Here, indoor / outdoor common objects refer to objects that are displayed in both the exterior model and the interior model, and for example, such indoor / outdoor common objects may be windows.

[0120] In the example of Fig. 11, Figure (b) illustrates an example of a volume model projected with a window, which is an indoor and outdoor common object.

[0121] The computing device (300) can set the alignment between volume models based on the location and relationship of indoor and outdoor common objects to perform alignment (S1040).

[0122] Figure 12 illustrates two situations in which the figure (b) of Figure 11 can be positionally aligned, and Figure 12 (a) illustrates an example in which two volume models are well aligned by projecting the position of an indoor-outdoor common object. That is, Figure 12 (b) illustrates a case of incorrect alignment that would occur if there were no indoor-outdoor common object, and if alignment is performed using more indoor-outdoor common objects, precise positional alignment can be achieved as shown in Figure 12 (a).

[0123] In one embodiment, the identification and alignment steps (S1030 to S1040) of the indoor and outdoor common objects described above may be performed only when the alignment direction is not unique. Since the process of identifying indoor and outdoor common objects requires significant computing resources and time, by executing steps S1030 to S1040 only when the alignment direction is not unique, the alignment can be performed more efficiently.

[0124] FIG. 13 is a flowchart illustrating a scaling method between heterogeneous digital models according to one embodiment disclosed in the present application.

[0125] Here, scaling refers to deriving a mutually corresponding match in size between two digital models.

[0126] Referring to FIG. 13, the computing device (300) can convert a three-dimensional volume model into a rectangular volume model of the same ratio (S1310).

[0127] That is, the computing device (300) can convert a three-dimensional interior volume model into an interior rectangular volume model of the same proportion based on the external size, and convert a three-dimensional exterior volume model into an exterior rectangular volume model of the same proportion based on the external size.

[0128] Figure 14 illustrates such an example. Figure 14 (a) illustrates a volume model, and Figure 14 (b) illustrates a rectangular volume model. Figure (b) shows an example of making a single rectangular prism while filling an empty area.

[0129] In this way, the reason for converting the 3D volume model into a 3D rectangular volume model of the same ratio is to calculate only the scale in this process, that is, to enable the comparison of scales without going through a separate alignment process.

[0130] The computing device (300) can calculate the size of the exterior rectangular volume model and the interior rectangular volume model (S1320), and based on this, calculate the scale ratio between the two rectangular volume models.

[0131] The computing device (300) can adjust the scale ratio of one volume model based on one volume model so that the exterior rectangular volume model and the interior rectangular volume model have volumes of equal range (S1330). The adjusted scale ratio can be stored and managed as scale ratio information between the two models.

[0132] In one embodiment, the computing device (300) can adjust the scale ratio of the interior rectangular volume model based on the exterior rectangular volume model. In this example, the exterior digital model can be configured based on absolute coordinates such as latitude and longitude information, and the interior digital model can be configured based on relative coordinates such as SLAM. That is, since the exterior digital model implements the digital model based on absolute coordinates, the size information of the model can be determined using the absolute coordinates. Therefore, the scale ratio of the interior rectangular volume model can be adjusted based on the exterior rectangular volume model.

[0133] Figure 15 (a) illustrates an example of an interior rectangular volume model, Figure 15 (b) illustrates an example of an exterior rectangular volume model, and Figure 16 illustrates an example of scaling between the two rectangular volume models of Figure 15. Figure 16 (a) illustrates an example before scaling, and Figure 16 (b) illustrates an example after scaling.

[0134] As shown in figure (b) of Fig. 16, the computing device (300) exemplifies an adjustment to scale up the interior rectangular volume model based on the exterior rectangular volume model so that the interior rectangular volume model and the exterior rectangular volume model become volumes of equal range.

[0135] Here, the term "equal range volumes" includes not only cases where the two volume models are identical, but also cases where a pre-set scale clearance is included between the two volume models. The scale clearance here is set considering the wall thickness of the building, and this can be set in advance or in proportion to the size of the volume models. Since this scale clearance is located outside the interior rectangular volume, a clearance may exist between the two rectangular volume models as shown in Figure 16 (b).

[0136] The computing device (300) can set the scale ratio of the interior and exterior digital models according to the scale ratio between the two rectangular volume models adjusted as above (S1340).

[0137] The scale ratio here is a value for the degree of scale of enlarging or reducing any one digital model, and the computing device (300) can individually store and manage the scale ratio between each model.

[0138] FIG. 17 is a flowchart illustrating a method for providing a seamless switching interface between heterogeneous digital models according to one embodiment disclosed in the present application.

[0139] The computing device (300) can adjust the 3D interior digital model and the 3D exterior digital model based on alignment and scaling to obtain a seamless switching interface. This seamless switching interface is an interface that provides seamless switching between the 3D interior digital model and the 3D exterior digital model, and by reflecting the positional alignment and scale adjustment between the two digital models during this switching, the user can be provided with the feeling of experiencing the same digital space even when switching between two different types of models.

[0140] Referring to FIG. 17, the computing device (300) can display a label for the interior digital model on the exterior digital model (S1710).

[0141] For example, a representation of an interior digital model can be made by displaying at least some three-dimensional space corresponding to the interior digital model in the exterior digital model.

[0142] For example, a cover for an interior digital model may be displayed by overlaying a 3D volume model for the interior digital model onto at least some 3D space of the exterior digital model.

[0143] In Fig. 18, an example is shown in which a blue rectangular prism is displayed as a marker for the interior digital model in a part of the three-dimensional space corresponding to the building in the exterior digital model.

[0144] When a user generates an event such as a click on an interior digital model cover while viewing an exterior digital model, the computing device (300) can load and display an interior digital model that is scaled and aligned with the corresponding 3D part of the exterior digital model to the user (S1720).

[0145] Figure 19 illustrates such an example. When a user clicks on the blue rectangular prism, which is the cover of the interior digital model, while viewing the exterior digital model on the left, the user's screen is switched to the interior digital model as shown in the image on the right. At that time, the transition can be made seamlessly, in that the scale is gradually expanded while the position is aligned.

[0146] For example, such a seamless transition can be implemented as a transition between different digital viewers displaying different objects; that is, the viewer of the interior digital model being transitioned can be seamlessly transitioned such that it is initially displayed with the scale and alignment matched, and then gradually enlarged to display the entire viewer.

[0147] In this way, the user can experience various interactions with the interior digital model displayed as a full viewer, and for this purpose, the computing device (300) can change the size or orientation of the interior digital model according to the user's operation (S1730).

[0148] FIG. 19 is a drawing illustrating an exemplary environment of a computing device according to one embodiment of the present invention.

[0149] FIG. 19 is intended to provide a general and simplified description of a suitable computing environment in which various embodiments of a computing device may be implemented. Referring to FIG. 19, a computing device (100) is illustrated.

[0150] The computing device (100) may include at least a processing unit (310) and a system memory (320).

[0151] The computing device may include multiple processing units that cooperate when executing a program. Depending on the exact configuration and type of the computing device, the system memory (301) may be volatile (e.g., RAM), non-volatile (e.g., ROM, flash memory, etc.), or a combination thereof. The system memory (320) includes a suitable operating system (330) for controlling the operation of the platform, which may be, for example, the WINDOWS operating system from Microsoft. The system memory (320) may include one or more software applications, such as program modules, applications, etc.

[0152] The computing device may include additional data storage devices (340), such as magnetic disks, optical disks, or tapes. These additional storage devices may be removable storage and / or fixed storage. Computer-readable storage media may include volatile and non-volatile, removable and fixed media implemented by any method or technique for storing information such as computer-readable instructions, data structures, program modules, or other data. System memory (320) and data storage devices (340) are merely examples of computer-readable storage media. Computer-readable storage media may include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory techniques, CD-ROM, DVD or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that stores desired information and can be accessed by the computing device (100).

[0153] Input devices (350) of a computing device may include, for example, a keyboard, a mouse, a pen, a voice input device, a touch input device, and comparable input devices. Output devices (360) may include, for example, a display, a speaker, a printer, and other types of output devices. Since these devices are widely known in the art, a detailed description is omitted.

[0154] The computing device may include a communication device (370) that allows the device to communicate with other devices through a network in a distributed computing environment, such as a wired / wireless network, a satellite link, a cellular link, a local area network, and a comparable mechanism. The communication device (370) is one example of a communication medium, and the communication medium may contain computer-readable instructions, data structures, program modules, or other data. For example, the communication medium includes, but is not limited to, wired media such as a wired network or direct wired connection, and wireless media such as acoustic, RF, infrared, and other wireless media.

[0155] Although the embodiments have been described above with reference to limited examples and drawings, those skilled in the art can make various modifications and variations from the description above. For example, suitable results can be achieved even if the described techniques are performed in a different order than described, and / or the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0156] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below.

[0157] Although specific embodiments have been described in the detailed description of this document, it will be obvious to those skilled in the art that various modifications are possible within the scope of this document.

[0158]

[0159] [Sasa]

[0160] This invention was filed overseas with the support of the following research project supported by the government of the Republic of Korea.

[0161] Project ID: 2710018107

[0162] Assignment No.: 00460360

[0163] Ministry Name: Ministry of Science and ICT

[0164] Project Management (Specialized) Agency Name: Korea Institute of Information & Communications Technology Planning & Evaluation

[0165] Research Project Name: Development of Source Technologies for the SW Computing Industry

[0166] Research Project Title: Development and Standardization of Spatial Convergence Digital Twin Technology Fusion of Wide-Area Exterior Digital Models and High-Density Interior Digital Models

[0167] Project Executing Organization Name: 3i Co., Ltd.

[0168] Research Period: July 1, 2024 – December 31, 2024

[0169]

[0170] According to one embodiment of the present invention, by expressing heterogeneous 3D models configured in different ways through a single interface, it is possible to provide a converged digital twin that offers heterogeneous digital models in a single converged environment, which has the effect of being highly applicable to industry.

[0171] In addition, by generating an outline-based volume model for heterogeneous 3D digital models configured in different ways and performing alignment, it is possible to align heterogeneous digital models efficiently and quickly with fewer resources, which has high potential for industrial application.

[0172] In addition, by performing scaling on heterogeneous 3D digital models configured in different ways based on an outline-based volume model, it is possible to perform accurate scaling efficiently and with minimal resources, which makes it highly suitable for industrial use.

[0173] Furthermore, it provides the effect of enabling the seamless switching between heterogeneous 3D digital models configured in different ways within a single interface. Additionally, by aligning and scaling the heterogeneous digital models, it offers a high level of user experience and allows more complex digital models to be displayed through a single user interface, thus demonstrating high potential for industrial application.

Claims

1. A three-dimensional alignment method performed on a computing device that provides a three-dimensional virtual model of real space, A step of generating a three-dimensional interior digital model of the indoor space using a plurality of data sets collected in the indoor space; A step of generating a three-dimensional exterior digital model of an outdoor space using a plurality of image sets collected in the outdoor space; A step of generating an outline-based 3D volume model for the above 3D interior digital model and the above 3D exterior digital model; and A step of performing alignment between the 3D interior digital model and the 3D exterior digital model based on external alignment between the 3D interior volume model and the 3D exterior volume model; comprising 3D registration method for heterogeneous 3D virtual models.

2. In paragraph 1, the plurality of data sets are, Each is individually acquired from multiple shooting points within the above indoor space, and 360-degree panoramic image; 360-degree panoramic depth data; and Indoor location information for each of the plurality of shooting points; comprising, 3D registration method for heterogeneous 3D virtual models.

3. In paragraph 2, the step of generating the three-dimensional interior digital model is, For each of the above plurality of data sets, a step of generating a plurality of individual 3D point clouds by reflecting depth data captured at a corresponding shooting point on individual 3D spatial coordinates; A step of generating a single integrated 3D point cloud by integrating the plurality of individual 3D point clouds by reflecting the individual 3D spatial coordinates into a single integrated 3D spatial coordinate based on the indoor location information; A step of generating a 3D mesh model based on the above integrated 3D point cloud; and The step of generating the 3D interior digital model by individually selecting a texturing image for each of the plurality of faces included in the 3D mesh model and performing texturing; 3D registration method for heterogeneous 3D virtual models.

4. In paragraph 3, the plurality of image sets are, A plurality of images acquired by a drone from a plurality of shooting points located above the outdoor space, 3D registration method for heterogeneous 3D virtual models.

5. In paragraph 4, the step of generating the three-dimensional exterior digital model is, A step of extracting feature points for each of the plurality of images included in the plurality of image sets; A step of estimating the camera pose for each of the plurality of images based on common feature point matching; A step of generating a 3D point cloud by mapping feature points of each image into a 3D space based on the estimated camera pose; A step of generating a 3D mesh model based on the above 3D point cloud; and The step of generating the 3D exterior digital model by individually selecting a texturing image for each of the plurality of faces included in the 3D mesh model and performing texturing; 3D registration method for heterogeneous 3D virtual models.

6. In paragraph 5, the step of generating the outline-based three-dimensional volume model is, A step of extracting multiple boundary outlines for a 3D digital model; A step of simplifying the extracted multiple boundary line outlines based on whether they are the outermost; and A step of generating a three-dimensional volume model using polyhedra and combinations of polyhedra based on a simplified outline; comprising 3D registration method for heterogeneous 3D virtual models.

7. In claim 6, the step of generating the outline-based three-dimensional volume model is, A step of generating a 3D interior volume model based on the above 3D interior digital model; and A step of generating a 3D exterior volume model based on the above 3D exterior digital model; comprising 3D registration method for heterogeneous 3D virtual models.

8. In claim 7, the step of performing alignment between the three-dimensional interior digital model and the three-dimensional exterior digital model is: A step of performing external fitting between the three-dimensional interior volume model and the three-dimensional exterior volume model based on planar alignment; A step of mutually aligning the three-dimensional interior volume model and the three-dimensional exterior volume model in the corresponding custom direction once the custom direction is specified; and A step of aligning the 3D interior digital model and the 3D exterior digital model in correspondence with the mutual alignment above; comprising 3D registration method for heterogeneous 3D virtual models.

9. In a text-based personal information identification device, At least one processor; and It includes memory for storing instructions, When the above instructions are executed individually or collectively by the at least one processor, the processor, A three-dimensional interior digital model of the indoor space is generated using multiple data sets collected from the indoor space, and A three-dimensional exterior digital model of the outdoor space is generated using a plurality of image sets collected from the outdoor space, and An outline-based 3D volume model is generated for the above 3D interior digital model and the above 3D exterior digital model, and Based on the external fitting between the 3D interior volume model and the 3D exterior volume model, performing alignment between the 3D interior digital model and the 3D exterior digital model, A computing device that performs 3D alignment for heterogeneous 3D virtual models.

10. In Paragraph 9, When the above instructions are executed individually or collectively by the at least one processor, the processor, In order to generate the above outline-based 3D volume model, Extract multiple boundary outlines for a 3D digital model, and Simplify the extracted multiple boundary line outlines based on whether they are the outermost edge, and Generating a 3D volume model using polyhedra and combinations of polyhedra based on simplified outlines, A computing device that performs 3D alignment for heterogeneous 3D virtual models.

11. In Paragraph 10, When the above instructions are executed individually or collectively by the at least one processor, the processor, In order to generate the above outline-based 3D volume model, Based on the above 3D interior digital model, a 3D interior volume model is generated, and Generating a 3D exterior volume model based on the above 3D exterior digital model, A computing device that performs 3D alignment for heterogeneous 3D virtual models.

12. In Paragraph 11, When the above instructions are executed individually or collectively by the at least one processor, the processor, In order to perform alignment between the above 3D interior digital model and the above 3D exterior digital model, Based on planar alignment, external fitting between the above 3D interior volume model and the above 3D exterior volume model is performed, and When a custom direction is determined, the above 3D interior volume model and the above 3D exterior volume model are aligned with each other in the corresponding custom direction, and A method for aligning the 3D interior digital model and the 3D exterior digital model in correspondence with the above mutual alignment, A computing device that performs 3D alignment for heterogeneous 3D virtual models.

Images