Virtual space providing method supporting partial re-photographing, and computing device therefor

Abstract

A virtual space providing method supporting partial re-photographing according to an embodiment of the present disclosure is performed by a computing device, and comprises the steps of: receiving a first data set acquired at a plurality of photographing points of an indoor space at a first time point; constructing a first virtual space survey for the indoor space at the first time point by using the first data set; receiving a second data set acquired in at least a partial area of the indoor space at a second time point; constructing a second virtual space survey of the second time point for the at least a partial area of the indoor space by using the second data set; and mapping photographing point information of the second data set to spatial information of the first virtual space survey, and matching the second virtual space survey in association with the first virtual space survey on the basis of a mapping relationship.

Description

Method for providing a virtual space supporting partial re-shooting and computing device for the same

[0001] The present invention relates to a method for providing a virtual space that supports partial re-shooting and a computing device for the same.

[0002] Virtual space refers to a technology that provides users with a virtualized version of real-world space in an online environment, thereby offering the sensation of being present in the real world.

[0003] Typically, virtual space is provided in ways such as by providing 360-degree images captured from various shooting points in real space through a VR (Virtual Reality) viewer, or by providing a 3D reconstructed digital model to the user.

[0004] When constructing such virtual spaces, it is crucial to reflect the locations of various shooting points in the real world within the virtual space. Particularly in environments like building interiors, where it is difficult to accurately obtain shooting location information, technologies such as SLAM (Simultaneous Localization and Mapping) are being applied to overcome this challenge.

[0005] As such, due to the difficulty of indoor positioning in indoor spaces, shooting for virtualization is performed on the entire indoor space. That is, if only a part of the indoor space is subsequently re-shot, there is a limitation in that a positional correlation with the previously shot data cannot be established. Consequently, there is a limitation in that it is not possible to re-shot only a part of the indoor space, and the entire indoor space must be shot to construct a new virtual space.

[0006] One technical aspect of the present application aims to solve the aforementioned problem by providing a virtual space providing technology that supports partial re-shooting, which enables the automatic construction of an entire survey even if only a partial area of ​​an indoor space is re-shot, by supporting the re-shooting and partial virtualization reconstruction of a partial area in a virtual space survey that has been previously captured and constructed.

[0007] One technical aspect of the present application is to provide a virtual space providing technology that supports partial re-shooting, which determines the similarity with the shooting location in the existing survey when performing reconstruction of a partially re-shot virtual space, and automatically determines the location within the survey of the partially re-shot virtual space based on this similarity.

[0008] However, the problems to be solved in this disclosure are not limited to those mentioned above, and may be expanded in various ways without departing from the spirit and scope of this disclosure.

[0009] One technical aspect of the present invention proposes a method for providing a virtual space that supports partial re-shooting. The method for providing a virtual space that supports partial re-shooting is performed on a computing device and includes the steps of: receiving a first data set acquired from a plurality of shooting points of an indoor space at a first time point; constructing a first virtual space survey for the indoor space at the first time point using the first data set; receiving a second data set acquired from at least a portion of the indoor space at a second time point; constructing a second virtual space survey for at least a portion of the indoor space at the second time point using the second data set; and mapping the shooting point information of the second data set to the spatial information of the first virtual space survey, and aligning the second virtual space survey with the first virtual space survey based on the mapping relationship.

[0010] Another technical aspect of the present invention proposes a computing device that performs the provision of a virtual space supporting partial re-shooting. The computing device includes at least one processor and a memory that stores instructions. When the instructions are executed individually or collectively by the at least one processor, the processor receives a first data set acquired from a plurality of shooting points of an indoor space at a first time point, constructs a first virtual space survey of the indoor space at the first time point using the first data set, receives a second data set acquired from at least a part area of ​​the indoor space at a second time point, constructs a second virtual space survey of at least a part area of ​​the indoor space at the second time point using the second data set, maps the shooting point information of the second data set to the spatial information of the first virtual space survey, and aligns the second virtual space survey with the first virtual space survey based on the mapping relationship.

[0011] Another technical aspect of the present invention proposes a storage medium. The storage medium is a storage medium that stores computer-readable instructions, wherein, when executed by a computing device, the instructions cause the computing device to perform the following operations: receiving a first data set acquired from a plurality of shooting points of an indoor space at a first time point; constructing a first virtual space survey of the indoor space at a first time point using the first data set; receiving a second data set acquired from at least a portion of the indoor space at a second time point; constructing a second virtual space survey of at least a portion of the indoor space at a second time point using the second data set; and mapping the shooting point information of the second data set to the spatial information of the first virtual space survey and associating the second virtual space survey with the first virtual space survey based on the mapping relationship.

[0012] The means for solving the above-mentioned problem do not enumerate all features of the present invention. Various means for solving the problem of the present invention may be understood in more detail by referring to specific embodiments in the following detailed description.

[0013] According to the present application, there is one or more of the following effects.

[0014] According to one embodiment of the present invention, by supporting re-shooting and partial virtualization reconstruction of partial areas in a previously captured and constructed virtual space survey, an entire survey can be automatically constructed even if only a partial area of ​​an indoor space is re-shot, thereby increasing the convenience of capturing the virtual space and expanding the usability for users.

[0015] According to one embodiment of the present invention, when performing reconstruction of a partially re-shot virtual space, the similarity with the shooting location in the existing survey is determined, and based on this similarity, the location of the partially re-shot virtual space within the survey is automatically determined, thereby enabling automatic mapping of the location of the partially re-shot virtual space even during partial re-shooting, which provides the user with convenience in the partial re-shooting environment while providing an accurate virtual space.

[0016] The effects of the present application are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art from the description in the claims.

[0017] FIG. 1 is a drawing illustrating a virtual space providing system according to one embodiment disclosed in the present application.

[0018] FIG. 2 is a drawing for explaining each component of a virtual space providing system according to one embodiment disclosed in the present application.

[0019] FIG. 3 is a flowchart illustrating a method for providing a virtual space that supports partial re-photography according to one embodiment disclosed in the present application.

[0020] FIG. 4 is an example screen for explaining the alignment process of a virtual space survey according to one embodiment disclosed in the present application.

[0021] FIG. 5 is a flowchart illustrating a procedure for mapping a shooting point of a second virtual space survey to a first virtual space survey according to one embodiment disclosed in the present application.

[0022] FIG. 6 is a flowchart illustrating a procedure for aligning a second virtual space survey with a first virtual space survey according to one embodiment.

[0023] FIG. 7 is a flowchart illustrating a procedure for reflecting a matched virtual space survey in a user interface according to one embodiment.

[0024] FIG. 8 is an example diagram illustrating a temporal comparison function according to one embodiment.

[0025] FIG. 9 is a drawing illustrating an exemplary environment of a computing device according to one embodiment of the present invention.

[0026] Hereinafter, embodiments of the present disclosure are described in detail with reference to the drawings so that those skilled in the art can easily practice them. However, the present disclosure may be embodied in various different forms and is not limited to the embodiments described herein. In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and brevity.

[0027] The various embodiments of this document and the terms used herein are not intended to limit the technical features described herein to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. This

[0028] In the document, each of the phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish a corresponding component from another corresponding component and do not limit the components in any other aspect (e.g., importance or order).

[0029] The term “module” as used in the various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0030] Various embodiments of this document may be implemented as software (e.g., a program) comprising one or more instructions stored in a storage medium (e.g., memory) readable by a machine or device. For example, the processor of the machine or device may call at least one of the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by a machine may be provided in the form of a non-transitory storage medium. Here, "non-transitory" simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and this term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily in the storage medium.

[0031] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0032] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0033] In this disclosure, the term "processor" may refer to hardware capable of performing functions and operations according to each designation described herein, computer program code capable of performing specific functions and operations, or an electronic recording medium loaded with computer program code capable of performing specific functions and operations. According to the embodiments, the operation of the processor may be defined and / or interpreted as the operation of a digital signage content providing device, but is not limited thereto. The term "processor" may refer to a functional and / or structural combination of hardware for carrying out the technical concept of this disclosure and / or software for driving said hardware.

[0034]

[0035] FIG. 1 is a drawing illustrating a virtual space providing system according to one embodiment disclosed in the present application.

[0036] A virtual space providing system may include a scanning device (100), a computing device (300), and a user terminal (500).

[0037] The scan device (100) is a portable electronic device comprising a camera and a distance measuring sensor, which is an electronic device for acquiring 360-degree panoramic images and 360-degree panoramic depth data at each shooting point in real space and generating a data set including indoor location information of the shooting point.

[0038] The scan device (100) is driven to scan at each of the multiple shooting points in real space, and can generate a data set for the multiple shooting points in real space.

[0039] The data set may include images and depth data acquired at each shooting point, and, depending on the embodiment, may further include indoor location data.

[0040] For example, the dataset is acquired individually for each of the multiple shooting points and may include not only images and depth data acquired at each shooting point but also relative position information—e.g., SLAM-based position data.

[0041] In one embodiment, the data set may include indoor location information mapped based on a floor plan of an indoor space. For example, a scan device (100) checks a floor plan of an indoor space, and the photographer of the scan device (100) may map the initial shooting point by marking it on the floor plan of the indoor space. Subsequently, based on the relative location information collected by the scan device (100), mapping between the coordinate information for the virtual survey and the floor plan of the indoor space may be performed by a computing device (300).

[0042] In one embodiment, the image and depth data acquired at the shooting point may be a 360-degree panoramic image acquired at the shooting point and 360-degree depth data acquired at the shooting point—e.g., a depth map.

[0043] The computing device (300) can construct a virtual model corresponding to the real space based on a data set collected from the scan device (100) and generate a virtual space survey based thereon.

[0044] Here, the virtual space survey is a virtual space that digitally reproduces a real indoor space using 360-degree panoramic images and / or depth data, and includes a structure in which position and direction information corresponding to multiple point-of-view points is aligned and mapped on a spatial coordinate system or a plan.

[0045] By selecting a shooting point displayed on the floor plan, the user can explore a 360-degree view (VR view) or a 3D mesh model view at that point, thereby experiencing the entire indoor space in an immersive way.

[0046] In addition, the virtual space survey enables partial re-capture and updating over time by linking the location information of the capture points with relative position coordinates (SLAM, etc.) or absolute position coordinates (plan map mapping, etc.), and allows the entire survey to be updated by registering or replacing only the changed parts.

[0047] To this end, the computing device (300) can receive a data set generated at each of several shooting points in the room from the scanning device (100). The computing device (300) can generate a virtual space survey corresponding to the real space using the data set, that is, images and depth data generated at each of several points in the room.

[0048] Here, the term "image" encompasses all images expressed in color and is not limited to images of a specific method of expression. Therefore, color images can be applied in various ways, such as RFG images expressed in RGB (Red Green Blue) as well as CMYK images expressed in CMYK (Cyan Magenta Yellow Key).

[0049] Depth data is a representation that encompasses data providing depth information about the subject space. For example, each pixel in the depth data may be distance information from the shooting point to each point in the subject space—a point in space corresponding to each pixel.

[0050] For example, the computing device (300) can construct a three-dimensional virtual model of an indoor space using a data set. The computing device (300) can generate a three-dimensional mesh model of the real space based on a data set for each of a plurality of shooting points in the real space received from the scanning device (100). In this way, the constructed three-dimensional virtual model can be provided to the user through a virtual space survey.

[0051] Here, the 3D mesh model is a 3D model that represents a spatial volume constituting real space as a set of polygons, and the 3D mesh model can be represented by multiple vertices, edges connecting two vertices, and faces divided by multiple edges.

[0052] Afterwards, the computing device (300) can perform texturing by mapping a portion of an image onto the surface of a three-dimensional mesh model to apply a texture similar to each region of real space, and by completing this texturing, can create a three-dimensional virtual model corresponding to real space.

[0053] The computing device (300) can provide the user with data sets acquired at different times for the same indoor space in association with each other. That is, it supports overlapping shooting of the same indoor space on different dates, or building a virtual space survey of the entire indoor space in the first shooting and building an additional virtual space survey by shooting only a part of the indoor space in the subsequent shooting. This is useful in terms of building and storing a historical virtual model that reflects the flow of time for the indoor space.

[0054] To this end, the computing device (300) receives a second data set captured at a second time point for a first virtual space survey constructed at a first time point, and maps the shooting point information of the second data set to the spatial information of the first virtual space survey, thereby associating and aligning the second virtual space survey with the first virtual space survey.

[0055] Accordingly, users can comprehensively explore past shooting results and partially updated shooting results on the same floor plan, which provides the advantage of being able to quickly respond to partial updates or remodeling situations of the indoor space.

[0056] The computing device (300) can provide the user terminal (500), etc., with the experience of a three-dimensional virtual space corresponding to real space by providing the three-dimensional virtual model created in this way.

[0057] The user terminal (500) is an electronic device that allows the user to experience a virtual space survey corresponding to the real space. The user can use the user terminal (500) to connect to the computing device (300) and experience a three-dimensional virtual model or 360-degree VR provided by the computing device (300).

[0058]

[0059] FIG. 2 is a drawing for explaining each component of a virtual space providing system according to one embodiment disclosed in the present application.

[0060] The computing device (300) may include a processing unit (310) and a system memory (320).

[0061] The processing unit (310) may include, as an example, at least one of a microprocessor, a central processing unit, a processor core, a multi-core processor, a multiprocessor, an ASIC (Application-Specific Integrated Circuit), or an FPGA (Field Programmable Gate Array), but is not limited thereto.

[0062] System memory (320) can store instructions (or programs) executable by the processing unit (310). System memory (320) may include volatile memory or non-volatile memory. Volatile memory may be implemented as dynamic random access memory (DRAM), static random access memory (SRAM), thyristor RAM (T-RAM), zero capacitor RAM (Z-RAM), or twin transistor RAM (TTRAM). Non-volatile memory can be implemented as EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory, MRAM (Magnetic RAM), Spin-Transfer Torque (STT)-MRAM, Conductive Bridging RAM (CBRAM), FeRAM (Ferroelectric RAM), PRAM (Phase change RAM), Resistive RAM (RRAM), Nanotube RRAM, Polymer RAM (PoRAM), Nano Floating Gate Memory (NFGM), holographic memory, Molecular Electronic Memory Device, or Insulator Resistance Change Memory.

[0063] The computing device (300) may be implemented as a server, but is not limited thereto.

[0064] The scan device (100) includes a camera for acquiring an image and a depth scanner for acquiring depth data.

[0065] A depth scanner may include a specific sensor for measuring distance, such as a LiDAR sensor, an infrared sensor, an ultrasonic sensor, etc. Alternatively, to acquire depth data, the depth scanner may include a stereocamera, a stereoscopic camera, a 3D depth camera (3D, depth camera), etc., which can measure distance information in place of a sensor.

[0066] The user terminal (500) is a device used by the user, and the user can be connected to the computing device (300) through the user terminal (500).

[0067] The user terminal (500) is an electronic device on which software such as an application runs, for example, a mobile phone, a smartphone, a laptop computer, a digital broadcasting terminal, a PDA (personal digital assistants), a PMP (portable multimedia player), a navigation device, a personal computer (PC), a tablet PC, an ultrabook, and a wearable device, for example, a smartwatch, a smart glass, a head-mounted display (HMD), a Virtual Reality (V) device, or an Augmented Reality (AR) device.

[0068]

[0069] FIG. 3 is a flowchart illustrating a method for providing a virtual space that supports partial re-photography according to one embodiment disclosed in the present application.

[0070] The scan device (100) can transmit a first data set of an indoor space for a first time point to a computing device (300).

[0071] In one embodiment, the first data set may include 360-degree panoramic images and 360-degree panoramic depth data acquired individually at a plurality of shooting points in an indoor space, and indoor location information for each shooting point. According to an embodiment, the indoor location information may include relative location information for the corresponding shooting point.

[0072] In step S302, the computing device (300) can construct a first virtual space survey at a first time point using the first data set.

[0073] For example, the computing device (300) receives a data set from the scanning device (100) and can use the data set to generate a three-dimensional space model or a 360-degree VR corresponding to real space to construct a first virtual space survey for a first viewpoint.

[0074] In one embodiment, the first virtual space survey may include a structure in which the result of virtualizing an indoor space based on a 360-degree VR or a 3D mesh model is linked with a floor plan showing the shooting point.

[0075] In one embodiment, the user can select a shooting point on a plan view to experience a 360-degree view of the point or a view of a 3D virtual model.

[0076] In step S303, the scan device (100) can generate a second data set for at least a portion of the indoor space at a second time point.

[0077] In one embodiment, the second data set is obtained by partial re-photography of an indoor space and may include a 360-degree panoramic image of the same format as the first data set, 360-degree panoramic depth data, and indoor location information by shooting point.

[0078] In step S304, the computing device (300) can construct a second virtual space survey at a second time point for a part area of ​​the indoor space using the second data set.

[0079] In step S305, the computing device (300) can map the shooting point information of the second data set to the spatial information of the first virtual space survey.

[0080] In one embodiment, the spatial information may include plan coordinate information constituting a virtual space survey. Additionally, the initial shooting point of the second data set may be mapped onto the plan of the first virtual space survey to establish a reference point for alignment.

[0081] For example, the second dataset can generate indoor location information based on a floor plan of an indoor space, and this floor plan of the indoor space can be used in common with the first virtual space survey generated at the first time point. Therefore, based on the floor plan of the indoor space, the first virtual space survey at the first time point and the second virtual space survey at the second time point can be mapped.

[0082] In one embodiment, the shooting point information may include pose information indicating the location and direction of the shooting point.

[0083] In one embodiment, a correspondence relationship can be determined by calculating the positional similarity between shooting points or the feature similarity of image and depth data.

[0084] In step S306, the computing device (300) can align the second virtual space survey with the first virtual space survey based on the mapping relationship.

[0085] In one embodiment, the second virtual space survey can be positioned on the spatial coordinate system of the first virtual space survey and positionally aligned.

[0086] In one embodiment, processing can be performed to replace and display at least some virtual space information of the first virtual space survey with virtual space information of the second virtual space survey according to the matched result.

[0087] In step S307, the computing device (300) can provide a user experience interface for the matched virtual space survey to the user terminal (500).

[0088] For example, a user terminal (500) may provide a UI to integrate the shooting result at a first point in time and the partial re-shooting result at a second point in time on the same planar view.

[0089] For example, the user terminal (500) may prioritize displaying the latest status by area, or provide functions such as time point switching, comparison view, and history timeline.

[0090]

[0091] FIG. 4 is an example screen for explaining the alignment process of a virtual space survey according to one embodiment disclosed in the present application.

[0092] Referring to Fig. 4, the virtual space survey includes a floor plan and a virtual space viewer, and in the illustrated example, a 360 VR screen is shown as the virtual space viewer.

[0093] Figure (a) is a screen illustrating a second virtual space survey. The second virtual space survey is constructed based on a dataset of a portion of the indoor space (401) captured at a second time point, and multiple shooting points (three shooting points in the illustrated example) may be displayed within the floor plan. By selecting a shooting point on the floor plan, the user can explore the indoor space at the time of shooting through a 360 VR viewer.

[0094] Figure (b) is a screen illustrating a first virtual space survey. The first virtual space survey is constructed based on a dataset of images taken of another part of the indoor space (402) at a first time point, and multiple shooting points (5 shooting points in the illustrated example) may be displayed within the floor plan. Likewise, the user can select a desired shooting point on the floor plan and experience the virtual space through a 360 VR viewer.

[0095] Figure (c) is a screen illustrating the result of aligning the second virtual space survey with the first virtual space survey. That is, the first virtual space survey constructed at the first time point and the second virtual space survey constructed at the second time point are aligned with each other based on a mapping relationship, so that Area 401 and Area 402 are displayed together in an integrated manner on a single virtual space survey.

[0096] Through this, users can continuously explore the two surveys as if they were a single virtual space survey, and achieve the effect of providing data captured at different points in time spatially within a single context.

[0097] In the example of FIG. 4, the first virtual space survey and the second virtual space survey cover different areas, but are not limited thereto and the two surveys may include a common area.

[0098] In such cases, results captured at different times for the same area are aligned and displayed, allowing users to compare and experience changes over time at the same location. This can be effectively utilized for management purposes such as remodeling, spatial changes, and facility replacement.

[0099] In addition, the two virtual space surveys are constructed based on datasets captured at different points in time rather than at the same point in time, and one embodiment of the present invention can provide spatiotemporally rich virtual space information by spatially aligning temporally different datasets.

[0100]

[0101] FIG. 5 is a flowchart illustrating a procedure for mapping a shooting point of a second virtual space survey to a first virtual space survey according to one embodiment disclosed in the present application.

[0102] An embodiment illustrated in FIG. 5 relates to a process of mapping the initial shooting point of a second data set onto a planar view to set a reference point for alignment, and automatically arranging all shooting points using SLAM-based relative position information.

[0103] In step S501, the computing device (300) may receive a second data set captured at a second point in time. The second data set may include a 360-degree panoramic image, 360-degree panoramic depth data, and indoor location information by shooting point.

[0104] At this time, the indoor location information for each shooting point may be relative coordinate information calculated as a result of SLAM-based sensor fusion. For example, the scan device (100) may record the relative movement trajectory between shooting points by performing an inertial sensor (IMU), vision SLAM, or LiDAR-based position estimation algorithm during the shooting process.

[0105] As such, the second data set has the same data structure as the first data set, but differs in that it is information acquired at a different point in time.

[0106] In step S502, the computing device (300) can map the first shooting point of the second data set onto the plan view of the first virtual space survey.

[0107] The photographer can select and input the actual location of the initial shooting point on the floor plan of the indoor space, and this input value can be used as a reference point to connect the relative coordinate system of the second data set to the absolute coordinate system of the first virtual space survey.

[0108] In step S503, the computing device (300) can automatically place the remaining shooting points in the coordinate system of the first virtual space survey using relative position information included in the second data set.

[0109] For example, SLAM-based relative position information may include relative distance traveled between shooting points, rotation angle, elevation change, etc., and these values ​​may be combined with the absolute coordinates of the first shooting point to calculate the coordinates of the entire second data set.

[0110] In one embodiment, the computing device (300) can extract feature points from a camera image and calculate a movement vector through feature point matching between consecutive frames. Subsequently, this movement vector is combined with the planar coordinates of the first shooting point so that all shooting points of the second data set can be placed on the planar plane.

[0111] In step S504, the computing device (300) can generate a mapping relationship between the shooting points of the second data set and the spatial information of the first virtual space survey.

[0112] This mapping relationship may be comprehensive mapping information that includes not only simple coordinate correspondence, but also orientation information of shooting points, distance information between shooting points, and feature similarity of image / depth data.

[0113] For example, if a specific shooting point in the second dataset is close to a specific point in the first virtual space survey, the two points can be mapped as the same node, and this can be used as a basis for aligning and integrating the two virtual space surveys in a subsequent step.

[0114] In one embodiment, the mapping relationship can be extended not only to a 1:1 correspondence method based on a single reference point, but also to a multi-reference matching method using multiple reference points. For example, not only the initial shooting point of the second data set but also a specific intermediate point can be additionally mapped on the plan view, in which case the matching accuracy can be improved.

[0115] In addition, during the mapping process, not only simple coordinate matching but also image-based similarity judgment and feature point-based ICP (Iterative Closest Point) matching algorithms can be applied, thereby correcting the accuracy of the coordinate arrangement of the second dataset.

[0116] Consequently, through the procedure illustrated in FIG. 5, the second virtual space survey is mapped to the spatial information of the first virtual space survey and reaches a state ready for alignment. This provides a basis for the two surveys to be combined into a single coordinate system in the subsequent alignment and integration stage.

[0117]

[0118] FIG. 6 is a flowchart illustrating a procedure for aligning a second virtual space survey with a first virtual space survey according to one embodiment.

[0119] An embodiment illustrated in FIG. 6 is for positionally aligning two virtual space surveys based on the mapping relationship described in FIG. 5, processing overlapping areas, and finally producing an integrated survey.

[0120] In step S601, the computing device (300) can verify the mapping relationship between the first virtual space survey and the second virtual space survey calculated as described above.

[0121] This mapping relationship may include corresponding information indicating where the shooting points of the second dataset are placed in the spatial coordinate system of the first virtual space survey. Since this mapping relationship is the result of transforming relative position information based on the absolute position of the initial shooting point, it can be expressed as a transformation matrix between the coordinate system of the second dataset and the coordinate system of the first virtual space survey.

[0122] In one embodiment, the mapping relationship may be defined as an affine transformation including a rotation matrix, a translation vector, and a scale factor.

[0123] In step S602, the computing device (300) can positionally align the second virtual space survey with the coordinate system of the first virtual space survey based on the mapping relationship.

[0124] This alignment process may include a procedure for projecting the shooting points of the second virtual space survey onto the planar coordinates or 3D coordinate system of the first virtual space survey and placing them at corresponding positions. For example, each shooting point of the second virtual space survey is moved to a corresponding node on the planar map, and at this time, through coordinate transformation, it can be represented in the same reference system as the first virtual space survey.

[0125] In step S603, if there is an overlapping area between the first virtual space survey and the second virtual space survey, the computing device (300) may perform a correction procedure to resolve the collision or overlap. Depending on the embodiment, this step S603 may be omitted.

[0126] In one embodiment, if data from two points in time for the same point exists, the computing device (300) may ensure that the second data acquired at the most recent point in time is reflected first.

[0127] In another embodiment, the computing device (300) may assign weights by considering image quality, the degree of alignment of depth data, or sensor reliability, and determine which data to retain based on the result. For example, it may select the data with the higher image feature matching result between two shooting points, or fuse the two data to generate averaged coordinates and textures.

[0128] In one embodiment, the computing device (300) can precisely correct the location and direction of duplicate points by applying an Iterative Closest Point (ICP) algorithm or a graph matching-based optimization technique.

[0129] In step S604, based on the matching and duplicate processing results, the computing device (300) can configure a single virtual space survey that integrates the first virtual space survey and the second virtual space survey.

[0130] This integrated survey combines and displays data from a first time point and a second time point, and can provide a spatially continuous and visually consistent exploration environment. For example, users can perceive it as a single virtual space survey on a floor plan, while being able to check the latest data from the second time point in specific areas.

[0131] In one embodiment, the integrated virtual space survey does not merely merge two sets of data but can be managed by considering the time axis. That is, for the same shooting point, both past (first point in time) and present (second point in time) data are preserved, and the user can selectively switch between them as needed. This can be very useful for recording and managing spatial changes over time, such as remodeling, interior modifications, or facility replacements.

[0132] In one embodiment illustrated in FIG. 6, the second virtual space survey is aligned with the coordinate system of the first virtual space survey, and overlapping areas are processed through correction, selection, and fusion, and can finally be completed as a single integrated virtual space survey.

[0133]

[0134] FIG. 7 is a flowchart illustrating a procedure for reflecting a matched virtual space survey in a user interface according to one embodiment.

[0135] An embodiment illustrated in FIG. 7 relates to updating a first virtual space survey based on matched results and providing it to a user terminal (500) so that data acquired at different points in time can be experienced in an integrated manner.

[0136] In step S701, the computing device (300) can replace or merge some virtual space information of the first virtual space survey with information of the second virtual space survey according to the matching result calculated as described above.

[0137] In one embodiment, mesh interpolation or texture blending can be performed on the updated region to ensure that the boundaries are smoothly connected by comparing it with data from a first time point. This allows the user to experience a virtual space with minimized visual dissonance.

[0138] In step S702, the computing device (300) may provide metadata for the replacement or merge processing to effectively display the replacement or merged result on a user terminal.

[0139] For example, an “update marker” may be added to a newly captured point at the second point in time, and on the plan view, it may be distinguished from the existing captured point by a different color or icon.

[0140] In one embodiment, the shooting point at the first time point may be displayed in gray and the shooting point at the second time point may be displayed in blue, and the user can intuitively identify which point is the latest data.

[0141] In step S703, the computing device (300) can provide a matched virtual space survey to the user terminal (500). The user terminal can explore the updated survey through a floor plan and a 360 VR viewer.

[0142] In one embodiment, when a user clicks a specific point on a plan view, the latest 360 VR view or 3D mesh view of that location may be displayed. Additionally, data from a previous point in time may be optionally switched to view as needed.

[0143] In step S704, the computing device (300) may provide the user terminal (500) with a multi-view-based interface function, such as comparing or switching data at different points in time at the same location. For example, the screen may be divided to simultaneously display a view of a first point in time on the left and a view of a second point in time on the right.

[0144] In one embodiment, the VR view of the same location can be moved to continuously explore how it changes over time by moving the timeline slider. This feature is useful for intuitively understanding temporal changes, such as before and after remodeling or before and after facility replacement.

[0145] In one embodiment, the interface of the user terminal may not be limited to simply switching viewpoints, but may provide a multi-layered comparison function. For example, the data difference between a first viewpoint and a second viewpoint may be visually highlighted (heatmap) to highlight the changed area, or the user may save a specific viewpoint as a “favorite” to utilize a history management function.

[0146] As described in FIG. 7, a virtual space survey aligned according to one embodiment is provided to a user terminal, thereby providing an environment in which the user can comprehensively experience and compare spatial data acquired at different points in time. This goes beyond simple virtual space exploration and possesses expanded utility value, such as managing temporal changes and long-term recording of spatial assets.

[0147]

[0148] FIG. 8 is an example diagram illustrating a temporal comparison function according to one embodiment.

[0149] Figure 8 <a1>is a representation of the first virtual space survey constructed at the first time point in a planar coordinate system, and of FIG. 8 <a2>This represents the second virtual space survey constructed at the second point in time on the same planar coordinate system.

[0150] Figure 8 <a1>In this, a plurality of shooting points V1 to V6 acquired at a first time point are arranged at coordinates on a planar view, and each shooting point corresponds to a 360-degree panoramic image and depth data. For example, point V3 located at coordinates (2,3) contains data captured at the first time point, and the view angle at that point may also be displayed.

[0151] Figure 8 <a2>In this case, multiple shooting points P1 to P6 acquired for the same space at the second time point are mapped to the same planar coordinate system. At this time, the first shooting point P1 at the second time point is set near coordinates (2,3), which may be the same or close to point V3 at the first time point. Subsequently, points P2 to P6 are arranged sequentially according to SLAM-based relative position information, and as a result, a virtual space survey at the second time point can be constructed on the coordinate system. In one embodiment, the virtual space survey at the first time point and the virtual space survey at the second time point can be compared with each other on the same planar coordinate system. The user can select the same location, for example, a point corresponding to coordinates (2,3), to compare the 360-degree VR views captured at the first time point and the second time point, respectively. Through this, the user can intuitively verify how the same space has changed over time.

[0152] The computing device (300) can receive input from the user for a first point of the first virtual space survey. The computing device (300) can receive input for a first point of the first virtual space survey, for example, <a1>A point on the second virtual space survey located closest to the V3 point can be selected, and in the illustrated example <a2>It is the same as P1. That is, the computing device (300) can select a point on the second virtual space survey closest to the first point of the first virtual space survey as the second point, and then simultaneously display an image of the first point and an image of the second point on the user interface. For example, the computing device (300) can display a first image of the first point and a second image of the second point on each part of the divided screen of the user interface.

[0153] According to an embodiment, the computing device (300) can set the viewpoints of the first image and the second image such that the viewpoints of the first image and the second image are the same. For example, the first image and the second image may be 360-degree panoramic images for 360-degree VR, and the viewpoint direction of the second image may be set to be the same as the viewpoint direction of the first image set by the user. In the example of FIG. 8 as well, the first image <a1>Viewpoint direction in V3 and the second image <a2>It can be seen that the viewpoint direction at P1 matches.

[0154] Furthermore, the first and second surveys may share some identical areas or include different areas. For instance, the entire space may have been captured at the first time point, while only a portion of the area may have been re-captured at the second time point. In this case, common points serve as reference points for comparison between the two time points, while points newly acquired only at the second time point can be displayed as updated data. Conversely, points included at the first time point but not re-captured at the second time point are preserved as history information, which users can selectively view when necessary.

[0155] In one embodiment, point-in-time comparison can be performed not only by simply switching between two points in time at the same coordinates, but also by using a split screen or a timeline UI for simultaneous comparison. For example, a user terminal can split the screen left and right to display the view of the first point in time on the left and the view of the second point in time on the right, or by moving a slider, the user can check how the VR view captured at the same point changes over time.

[0156] Accordingly, FIG. 8 illustrates an exemplary process of mapping the first virtual space survey and the second virtual space survey in the same planar coordinate system, enabling the comparison and experience of data from two points in time regarding the same point. Through this, users can efficiently explore spatial changes over time, remodeling results, and additions or changes to facilities, and the virtual space survey can be expanded beyond simple static representation to become a means of spatiotemporal recording and management.

[0157]

[0158] FIG. 9 is a drawing illustrating an exemplary environment of a computing device according to one embodiment of the present invention.

[0159] FIG. 9 is intended to provide a general and simplified description of a suitable computing environment in which various embodiments of a computing device may be implemented. Referring to FIG. 9, a computing device (100) is illustrated.

[0160] The computing device (100) may include at least a processing unit (310) and a system memory (320).

[0161] The computing device may include multiple processing units that cooperate when executing a program. Depending on the exact configuration and type of the computing device, the system memory (301) may be volatile (e.g., RAM), non-volatile (e.g., ROM, flash memory, etc.), or a combination thereof. The system memory (320) includes a suitable operating system (330) for controlling the operation of the platform, which may be, for example, the WINDOWS operating system from Microsoft. The system memory (320) may include one or more software applications, such as program modules, applications, etc.

[0162] The computing device may include additional data storage devices (340), such as magnetic disks, optical disks, or tapes. These additional storage devices may be removable storage and / or fixed storage. Computer-readable storage media may include volatile and non-volatile, removable and fixed media implemented by any method or technique for storing information such as computer-readable instructions, data structures, program modules, or other data. System memory (320) and data storage devices (340) are merely examples of computer-readable storage media. Computer-readable storage media may include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory techniques, CD-ROM, DVD or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that stores desired information and can be accessed by the computing device (100).

[0163] Input devices (350) of a computing device may include, for example, a keyboard, a mouse, a pen, a voice input device, a touch input device, and comparable input devices. Output devices (360) may include, for example, a display, a speaker, a printer, and other types of output devices. Since these devices are widely known in the art, a detailed description is omitted.

[0164] The computing device may include a communication device (370) that allows the device to communicate with other devices through a network in a distributed computing environment, such as a wired / wireless network, a satellite link, a cellular link, a local area network, and a comparable mechanism. The communication device (370) is one example of a communication medium, and the communication medium may contain computer-readable instructions, data structures, program modules, or other data. For example, the communication medium includes, but is not limited to, wired media such as a wired network or direct wired connection, and wireless media such as acoustic, RF, infrared, and other wireless media.

[0165]

[0166] Although the embodiments have been described above with reference to limited examples and drawings, those skilled in the art can make various modifications and variations from the description above. For example, suitable results can be achieved even if the described techniques are performed in a different order than described, and / or the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0167] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below.

[0168] Although specific embodiments have been described in the detailed description of this document, it will be obvious to those skilled in the art that various modifications are possible within the scope of this document.

[0169] According to one embodiment of the present invention, by supporting re-photography and partial virtualization reconstruction of partial areas in a previously captured and constructed virtual space survey, the entire survey can be automatically constructed even if only a partial area of ​​an indoor space is re-photographed, thereby offering high potential for industrial application.

[0170] Furthermore, according to one embodiment of the present invention, when performing reconstruction of a partially re-captured virtual space, the similarity to the shooting location in the existing survey is determined, and based on this similarity, the location within the survey of the partially re-captured virtual space can be automatically determined, which has the effect of having high potential for industrial application.

Claims

1. A method for providing a virtual space for an indoor space performed on a computing device, A step of receiving a first data set acquired from multiple shooting points in an indoor space at a first point in time; A step of constructing a first virtual space survey of the indoor space at a first time point using the first data set; A step of receiving a second data set acquired in at least a portion of the indoor space at a second time point; A step of constructing a second virtual space survey at a second time point for at least a portion of the indoor space using the second data set; and A step comprising: mapping the shooting point information of the second data set to the spatial information of the first virtual space survey, and aligning the second virtual space survey with the first virtual space survey based on the mapping relationship; Method for providing a virtual space that supports partial reshooting.

2. In paragraph 1, the first data set is, A 360-degree panoramic image, 360-degree panoramic depth data, and indoor location information for each of the plurality of shooting points individually acquired at each of the plurality of shooting points within the indoor space, Method for providing a virtual space that supports partial reshooting.

3. In paragraph 2, the indoor location information for each of the plurality of shooting points is, including relative position information automatically generated based on movement paths between shooting points in the above indoor space, Method for providing a virtual space that supports partial reshooting.

4. In paragraph 3, the above relative position information is, The initial shooting point in the above indoor space is set by mapping it onto a floor plan of the above indoor space, Method for providing a virtual space that supports partial reshooting.

5. In paragraph 4, the step of associating the second virtual space survey with the first virtual space survey based on the mapping relationship is, A step comprising: mapping the initial shooting point of the second data set onto a plan view of the first virtual space survey to perform positional alignment between the second virtual space survey and the first virtual space survey; Method for providing a virtual space that supports partial reshooting.

6. In Paragraph 1, The method further comprises the step of replacing and displaying at least some virtual space information of the first virtual space survey with virtual space information of the second virtual space survey according to the matched result. Method for providing a virtual space that supports partial reshooting.

7. At least one processor; and It includes memory for storing instructions, When the above instructions are executed individually or collectively by the at least one processor, the processor, Receive a first data set acquired from multiple shooting points in an indoor space at a first point in time, and Using the above first data set, a first virtual space survey for the indoor space at a first time point is constructed, and Receiving a second data set acquired from at least a portion of the indoor space at a second time point, and Using the second data set above, a second virtual space survey at a second time point for at least a portion of the indoor space is constructed, and Mapping the shooting point information of the second data set to the spatial information of the first virtual space survey, and aligning the second virtual space survey with the first virtual space survey based on the mapping relationship. A computing device that provides a virtual space supporting partial re-shooting.

8. In paragraph 7, the first data set is, A 360-degree panoramic image, 360-degree panoramic depth data, and indoor location information for each of the plurality of shooting points individually acquired at each of the plurality of shooting points within the indoor space, A computing device that provides a virtual space supporting partial re-shooting.

9. In paragraph 8, the indoor location information for each of the plurality of shooting points is, including relative position information automatically generated based on movement paths between shooting points in the above indoor space, A computing device that provides a virtual space supporting partial re-shooting.

10. In paragraph 9, the above relative position information is, The initial shooting point in the above indoor space is set by mapping it onto a floor plan of the above indoor space, A computing device that provides a virtual space supporting partial re-shooting.

11. In Paragraph 10, The above instructions cause the processor, In order to associate the second virtual space survey with the first virtual space survey based on the above mapping relationship, Mapping the initial shooting point of the second data set onto a plan view of the first virtual space survey to perform positional alignment between the second virtual space survey and the first virtual space survey. A computing device that provides a virtual space supporting partial re-shooting.

12. In Paragraph 7, The above instructions cause the processor, According to the matched result, at least some virtual space information of the first virtual space survey is replaced with virtual space information of the second virtual space survey and displayed. A computing device that provides a virtual space supporting partial re-shooting.

13. In Paragraph 7, The above instructions cause the processor, A method for receiving a first point of the first virtual space survey from a user, selecting a point on the second virtual space survey closest to the first point as the second point, and then displaying a first image of the first point and a second image of the second point on respective parts of a divided screen of a user interface. A computing device that provides a virtual space supporting partial re-shooting.

14. In a storage medium storing computer-readable instructions, When the above instructions are executed by a computing device, the computing device, The operation of receiving a first data set acquired from multiple shooting points in an indoor space at a first point in time; The operation of constructing a first virtual space survey of the indoor space at a first time point using the first data set; The operation of receiving a second data set acquired in at least a portion of the indoor space at a second point in time; The operation of constructing a second virtual space survey at a second time point for at least a portion of the indoor space using the second data set; and The operation of mapping the shooting point information of the second data set to the spatial information of the first virtual space survey, and associating the second virtual space survey with the first virtual space survey based on the mapping relationship; is performed. Computer-readable storage media.

Images