Device, method, and system for providing metaverse-based virtual experience service linked to real-world space

Abstract

Disclosed are a device, a method, and a system for providing a metaverse-based virtual experience service linked to a real-world space. According to an embodiment of the present application, the method for providing a metaverse-based virtual experience service linked to a real-world space may comprise the steps of: (a) collecting point cloud data in a target space by using a depth sensor; (b) constructing a three-dimensional model corresponding to the target space by using the point cloud data; (c) identifying a semantic element of a real object provided in the target space, and generating a virtual object corresponding to the real object by using the semantic element; and (d) constructing a virtual environment matching a predetermined theme from the three-dimensional model by changing a concept element of the virtual object.

Description

Device, method, and system for providing metaverse-based virtual experience services linked to real space

[0001] The present invention relates to an apparatus, method, and system for providing virtual experience services based on a metaverse linked to real space.

[0002] With the advancement of virtual reality technology, research on 3D spatial modeling is actively underway. In particular, attempts to improve digital reproducibility are being made for indoor environments through various techniques such as 3D scanning, image-based modeling, and AI-based modeling.

[0003] In particular, these indoor environment-based models are being utilized in various VR-based applications such as architecture, interior design, and indoor navigation, and with the rise of the metaverse, the scope of VR applications is further expanding to include social interaction, cultural experiences, and remote collaboration.

[0004] Recently, advancements in AI-based content generation technology have made it possible to generate 3D models using only text input; however, this requires significant computing resources and vast amounts of training data, and very detailed descriptions are required to obtain accurate modeling results. Therefore, traditional point cloud-based 3D modeling techniques are still being utilized as an effective method for obtaining accurate modeling results.

[0005] However, although current 3D modeling technology can accurately replicate real space into a digital twin, there are still significant challenges in semantic modeling that is perfectly aligned with the virtual scene. Semantic modeling is a crucial element for giving meaning and functional interactivity to virtual objects and environments, and for enabling them to behave like the real world.

[0006] Furthermore, in the case of virtual reality content generated based on real-world spaces, diversity is currently lacking because most of it has been activated and developed primarily around games. In the case of indoor spaces, where users spend a significant amount of time, virtual reality experiences based on them can be considered limited, even though various spatial elements and environments have become an important part of life.

[0007] Accordingly, as interest in the metaverse—which allows for virtual reality experiences based on indoor spaces—grows, advancements are being made in various fields such as 3D scanning, 3D modeling, and 3D reconstruction. However, achieving a balance between the familiarity of the real environment and the flexibility of the virtual environment within a limited indoor space remains an important challenge, and there is a demand for the development of technology that enables diverse virtual experiences while providing users with a high sense of presence.

[0008] The technology forming the background of this invention is disclosed in Korean Registered Patent Publication No. 10-2402580.

[0009] The present invention aims to solve the problems of the aforementioned prior art by providing a device, method, and system for providing a metaverse-based virtual experience service linked to real space, which can provide a unique virtual experience to the user by transforming a limited indoor space into a virtual environment capable of various metaverse experiences.

[0010] The present invention aims to solve the problems of the aforementioned conventional technology by constructing a virtual environment based on real space, providing virtual experiences of various concepts within a single space by modifying the semantic elements of said space, and enhancing user immersion through the alignment of real objects and virtual objects.

[0011] However, the technical problems that the embodiments of the present invention aim to solve are not limited to the technical problems described above, and other technical problems may exist.

[0012] As a technical means for achieving the above-mentioned technical task, a method for providing a metaverse-based virtual experience service linked to a real space according to one embodiment of the present invention may include: (a) collecting point cloud data in a target space using a depth sensor; (b) constructing a three-dimensional model corresponding to the target space using the point cloud data; (c) identifying semantic elements of a real object provided in the target space and creating a virtual object corresponding to the real object using the semantic elements; and (d) constructing a virtual environment corresponding to a predetermined theme from the three-dimensional model by changing the concept elements of the virtual object.

[0013] In addition, a method for providing a metaverse-based virtual experience service linked to a real space according to one embodiment of the present invention may include the step of (e) providing metaverse-based content using the virtual environment to a user terminal.

[0014] Additionally, the above step (e) may include a step of replacing the user's actual interaction data performed on the actual object with virtual interaction data performed on the virtual object through the virtual environment.

[0015] In addition, a method for providing a virtual experience service based on a metaverse linked to a real space according to one embodiment of the present invention may include, after step (a), a step of performing preprocessing that removes outlier data included in the point cloud data and applies segmentation that divides a plurality of points included in the point cloud data into point groups corresponding to each of the actual objects.

[0016] In addition, the above-mentioned target space may include an indoor space.

[0017] Additionally, the above step (b) may include the step of modeling the basic structure of the target space by identifying the floor surface, wall surface, and ceiling surface forming the indoor space from the point cloud data, the step of identifying the geometric characteristics of the actual object from the point cloud data and generating a bounding box for each of the actual objects, and the step of generating the three-dimensional model by combining the basic structure and the bounding box.

[0018] Additionally, the above step (c) may include the step of identifying the semantic elements, including the type, size, and shape of the actual object, using a pre-trained artificial intelligence-based detection model; the step of selecting the virtual object that matches the actual object from among a plurality of candidate models based on the identified semantic elements; and the step of aligning the position and orientation of the selected virtual object with the actual object.

[0019] Additionally, the above step (d) may include changing at least one of the size, material, and shape of the virtual object according to the theme, adding a background effect associated with the theme to the virtual environment, and adjusting environment settings for harmony between the virtual object and the background effect.

[0020] In addition, the semantic element may include at least one of the type, physical size, shape, functional attributes, and spatial relationship with other actual objects of the actual object.

[0021] Meanwhile, a device for providing a virtual experience service based on a metaverse linked to a real space according to one embodiment of the present invention may include a data collection unit that collects point cloud data in a target space using a depth sensor, a 3D model generation unit that constructs a 3D model corresponding to the target space using the point cloud data, an object generation unit that identifies semantic elements of a real object provided in the target space and creates a virtual object corresponding to the real object using the semantic elements, and a virtual environment implementation unit that constructs a virtual environment corresponding to a predetermined theme from the 3D model by changing concept elements of the virtual object.

[0022] In addition, a device for providing a virtual experience service based on a metaverse linked to a real space according to one embodiment of the present invention may include a content providing unit that provides metaverse-based content using the virtual environment to a user terminal.

[0023] In addition, the content providing unit can replace the user's actual interaction data performed on the actual object with virtual interaction data performed on the virtual object through the virtual environment.

[0024] In addition, a device for providing a virtual experience service based on a metaverse linked to a real space according to one embodiment of the present invention may include a data preprocessing unit that performs preprocessing by removing outlier data included in the point cloud data and applying segmentation to divide a plurality of points included in the point cloud data into point groups corresponding to each of the actual objects.

[0025] In addition, the object generation unit can identify the semantic elements, including the type, size, and shape of the actual object, using a pre-trained artificial intelligence-based detection model, select the virtual object that matches the actual object from among a plurality of candidate models based on the identified semantic elements, and align the position and orientation of the selected virtual object with the actual object.

[0026] In addition, the virtual environment implementation unit may change at least one of the size, material, and shape of the virtual object according to the theme, add a background effect associated with the theme to the virtual environment, and adjust environment settings for harmony between the virtual object and the background effect.

[0027] The means for solving the problem described above are merely exemplary and should not be interpreted as intended to limit the present invention. In addition to the exemplary embodiments described above, additional embodiments may exist in the drawings and the detailed description of the invention.

[0028]

[0029] According to the means for solving the problem of the present invention described above, it is possible to provide a device, method, and system for providing a metaverse-based virtual experience service linked to a real space, which can provide a unique virtual experience to a user by converting a limited indoor space into a virtual environment capable of various metaverse experiences.

[0030] According to the solution to the problem of the present invention described above, a virtual environment is constructed based on real space, and by changing the semantic elements of said space, various conceptual virtual experiences can be provided within a single space, and the user's sense of immersion can be enhanced through the alignment between real objects and virtual objects.

[0031] According to the solution to the problem of the present invention described above, by transforming an indoor space that provides only limited experiences into a virtual environment of various concepts, not only is it possible to have various metaverse experiences in a single space, but a high sense of presence and immersion can also be provided through the interaction between physical objects in the actual space and the virtual environment.

[0032] However, the effects obtainable from this invention are not limited to those described above, and other effects may exist.

[0033] FIG. 1 is a schematic diagram of a metaverse-based virtual experience service provision system according to one embodiment of the present invention.

[0034] FIG. 2 is a conceptual diagram illustrating the process of merging real space and virtual space through a metaverse-based virtual experience service provision system according to one embodiment of the present invention.

[0035] FIG. 3 is a conceptual diagram illustrating a semantic modeling process through a metaverse-based virtual experience service provision system according to one embodiment of the present invention.

[0036] Figure 4 is a diagram illustrating an exemplary result of applying completion processing to a point cloud.

[0037] FIG. 5 is a diagram illustrating an exemplary virtual reality environment provided through a metaverse-based virtual experience service provision system according to one embodiment of the present invention.

[0038] Figure 6 is a diagram illustrating an exemplary user interaction area for switching between virtual environments.

[0039] Figure 7 is a conceptual diagram illustrating a generalization process that enhances immersion through real-space-based modeling.

[0040] FIG. 8 is a schematic diagram of a device for providing virtual experience services based on a metaverse linked to real space according to one embodiment of the present invention.

[0041] FIG. 9 is a flowchart of the operation of a method for providing a metaverse-based virtual experience service linked to a real space according to one embodiment of the present invention.

[0042] Figure 10 is a detailed flowchart of the process for building a 3D model of a target space.

[0043] Figure 11 is a detailed flowchart of the process for creating a virtual object corresponding to a real object.

[0044] Figure 12 is a detailed flowchart of the process for building a virtual environment of a specific theme through the change of concept elements.

[0045] Embodiments of the present invention are described below with reference to the attached drawings to enable those skilled in the art to easily implement the invention. However, the present invention may be embodied in various different forms and is not limited to the embodiments described herein. Furthermore, in order to clearly explain the present invention in the drawings, parts unrelated to the explanation have been omitted, and similar parts throughout the specification are denoted by similar reference numerals.

[0046] Throughout this specification, when a part is described as being "connected" to another part, this includes not only cases where they are "directly connected," but also cases where they are "electrically connected" or "indirectly connected" with other elements interposed between them.

[0047] Throughout the entire specification, when a component is described as being located "on," "on top," "on top," "under," "on bottom," or "on bottom" of another component, this includes not only cases where the component is in contact with the other component but also cases where another component exists between the two components.

[0048] Throughout this specification, when a part is described as "comprising" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components.

[0049] The present invention relates to an apparatus, method, and system for providing virtual experience services based on a metaverse linked to real space.

[0050] FIG. 1 is a schematic diagram of a metaverse-based virtual experience service provision system according to one embodiment of the present invention.

[0051] Referring to FIG. 1, a metaverse-based virtual experience service providing system (10) (hereinafter referred to as the 'virtual experience service providing system (10)') according to one embodiment of the present invention may include a metaverse-based virtual experience service providing device (100) (hereinafter referred to as the 'service providing device (100)') connected to a real space according to one embodiment of the present invention, a measuring device (200), a database (300), and a user terminal (400).

[0052] A service provider (100), a measuring device (200), a database (300), and a user terminal (400) can communicate with each other through a network (20). The network (20) refers to a connection structure that enables information exchange between each node, such as terminals and servers. Examples of such a network (20) include, but are not limited to, a 3GPP (3rd Generation Partnership Project) network, an LTE (Long Term Evolution) network, a 5G network, a WIMAX (World Interoperability for Microwave Access) network, the Internet, a LAN (Local Area Network), a Wireless LAN (Wireless Local Area Network), a WAN (Wide Area Network), a PAN (Personal Area Network), a Wi-Fi network, a Bluetooth network, a satellite broadcasting network, an analog broadcasting network, and a DMB (Digital Multimedia Broadcasting) network.

[0053] The user terminal (400) can be any type of wireless communication device, such as a smartphone, smartpad, tablet PC, PCS (Personal Communication System), GSM (Global System for Mobile communication), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), Wibro (Wireless Broadband Internet) terminal.

[0054] In particular, according to one embodiment of the present invention, a user terminal (400) may include a VR device capable of playing virtual environment-based metaverse content or virtual reality content of the present invention. Specifically, the VR device may refer to a head-mounted display (410) terminal. In this regard, the user terminal (400) may output content that reflects information about the user's movements, interactions, etc., in a virtual environment corresponding to a predetermined theme provided by a service provider (100).

[0055] Additionally, in the description of the embodiments of the present invention, the measuring equipment (200) may be a measuring device equipped with a depth sensor, such as a ToF (Time of Flight) Camera, a LiDAR sensor, etc., for acquiring point cloud data including a plurality of three-dimensional points measured to correspond to actual objects existing in the target space. For example, the measuring equipment (200) may include a mobile device including a depth sensor such as a ToF camera or a user terminal (400) that collects point clouds using a LiDAR sensor mounted in the target space, but is not limited thereto.

[0056] In addition, in the description of the embodiments of the present invention, the database (300) may be a server or device for storing semantic modeling data using a point cloud constructed by a service providing device (100) as described in detail below, virtual objects by object type used when modeling a target space, etc.

[0057] FIG. 2 is a conceptual diagram illustrating the process of merging real space and virtual space through a metaverse-based virtual experience service provision system according to one embodiment of the present invention.

[0058] Referring to FIG. 2, the service providing device (100) disclosed herein can convert a target space into a virtual environment through the following process. First, the service providing device (100) can collect point cloud data using a depth sensor in the target space and model the target space using the data (Fig. 2, 'a'), and can construct a virtual reality environment representing the target space using the point cloud data (Fig. 2, 'b').

[0059] Additionally, the service providing device (100) can implement a virtual environment in different ways depending on whether the actual target space is an empty space ('c' in FIG. 2) or a space where actual objects exist ('d' in FIG. 2), based on the constructed virtual reality environment.

[0060] Specifically, the service providing device (100) can operate to build a virtual environment that matches a predetermined theme by identifying semantic elements of a real object, using them to create a virtual object corresponding to the real object, and then changing the concept elements of the virtual object.

[0061] Hereinafter, the specific functions and operations of the service providing device (100) will be described in detail with reference to FIGS. 3 to 7.

[0062] FIG. 3 is a conceptual diagram illustrating a semantic modeling process through a metaverse-based virtual experience service provision system according to one embodiment of the present invention.

[0063] Referring to FIG. 3, a service providing device (100) can collect point cloud data for a target space using a depth sensor (not shown) ('Point Cloud' of FIG. 3).

[0064] Specifically, the service providing device (100) can acquire point cloud data including a plurality of three-dimensional points measured to correspond to actual objects existing in the target space. For example, the service providing device (100) can acquire point cloud data that captures the target space corresponding to the indoor space using a LiDAR sensor (not shown) mounted on a user terminal (400) and a predetermined data collection application (e.g., an application such as SiteScape) as input data. For example, the coordinate information and color information of each captured point included in the point cloud data may be reflected in the form of a ply file in the form of x, y, z, r, g, b.

[0065] Additionally, the service providing device (100) can perform preprocessing to remove outlier data included in the collected point cloud data.

[0066] For example, if the coordinate system of the data set used in the object detection algorithm described below and the coordinate system of the measured point cloud data do not match, the service providing device (100) can perform preprocessing to match the coordinate system by rotating the data included in the collected point cloud data (e.g., rotating 90 degrees relative to the X-axis, etc.) and removing noise through filtering.

[0067] Additionally, the service providing device (100) can apply filtering based on a Statistical Outlier Removal algorithm, which is a technique for detecting noise using statistical information, and calculates average distance information with neighboring points and considers points above a certain distribution as noise. Specifically, the service providing device (100) can set the number of neighboring points to be considered when analyzing for filtering point cloud data to 50, and the distance information to be treated as outliers to 3, so that points located at a distance where the standard deviation of the average distance of the 50 points located near the reference point is 3 or greater can be considered as noise and removed, but is not limited to this.

[0068] That is, according to one embodiment of the present invention, the service providing device (100) can remove noise points unrelated to actual objects existing in the target space among a plurality of points included in the point cloud data.

[0069] Additionally, referring to FIG. 3, the service providing device (100) may apply segmentation to divide a plurality of points included in the point cloud data into point groups corresponding to each actual object provided in the target space (‘Object Detection’ of FIG. 3).

[0070] Specifically, the service providing device (100) can detect object extraction information including type information, location information, and size information corresponding to an actual object from the acquired point cloud data.

[0071] Specifically, the service providing device (100) can derive object extraction information by applying a pre-built artificial intelligence-based 3D object detection algorithm to identify bounding boxes for each object existing within the point cloud data using point cloud data as input and to estimate the type information for each bounding box.

[0072] In this regard, according to one embodiment of the present invention, in order to create a virtual space by replacing objects in a scene reflected in point cloud data with a 3D model (detection object), the accurate 3D location and size of the objects in the scene reflected in point cloud data must be determined, and in order to determine the location and size of the objects in the point cloud scene, the service providing device (100) can estimate the location and size of the objects by applying a 3D object detection algorithm that performs 3D object detection on the point cloud data.

[0073] For example, the location and size of each actual object identified from point cloud data can be represented as a 3D bounding box.

[0074] In other words, the service providing device (100) can use object extraction information to separate and extract point clouds corresponding to each object among the point cloud data ('Object Identification' in FIG. 3). For example, the service providing device (100) can apply a knowledge-driven method that extracts objects by utilizing information and geometric characteristics of objects or by setting constraints, or a data-driven method that performs learning based on vast amounts of labeled learning data by designing and applying a series of functions.

[0075] More specifically, among knowledge-based methods, the matching-based approach is a technique that extracts regions exhibiting similar characteristics by comparing objects with pre-selected models, while the rule-based approach uses geometric features (location, size, orientation, shape) of each object to remove parts that are not the object to be extracted, leaving only the object to be extracted. Additionally, methods such as filtering non-extraction parts based on various parameters (location, size) or using a slicing method—which divides the scene to extract objects and merges parts containing rules—can be employed.

[0076] In addition, among data-driven methods, localization techniques based on the integration of 2D images and point clouds are methods that learn local and global features related to location and size, and pixel intensity related to location maps, through SVM and Random Forest classifiers. When learning object features using ESF (Ensemble of Shape Function) and geometric features, ESF defines three shape functions (distance between points, area, and angle) and a ratio function, and ESF includes geometric features such as linearity, planarity, scattering of the covariance matrix, and the ratio of the bounding box, and learning can proceed through Random Forest classifiers.

[0077] In summary, the service providing device (100) can extract objects detected through 3D object detection in a point cloud scene in order to compare the 3D model with the objects in the point cloud during the object extraction step. Specifically, the objects in the point cloud can be extracted by obtaining the vertices and center points of the bounding box of each detected object from the object detection information derived from the 3D object detection result, and then extracting points inside the object using the equation of the plane of each face of the bounding box.

[0078] Additionally, the service providing device (100) can perform a completion process by adding missing loss points to the point cloud data in relation to the actual object, taking into account the object reference shape identified in advance according to object-specific type information ('Completion' in FIG. 3). Additionally, in the completion process step, the service providing device (100) can remove noise points unrelated to the actual object existing in the target space among the multiple points included in the point cloud data.

[0079] In this regard, if alignment is applied directly using only the 3D object detection results without applying completion, the detected objects may not correspond to the state in which they are placed in actual space and may not be properly aligned. This may be caused by point cloud loss due to occlusion that occurs during point cloud data measurement, measurement noise, etc.

[0080] Accordingly, the service providing device (100) can solve the noise problem by sampling the extracted object and compensate for the point cloud loss caused by the occlusion phenomenon through a completion operation.

[0081] More specifically, the service providing device (100) may apply a pre-built artificial intelligence-based completion network, and such completion network may include, for example, a PoinTr network, but is not limited thereto. For example, in the case of a PoinTr network, ShapeNetCore may be utilized as training data by considering similarity with input data.

[0082] In this regard, according to one embodiment of the present invention, the PoinTr network can operate to downsample an input point cloud to obtain center points of an object and extract local features around the center points, add location information to local features through positional embedding, which is a method for capturing unique relationships between locations, predict points for missing parts using an encoder-decoder structure, and restore the point cloud based on the predicted points by utilizing FoldingNet, a technique for reconstructing a 2D grid into a point cloud.

[0083] In this regard, Figure 4 is a diagram illustrating an exemplary result of applying completion processing to a point cloud.

[0084] Referring to FIG. 4, it can be seen that the completion processing performed by the service providing device (100) can obtain relatively accurate results as the object has a structured and symmetrical shape.

[0085] Meanwhile, according to one embodiment of the present invention, the service providing device (100) can perform completion processing in various ways, including geometry-based methods such as Surface Reconstruction Methods that create a complete shape through geometric signals of input partial data without external data, alignment-based methods that restore a shape through matching input data with template data in a database, and learning-based methods that voxelize a point cloud and perform 3D convolution operations.

[0086] Additionally, the service providing device (100) can perform alignment processing to determine the placement direction of the detection object by comparing the bounding box containing the target point cloud to which completion processing has been applied with the detection object generated in correspondence with the actual object ('Alignment' in FIG. 3).

[0087] In this regard, when modeling is performed based on 3D object detection results without an alignment process, the position and size of the modeled objects are positioned to correspond to the actual objects to some extent, but the orientation of each modeled object in the 3D models is all facing the same direction, so it may look different from the actual scene. This can be interpreted as being because the orientation value derived from the 3D object detection results only refers to how much the detected bounding box is rotated with respect to the Y-axis, and does not include information about the orientation of the model inside the bounding box.

[0088] That is, since the three-dimensional models are positioned in the same direction within their respective model coordinate systems, when modeling using the same model, they are all positioned in the same direction. To solve this problem, the service providing device (100) disclosed herein can perform alignment processing to position the model (detection object) corresponding to each object in the same direction as the object in the actual scene.

[0089] Additionally, in the alignment task (processing) stage, the service providing device (100) can determine the orientation of the extracted object within the target point cloud (point cloud data on which completion processing has been performed) corresponding to the extracted object. Specifically, the service providing device (100) can voxelize both the actual object and the model, rotate the voxelized model in various directions, and measure similarity through comparison with the voxelized object at the voxel level.

[0090] Additionally, the service providing device (100) can normalize the size of the detected object based on the size information derived as object detection information.

[0091] Additionally, the service providing device (100) can generate an undetected object corresponding to each cluster of remaining points from which object extraction information was not detected in the point cloud input ('Postprocessing' of FIG. 3).

[0092] In this regard, in the alignment stage, a semantic virtual space can be created by generating a model (detected object) that corresponds to the size of the detected object at the location of the object detected through 3D object detection in point cloud data. However, objects that were not detected due to the accuracy of the 3D object detection network or were not detected because the corresponding object type was not included as a label in the training dataset must also be modeled in the virtual space. To this end, in the postprocessing stage, post-processing can be applied to model objects that were not detected in the 3D object detection stage (in other words, to create undetected objects).

[0093] In addition, the service providing device (100) can construct a three-dimensional model corresponding to the target space using preprocessed point cloud data ('Semantic Model' of FIG. 3).

[0094] In summary, the service providing device (100) can model the basic structure of the target space by identifying the floor surface, wall surface, and ceiling surface forming the indoor space from the point cloud data measured in the target space corresponding to the indoor space.

[0095] In this regard, according to one embodiment of the present invention, a service providing device (100) can remove noise within a scene by first performing filtering and sampling steps in a scene where objects detected in a point cloud input are extracted and the remaining, in other words, only objects that were not detected remain.

[0096] Additionally, the service providing device (100) can create a scene in which only the undetected objects (in other words, points to which object detection information has not been assigned) remain by applying RANSAC (RANdom SAmple consensus)-based planar extraction, etc., thereby removing points corresponding to the floor or wall among the remaining points.

[0097] Additionally, the service providing device (100) can cluster points that can be treated as the same object by performing clustering using Euclidean distance in a scene consisting of remaining points from which points corresponding to the floor or wall surface have been removed. That is, the service providing device (100) can classify remaining points in which object extraction information was not detected in the point cloud input into cluster units.

[0098] Additionally, the service providing device (100) can create a cube of the size of the corresponding cluster at the cluster location of each classified cluster. That is, the service providing device (100) can create an undetected object corresponding to each classified cluster, and specifically, according to one embodiment of the present invention, the service providing device (100) can create an undetected object that covers a space in which a plurality of points included in each cluster are distributed and has a preset shape.

[0099] Additionally, the service providing device (100) can model the undetected object based on the clustering results, and then model the wall and floor by distinguishing them through a rule-based method in the extracted plane.

[0100] Additionally, the service providing device (100) can identify the geometric characteristics of the actual object from the collected point cloud data and use them to generate a bounding box for each of the actual objects.

[0101] Additionally, the service providing device (100) can create a three-dimensional model by combining a modeled basic structure and a bounding box created for an actual object. Specifically, the service providing device (100) can place detected objects and undetected objects created in a virtual space that is virtually simulated to correspond to a target space.

[0102] At this time, the service providing device (100) can determine a texture corresponding to each of the detected object and the undetected object by projecting a point cloud input as an image onto the generated detected object and the undetected object.

[0103] Meanwhile, if there are empty pixels in the projected image, the service providing device (100) can perform interpolation for the empty pixels.

[0104] Additionally, the service providing device (100) can create a realistic model closer to the shape of the actual target space by mapping textures to the object created in the shape of a cube and to the floor and walls through texturing, and to this end, the service providing device (100) can create textures by projecting the point cloud that makes up the floor surface, the wall surface and the object.

[0105] At this time, the wider and more evenly the point cloud is distributed relative to the projection surface, the more accurate the texture can be obtained, and the generated texture can be normalized to the size of each model. Since the point cloud is not uniform like the pixels of an image, when the point cloud is projected onto an image, empty pixels may occur within the image. To resolve this, the service providing device (100) can perform interpolation on the empty pixels, and such interpolation may apply a method such as Nearest-neighbor Interpolation, which uses the average of the colors of surrounding pixels, but is not limited to this.

[0106] Additionally, the service providing device (100) can identify semantic elements of a real object provided in a target space and create a virtual object corresponding to the real object using the identified semantic elements.

[0107] Specifically, the service providing device (100) can identify semantic elements including the type, size, and shape of an actual object provided in a target space using a pre-trained artificial intelligence-based detection model.

[0108] Additionally, the service providing device (100) can identify semantic elements including at least one of the type, physical size, shape, functional attributes, and spatial relationship with other actual objects of the actual object through analysis of the collected point cloud data.

[0109] Additionally, the service providing device (100) can select a virtual object that matches a real object from among a plurality of candidate models based on identified semantic elements.

[0110] Additionally, the service providing device (100) can align the position and orientation of the selected virtual object with the actual object.

[0111] FIG. 5 is a diagram illustrating an exemplary virtual reality environment provided through a metaverse-based virtual experience service provision system according to one embodiment of the present invention. Specifically, FIG. 5(a) shows a base model in which basic modeling is applied to a target space, FIG. 5(b) shows a virtual environment with a table tennis court theme in which an office table corresponding to a real object in the base model is replaced by a virtual object, a table tennis table, and props related to the replaced virtual object are placed, FIG. 5(c) shows a virtual environment with a bar theme in which a bar table, wine bottles, lighting, etc. are placed and a soft lighting effect is applied, FIG. 5(d) shows a virtual environment with an outdoor cafeteria theme in which outdoor cafe tables and chairs are placed and a background with a forest-like atmosphere is applied.

[0112] Referring to FIG. 5, the service providing device (100) can construct a virtual environment corresponding to a predetermined theme using a pre-built 3D model by changing the concept elements of the generated virtual object. For example, the service providing device (100) can implement virtual environments of different themes, such as a table tennis hall, a bar, and an outdoor cafeteria, targeting the same indoor space.

[0113] To this end, the service providing device (100) may change at least one of the size, material, and shape of a virtual object according to a set theme. For example, in the case of a table tennis theme, the service providing device (100) may replace a virtual object corresponding to an actual office table with a table tennis table and additionally place a shelf for storing table tennis rackets and balls around it. In addition, in the case of a bar theme, the service providing device (100) may replace a virtual object corresponding to an actual table with a bar table and place props such as wine bottles and glasses, and in the case of an outdoor cafeteria theme, may replace a virtual object corresponding to an actual table and chair with an outdoor cafe table and chair and place props such as coffee cups.

[0114] Additionally, the service providing device (100) may add background effects associated with the theme to the virtual environment. In this regard, according to one embodiment of the present invention, the service providing device (100) may reflect a background effect, including at least one of lighting effects, sound effects, and atmosphere effects, into the virtual environment to suit the set theme.

[0115] For example, the service providing device (100) can create a bar atmosphere by providing background music along with soft lighting effects in the case of a bar theme, and can create an outdoor atmosphere by adding natural objects such as trees and plants as a background and providing environmental sounds such as birdsong in the case of an outdoor cafeteria theme. In this regard, according to one embodiment of the present invention, the service providing device (100) can reflect a background effect including at least one of lighting effects, sound effects, and atmosphere effects in a virtual environment suitable for the set theme.

[0116] For example, the service providing device (100) disclosed herein may implement a virtual environment using the Unity 3D engine, but is not limited thereto. For example, the service providing device (100) may implement interaction with virtual objects by utilizing the XR Interaction Toolkit of Unity 3D, and may apply visual effects such as lighting, shadows, and colors suitable for each theme by utilizing the Post Processing Stack of Unity. In addition, a more realistic virtual environment may be provided by implementing 3D spatial sound by utilizing the audio system of Unity.

[0117] Additionally, the service providing device (100) can adjust environment settings for harmony between virtual objects and background effects. In this regard, according to one embodiment of the present invention, the service providing device (100) can reflect environment settings, including at least one of illuminance, volume, and color, in a virtual environment suitable for a set theme.

[0118] For example, in the case of a Bar theme, the service providing device (100) can lower the overall illumination and brighten the lighting on the table to create a soft atmosphere, and can maintain a level where conversation is possible by appropriately adjusting the volume of the background music. In addition, in the case of an outdoor cafeteria theme, it can set a bright overall illumination and emphasize natural colors to provide the feeling of an outdoor space. In this regard, according to one embodiment of the present invention, the service providing device (100) can reflect an environment setting including at least one of illumination, volume, and color in a virtual environment suitable for the set theme.

[0119] Meanwhile, the service providing device (100) disclosed herein can analyze the spatial characteristics of the target space and the type and specifications of actual objects to propose a theme of a virtual environment suitable for changing the target space. To this end, the service providing device (100) can analyze spatial characteristics such as the area, height, and arrangement pattern of actual objects of the target space, and object characteristics such as the type, size, and number of actual objects. For example, in the case of an office space where desks and chairs are arranged at regular intervals and storage cabinets are installed along the wall, the service providing device (100) can generate recommendation information that prioritizes the proposal of themes such as a study cafe, library, or exhibition hall to implement a virtual environment by considering the area and arrangement of objects of the space, and transmit it to a user terminal (400).

[0120] Additionally, the service providing device (100) can select a suitable theme by matching the attributes of an actual object with the functional role that the object performs in each theme. For example, the service providing device (100) can analyze the size, height, shape, etc. of a table to determine whether the table is suitable for use as a dining table, work table, conference table, etc., and based on this, evaluate the suitability of themes such as a restaurant, workshop, conference room, etc.

[0121] Meanwhile, if the service providing device (100) determines that the characteristics of the target space and the configuration of the actual object are unsuitable for a specific theme, it may recommend a more suitable alternative theme instead of the theme. For example, if a user selects a basketball court theme for a space that is narrow in area or has structures such as pillars, the service providing device (100) may analyze the constraints of the space and recommend an alternative theme that can be implemented in a relatively narrow space, such as a table tennis room or a billiards room.

[0122] Additionally, the service providing device (100) can identify in advance any constraints that may occur during conversion to a virtual environment by considering the type and placement status of the actual object, and can recommend an optimal theme that can resolve these constraints. For example, if the actual object is fixed or difficult to move, the service providing device (100) can prioritize recommending a theme that allows for a natural replacement with a virtual object by considering the position and size of the object.

[0123] In this way, the service providing device (100) does not merely apply a theme selected by the user, but also comprehensively analyzes the characteristics of the target space and the actual object to suggest an optimal theme or present an alternative to an inappropriate theme, thereby enabling the realization of a more realistic and natural virtual environment.

[0124] Additionally, the service providing device (100) can provide metaverse-based content using a virtual environment to a user terminal (400).

[0125] In this regard, according to one embodiment of the present invention, a service providing device (100) can replace actual user interaction data performed on an actual object provided in a target space with virtual interaction data performed on a virtual object through a constructed virtual environment.

[0126] Specifically, the service providing device (100) can output an interface through a user terminal (400) that outputs a virtual environment in which a virtual space modeled to correspond to a target space has been modified to conform to a predetermined theme. In this regard, through the interface to the virtual space, the user of the user terminal (400) can perform free interaction with the virtual environment by authorizing user input for individual operation (e.g., changing position, direction, etc.) of each virtual object placed within the virtual environment.

[0127] Figure 6 is a diagram illustrating an exemplary user interaction area for switching between virtual environments.

[0128] Referring to FIG. 6, the service providing device (100) may provide a user interaction area for switching between multiple virtual environments implemented for a single target space. Specifically, the service providing device (100) may place an interaction button ('A' in FIG. 6) for switching virtual environments at a specific location within the virtual environment, for example, near an entrance.

[0129] In this regard, according to one embodiment of the present invention, a service providing device (100) can provide content that allows a user to interact using an interaction button and a virtual hand while experiencing a virtual environment through a head-mounted display (410). For example, if a user operates an interaction button placed near an entrance using a virtual hand in a current table tennis room theme environment, the service providing device (100) can switch the space to a bar theme or an outdoor cafeteria theme virtual environment.

[0130] Meanwhile, the service providing device (100) can provide a natural transition experience by providing visual effects when switching virtual environments. For example, the service providing device (100) can cause the current virtual environment to gradually fade out while the new virtual environment fades in, or apply visual effects suitable for the transition effect. In addition, the service providing device (100) can control background sounds or environmental sounds suitable for each theme to transition naturally.

[0131] Figure 7 is a conceptual diagram illustrating a generalization process that enhances immersion through real-space-based modeling.

[0132] Referring to FIG. 7, the service providing device (100) can perform a generalized process to enhance user immersion through real space-based modeling. Specifically, the service providing device (100) can perform accurate alignment between real objects and virtual objects based on actual measurement data obtained from a point cloud.

[0133] According to one embodiment of the present invention, when creating a virtual object that matches a real object, the service providing device (100) can select the model most similar to the geometric shape of the real object from among a plurality of candidate models. For example, the service providing device (100) can improve the consistency between the real space and the virtual environment by selecting a virtual model most similar to the geometric shape of the object for real objects such as bookshelves and chairs.

[0134] Additionally, the service providing device (100) can adjust the size or placement of the virtual object to match the actual object in order to prevent collisions between the actual object and the virtual object. For example, the service providing device (100) can expand or adjust the size of the virtual table by taking into account the position of the actual chair to prevent a user from physically colliding with the virtual object near an area where the actual chair is placed.

[0135] In addition, the service providing device (100) can set an interaction area to enable direct interaction with virtual objects. For example, the service providing device (100) can provide a more realistic user experience in a virtual environment by allowing the user to pick up or manipulate props placed on a shelf or objects placed on a table using a virtual hand.

[0136] In this way, the service providing device (100) can provide the user with a more enhanced sense of immersion by accurately reflecting the physical characteristics of the actual space while enabling natural interaction in the virtual environment.

[0137] Hereinafter, an experimental example linked to a metaverse-based virtual experience service provision technique linked to real space according to one embodiment of the present invention will be described.

[0138] In an experimental example linked to a metaverse-based virtual experience service provision method linked to a real space according to one embodiment of the present invention, an experiment was conducted over approximately one month with a total of 31 participants (16 women, 15 men, average age 23.3 years). Specifically, the experiment was conducted by dividing the space into an alternative space equipped with actual objects (hereinafter, 'alternative space') and an empty space equipped with no objects (hereinafter, 'empty space'), and a head-mounted display (410) was used as the user terminal (400).

[0139] In this experimental example, user experience was quantitatively evaluated through three indicators: the System Usability Scale (hereinafter 'SUS'), the Replacement Effect (hereinafter 'RE'), and the Presence Questionnaire (hereinafter 'PQ'). Specifically, SUS is an indicator that evaluates the overall usability of the system, RE is an indicator that evaluates the effect of replacing virtual objects with real objects, and PQ is an indicator that evaluates presence and immersion in a virtual environment.

[0140] As a result of the experiment, in the SUS evaluation, both the substitute space and the empty space recorded a "Good" level score of 70 or higher, and no significant difference was found between the two spaces (p=0.76 > 0.05). In the RE evaluation, the substitute space recorded a score approximately 30% higher than the empty space, and statistically significant differences were observed particularly in the cognitive (p=0.022 < 0.05) and emotional (p=0.007 < 0.05) domains. In the PQ evaluation as well, the substitute space recorded a higher average score compared to the empty space, and this difference was also statistically significant (p=0.01 < 0.05).

[0141] Through these experimental results, it can be confirmed that the alignment and interaction between the actual object and the virtual object provided by the service providing device (100) disclosed herein can effectively enhance the user's sense of presence and immersion. Specifically, in the alternative space, physical interaction with the actual object was possible, which provided a higher sense of presence, and additional elements such as background music or sound effects were found to further enhance emotional immersion. Meanwhile, in the case of the empty space, although there was an advantage of allowing free movement due to the absence of physical constraints, it showed a relatively low sense of presence due to the lack of interaction with the actual object.

[0142] FIG. 8 is a schematic diagram of a device for providing virtual experience services based on a metaverse linked to real space according to one embodiment of the present invention.

[0143] Referring to FIG. 8, the service providing device (100) may include a data collection unit (110), a data preprocessing unit (120), a three-dimensional model creation unit (130), an object creation unit (140), a virtual environment implementation unit (150), and a content providing unit (160).

[0144] The data collection unit (110) can collect point cloud data in the target space using a depth sensor (not shown).

[0145] The data preprocessing unit (120) can perform preprocessing by removing outlier data included in the collected point cloud data and applying segmentation to divide multiple points included in the point cloud data into point groups corresponding to each actual object provided in the target space.

[0146] The 3D model generation unit (130) can build a 3D model corresponding to the target space using preprocessed point cloud data.

[0147] Specifically, the 3D model generation unit (130) can model the basic structure of the target space by identifying the floor surface, wall surface, and ceiling surface that make up the target space from the point cloud data measured in the target space corresponding to the indoor space.

[0148] In addition, the 3D model generation unit (130) can identify the geometric characteristics of the actual object from the collected point cloud data and use them to generate a bounding box for each of the actual objects.

[0149] In addition, the 3D model generation unit (130) can generate a 3D model by combining the modeled basic structure and the bounding box generated for the actual object.

[0150] The object creation unit (140) can identify semantic elements of an actual object provided in a target space and create a virtual object corresponding to the actual object using the identified semantic elements.

[0151] Specifically, the object generation unit (140) can identify semantic elements including the type, size, and shape of an actual object provided in the target space using a pre-trained artificial intelligence-based detection model.

[0152] Additionally, the object creation unit (140) can identify semantic elements including at least one of the type, physical size, shape, functional attributes, and spatial relationship with other actual objects of the actual object through analysis of the collected point cloud data.

[0153] Additionally, the object generation unit (140) can select a virtual object that matches the actual object from among a plurality of candidate models based on identified semantic elements.

[0154] Additionally, the object creation unit (140) can align the position and orientation of the selected virtual object with the actual object.

[0155] The virtual environment implementation unit (150) can construct a virtual environment that matches a predetermined theme by changing the concept elements of the created virtual object using a pre-built 3D model.

[0156] Specifically, the virtual environment implementation unit (150) can change at least one of the size, material, and shape of the virtual object according to the set theme.

[0157] Additionally, the virtual environment implementation unit (150) may add background effects associated with the theme to the virtual environment. In this regard, according to one embodiment of the present invention, the virtual environment implementation unit (150) may reflect a background effect, including at least one of lighting effects, sound effects, and atmosphere effects, into the virtual environment to suit the set theme.

[0158] Additionally, the virtual environment implementation unit (150) can adjust environment settings for harmony between virtual objects and background effects. In this regard, according to one embodiment of the present invention, the virtual environment implementation unit (150) can reflect environment settings, including at least one of illuminance, volume, and color, in the virtual environment to suit the set theme.

[0159] The content provider (160) can provide metaverse-based content using a virtual environment to a user terminal (400).

[0160] According to one embodiment of the present invention, the content providing unit (160) can replace actual user interaction data performed on an actual object provided in a target space with virtual interaction data performed on a virtual object through a constructed virtual environment.

[0161] Below, based on the details described above, we will briefly examine the operation flow of the present invention.

[0162] FIG. 9 is a flowchart of the operation of a method for providing a metaverse-based virtual experience service linked to a real space according to one embodiment of the present invention.

[0163] The method for providing a metaverse-based virtual experience service linked to a real space illustrated in FIG. 9 can be performed by the service providing device (100) described above. Therefore, even if the content is omitted below, the description of the service providing device (100) can be equally applied to the description of the method for providing a metaverse-based virtual experience service linked to a real space.

[0164] Referring to FIG. 9, in step S11, the data collection unit (110) can collect point cloud data in the target space using a depth sensor (not shown).

[0165] Next, in step S12, the data preprocessing unit (120) can perform preprocessing that removes outlier data included in the collected point cloud data and applies segmentation to divide multiple points included in the point cloud data into point groups corresponding to each actual object provided in the target space.

[0166] Next, in step S13, the 3D model generation unit (130) can build a 3D model corresponding to the target space using the preprocessed point cloud data.

[0167] Next, in step S14, the object creation unit (140) can identify semantic elements of an actual object provided in the target space and create a virtual object corresponding to the actual object using the identified semantic elements.

[0168] Next, in step S15, the virtual environment implementation unit (150) can construct a virtual environment corresponding to a predetermined theme from the 3D model constructed through step S13 by changing the concept elements of the created virtual object.

[0169] Next, in step S16, the content provider (160) can provide metaverse-based content using a virtual environment to a user terminal (400).

[0170] Specifically, in step S16, the content providing unit (160) can replace the actual user interaction data performed on the actual object provided in the target space with virtual interaction data performed on the virtual object through the constructed virtual environment.

[0171] In the description above, steps S11 through S16 may be further divided into additional steps or combined into fewer steps according to an embodiment of the present invention. Additionally, some steps may be omitted as necessary, and the order of the steps may be changed.

[0172] Figure 10 is a detailed flowchart of the process for building a 3D model of a target space.

[0173] The three-dimensional model construction process illustrated in FIG. 10 can be performed by the service providing device (100) described above. Therefore, even if the details are omitted below, the description of the service providing device (100) can be applied equally to the description of FIG. 10.

[0174] Referring to FIG. 10, in step S131, the 3D model generation unit (130) can model the basic structure of the target space by identifying the floor surface, wall surface, and ceiling surface that make up the indoor space from the point cloud data measured in the target space corresponding to the indoor space.

[0175] Next, in step S132, the 3D model generation unit (130) can identify the geometric characteristics of the actual object from the collected point cloud data and use them to generate a bounding box for each of the actual objects.

[0176] Next, in step S133, the 3D model generation unit (130) can generate a 3D model by combining the modeled basic structure and the bounding box generated for the actual object.

[0177] In the description above, steps S131 to S133 may be further divided into additional steps or combined into fewer steps according to an embodiment of the present invention. Additionally, some steps may be omitted as necessary, and the order of the steps may be changed.

[0178] Figure 11 is a detailed flowchart of the process for creating a virtual object corresponding to a real object.

[0179] The virtual object creation process illustrated in FIG. 11 can be performed by the service providing device (100) described above. Therefore, even if the details are omitted below, the description of the service providing device (100) can be applied equally to the description of FIG. 11.

[0180] Referring to FIG. 11, in step S141, the object generation unit (140) can identify semantic elements including the type, size, and shape of an actual object provided in the target space using a pre-trained artificial intelligence-based detection model.

[0181] Specifically, in step S141, the object creation unit (140) can identify semantic elements including at least one of the type, physical size, shape, functional attributes, and spatial relationship with other actual objects of the actual object through analysis of the collected point cloud data.

[0182] Next, in step S142, the object creation unit (140) can select a virtual object that matches the actual object from among a plurality of candidate models based on the identified semantic elements.

[0183] Next, in step S143, the object creation unit (140) can align the position and orientation of the selected virtual object with the actual object.

[0184] In the description above, steps S141 to S143 may be further divided into additional steps or combined into fewer steps according to an embodiment of the present invention. Additionally, some steps may be omitted as necessary, and the order of the steps may be changed.

[0185] Figure 12 is a detailed flowchart of the process for building a virtual environment of a specific theme through the change of concept elements.

[0186] The virtual environment construction process illustrated in FIG. 12 can be performed by the service providing device (100) described above. Therefore, even if the details are omitted below, the description of the service providing device (100) can be applied equally to the description of FIG. 12.

[0187] Referring to FIG. 12, in step S151, the virtual environment implementation unit (150) can change at least one of the size, material, and shape of the virtual object according to the set theme.

[0188] Next, in step S152, the virtual environment implementation unit (150) can add background effects associated with the theme to the virtual environment.

[0189] Specifically, in step S152, the virtual environment implementation unit (150) can reflect a background effect, including at least one of lighting effect, sound effect, and atmosphere effect, into the virtual environment to suit the set theme.

[0190] Next, in step S153, the virtual environment implementation unit (150) can adjust the environment settings for harmony between the virtual object and the background effect.

[0191] Specifically, in step S153, the virtual environment implementation unit (150) can reflect an environment setting, including at least one of illuminance, volume, and color, in the virtual environment to suit the set theme.

[0192] In the description above, steps S151 to S153 may be further divided into additional steps or combined into fewer steps according to an embodiment of the present invention. Additionally, some steps may be omitted as necessary, and the order of the steps may be changed.

[0193] A method for providing a virtual experience service based on a metaverse linked to real space according to one embodiment of the present invention may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either individually or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the present invention, or they may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The above-described hardware device may be configured to operate as one or more software modules to perform the operation of the present invention, and vice versa.

[0194] In addition, the aforementioned method may also be implemented in the form of a computer program or application executed by a computer that is stored on a recording medium.

[0195] The foregoing description of the present invention is for illustrative purposes only, and those skilled in the art will understand that other specific forms can be easily modified without altering the technical concept or essential features of the present invention. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. For example, each component described as a single unit may be implemented in a distributed manner, and components described as distributed may likewise be implemented in a combined form.

[0196] The scope of the present invention is defined by the claims set forth below rather than by the detailed description above, and all modifications or variations derived from the meaning and scope of the claims and the concept of equivalents thereof should be interpreted as being included within the scope of the present invention.

Claims

1. In a method for providing a metaverse-based virtual experience service linked to real space, (a) A step of collecting point cloud data in the target space using a depth sensor; (b) a step of constructing a three-dimensional model corresponding to the target space using the point cloud data; (c) identifying semantic elements of an actual object provided in the target space and generating a virtual object corresponding to the actual object using the semantic elements; and (d) a step of constructing a virtual environment corresponding to a predetermined theme from the 3D model by changing the concept elements of the virtual object, A method for providing a virtual experience service including 2. In Paragraph 1, (e) a step of providing metaverse-based content using the above virtual environment to a user terminal, A method for providing a virtual experience service that further includes 3. In Paragraph 2, The above step (e) is, A step of replacing actual user interaction data performed on the actual object with virtual interaction data performed on the virtual object through the virtual environment, A method for providing a virtual experience service that includes 4. In Paragraph 1, After step (a) above, A step of performing preprocessing that removes outlier data included in the point cloud data and applies segmentation to divide a plurality of points included in the point cloud data into point groups corresponding to each of the actual objects. A method for providing a virtual experience service that further includes 5. In Paragraph 1, The above-mentioned target space includes an indoor space, and The above step (b) is, A step of modeling the basic structure of the target space by identifying the floor surface, wall surface, and ceiling surface forming the indoor space from the above point cloud data; The step of identifying geometric characteristics of the actual objects from the point cloud data and generating bounding boxes for each of the actual objects; and A step of generating the three-dimensional model by combining the basic structure and the bounding box, A method for providing a virtual experience service that includes 6. In Paragraph 1, The above step (c) is, A step of identifying the semantic elements, including the type, size, and shape of the actual object, using a pre-trained artificial intelligence-based detection model; A step of selecting the virtual object that matches the actual object from among a plurality of candidate models based on the identified semantic elements; and A step of aligning the position and orientation of the selected virtual object with the actual object, A method for providing a virtual experience service that includes 7. In Paragraph 1, The above step (d) is, A step of changing at least one of the size, material, and shape of the virtual object according to the above theme; A step of adding background effects associated with the above theme to the above virtual environment; and A step of adjusting environment settings for harmony between the above virtual object and the above background effect, A method for providing a virtual experience service that includes 8. In Paragraph 7, The above background effect includes at least one of lighting effects, sound effects, and atmosphere effects, and A method for providing a virtual experience service, wherein the above environment settings include at least one of illuminance, volume, and color.

9. In Paragraph 1, The above semantic element is, A method for providing a virtual experience service, comprising at least one of the type, physical size, shape, functional attributes, and spatial relationship with other real objects of the above-mentioned actual object.

10. In a device for providing virtual experience services based on a metaverse linked to real space, A data collection unit that collects point cloud data in a target space using a depth sensor; A 3D model generation unit that constructs a 3D model corresponding to the target space using the above point cloud data; An object generation unit that identifies semantic elements of an actual object provided in the above target space and generates a virtual object corresponding to the actual object using the semantic elements; and A virtual environment implementation unit that constructs a virtual environment conforming to a predetermined theme from the 3D model by changing the concept elements of the virtual object, A virtual experience service providing device including 11. In Paragraph 10, A content providing unit that provides metaverse-based content using the above-mentioned virtual environment to a user terminal, A virtual experience service providing device further comprising 12. In Paragraph 11, The above-mentioned content provider is, A virtual experience service providing device that replaces actual user interaction data performed on the actual object with virtual interaction data performed on the virtual object through the virtual environment.

13. In Paragraph 10, A data preprocessing unit that performs preprocessing by removing outlier data included in the above point cloud data and applying segmentation to divide a plurality of points included in the above point cloud data into point groups corresponding to each of the actual objects, A virtual experience service providing device that further includes 14. In Paragraph 10, The above object creation unit is, A virtual experience service providing device that identifies semantic elements including the type, size, and shape of the actual object using a pre-trained artificial intelligence-based detection model, selects a virtual object that matches the actual object from among a plurality of candidate models based on the identified semantic elements, and aligns the position and orientation of the selected virtual object with the actual object.

15. In Paragraph 10, The virtual environment implementation unit mentioned above is, A virtual experience service providing device that changes at least one of the size, material, and shape of the virtual object according to the above theme, adds a background effect associated with the above theme to the virtual environment, and adjusts environment settings for harmony between the virtual object and the background effect.

Images