Method and system for controlling virtual object in XR environment

Abstract

According to one aspect of the present invention, provided is a method for controlling a virtual object in an eXtended reality (XR) environment, the method comprising the steps of: recognizing a preset user gesture by using an XR device; and in response to the recognition of the preset user gesture, arranging, in a second scale in a real space, at least one 3D virtual object arranged in a first scale in a 3D virtual space, or arranging, in the first scale in the 3D virtual space, the at least one 3D virtual object arranged in the second scale in the real space.

Description

Method and system for controlling virtual objects in an XR environment

[0001] The present invention relates to a method and system for controlling virtual objects in an XR environment.

[0002] [This work was supported by the Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. RS-2024-00397663, Real-time XR Interface Technology Development for Environmental Adaptation); This patent is based on research conducted with support from the Institute of Information & Communications Technology Planning & Evaluation, funded by the government (Ministry of Science and ICT) in 2024 (No. RS-2024-00397663, Development of Real-time XR Interface Technology Adapting to the Environment)]

[0003] Recently, with the proliferation of glasses-type XR devices such as Apple’s Vision Pro and Meta’s Quest 3, there has been a growing demand for multi-party immersive remote experiences, such as remote work, metaverse performances, arts, and shopping, as well as economic platforms based on them. Consequently, there is also a growing need for adaptive, realistic XR interface technology for the metaverse that allows one to see, hear, and touch as if it were reality.

[0004] While glasses-type XR devices allow users to view and interact with 3D virtual objects as if they were actually right in front of their eyes, current commercialization is limited to providing visual experiences such as touching, moving, or changing the size of 3D virtual objects, and thus fails to fully utilize metaverse platforms that overcome spatial limitations.

[0005] In particular, glasses-type XR devices primarily rely on the user's hands as an input method; however, the reality is that detailed interaction with 3D virtual objects is difficult because they fail to accurately recognize gestures when the hands move quickly and cannot detect the contact surfaces or minute displacements intended by the user. Furthermore, since they remain based on 2D screen interfaces, the load is heavy when switching between reality and XR tasks, and it is difficult to encourage the continued use of wearable XR devices because they fail to provide users with audio-tactile feedback corresponding to reality when interacting with 3D virtual objects. There is also the problem that not only is it difficult to protect the copyright of meta-objects (virtual objects in the metaverse environment) created by users, but the continuous production and management of meta-objects is also difficult due to the absence of an incentive system.

[0006] Furthermore, currently commercialized XR applications are unable to enable users connected from geographically separated locations to have a sense of co-existence through shared experiences due to factors such as insufficient real-time 3D interaction synchronization and the absence of realistic feedback. Although research on multi-party XR interaction and object experiences is underway, there is still a lack of cases that provide a proper user experience by integrating various elemental technologies for realistic experiences into actual XR platforms. To address this issue, it is necessary to demonstrate the effectiveness of XR immersive experiences as a metaverse economy platform by linking XR immersive experience platforms with industries that have actual demand, such as XR immersive commerce and remote science museum visits.

[0007] Accordingly, the inventor(s) propose a technology that enables users to have various experiences in which reality and virtuality are fused beyond spatial constraints through meta-objects by recognizing a preset user gesture using an XR device and, in response to the recognition of the preset user gesture, placing at least one 3D virtual object placed at a first scale in a 3D virtual space into a real space at a second scale, or placing at least one 3D virtual object placed at a second scale in a real space into a 3D virtual space at a first scale.

[0008] <Prior Art Literature>

[0009] <Patent Literature>

[0010] (Patent Document 1) Korean Published Patent Application No. 10-2020-0076647 (June 29, 2020)

[0011] The present invention aims to solve all the problems of the aforementioned prior art.

[0012] In addition, the present invention has another purpose of recognizing a preset user gesture using an XR device and, in response to the recognition of the preset user gesture, placing at least one 3D virtual object placed at a first scale in a 3D virtual space into a real space at a second scale, or placing at least one 3D virtual object placed at a second scale in a real space into a 3D virtual space at a first scale.

[0013] In addition, another objective of the present invention is to provide a metaverse economic platform that enables users to have various experiences where reality and virtuality are fused, transcending spatial limitations through meta-objects.

[0014] In addition, another objective of the present invention is to support the creation (authoring) and management of meta-objects having various inherent attributes.

[0015] In addition, another objective of the present invention is to provide realistic tactile feedback based on interaction by measuring the user's rapid gestures and fine displacements.

[0016] In addition, another objective of the present invention is to provide technology regarding an intelligent simulation-based environment-adaptive real-time XR interface and META OBJ (Metaverse Experience with Tangible Attributes and Object Behavior Junction) for real-time on-site-remote multi-party joint experience.

[0017] A representative configuration of the present invention for achieving the above objective is as follows.

[0018] According to one aspect of the present invention, a method is provided comprising the steps of: recognizing a preset user gesture using an XR device; and, in response to the recognition of the preset user gesture, placing at least one 3D virtual object placed at a first scale in a 3D virtual space into a real space at a second scale, or placing the at least one 3D virtual object placed at the second scale in the real space into the 3D virtual space at the first scale.

[0019] According to another aspect of the present invention, a system is provided comprising a gesture recognition unit that recognizes a preset user gesture using an XR device, and, in response to the recognition of the preset user gesture, at least one 3D virtual object placed at a first scale in a 3D virtual space is placed in a real space at a second scale, or at least one 3D virtual object placed at the second scale in the real space is placed at the first scale in the 3D virtual space.

[0020] In addition to this, other methods for implementing the present invention, other systems, and non-transient computer-readable recording media for recording a computer program for executing said methods are further provided.

[0021] According to the present invention, a preset user gesture is recognized using an XR device, and in response to the recognition of the preset user gesture, at least one 3D virtual object placed at a first scale in a 3D virtual space is placed in a real space at a second scale, or at least one 3D virtual object placed at a second scale in a real space is placed in a 3D virtual space at a first scale.

[0022] In addition, according to the present invention, it is possible to provide users with a metaverse economic platform that enables various experiences where reality and virtuality are fused, transcending spatial constraints through meta-objects.

[0023] FIG. 1 is a diagram showing the schematic configuration of an overall system for controlling a virtual object in an XR environment according to one embodiment of the present invention.

[0024] FIG. 2 is a drawing illustrating in detail the internal configuration of an XR providing system (a system corresponding to 200) according to one embodiment of the present invention.

[0025] FIGS. 3 to 13 are drawings illustrating an exemplary description of a process or method of controlling a virtual object according to an embodiment of the present invention.

[0026] <Explanation of Symbols>

[0027] 100: Communication network

[0028] 200: XR providing system

[0029] 210: Gesture recognition unit

[0030] 220: XR Management Department

[0031] 230: Communications Department

[0032] 240: Control unit

[0033] 300: Device

[0034] The following detailed description of the invention refers to the accompanying drawings, which illustrate specific embodiments in which the invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention. It should be understood that various embodiments of the invention are different but need not be mutually exclusive. For example, specific shapes, structures, and characteristics described herein may be modified from one embodiment to another without departing from the spirit and scope of the invention. It should also be understood that the location or arrangement of individual components within each embodiment may be modified without departing from the spirit and scope of the invention. Accordingly, the following detailed description is not meant to be limiting, and the scope of the invention should be understood to encompass the scope claimed by the claims and all equivalents thereof. Similar reference numerals in the drawings indicate identical or similar components across various aspects.

[0035] Hereinafter, in order to enable a person skilled in the art to easily practice the present invention, various preferred embodiments of the present invention will be described in detail with reference to the attached drawings.

[0036] Configuration of the entire system

[0037] FIG. 1 is a diagram showing the schematic configuration of an overall system for controlling a virtual object in an XR environment according to one embodiment of the present invention.

[0038] As illustrated in FIG. 1, the entire system according to one embodiment of the present invention may include a communication network (100), an XR providing system (200), and an XR device (300).

[0039] First, a communication network (100) according to one embodiment of the present invention can be configured regardless of the mode of communication, such as wired communication or wireless communication, and can be configured as various communication networks such as a Local Area Network (LAN), a Metropolitan Area Network (MAN), or a Wide Area Network (WAN). Preferably, the communication network (100) referred to in this specification may be the known Internet or the World Wide Web (WWW). However, the communication network (100) may include at least a known wired / wireless data communication network, a known telephone network, or a known wired / wireless television communication network, without being limited thereto.

[0040] For example, the communication network (100) may be a wireless data communication network and may implement conventional communication methods such as WiFi communication, WiFi-Direct communication, Long Term Evolution (LTE) communication, 5G communication, Bluetooth communication (including Bluetooth Low Energy (BLE) communication), infrared communication, ultrasonic communication, etc., in at least a part thereof. As another example, the communication network (100) may be an optical communication network and may implement conventional communication methods such as Light Fidelity (LiFi), etc., in at least a part thereof.

[0041] Next, an XR providing system (200) according to one embodiment of the present invention can recognize a preset user gesture using an XR device, and in response to the recognition of the preset user gesture, can perform the function of placing at least one 3D virtual object placed at a first scale in a 3D virtual space into a real space at a second scale, or placing at least one 3D virtual object placed at a second scale in a real space into a 3D virtual space at a first scale.

[0042] The configuration and function of the XR providing system (200) according to the present invention will be examined in detail through the following detailed description.

[0043] Next, the XR device (300) according to one embodiment of the present invention is a digital device that includes a function to communicate after connecting to the XR providing system (200), and any digital device equipped with memory means and equipped with a microprocessor to have computational capabilities, such as a smartphone, tablet, smart watch, smart band, smart glasses, desktop computer, laptop computer, workstation, PDA, web pad, mobile phone, etc., can be adopted as the XR device (300) according to the present invention.

[0044] In particular, the XR device (300) may include an application (not shown) that enables a user to receive a service according to the present invention from the XR providing system (200). Such an application may be downloaded from the XR providing system (200) or an external application distribution server (not shown). Meanwhile, the nature of such an application may generally be similar to the gesture recognition unit (210), XR management unit (220), communication unit (230), and control unit (240) of the XR providing system (200) as described below. Here, at least a part of the application may be replaced with a hardware device or firmware device capable of performing substantially the same or equivalent functions as needed.

[0045] Configuration of the XR provision system

[0046] Below, we will examine the internal configuration of the XR providing system (200) that performs important functions for the implementation of the present invention and the functions of each component.

[0047] FIG. 2 is a drawing illustrating in detail the internal configuration of an XR providing system (200) according to one embodiment of the present invention.

[0048] As illustrated in FIG. 2, an XR providing system (200) according to one embodiment of the present invention may be configured to include a gesture recognition unit (210), an XR management unit (220), a communication unit (230), and a control unit (240). According to one embodiment of the present invention, the gesture recognition unit (210), the XR management unit (220), the communication unit (230), and the control unit (240) may be program modules, at least some of which communicate with an external system (not shown). Such program modules may be included in the XR providing system (200) in the form of an operating system, an application program module, or other program modules, and may be physically stored in various known storage devices. Additionally, such program modules may be stored in a remote storage device capable of communicating with the XR providing system (200). Meanwhile, such program modules encompass, but are not limited to, routines, subroutines, programs, objects, components, data structures, etc., that perform specific tasks or execute specific abstract data types as described below according to the present invention.

[0049] Meanwhile, although the XR providing system (200) has been described as above, this description is exemplary, and it is obvious to those skilled in the art that at least some of the components or functions of the XR providing system (200) may be realized within an XR device (300) or server (not shown) or included within an external system (not shown) as needed.

[0050] First, a gesture recognition unit (210) according to one embodiment of the present invention can perform the function of recognizing a preset user gesture using an XR device.

[0051] Specifically, a gesture recognition unit (210) according to one embodiment of the present invention can recognize user gestures, particularly user gestures using a hand, by utilizing an artificial intelligence-based behavior recognition model. Such user gestures may be pre-set, for example, pinching, moving while pinching or releasing the pinched state, turning the palm upward or downward, hand camera movements, pinching after turning the palm in a specific direction, etc., may correspond to such user gestures. However, the types of user gestures according to one embodiment of the present invention are not limited to those listed above and may be varied within the scope of achieving the purpose of the present invention.

[0052] Next, an XR management unit (220) according to one embodiment of the present invention may perform the function of placing at least one 3D virtual object placed at a first scale in a 3D virtual space into a real space at a second scale in response to the recognition of a preset user gesture, or placing at least one 3D virtual object placed at a second scale in a real space into a 3D virtual space at a first scale. Here, according to one embodiment of the present invention, the first scale may be a reduced scale (small-scale), and the second scale may be a full scale or an original scale.

[0053] FIGS. 3 to 13 are drawings illustrating an exemplary description of a process or method of controlling a virtual object according to an embodiment of the present invention.

[0054] For example, referring to FIGS. 3 and 4, an XR management unit (220) according to one embodiment of the present invention may begin preparations to place 3D virtual objects placed at a first scale in a 3D virtual space (400; e.g., a 3D room) into a real space at a second scale in response to the recognition of a pre-set user gesture, such as a hand camera motion. Furthermore, referring to FIGS. 5 and 6, an XR management unit (220) according to one embodiment of the present invention may place 3D virtual objects (500) placed at a first scale in a 3D virtual space (400) into a real space at a second scale in response to the recognition of a pre-set user gesture, such as a pinch after facing a specific direction.

[0055] An XR management unit (220) according to one embodiment of the present invention can support a user to edit 3D virtual objects (500) placed in real space at a second scale (e.g., adding / deleting / changing attributes of selected 3D virtual objects). FIG. 7 illustrates, exemplarily, a scene in which a globe, which is a selected 3D virtual object, is deleted.

[0056] For another example, referring to FIG. 8, an XR management unit (220) according to one embodiment of the present invention can place 3D virtual objects (500; which may mean objects with the user's edits reflected if there were edits made while they were placed in real space) placed in a 3D virtual space (400) at a first scale in response to the recognition of a hand camera motion, which is a preset user gesture.

[0057] An XR management unit (220) according to one embodiment of the present invention can support a user in editing 3D virtual objects placed at a first scale in a 3D virtual space (e.g., adding / deleting / changing attributes of selected 3D virtual objects). FIGS. 9 to 11 illustrate, exemplarily, scenes in which a user edits 3D virtual objects placed at a first scale in a 3D virtual space.

[0058]

[0059] According to the present invention, the following functions or experiences can be provided to the user or effects can be achieved:

[0060]

[0061] 1. Creation of Meta-objects and Conversion into Assets / Authoring Tools

[0062] Technologies and interfaces for meta-object creation (authoring) and management are provided to induce continuous meta-object generation by applying an appropriate reward system for their creation, thereby establishing a sustainable metaverse economic platform. To provide a realistic interaction experience, users are enabled to interact with objects that embed attributes reflecting temperature and physical properties, as well as reversible and irreversible actions such as disassembly, assembly, transformation, and destruction. Furthermore, meta-objects are created using incentive-based, user-participatory meta-object authoring tools, which are then registered, tracked, and managed within an asset store.

[0063]

[0064] 2. Human Gesture Responsive Visual / Auditory / Tactile Feedback Technology

[0065] It tracks users' rapid gestures and minute displacements through sensor fusing, and enables a realistic immersive experience based on user-object-environmental interactions through visual (visual feedback technology based on meta-object attributes and user task context), auditory (3D acoustic technology adaptive to the time and position of interaction events), and tactile (vibration / temperature / force hardware technology for experiencing realistic meta-object physical properties) feedback.

[0066]

[0067] 3. Intelligent Simulation and Wearable XR Experience Platform

[0068] To facilitate intelligent interaction between meta-objects, environments, and users, it provides intelligent simulation technology that learns by observing interactions with meta-objects in virtual space and responds adaptively based on the Scene Graph (SG); a data structure used in 3D computer graphics that hierarchically represents the relationships between objects within a scene, including various elements such as the spatial arrangement, attributes, transformations, and interactions of objects. Furthermore, by utilizing a wearable XR platform integrating a meta-object authoring interface and audio-tactile XR interaction, it enables low-latency, realistic collaborative interaction and experiences between multiple on-site and remote users via 3D avatars.

[0069]

[0070] Next, the communication unit (230) according to one embodiment of the present invention can perform the function of enabling data transmission and reception from / to the gesture recognition unit (210) and the XR management unit (220).

[0071] Finally, a control unit (240) according to one embodiment of the present invention can perform the function of controlling the flow of data between a gesture recognition unit (210), an XR management unit (220), and a communication unit (230). That is, by controlling the flow of data from / to / from the outside of the XR providing system (200) or the flow of data between each component of the XR providing system (200), the control unit (240) according to one embodiment of the present invention can control the gesture recognition unit (210), the XR management unit (220), and the communication unit (230) to perform their respective unique functions.

[0072] The embodiments according to the present invention described above may be implemented in the form of program instructions that can be executed through various computer components and recorded on a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc., either individually or in combination. The program instructions recorded on the computer-readable recording medium may be those specifically designed and configured for the present invention or those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. Hardware devices may be modified into one or more software modules to perform processing according to the present invention, and vice versa.

[0073] Although the present invention has been described above with reference to specific details such as specific components, limited embodiments, and drawings, this is provided only to aid in a more comprehensive understanding of the invention, and the invention is not limited to the above embodiments, and a person skilled in the art to which the invention belongs can make various modifications and changes from this description.

[0074] Accordingly, the scope of the present invention should not be limited to the embodiments described above, and all scopes equivalent to or equivalently modified from the claims set forth below, as well as the claims set forth below, shall be considered to fall within the scope of the concept of the present invention.

Claims

1. As a method for controlling virtual objects in an XR (eXtented Reality) environment, A step of recognizing a preset user gesture using an XR device, and In response to the recognition of the above-mentioned preset user gesture, the method comprises the step of placing at least one 3D virtual object placed at a first scale in a 3D virtual space into a real space at a second scale, or placing the at least one 3D virtual object placed at the second scale in the real space into the 3D virtual space at the first scale. method.

2. A non-transient computer-readable recording medium for recording a computer program for executing the method according to paragraph 1.

3. As a system for controlling virtual objects in an XR (eXtented Reality) environment, A gesture recognition unit that recognizes preset user gestures using an XR device, and An XR management unit comprising, in response to the recognition of the above-mentioned preset user gesture, placing at least one 3D virtual object placed at a first scale in a 3D virtual space into a real space at a second scale, or placing the at least one 3D virtual object placed at the second scale in the real space into the 3D virtual space at the first scale. System.

Images