Wearable device for sharing real object or background in xr environment, and operating method thereof

Abstract

The present disclosure provides a head mounted display (HMD) device. The HDM device can: receive a user input of requesting sharing of a first object in a first scene acquired through at least one camera; generate three-dimensional object information corresponding to the first object on the basis of one or more images of the first scene; share the three-dimensional object information with an external electronic device through a communication circuit; receive, through the communication circuit, three-dimensional background information from the external electronic device in response to sharing the three-dimensional object information; generate a three-dimensional virtual space on the basis of the three-dimensional background information; and display the three-dimensional virtual space in an area that excludes the first object in the first scene, so as to output same through a display.

Description

Wearable device sharing real-world objects or backgrounds in an XR environment and method of operation thereof

[0001] The present disclosure relates to a wearable device that shares a real-world object or background in an XR environment and a method of operating the same.

[0002] XR (Extended Reality) encompasses extended reality technologies including VR (Virtual Reality), AR ( Augmented Reality), and MR (Mixed Reality). XR fuses reality with digital environments to provide an immersive environment for user interaction. VR creates a completely virtualized space that separates the user from the physical world, while AR overlays digital information onto the real world. MR provides an environment where real and virtual elements interact in real time.

[0003] XR technology utilizes various devices such as immersive headsets (HMDs), smartphones, and AR classes, and applies 3D rendering, sensor data processing, and computer vision technologies to implement them. It is being applied in various fields such as gaming, education, healthcare, architectural design, and remote collaboration.

[0004] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art related to the present disclosure.

[0005] A head-mounted display (HMD) device according to one embodiment of the present disclosure comprises at least one camera; a display; at least one processor including a communication circuit and a processing circuit; and a memory including at least one storage medium for storing instructions, wherein when the instructions are executed individually or collectively by the at least one processor, the HMD device may be caused to: receive a user input requesting sharing of a first object in a first scene obtained through the at least one camera; generate three-dimensional object information corresponding to the first object based on one or more images of the first scene; share the three-dimensional object information with an external electronic device through the communication circuit; receive three-dimensional background information from the external electronic device in response to sharing the three-dimensional object information through the communication circuit; generate a three-dimensional virtual space based on the three-dimensional background information; and display the three-dimensional virtual space in an area excluding the first object in the first scene and output it through the display.

[0006] An HMD device according to another embodiment of the present disclosure comprises at least one camera; a display; at least one processor including a communication circuit and a processing circuit; and a memory including at least one storage medium for storing instructions, wherein the instructions, when executed individually or collectively by the at least one processor, may cause the HMD device to: receive three-dimensional object information sharing from an external electronic device through the communication circuit, generate three-dimensional background information from images acquired for a first scene through the at least one camera, share the three-dimensional background information with the external electronic device through the communication circuit, and place a first virtual object corresponding to the three-dimensional object information in the first scene and output it through the display.

[0007] An HMD device according to another embodiment of the present disclosure comprises at least one camera; a display; a communication circuit; and at least one processor including a processing circuit; and a memory including at least one storage medium for storing instructions, wherein the instructions, when executed individually or collectively by the at least one processor, may cause the HMD device to: receive first virtual object information or first virtual space information sharing from an external electronic device through the communication circuit; in response to receiving the first virtual object information, place the first virtual object in a first scene obtained through the at least one camera and output it through the display; and in response to receiving the first virtual space information, superimpose the first virtual space in a second scene obtained through the at least one camera and output it through the display.

[0008] However, the problems to be solved in this disclosure are not limited to those mentioned above, and may be determined in various ways without departing from the scope and subject matter of this disclosure.

[0009] FIG. 1 is a block diagram of an electronic device in a network environment according to various embodiments.

[0010] FIG. 2a illustrates an example of a perspective view of a wearable device according to one embodiment.

[0011] FIG. 2b illustrates an example of one or more hardware components disposed within a wearable device according to one embodiment.

[0012] FIGS. 3a and 3b illustrate an example of the appearance of a wearable device according to one embodiment.

[0013] FIG. 4 illustrates an example of a block diagram of a wearable device according to one embodiment.

[0014] FIG. 5a illustrates an example in which wearable devices share a real object and a background in a spaced-apart space, according to one embodiment of the present disclosure.

[0015] FIG. 5b illustrates an example in which wearable devices share a real object and a background in a spaced-apart space using a pairing terminal, according to one embodiment of the present disclosure.

[0016] FIG. 6 is a flowchart illustrating the operation of a wearable device sharing a real object with an external electronic device according to one embodiment of the present disclosure.

[0017] FIG. 7 is a flowchart illustrating the operation of a wearable device receiving an external object from an external electronic device according to one embodiment of the present disclosure.

[0018] FIG. 8a is an example of a wearable device sharing a real object according to one embodiment of the present disclosure.

[0019] FIG. 8b is an example of a wearable device sharing a real-world background according to one embodiment of the present disclosure.

[0020] FIG. 9a is an example of a wearable device sharing a real-world background according to one embodiment of the present disclosure.

[0021] FIG. 9b is an example of a wearable device receiving a real-world object according to one embodiment of the present disclosure.

[0022] FIG. 10 is an example of a wearable device recommending a shared object according to one embodiment of the present disclosure.

[0023] FIG. 11a is an example of manually adjusting the placement of a shared object of a wearable device according to one embodiment of the present disclosure.

[0024] FIG. 11b is an example of automatically adjusting the placement of shared objects of a wearable device according to one embodiment of the present disclosure.

[0025] FIG. 12 is an example of a display screen that displays information of an object shared by a wearable device according to one embodiment of the present disclosure.

[0026] FIG. 13 is an example of a display screen in which a wearable device changes and displays the attributes of an object received by a wearable device according to one embodiment of the present disclosure.

[0027] FIG. 14 is an example of a wearable device editing a display screen displaying a real-world scene according to one embodiment of the present disclosure.

[0028] In the following description, the attached drawings are referenced, and specific examples of implementation are illustrated within the drawings. Additionally, other examples may be used and structural modifications may be made without departing from the scope of the various examples.

[0029] The terms used in this disclosure are used merely to describe specific embodiments and are not intended to limit the technical features of this disclosure. For example, a component expressed in the singular form should be understood as a concept including singular or plural components unless the context clearly indicates only the singular form.

[0030] In the present disclosure, each of the phrases such as “A or B,” “at least one of A and B,” “at least one of A or B,” “A, B or C,” “at least one of A, B and C,” and “at least one of A, B, or C” may include any one of the items listed with the corresponding phrase, or all possible combinations thereof. The term “and / or” as used in the present disclosure should be understood to encompass any possible combination by one or more of the plurality of items listed with the corresponding term. Terms such as “first,” “second,” “first,” or “second” as used in the present disclosure may be used merely to distinguish a component from another component and do not limit the components in any other aspect (e.g., importance or order).

[0031] Where it is stated that any (e.g., 1st) component is “coupled,” “connected,” “linked,” “coupled,” “supported,” “connected,” or “contacted” with or without the terms “functionally” or “communicationly,” it includes not only cases where the component is directly coupled, connected, linked, coupled, supported, or contacted with the other component, but also cases where it is indirectly coupled, connected, linked, coupled, supported, or contacted through a third component.

[0032] Terms such as "include" or "have" are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in this disclosure, and do not preclude the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof. When a component is described as being located "on" another component, this includes not only cases where the component is in contact with the other component, but also cases where another component exists between the two components.

[0033] The expression “configured to” as used in this disclosure may be appropriately substituted, depending on the context, for example, “suitable for,” “capable of,” “designed to,” “modified to,” “made to,” or “capable of.” The term “configured to” does not necessarily mean only that which is “specially designed” in hardware. Instead, in some situations, the expression “device configured to” may mean that the device is “capable of” together with other devices or components. For example, the phrase “device configured (or set) to perform A, B, and C” may mean a device dedicated to performing the said operation, or a general-purpose device capable of performing various operations including said operation.

[0034] Terms used in this disclosure, such as "upper side," "lower side," and "front-rear direction," are defined based on the drawings, and the shape and position of each component are not limited by these terms.

[0035] Although the description in this disclosure is centered on specific embodiments, this disclosure is not limited to such specific embodiments and should be understood to encompass all various modifications, equivalents, and / or substitutions of the various embodiments described in this disclosure. In connection with the description of the drawings, similar reference numerals may be used for similar or related components.

[0036] FIG. 1 is a block diagram of an electronic device in a network environment according to various embodiments.

[0037] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or with at least one of an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through a server (108). According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (160), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)).

[0038] The processor (120) can control at least one other component (e.g., hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., program (140)), for example, and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., sensor module (176) or communication module (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., central processing unit or application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., graphics processing unit, neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use lower power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.

[0039] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0040] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, input data or output data for software (e.g., program (140)) and related commands. The memory (130) may include volatile memory (132) or non-volatile memory (134).

[0041] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0042] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0043] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0044] The display module (160) can visually provide information to an external (e.g., user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0045] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (101).

[0046] The sensor module (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0047] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multi-media interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0048] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0049] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0050] The camera module (180) can capture still images and video. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0051] The power management module (188) can manage the power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0052] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0053] The communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).

[0054] The wireless communication module (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or largescale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), external electronic device (e.g., electronic device (104)), or network system (e.g., second network (199)). According to one embodiment, the wireless communication module (192) can support a Peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mMTC, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for realizing URLLC.

[0055] An antenna module (197) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197).

[0056] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0057] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0058] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (104) or the server (108) may be included within the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0059] The electronic device according to the various embodiments disclosed in this document may be of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. The electronic device according to the embodiments of this document is not limited to the devices described above.

[0060] FIG. 2a illustrates an example of a perspective view of a wearable device according to one embodiment.

[0061] FIG. 2b illustrates an example of one or more hardware components disposed within a wearable device according to one embodiment.

[0062] According to one embodiment, the wearable device (103) may have the form of glasses that are wearable on a part of a user's body (e.g., head). The wearable device (103) of FIGS. 2a and 2b may be an example of the electronic device (101) of FIG. 1. The wearable device (103) may include a head-mounted display (HMD). For example, the housing of the wearable device (103) may include a flexible material such as rubber and / or silicone that has a shape that adheres to a part of the user's head (e.g., a part of the face covering both eyes). For example, the housing of the wearable device (103) may include one or more straps that can be twined around the user's head and / or one or more temples that are attachable to the ears of the head.

[0063] Referring to FIG. 2a, a wearable device (103) according to one embodiment may include at least one display (250) and a frame (200) supporting at least one display (250).

[0064] According to one embodiment, a wearable device (103) may be worn on a part of a user's body. The wearable device (103) may provide augmented reality (AR), virtual reality (VR), or mixed reality (MR) that combines augmented reality and virtual reality to a user wearing the wearable device (103). For example, the wearable device (103) may display a virtual reality image provided by at least one optical device (282, 284) of FIG. 2b on at least one display (250) in response to a specified gesture of the user obtained through a motion recognition camera (or motion tracking camera) (260-2, 260-3) of FIG. 2b.

[0065] According to one embodiment, at least one display (250) can provide visual information to a user. For example, at least one display (250) may include a transparent or translucent lens. At least one display (250) may include a first display (250-1) and / or a second display (250-2) spaced apart from the first display (250-1). For example, the first display (250-1) and the second display (250-2) may be positioned at locations corresponding to the user's left eye and right eye, respectively.

[0066] Referring to FIG. 2b, at least one display (250) may provide visual information transmitted from external light to a user through a lens included in at least one display (250) and other visual information distinct from said visual information. The lens may be formed based on at least one of a Fresnel lens, a pancake lens, or a multi-channel lens. For example, at least one display (250) may include a first surface (231) and a second surface (232) opposite to the first surface (231). A display area may be formed on the second surface (232) of at least one display (250). When a user wears the wearable device (103), external light may be transmitted to the user by being incident on the first surface (231) and transmitted through the second surface (232). As another example, at least one display (250) can display an augmented reality image combined with a virtual reality image provided by at least one optical device (282, 284) on a real image transmitted through external light in a display area formed on the second surface (232).

[0067] In one embodiment, at least one display (250) may include at least one waveguide (233, 234) that diffracts light emitted from at least one optical device (282, 284) and transmits it to a user. At least one waveguide (233, 234) may be formed based on at least one of glass, plastic, or polymer. A nano pattern may be formed on the exterior or at least a portion of the interior of at least one waveguide (233, 234). The nano pattern may be formed based on a polygonal and / or curved grating structure. Light incident on one end of at least one waveguide (233, 234) may be propagated to the other end of at least one waveguide (233, 234) by the nano pattern. At least one waveguide (233, 234) may include at least one diffractive element (e.g., DOE (diffractive optical element), HOE (holographic optical element)) and at least one reflective element (e.g., a reflective mirror). For example, at least one waveguide (233, 234) may be placed within a wearable device (103) to guide a screen displayed by at least one display (250) to the user's eye. For example, the screen may be transmitted to the user's eye based on total internal reflection (TIR) ​​occurring within at least one waveguide (233, 234).

[0068] A wearable device (103) can analyze an object included in a real-world image collected through a camera (260-4), combine a virtual object corresponding to an object among the analyzed objects that is the target of augmented reality provision, and display it on at least one display (250). The virtual object may include at least one of text and an image regarding various information related to the object included in the real-world image. The wearable device (103) can analyze the object based on a multi-camera such as a stereo camera. For the object analysis, the wearable device (103) can perform spatial recognition (e.g., SLAM (simultaneous localization and mapping)) using a multi-camera and / or time-of-flight (ToF). A user wearing the wearable device (103) can view the image displayed on at least one display (250).

[0069] According to one embodiment, the frame (200) may be formed as a physical structure that allows the wearable device (103) to be worn on the user's body. According to one embodiment, the frame (200) may be configured so that when the user wears the wearable device (103), the first display (250-1) and the second display (250-2) can be positioned corresponding to the user's left and right eyes. The frame (200) may support at least one display (250). For example, the frame (200) may support the first display (250-1) and the second display (250-2) so that they are positioned corresponding to the user's left and right eyes.

[0070] Referring to FIG. 2a, the frame (200) may include an area (220) in which at least a portion of the frame contacts a part of the user's body when the user wears the wearable device (103). For example, the area (220) of the frame (200) in contact with a part of the user's body may include an area in contact with a part of the user's nose, a part of the user's ear, and a part of the side of the user's face that the wearable device (103) contacts. According to one embodiment, the frame (200) may include a nose pad (210) that contacts a part of the user's body. When the wearable device (103) is worn by the user, the nose pad (210) may contact a part of the user's nose. The frame (200) may include a first temple (204) and a second temple (205) that contact a different part of the user's body distinct from the part of the user's body.

[0071] For example, the frame (200) may include a first rim (201) covering at least a portion of a first display (250-1), a second rim (202) covering at least a portion of a second display (250-2), a bridge (203) positioned between the first rim (201) and the second rim (202), a first pad (211) positioned along a portion of the edge of the first rim (201) from one end of the bridge (203), a second pad (212) positioned along a portion of the edge of the second rim (202) from the other end of the bridge (203), a first temple (204) extending from the first rim (201) and fixed to a portion of the wearer's ear, and a second temple (205) extending from the second rim (202) and fixed to a portion of the ear opposite to the first. The first pad (211) and the second pad (212) may come into contact with a part of the user's nose, and the first temple (204) and the second temple (205) may come into contact with a part of the user's face and a part of the ear. The temples (204, 205) may be rotatably connected to the rim through the hinge units (206, 207) of FIG. 2B. The first temple (204) may be rotatably connected to the first rim (201) through a first hinge unit (206) positioned between the first rim (201) and the first temple (204). The second temple (205) may be rotatably connected to the second rim (202) through a second hinge unit (207) positioned between the second rim (202) and the second temple (205). According to one embodiment, a wearable device (103) can identify an external object (e.g., a user's fingertip) touching the frame (200) and / or a gesture performed by said external object by using a touch sensor, a grip sensor, and / or a proximity sensor formed on at least a portion of the surface of the frame (200).

[0072] According to one embodiment, the wearable device (103) may include hardware that performs various functions (e.g., hardware to be described later based on the block diagram of FIG. 4). For example, the hardware may include a battery module (270), an antenna module (275), at least one optical device (282, 284), speakers (e.g., speakers (255-1, 255-2)), a microphone (e.g., microphones (265-1, 265-2, 265-3)), a light-emitting module (not shown), and / or a PCB (printed circuit board) (290) (e.g., a printed circuit board). The various hardware may be placed within a frame (200).

[0073] According to one embodiment, a microphone (e.g., microphones (265-1, 265-2, 265-3)) of a wearable device (103) is positioned on at least a portion of a frame (200) to acquire a sound signal. A first microphone (265-1) positioned on a bridge (203), a second microphone (265-2) positioned on a second rim (202), and a third microphone (265-3) positioned on a first rim (201) are shown in FIG. 2b, but the number and position of the microphones (265) are not limited to the embodiment of FIG. 2b. If there are two or more microphones (265) included in the wearable device (103), the wearable device (103) can identify the direction of the sound signal by using a plurality of microphones positioned on different portions of the frame (200).

[0074] According to one embodiment, at least one optical device (282, 284) may project a virtual object onto at least one display (250) to provide various image information to a user. For example, at least one optical device (282, 284) may be a projector. At least one optical device (282, 284) may be disposed adjacent to at least one display (250) or included within at least one display (250) as part of at least one display (250). According to one embodiment, a wearable device (103) may include a first optical device (282) corresponding to a first display (250-1) and a second optical device (284) corresponding to a second display (250-2). For example, at least one optical device (282, 284) may include a first optical device (282) positioned at the edge of a first display (250-1) and a second optical device (284) positioned at the edge of a second display (250-2). The first optical device (282) may transmit light to a first waveguide (233) positioned on the first display (250-1), and the second optical device (284) may transmit light to a second waveguide (234) positioned on the second display (250-2).

[0075] In one embodiment, the camera (260) may include a shooting camera (260-4), an eye tracking camera (ET CAM) (260-1), and / or a motion recognition camera (260-2, 206-3). The shooting camera (260-4), the eye tracking camera (260-1), and the motion recognition camera (260-2, 260-3) may be positioned at different locations on the frame (200) and may perform different functions. The eye tracking camera (260-1) may output data indicating the position of the eyes or the gaze of a user wearing the wearable device (103). For example, the wearable device (103) may detect the gaze from an image containing the user's pupils obtained through the eye tracking camera (260-1). A wearable device (103) can identify an object focused by a user (e.g., a real object, and / or a virtual object) by using the user's gaze obtained through an eye-tracking camera (260-1). Upon identifying the focused object, the wearable device (103) can perform a function for interaction between the user and the focused object (e.g., gaze interaction). The wearable device (103) can represent a portion corresponding to the eyes of an avatar representing the user in a virtual space by using the user's gaze obtained through the eye-tracking camera (260-1). The wearable device (103) can render an image (or screen) displayed on at least one display (250) based on the position of the user's eyes. For example, the visual quality of a first area related to the gaze within the image and the visual quality of a second area distinguished from the first area (e.g., resolution, brightness, saturation, grayscale, or PPI (pixels per inch)) may differ from each other.The wearable device (103) can acquire an image (or screen) having a visual quality of a first area that matches the user's gaze and a visual quality of a second area by using foveated rendering. For example, if the wearable device (103) supports an iris recognition function, user authentication can be performed based on iris information acquired using an eye-tracking camera (260-1). An example in which the eye-tracking camera (260-1) is positioned toward both of the user's eyes is shown in FIG. 2b, but the embodiment is not limited thereto, and the eye-tracking camera (260-1) may be positioned solely toward the user's left eye or right eye.

[0076] In one embodiment, the camera (260-4) can capture a real image or background to be matched with a virtual image in order to implement augmented reality or mixed reality content. The camera (260-4) can be used to acquire high-resolution images based on HR (high resolution) or PV (photo video). The camera (260-4) can capture an image of a specific object located at the position viewed by the user and provide the image to at least one display (250). The at least one display (250) can display a single image in which information regarding a real image or background including the image of the specific object acquired using the camera (260-4) and a virtual image provided through at least one optical device (282, 284) are superimposed. The wearable device (103) can compensate for depth information (e.g., the distance between the wearable device (103) and an external object acquired through a depth sensor) using the image acquired through the camera (260-4). The wearable device (103) can perform object recognition through an image acquired using a shooting camera (260-4). The wearable device (103) can perform a function of focusing on an object (or subject) within an image (e.g., auto focus) and / or an optical image stabilization (OIS) function (e.g., anti-shake function) using the shooting camera (260-4). The wearable device (103) can perform a pass-through function to superimpose an image acquired through the shooting camera (260-4) onto at least a portion of the screen while displaying a screen representing a virtual space on at least one display (250). The shooting camera (260-4) may be referred to as a high resolution (HR) camera or a photo-video (PV) camera.The shooting camera (260-4) may provide an autofocus (AF) function and an optical image stabilization (OIS) function. The shooting camera (260-4) may include a global shutter (GS) camera and / or a rolling shutter (RS) camera. In one embodiment, the shooting camera (260-4) may be placed on a bridge (203) positioned between the first rim (201) and the second rim (202).

[0077] The eye tracking camera (260-1) can achieve more realistic augmented reality by tracking the gaze of a user wearing the wearable device (103), thereby matching the user's gaze with visual information provided to at least one display (250). For example, when the user looks straight ahead, the wearable device (103) can naturally display environmental information related to the user's front on at least one display (250) at the location where the user is situated. The eye tracking camera (260-1) may be configured to capture an image of the user's pupil to determine the user's gaze. For example, the eye tracking camera (260-1) may receive a gaze detection light reflected from the user's pupil and track the user's gaze based on the position and movement of the received gaze detection light. In one embodiment, the eye tracking camera (260-1) may be positioned at locations corresponding to the user's left and right eyes. For example, the eye-tracking camera (260-1) may be positioned within the first rim (201) and / or the second rim (202) to face the direction in which the user wearing the wearable device (103) is located.

[0078] A motion recognition camera (260-2, 260-3) can provide a specific event to a screen provided on at least one display (250) by recognizing the movement of the user's entire body or part thereof, such as the user's torso, hands, or face. A motion recognition camera (260-2, 260-3) can recognize the user's gesture, acquire a signal corresponding to the gesture, and provide a display corresponding to the signal to at least one display (250). A processor can identify the signal corresponding to the gesture and, based on the identification, perform a designated function. A motion recognition camera (260-2, 260-3) can be used to perform spatial recognition functions using SLAM and / or depth maps for a 6-degrees-of-freedom pose (6 DOF pose). A processor can use the motion recognition camera (260-2, 260-3) to perform gesture recognition functions and / or object tracking functions. In one embodiment, a motion recognition camera (260-2, 260-3) may be positioned on the first rim (201) and / or the second rim (202). The motion recognition camera (260-2, 260-3) may include a global shutter (GS) camera (e.g., a global shutter (GS) camera) used for head tracking, hand tracking, and / or spatial recognition based on one of a 3-degree-of-freedom pose or a 6-degree-of-freedom pose. The GS camera may include two or more stereo cameras to track fine movements. As an example, the GS camera may be included in an eye-tracking camera (260-1) for tracking the user's gaze.

[0079] The camera (260) included in the wearable device (103) is not limited to the eye-tracking camera (260-1) and motion recognition camera (260-2, 260-3) described above. For example, the wearable device (103) can identify external objects included within the FoV by using a camera positioned toward the user's FoV. The identification of external objects by the wearable device (103) can be performed based on a sensor for identifying the distance between the wearable device (103) and the external object, such as a depth sensor and / or a time of flight (ToF) sensor. The camera (260) positioned toward the FoV may support an autofocus function and / or an optical image stabilization (OIS) function. For example, the wearable device (103) may include a camera (260) (e.g., a face tracking camera) positioned toward the face to obtain an image including the face of a user wearing the wearable device (103).

[0080] Although not illustrated, according to one embodiment, the wearable device (103) may further include a light source (e.g., LED) that emits light toward a subject (e.g., user's eye, face, and / or an object outside the FoV) being photographed using a camera (260). The light source may include an LED of infrared wavelength. The light source may be placed in at least one of the frame (200) and hinge units (206, 207).

[0081] According to one embodiment, the battery module (270) can supply power to the electronic components of the wearable device (103). In one embodiment, the battery module (270) may be placed within the first temple (204) and / or the second temple (205). For example, the battery module (270) may be a plurality of battery modules (270). The plurality of battery modules (270) may each be placed in the first temple (204) and the second temple (205). In one embodiment, the battery module (270) may be placed at the end of the first temple (204) and / or the second temple (205).

[0082] The antenna module (275) can transmit a signal or power to the outside of the wearable device (103) or receive a signal or power from the outside. In one embodiment, the antenna module (275) may be placed within the first temple (204) and / or the second temple (205). For example, the antenna module (275) may be placed close to one side of the first temple (204) and / or the second temple (205).

[0083] The speaker (255) can output an acoustic signal to the outside of the wearable device (103). The acoustic output module may be referred to as the speaker. In one embodiment, the speaker (255) may be placed within a first temple (204) and / or a second temple (205) to be positioned adjacent to the ear of a user wearing the wearable device (103). For example, the speaker (255) may include a second speaker (255-2) positioned adjacent to the user's left ear by being placed within the first temple (204), and a first speaker (255-1) positioned adjacent to the user's right ear by being placed within the second temple (205).

[0084] A light-emitting module (not shown) may include at least one light-emitting element. The light-emitting module may emit light of a color corresponding to a specific state or emit light with an action corresponding to a specific state in order to visually provide information regarding a specific state of the wearable device (103) to the user. For example, if the wearable device (103) requires charging, it may emit red light at a constant frequency. In one embodiment, the light-emitting module may be placed on the first rim (201) and / or the second rim (202).

[0085] Referring to FIG. 2b, according to one embodiment, a wearable device (103) may include a printed circuit board (PCB) (290). The PCB (290) may be included in at least one of a first temple (204) or a second temple (205). The PCB (290) may include an interposer disposed between at least two sub-PCBs. On the PCB (290), one or more hardware components included in the wearable device (103) (e.g., hardware components illustrated by different blocks in FIG. 4) may be disposed. The wearable device (103) may include a flexible PCB (FPCB) for interconnecting the hardware components.

[0086] According to one embodiment, a wearable device (103) may include at least one of a gyroscope sensor, a gravity sensor, and / or an acceleration sensor for detecting the posture of the wearable device (103) and / or the posture of a body part (e.g., head) of a user wearing the wearable device (103). Each of the gravity sensor and the acceleration sensor may measure gravitational acceleration and / or acceleration based on designated three-dimensional axes (e.g., x-axis, y-axis, and z-axis) that are perpendicular to each other. The gyroscope sensor may measure the angular velocity of each of the designated three-dimensional axes (e.g., x-axis, y-axis, and z-axis). At least one of the gravity sensor, the acceleration sensor, and the gyroscope sensor may be referred to as an inertial measurement unit (IMU). According to one embodiment, the wearable device (103) can identify a user's motion and / or gesture performed to execute or stop a specific function of the wearable device (103) based on an IMU.

[0087] FIGS. 3a and 3b illustrate an example of the appearance of a wearable device according to one embodiment.

[0088] The wearable device (103) of FIGS. 3a and 3b may be an example of the electronic device (101) of FIG. 1. According to one embodiment, an example of the appearance of a first surface (310) of the housing of the wearable device (103) is shown in FIG. 3a, and an example of the appearance of a second surface (320) opposite to the first surface (310) may be shown in FIG. 3b.

[0089] Referring to FIG. 3a, according to one embodiment, a first surface (310) of a wearable device (103) may have a shape that is attachable to a part of a user's body (e.g., the user's face). Although not illustrated, the wearable device (103) may further include a strap for fixing to a part of a user's body and / or one or more temples (e.g., a first temple (204) and / or a second temple (205) of FIG. 2a and FIG. 2b). A first display (250-1) for outputting an image to the left eye among the user's two eyes and a second display (250-2) for outputting an image to the right eye among the two eyes may be disposed on the first surface (310). The wearable device (103) may further include rubber or silicone packing formed on the first surface (310) to prevent interference by light different from light emitted from the first display (250-1) and the second display (250-2) (e.g., ambient light).

[0090] According to one embodiment, the wearable device (103) may include cameras (260-1) for photographing and / or tracking both eyes of a user adjacent to each of the first display (250-1) and the second display (250-2). The cameras (260-1) may be referenced to the eye-tracking camera (260-1) of FIG. 2B. According to one embodiment, the wearable device (103) may include cameras (260-5, 260-6) for photographing and / or recognizing the user's face. The cameras (260-5, 260-6) may be referenced to face tracking (FT) cameras. The wearable device (103) may control an avatar representing the user in a virtual space based on the motion of the user's face identified using the cameras (260-5, 260-6). For example, the wearable device (103) can change the texture and / or shape of a part of an avatar (e.g., a part of an avatar representing a human face) by using information obtained by cameras (260-5, 260-6) (e.g., FT cameras) and representing the facial expression of a user wearing the wearable device (103).

[0091] Referring to FIG. 3b, on a second surface (320) opposite to the first surface (310) of FIG. 3a, a camera (e.g., cameras (260-7, 260-8, 260-9, 260-10, 260-11, 260-12)), and / or a sensor (e.g., a depth sensor (330)) may be placed to acquire information related to the external environment of the wearable device (103). For example, cameras (260-7, 260-8, 260-9, 260-10) may be placed on the second surface (320) to recognize external objects. The cameras (260-7, 260-8, 260-9, 260-10) may be referenced to the motion recognition cameras (260-2, 260-3) of FIG. 2b.

[0092] For example, using cameras (260-11, 260-12), the wearable device (103) can acquire images and / or videos to be transmitted to each of the user's two eyes. Camera (260-11) may be placed on the second surface (320) of the wearable device (103) to acquire an image to be displayed through a second display (250-2) corresponding to the right eye among the two eyes. Camera (260-12) may be placed on the second surface (320) of the wearable device (103) to acquire an image to be displayed through a first display (250-1) corresponding to the left eye among the two eyes. As an example, the wearable device (103) can acquire a single screen using multiple images acquired through the cameras (260-11, 260-12). The cameras (260-11, 260-12) can be referenced to the shooting camera (260-4) of FIG. 2b.

[0093] According to one embodiment, the wearable device (103) may include a depth sensor (330) disposed on a second surface (320) to identify the distance between the wearable device (103) and an external object. Using the depth sensor (330), the wearable device (103) may obtain spatial information (e.g., a depth map) for at least a portion of the field of view (FoV) of a user wearing the wearable device (103). Although not illustrated, a microphone may be disposed on the second surface (320) of the wearable device (103) to obtain sound output from an external object. The number of microphones may be one or more, depending on the embodiment.

[0094] FIG. 4 illustrates an example of a block diagram of a wearable device according to one embodiment.

[0095] Referring to FIG. 4, a wearable device (103) according to one embodiment may include at least one of a processor (410), memory (415), display (420), camera (425), sensor (430), or communication circuit (435). The processor (410), memory (415), display (420), camera (425), sensor (430), and communication circuit (435) may be electrically and / or operably coupled with each other by an electronic component such as a communication bus (402). The type and / or number of hardware components included in the wearable device (103) are not limited to those shown in FIG. 4. For example, the wearable device (103) may include only some of the hardware components shown in FIG. 4. The elements within the memory (e.g., layers and / or modules) described below may be in a logically separated state. The elements within the memory (415) may be included in a hardware component distinct from the memory (415). An operation performed by the processor (410) using each of the elements within the memory (415) is one embodiment, and the processor (410) may perform a different operation different from the above operation through at least one of the elements within the memory (415).

[0096] A processor (410) of a wearable device (103) according to one embodiment may include a hardware component for processing data based on one or more instructions. The hardware component for processing data may include, for example, an arithmetic and logic unit (ALU), a field programmable gate array (FPGA), and / or a central processing unit (CPU). The number of processors (410) may be one or more. For example, the processor (410) may have the structure of a multi-core processor such as a dual core, a quad core, or a hexa core.

[0097] A memory (415) of a wearable device (103) according to one embodiment may include a hardware component for storing data and / or instructions that are input and / or output to a processor (410). The memory (415) may include, for example, volatile memory such as random-access memory (RAM) and / or non-volatile memory such as read-only memory (ROM). Volatile memory may include, for example, at least one of dynamic RAM (DRAM), static RAM (SRAM), cache RAM, and pseudo SRAM (PSRAM). Non-volatile memory may include, for example, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, hard disk, compact disk, and embedded multi-media card (eMMC).

[0098] In one embodiment, a display (420) of a wearable device (103) can output visualized information to a user of the wearable device (103). For example, the display (420) can be controlled by a processor (410) including a circuit such as a GPU (graphic processing unit) to output visualized information to a user. The display (420) may include a flat panel display (FPD) and / or electronic paper. The FPD may include a liquid crystal display (LCD), a plasma display panel (PDP), and / or one or more light emitting diodes (LEDs). The LED may include an organic LED (OLED).

[0099] In one embodiment, the camera (425) of the wearable device (103) may include one or more light sensors (e.g., a CCD (charged coupled device) sensor, a CMOS (complementary metal oxide semiconductor) sensor) that generate an electrical signal indicating the color and / or brightness of light. The plurality of light sensors included in the camera (425) may be arranged in the form of a two-dimensional grid (2 dimensional array). The camera (425) may acquire the electrical signals of each of the plurality of light sensors substantially simultaneously to generate two-dimensional frame data corresponding to the light reaching the light sensors of the two-dimensional grid. For example, photo data captured using the camera (425) may mean one (a) two-dimensional frame data acquired from the camera (425). For example, video data captured using the camera (425) may mean a sequence of multiple two-dimensional frame data acquired from the camera (425) along a frame rate. The camera (425) may further include a flash light for outputting light in the direction in which the camera (425) receives light.

[0100] According to one embodiment, the wearable device (103) may include a plurality of cameras arranged facing different directions as an example of a camera (425). Among the plurality of cameras, the first camera may be referred to as a motion recognition camera (e.g., the motion recognition camera of FIG. 2b (260-2, 260-3) or the cameras of FIG. 3b (260-7, 260-8, 260-9, 260-10)), the second camera may be referred to as an eye-tracking camera (e.g., the eye-tracking camera of FIG. 2b (260-1) or the camera of FIG. 3a (260-1)), and the third camera may be referred to as a shooting camera (e.g., the shooting camera of FIG. 2b (260-4) or the cameras of FIG. 3b (260-11, 260-12)). The wearable device (103) can identify the position, shape, and / or gesture of the hand using an image acquired using a first camera. The wearable device (103) can identify the direction of gaze of the user wearing the wearable device (103) using an image acquired using a second camera. For example, the direction in which the first camera is facing and the direction in which the second camera is facing may be opposite.

[0101] In one embodiment, a recognition camera (e.g., a first camera (260-2, 260-3)) may be used for 3DoF and 6DoF head tracking, hand detection and tracking, and spatial recognition. The first camera (260-2, 260-3) may be utilized for SLAM for 6DoF and for performing spatial recognition functions through depth imaging. Additionally, it may be utilized for performing user gesture recognition and object tracking functions.

[0102] In one embodiment, a gaze tracking camera (e.g., a second camera (260-1)) may be used to track the position and direction of the user's eyes. The wearable device (103) can render a 3D image displayed on a VST (video see through) according to the position of the eyes obtained through the second camera (260-1). The wearable device (103) can perform foveated rendering by using the second camera (260-1) to render only the display area corresponding to the user's gaze, that is, the user's gaze, in high resolution. Based on the iris information obtained by the second camera (260-1), the wearable device (103) can perform user authentication based on iris recognition and perform functions such as account login or payment.

[0103] In one embodiment, a camera for capturing images (e.g., a third camera (260-4)) may be used to preview the real world (the scene being captured) for video see-through (VST) operation. The camera for capturing images may be used to acquire depth information and may be used for object recognition through a 2D image obtained by capturing the screen.

[0104] According to one embodiment, a sensor (430) of a wearable device (103) can generate electrical information that can be processed by a processor (410) and / or memory (415) of the wearable device (103) from non-electronic information associated with the wearable device (103). The information may be referred to as sensor data. The sensor (430) may include a global positioning system (GPS) sensor for detecting the geographic location of the wearable device (103), an image sensor, an illuminance sensor and / or a time-of-flight (ToF) sensor, and an inertial measurement unit (IMU) for detecting physical motion of the wearable device (103).

[0105] In one embodiment, the communication circuit (435) of the wearable device (103) may include hardware components to support the transmission and / or reception of electrical signals between the wearable device (103) and an external electronic device. The communication circuit (435) may include, for example, at least one of a modem, an antenna, and an optic / electronic converter. The communication circuit (435) may support the transmission and / or reception of electrical signals based on various types of protocols such as Ethernet, LAN (local area network), WAN (wide area network), WiFi (wireless fidelity), Bluetooth, BLE (Bluetooth low energy), ZigBee, LTE (long term evolution), 5G NR (new radio) and / or 6G.

[0106] According to one embodiment, within the memory (415) of the wearable device (103), one or more instructions (or commands) representing operations and / or operations to be performed on data by the processor (410) of the wearable device (103) may be stored. A set of one or more instructions may be referred to as firmware, an operating system, a process, a routine, a sub-routine, and / or an application. For example, the wearable device (103) and / or the processor (410) may perform at least one of the operations of FIG. 6 or FIG. 7 when a set of a plurality of instructions distributed in the form of an operating system, firmware, a driver, and / or an application is executed. In the following, the statement that an application is installed in the wearable device (103) may mean that one or more instructions provided in the form of an application are stored in memory (415), and that the one or more applications are stored in an executable format (e.g., a file having an extension specified by the operating system of the wearable device (103)) by the processor (410). For example, the application may include a program and / or library related to a service provided to the user.

[0107] Referring to FIG. 4, programs installed on a wearable device (103) may be classified into any one of different layers based on the target, including an application layer (440), a framework layer (460), and / or a hardware abstraction layer (HAL) (490). For example, within the hardware abstraction layer (490), programs (e.g., modules, or drivers) designed to target the hardware of the wearable device (103) (e.g., a display (420), a camera (420), and / or a sensor (430)) may be classified. The framework layer (460) may be referred to as an XR framework layer in that it contains one or more programs for providing XR (extended reality) services. For example, FIG. 4 illustrates the layers separated within memory (415), but the layers may be logically separated. However, it is not limited thereto. According to an embodiment, the layers may be stored in a designated area within memory (415).

[0108] For example, within the framework layer (460), programs designed to target at least one of the hardware abstraction layer (490) and / or the application layer (440) (e.g., location tracker (481), spatial recognizer (482), gesture tracker (483), and / or eye tracker (484), face tracker (485)) may be classified. Programs classified into the framework layer (460) may provide an application programming interface (API) that is executable based on other programs.

[0109] For example, within the application layer (440), programs designed to target a user controlling a wearable device (103) may be classified. Examples of programs classified into the application layer (440) include an XR (extended reality) system UI (user interface) and / or an XR application (442), but embodiments are not limited thereto. For example, programs classified into the application layer (440) (e.g., software applications) may call an API (application programming interface) to cause the execution of functions supported by programs classified into the framework layer (460).

[0110] For example, the wearable device (103) may display one or more visual objects on the display (420) to perform interaction with a user for using a virtual space based on the execution of the XR system UI (441). A visual object may mean an object that can be deployed on the screen for the transmission of information and / or interaction, such as text, images, icons, videos, buttons, checkboxes, radio buttons, text boxes, sliders, and / or tables. A visual object may be referred to as a visual guide, a virtual object, a visual element, a UI element, a view object, and / or a view element. The wearable device (103) may provide the user with a service to control functions available in the virtual space based on the execution of the XR system UI (441).

[0111] Referring to FIG. 4, a lightweight renderer and / or XR plugin may be included within the XR system UI (441). For example, the XR system UI (441) may cause the execution of functions supported by the lightweight renderer and / or XR plugin included within the application layer (440).

[0112] For example, a wearable device (103) may acquire resources (e.g., APIs, system processes and / or libraries) used to define, create, and / or execute a rendering pipeline that is partially modified, based on the execution of a lightweight renderer. The lightweight renderer may be referred to as a lightweight render pipeline in terms of defining a rendering pipeline that is partially modified. The lightweight renderer may include a renderer built prior to the execution of a software application (e.g., a pre-built renderer). For example, the wearable device (103) may acquire resources (e.g., APIs, system processes and / or libraries) used to define, create, and / or execute the entire rendering pipeline based on the execution of an XR plugin. The XR plugin may be referred to as an open XR native client in terms of defining (or setting) the entire rendering pipeline.

[0113] For example, the wearable device (103) may display a screen representing at least a portion of a virtual space on the display (420) based on the execution of the XR application (442). The XR plugin included in the XR application (442) may be referenced in the XR plugin of the XR system UI (441). Descriptions of the XR plugin that overlap with descriptions of the XR plugin may be omitted. The wearable device (103) may cause the execution of a screen composition layer (461) based on the execution of the XR application (442).

[0114] According to one embodiment, the wearable device (103) can integrate various spatial elements into a virtual environment based on the execution of a spatialization layer (450). If the spatialization layer (450) is not an immersive application or object with 3D information, it can convert and display the application or object with binocular parallax through a space flinger (451). The space flinger (451) can display the object or application by taking depth information into account and may include a renderer for this purpose. According to one embodiment, the wearable device (103) can provide a virtual space service based on the execution of a screen composition layer (461). For example, the screen composition layer (461) may include a platform (e.g., an Android platform) to support the virtual space service. The wearable device (103) can display on the display the posture of a virtual object representing the user’s posture rendered using data obtained through the sensor (430) based on the execution of the screen composition layer (461). The screen composition layer (461) may be referred to as a composition presentation manager (CPM).

[0115] For example, the screen composition layer (461) may include a runtime service (462). In one example, the runtime service (462) may be referred to as an OpenXR runtime module. A wearable device (103) may be used to provide at least one of a user’s pose prediction function, frame timing function, and / or spatial input function through the wearable device (103) based on the execution of the runtime service (462). In one example, the wearable device (103) may be used to perform rendering for a virtual space service for the user based on the execution of the runtime service (462). For example, an application (e.g., Unity or OpenXR native application) may be implemented based on the execution of the runtime service (462).

[0116] For example, the screen composition layer (461) may include a pass-through library (463). The wearable device (103) may display another screen representing real space acquired through a camera (425) superimposed on at least a portion of the screen while displaying a screen representing virtual space on the display (420) based on the execution of the pass-through library (463).

[0117] For example, the screen composition layer (461) may include a renderer. The wearable device (101) can render a screen to be displayed on a display by compositing virtual layers (or virtual nodes) rendered based on sensor data (e.g., sensing data obtained through a camera (425) or sensor (430)) and pass-through layers (or pass-through nodes) obtained through a pass-through library (463) through the screen composition layer (461) using the renderer. The virtual layers may be referred to as virtual nodes and / or virtual surfaces. The wearable device (101) can render each of the virtual layers or render all of the virtual layers through the screen composition layer (461).

[0118] For example, the screen composition layer (461) may include a compositor (464). The compositor (464) can provide an XR environment to the user by compositing a virtual node rendered based on recognition / tracking data obtained through the input manager (465) with a pass-through node obtained through the pass-through library (463). The compositor (464) may include a renderer.

[0119] For example, the screen configuration layer (461) may include an input manager (465). Based on the execution of the input manager (465), the wearable device (103) may execute one or more programs included in the recognition service layer (480) to identify acquired data (e.g., sensor data). The wearable device (103) may use the acquired data to initiate the execution of at least one of the functions of the wearable device (103).

[0120] For example, the perception abstract layer (470) may be used for data exchange between the screen composition layer (461) and the perception service layer (480). In terms of being used for data exchange between the screen composition layer (461) and the perception service layer (480), the perception abstract layer (470) may be referred to as an interface. As an example, the perception abstract layer (470) may be referred to as OpenPX and / or PPAL (perception platform abstract layer). The perception abstract layer (470) may be used for a perception client and a perception service.

[0121] According to one embodiment, the recognition service layer (480) may include one or more programs for processing data obtained from a sensor (430) (or a camera (425)). The one or more programs may include at least one of a location tracker (481), a spatial recognizer (482), a gesture tracker (483), an eye tracker (484), and / or a face tracker (485). The type and / or number of the one or more programs included in the recognition service layer (480) are not limited to those shown in FIG. 4.

[0122] For example, the wearable device (103) can identify the posture of the wearable device (103) using the sensor (430) based on the operation of the position tracker (481). The wearable device (103) can identify the 6 degrees of freedom pose (6 DOF pose) of the wearable device (103) using data acquired using the camera (425) and the IMU based on the operation of the position tracker (481). The position tracker (481) may be referred to as a head tracking (HeT) module.

[0123] For example, the wearable device (103) may be used to construct the surrounding environment of the wearable device (103) (or the user of the wearable device (103)) into a three-dimensional virtual space based on the execution of the space recognizer (482). The wearable device (103) may reconstruct the surrounding environment of the wearable device (103) in three dimensions using data acquired through the camera (425) based on the execution of the space recognizer (482). The wearable device (103) may identify at least one of a plane, an incline, or a staircase based on the surrounding environment of the wearable device (103) reconstructed in three dimensions based on the execution of the space recognizer (482). The space recognizer (482) may be referred to as a scene understanding (SU) module.

[0124] For example, the wearable device (103) may be used to identify (or recognize) the pose and / or gesture of the user's hand of the wearable device (103) based on the execution of the gesture tracker (483). For example, the wearable device (103) may identify the pose and / or gesture of the user's hand using data acquired from the sensor (430) based on the execution of the gesture tracker (483). For example, the wearable device (103) may identify the pose and / or gesture of the user's hand based on data (or images) acquired using the camera (425) based on the execution of the gesture tracker (473). The gesture tracker (473) may be referred to as a hand tracking (HaT) module and / or a gesture tracking module.

[0125] For example, the wearable device (103) can identify (or track) the movement of the user's eyes of the wearable device (103) based on the execution of the eye tracker (484). For example, the wearable device (103) can identify the movement of the user's eyes using data obtained from at least one sensor based on the execution of the eye tracker (484). For example, the wearable device (103) can identify the movement of the user's eyes based on data obtained using a camera (425) (e.g., the eye tracking camera (260-1) of FIG. 2a and FIG. 2b) and / or an IR LED (infrared light emitting diode) based on the execution of the eye tracker (484). The eye tracker (484) may be referred to as an eye tracking (ET) module and / or a gaze tracking module.

[0126] For example, the recognition service layer (480) of the wearable device (103) may further include a face tracker (485) for tracking the user's face. For example, the wearable device (103) may identify (or track) the movement of the user's face and / or the user's facial expression based on the execution of the face tracker (485). The wearable device (103) may estimate the user's facial expression based on the movement of the user's face based on the execution of the face tracker (485). For example, the wearable device (103) may identify the movement of the user's face and / or the user's facial expression based on data (e.g., an image) acquired using a camera based on the execution of the face tracker (485).

[0127] FIG. 5a illustrates an example in which wearable devices share a real object and a background in a spaced-apart space, according to one embodiment of the present disclosure.

[0128] According to one embodiment, wearable devices (e.g., the wearable device (103) of FIG. 2a to 2b, FIG. 3a to 3b, and FIG. 4) may be in the form of a headset worn on the head or face. The wearable device in the form of a headset may include a display. In one embodiment, the first wearable device (103a) may be located in a first space (510), and the second wearable device (103b) may be located in a second space (520). The first space (510) and the second space (520) are physically separated spaces, and the first wearable device (103a) and the second wearable device (103b) may be worn by their respective users.

[0129] In one embodiment, when the first wearable device (103a) shares a real-world object within a scene acquired through a camera with the second wearable device (103b) in real time, the second wearable device (103b) can share a real-world background acquired through the camera with the first wearable device (103a). The second wearable device (103b) can display a video captured through the camera in real time on a display, and overlay a virtual object corresponding to the real-world object shared from the first wearable device (103a) on top of it. On the other hand, the first wearable electronic device (103a) can display a real-world object shared with the second wearable device (103b) in real time on a display among the video captured through the camera, and display a virtual space corresponding to the real-world background shared from the second wearable device (103b) in an area excluding the real-world object. As a result, the real-world objects of the first wearable device (103a) and the real-world backgrounds of the second wearable device (103b) can be shared in real time by the two wearable devices. The first wearable device (103a) displays real-world objects and a virtual space through a display, and the second wearable device (103b) displays virtual objects and a real-world background through a display.

[0130] In one embodiment, the first wearable device (103a) can use an AI model to acquire recommended objects suitable for a virtual space (real background of the second wearable device (103b)) by using scenes acquired through a camera by the movement of the first wearable device (103a) as input, and can share the recommended objects with the second wearable device (103b). In one embodiment, the second wearable device (103b) can use an AI model to acquire recommended locations for virtual objects (real objects of the first wearable device (103a)) shared within the real background, display virtual objects at the recommended locations, and share the recommended locations with the first wearable device (103a). The AI ​​model may be a model trained to provide interior layouts. For example, an AI model can be trained based on indoor space layout data, furniture and prop information, design trends, design case data, user preference data, 3D or AR model data, lighting or color data, and data reflecting user preferences obtained from social media or online platforms.

[0131] In one embodiment, when the second wearable device (103b) shares a real-time real-time background acquired through a camera with the first wearable device (103a) in real time, the first wearable device (103a) can select a recommended object that matches the shared real-time background within the scene acquired through the camera and share the recommended object with the second wearable device (103b) in real time.

[0132] An HMD device according to one embodiment of the present disclosure comprises at least one camera; a display; a communication circuit; and at least one processor including a processing circuit; and a memory including at least one storage medium for storing instructions, wherein the instructions, when executed individually or collectively by the at least one processor, may cause the HMD device to: receive first virtual object information or first virtual space information sharing from an external electronic device through the communication circuit; in response to receiving the first virtual object information, place the first virtual object in a first scene obtained through the at least one camera and output it through the display; and in response to receiving the first virtual space information, superimpose the first virtual space in a second scene obtained through the at least one camera and output it through the display.

[0133] According to one embodiment, in response to receiving the first virtual object information, three-dimensional background information for the first scene is generated, and the three-dimensional background information can be shared with the external electronic device through the communication circuit.

[0134] According to one embodiment, in response to receiving the first virtual space information, a first object to be placed in the first virtual space in the second scene is selected, and three-dimensional object information for the first object is generated and shared with the external electronic device through the communication circuit.

[0135] According to one embodiment, based on an image analysis AI model, an object suitable for the first virtual space among one or more objects included in the second scene can be selected as the first object.

[0136] FIG. 5b illustrates an example in which wearable devices share a real object and a background in a spaced-apart space using a pairing terminal, according to one embodiment of the present disclosure.

[0137] According to one embodiment, wearable devices (e.g., the wearable device (103) of FIG. 2a to 2b, FIG. 3a to 3b, and FIG. 4) may be in the form of glasses worn on the head or face and, unlike a headset, may not include a display.

[0138] The first wearable device (103a) can be paired with the first terminal (101a), and the second wearable device (103b) can be paired with the second terminal (101b). The first wearable device (103a) and the first terminal (101a) may be devices of the same user, and a pairing connection may be established in advance. The same may be true for the second wearable device (103b) and the second terminal (101b).

[0139] The wearable devices of FIG. 5b are located in a third space (530) and a fourth space (540) that are physically separated as in FIG. 5a, respectively, and each wearable device (103a, 103b) may be worn by a user.

[0140] In one embodiment, a first wearable device (103a) can share a selected real-world object within a scene acquired through a camera with a second wearable device (103b) in real time. The second wearable device (103b) can output the shared real-world object through an output device (e.g., a speaker) included in the second wearable device (103b) or through an output device (e.g., a display, a speaker) of a second terminal (101b) (e.g., a smartphone, tablet, TWS) paired with the second wearable device (103b). For example, analysis information regarding the shared real-world object can be output as voice through an output device (e.g., a speaker) included in the second wearable device (103b). Alternatively, the second wearable device (103b) can output the real-world object through the display of the second terminal (101b). At this time, the second wearable device (103b) can display a video captured through the camera of the second wearable device (103b) in real time on the display of the second terminal (101b), and can overlay a virtual object corresponding to a real object shared from the first wearable device (103a) on top of it. The second wearable device (103b) can share a real background captured through the camera with the first wearable device (103a).

[0141] The first wearable device (103a) can also output information about a shared real-world object through an output device (e.g., a speaker) of the first wearable device (103a) or an output device (e.g., a display, a speaker) of the first terminal (101a) paired with the first wearable device (103a). Additionally, the first wearable device (103a) can also output information analyzing a selected real-world object among images captured through the camera of the first wearable device (103a) along with the shared information. For example, the first wearable device (103a) can output the analysis information about the shared real-world background and the shared real-world object as voice information through the speaker of the first wearable device (103a). Alternatively, the first wearable device (103a) can superimpose a real-world object acquired through a camera onto a virtual space corresponding to the shared real-world background and output it through the display of the first terminal (101a).

[0142] The first wearable device (103a) can transmit information about an object or background acquired through a camera and shared with another device to the paired first terminal (101a), output it through the display of the first terminal (101a), and output it as voice information through the speaker of the first wearable device (103a). At the same time, the first wearable device (103a) can transmit information about an object or background shared from another device to the paired first terminal (101a), output it through the display of the first terminal (101a), and output it as voice information through the speaker of the first wearable device (103a). As a result, the first wearable device (103a) can analyze the shared information (background or object) and the shared information (object or background) and output it through the speaker, and at the same time, display the shared real object superimposed on a virtual space corresponding to the shared real background through the display of the paired first terminal (101a). The user of the first wearable device (103a) and the first terminal (101a) can receive information about an object or background shared with another electronic device in a spaced-away space through multiple devices. The same applies to the second wearable device (103b).

[0143] Each output device and output method in the first wearable device (103a) and the second wearable device (103b) can be pre-set by the user, and the output method or output device can be changed during use.

[0144] Although FIGS. 5a and FIGS. 5b have been described separately, in one embodiment, the first wearable device (103a) may be a headset-type wearable device and the second wearable device (103b) may be a glasses-type wearable device. The two wearable devices sharing a real object / background acquired through a camera in a separated space may be of the same type (e.g., AR-AR, VST-VST) or may be of different types (e.g., AR-VST, VR-AR). In various embodiments, each wearable device may use the output device of another paired device depending on the type of device.

[0145] FIG. 6 is a flowchart illustrating the operation of a wearable device sharing a real object with an external electronic device according to one embodiment of the present disclosure.

[0146] According to one embodiment, a wearable device (e.g., the electronic device (101) of FIG. 1, the wearable device (103) of FIG. 2a and 2b, or the first wearable device (103a) of FIG. 5) can share a real-world object acquired through a camera in real time with an external electronic device (e.g., the second wearable device (103b) of FIG. 5) and receive a real-world background from the external electronic device in real time and display it on a display in real time. In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0147] In operation 610, a wearable device (103) according to one embodiment may receive user input requesting sharing of a first object in a first scene acquired through a camera (e.g., a shooting camera (260-4) in FIG. 2b). The wearable device (103) may acquire a first scene corresponding to the user's field of view through the camera while being worn by a user. The wearable device (101) may receive user input requesting sharing of a first object in the first scene. The user input may be voice containing a user's intention to share a real object. For example, the wearable device (101) may detect a request to share a real object based on keywords. Alternatively, the wearable device (101) may understand the user's intention regarding the request to share a real object from the user's voice input based on a large language model (LM). The wearable device (101) can recognize a real object sharing request when it detects a trigger condition specified for the real object sharing function.

[0148] To identify a first object in a first scene, the wearable device (103) may further consider gaze information, gesture information, and separate controller control information at the time of receiving user voice input. For example, the wearable device (103) can identify a first object in a first scene based on user voice input and gaze information. The wearable device (103) can acquire user gaze information using a gaze tracking camera (e.g., the gaze tracking camera (260-1) of FIGS. 2a and 2b). The wearable device (101) can receive user voice input, for example, "I want to put that gray armchair in my house. Connect me with my wife," through a microphone (e.g., the microphone (265) of FIGS. 2a and 2b). The wearable device (103) can identify the object referred to by the user in the first scene from the user gaze information and user voice information using an image analysis AI model.

[0149] For example, the wearable device (103) can identify a first object in a first scene based on a user gesture. The wearable device (103) can identify the user gesture from an image obtained through a recognition camera (e.g., the motion recognition camera (260-2, 260-3) of FIG. 2b) and obtain gesture information. The wearable device (101) can identify an object referred to by the user in the first scene by using an image analysis AI model on user voice input and user gesture information obtained through a microphone (265).

[0150] For example, the wearable device (103) can detect a user selection for a first object in a first scene through an input device (e.g., a controller).

[0151] In operation 620, a wearable device (103) according to one embodiment can generate three-dimensional object information corresponding to a first object. The wearable device (103) can extract information about the first object from one or more images consecutive to a first scene acquired through a shooting camera (260-4). The information about the first object may include at least one of shape, material, color, size, location, or depth data. The wearable device (103) can generate 3D virtual object information based on the information about the first object. The wearable device (103) can acquire additional data about the first object and include it in the 3D virtual object information. For example, the additional data may include a product name, model name, manufacturer, product specifications, color attributes, material attributes, price, place of purchase, or sales link. The wearable device (103) can acquire additional data through a database or web search.

[0152] In operation 630, a wearable device (103) according to one embodiment may share three-dimensional object information with an external electronic device. In one embodiment, the external electronic device may be identified by user input requesting sharing of the first object. For example, in user input such as "I want to put that gray armchair in my house. Connect me with my wife," the electronic device designated as the user's 'wife' in the contacts may be identified as the external electronic device. Alternatively, if information about the external electronic device is missing from the user input, the wearable device (103) may induce user input to select a target device for transmission in response to the object sharing request. Alternatively, the wearable device (103) may allow information about the external electronic device to be shared to be set in advance for the object sharing function.

[0153] In operation 640, a wearable device (103) according to one embodiment may receive three-dimensional background information from an external electronic device. The three-dimensional background information may correspond to a real background obtained through a camera of the external electronic device. The three-dimensional background information may include information about a certain area and information about one or more objects included in the certain area. For example, the three-dimensional background information may include information about an area corresponding to the user's field of view obtained through a camera of the external electronic device in an indoor space (e.g., living room, kitchen) where a user wearing the external electronic device is located. The external electronic device may recognize the background through spatial recognition (e.g., SLAM). The external electronic device may generate three-dimensional background information based on the recognized information and share it with the wearable device (103).

[0154] In operation 650, a wearable device (103) according to one embodiment can generate a three-dimensional virtual space based on three-dimensional background information. For example, the wearable device (103) can render a three-dimensional virtual space from the three-dimensional background information using a screen composition layer (e.g., the screen composition layer (461) of FIG. 4).

[0155] In operation 660, a wearable device (103) according to one embodiment can display a three-dimensional virtual space in an area excluding a first object shared with an external electronic device in a first scene acquired through a camera, and output it through a display. The wearable device (103) can display the virtual space semi-transparently. The wearable device (103) can adjust the immerse level as transparency based on user input.

[0156] In operation 670, a wearable device (103) according to one embodiment can recommend a second object suitable for a three-dimensional virtual space from images acquired through a camera as the user moves. The wearable device (103) can segment real-time images and analyze surrounding objects corresponding to a category similar to the first object (e.g., furniture, sofa) in the segmentation results to determine whether they are suitable for the three-dimensional virtual space. The wearable device (103) can receive real-time images and analyze the scene through a spatial recognition device (e.g., the spatial recognition device (482) of FIG. 4). Based on the analysis results, the wearable device (103) can recognize objects included in the scene. The wearable device (103) can select an object to be placed in the three-dimensional virtual space from among the objects included in the scene. For example, the wearable device (103) can use an image analysis AI model to place an object in the space or select an object suitable for the space. The image analysis AI model may be trained to receive spatial information and object information as input, and to infer the result of harmoniously placing the object in the space (e.g., coordinates within the space, direction of placement of the object). Alternatively, the image analysis AI model may receive real-time images acquired through the camera of the wearable device (103) and a 3D virtual space received from an external electronic device as input, and select an object that fits the 3D virtual space from among the objects included in the real-time images. The image analysis AI model may be trained to receive real images containing multiple objects and a virtual space as input, and to select the optimal object that fits the virtual space from among the objects included in the real images. Alternatively, the image analysis AI model may be trained to receive real images, recommended object information, and a virtual space as input, select objects similar to the recommended object information from among the objects included in the real images, and select the optimal object that fits the virtual space from among the similar objects.For example, the wearable device (103) can use an image analysis AI model to select an optimal second object that fits into a three-dimensional virtual space from among the surrounding objects included in images acquired through a camera according to the user's movement.

[0157] The wearable device (103) outputs a message recommending a second object and can share the second object with an external electronic device sharing a three-dimensional virtual space according to the user's final decision. For example, the wearable device (103) can output a recommendation message such as, "There is a brown sofa on the left, and I think it would look good in the living room. Would you like to try placing it?" In response to the recommendation message, the wearable device (103) can receive a consent from the user input such as, "Yes, I will try placing it." In response to the consent user input, the wearable device (103) can generate and transmit three-dimensional object information regarding the second object to the external electronic device. That is, if the wearable device (103) selects an additional object to share, it can repeat operations 620 to 630.

[0158] The wearable device (103) can continue sharing a manually or automatically selected object with an external electronic device that is sharing a three-dimensional virtual space. The wearable device (103) can receive three-dimensional background information for the new space again if the space where the external electronic device is located changes as the user wearing the external electronic device moves. In this case, the wearable device (103) can repeat operations 640 to 670.

[0159] When the wearable device (103) receives a request to stop sharing via user input, it may stop sharing the object and background with an external electronic device and stop the communication connection.

[0160] A head-mounted display (HMD) device according to one embodiment of the present disclosure comprises at least one camera; a display; at least one processor including a communication circuit and a processing circuit; and a memory including at least one storage medium for storing instructions, wherein when the instructions are executed individually or collectively by the at least one processor, the HMD device may be caused to: receive a user input requesting sharing of a first object in a first scene obtained through the at least one camera; generate three-dimensional object information corresponding to the first object based on one or more images of the first scene; share the three-dimensional object information with an external electronic device through the communication circuit; receive three-dimensional background information from the external electronic device in response to sharing the three-dimensional object information through the communication circuit; generate a three-dimensional virtual space based on the three-dimensional background information; and display the three-dimensional virtual space in an area excluding the first object in the first scene and output it through the display.

[0161] According to one embodiment, the user input includes at least one of user voice, user gesture, or gaze information, and based on the user input, the first object within the first scene can be identified.

[0162] According to one embodiment, the external electronic device may be identified by the user input or identified as a device designated in advance corresponding to a shared function.

[0163] According to one embodiment, the three-dimensional virtual space is displayed semi-transparently by overlaying it on the first scene, and the transparency of the three-dimensional virtual space can be adjusted according to the degree of immersion by user input.

[0164] According to one embodiment, based on an image analysis AI model, a second object suitable for the three-dimensional virtual space can be selected from among one or more objects included in images acquired through at least one camera, and the second object can be placed in the three-dimensional virtual space.

[0165] According to one embodiment, the second object may be included in the same or similar category as the first object.

[0166] According to one embodiment, the second object can be placed in the three-dimensional virtual space by replacing the first object.

[0167] According to one embodiment, the three-dimensional object information further includes additional information about the first object, and the additional information may include data obtained through a search for the first object or data extracted from one or more images of the first scene.

[0168] FIG. 7 is a flowchart illustrating the operation of a wearable device receiving an external object from an external electronic device according to one embodiment of the present disclosure.

[0169] According to one embodiment, a wearable device (e.g., the electronic device (101) of FIG. 1 or the wearable device (103) of FIG. 2a and 2b, and the second wearable device (103b) of FIG. 5) may share a real-world object from an external electronic device (e.g., the first wearable device (103a) of FIG. 5) and display it in real-time through a display, and may share a real-world background acquired through a camera with the external electronic device. In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0170] In operation 710, a wearable device (103) according to one embodiment may receive three-dimensional object information sharing from an external electronic device. The wearable device (103) may not simply receive three-dimensional object information, but may share background information about the space where the wearable device (103) is located with the external electronic device by being requested to share three-dimensional object information. For example, the wearable device (103) may receive a message along with the three-dimensional object information, such as "I want to place that gray armchair in my house. Connect me with my wife." The wearable device (103) may identify the user's sharing intention (e.g., to place in my house) and share the background scene obtained through the camera of the wearable device (103) with the external electronic device.

[0171] In operation 720, a wearable device (103) according to one embodiment can generate three-dimensional background information from images of a user's field of view obtained through a camera. In response to receiving three-dimensional object information sharing from an external electronic device, the wearable device (103) can acquire real-time two-dimensional images of the field of view of a user wearing the wearable device (103) using a camera (e.g., cameras (206-11, 260-12) of FIG. 3B). The wearable device (103) can generate three-dimensional reconstruction data that reproduces the real-time surrounding space of the user from the two-dimensional images obtained from the cameras (260-11, 260-12). The wearable device (103) can recognize objects or objects through spatial recognition (e.g., SLAM) with respect to real-time scene images obtained from the cameras. The wearable device (103) can generate three-dimensional background information through spatial recognition of a real-time scene in response to the sharing of three-dimensional object information from an external electronic device. Since the wearable device (103) basically performs spatial recognition while the cameras for shooting are operating, it can generate three-dimensional background information immediately. When the wearable device (103) is performing real-time spatial recognition through SLAM, the 720 operation may be omitted.

[0172] In operation 730, a wearable device (103) according to one embodiment can share three-dimensional background information in real time with an external electronic device. When the wearable device (103) performs real-time spatial recognition through SLAM, it can immediately transmit three-dimensional background information to an external electronic device in response to the third object information sharing without operation 720. When the scene acquired through the camera changes according to the movement of the user wearing the wearable device (103), the wearable device (103) can continuously generate background information for the changing scene and continue to share it with the external electronic device. The wearable device (103) can stop sharing real-time background information when it receives a request to stop sharing via user input or receives a communication connection interruption from the external electronic device. While sharing an object or background with the external electronic device, the wearable device (103) can update the three-dimensional background information according to the change in the real-time scene acquired through the camera and continue to share it with the external electronic device.

[0173] In operation 740, a wearable device (103) according to one embodiment may display a first virtual object corresponding to three-dimensional object information by superimposing it on a real scene acquired through a camera. The wearable device (103) may generate the first virtual object based on the three-dimensional object information. The three-dimensional object information may include at least one of shape, material, color, size, position, or depth data. The wearable device (103) may display the first virtual object by superimposing it according to the conditions of the real scene. For example, the wearable device (103) may adjust the ratio of the first virtual object to match the size of the real scene, or adjust the color of the first virtual object by considering the position of the lighting. The wearable device (103) may position the first virtual object according to a default setting value. Alternatively, the wearable device (103) may recommend the optimal placement of the first virtual object by analyzing the real scene acquired through a camera via a spatial recognition device. For example, a wearable device (103) can place an object in space using an image analysis AI model. The image analysis AI model may be trained to receive spatial information and object information as input and to infer the result of harmoniously placing the object in the space (e.g., coordinates within the space, direction of placement of the object). The wearable device (103) can determine the location and placement direction of the first virtual object by using the image analysis AI model with a real scene acquired through a camera and the first virtual object as input. The wearable device (103) can display the first virtual object based on the determined location information. The wearable device (103) can share the location information of the first virtual object with an external electronic device. Based on the relative location information of the first virtual object and the space received, the external electronic device can reposition the location of the virtual space displayed on the external electronic device's display.

[0174] In operation 750, a wearable device (103) according to one embodiment can recommend the placement of a first virtual object within a real scene acquired through a camera. The wearable device (103) can determine the position and orientation of the first virtual object that best fits the images of the real scene using an image analysis AI model. The wearable device (103) outputs a message suggesting the recommended placement of the first virtual object and can change the position of the first virtual object according to the user's final decision. The wearable device (103) can share the recommended placement of the first virtual object with an external electronic device.

[0175] An HMD device according to one embodiment of the present disclosure comprises at least one camera; a display; at least one processor including a communication circuit and a processing circuit; and a memory including at least one storage medium for storing instructions, wherein the instructions, when executed individually or collectively by the at least one processor, may cause the HMD device to: receive three-dimensional object information sharing from an external electronic device through the communication circuit, generate three-dimensional background information from images acquired for a first scene through the at least one camera, share the three-dimensional background information with the external electronic device through the communication circuit, and place a first virtual object corresponding to the three-dimensional object information in the first scene and output it through the display.

[0176] According to one embodiment, spatial recognition of a user's field of view within a first scene is performed based on images acquired through at least one camera, and the three-dimensional background information may include one or more object information recognized in the field of view.

[0177] According to one embodiment, an empty area within the first scene can be identified, and the first virtual object can be placed within the empty area.

[0178] According to one embodiment, the placement of the first virtual object within the first scene can be determined based on an image analysis AI model.

[0179] According to one embodiment, in response to a change in the scene acquired through the at least one camera due to movement of the HMD device, three-dimensional background information according to the scene change can be updated, and the updated three-dimensional background information can be shared with the external electronic device.

[0180] According to one embodiment, additional information related to the three-dimensional object is further received from an external electronic device through the communication circuit, and additional information regarding the first virtual object is output through the display, and the additional information may include data obtained through a search for the first object or data extracted from one or more images of the first scene.

[0181] According to one embodiment, a UI for changing attributes of the first virtual object is output, user input for the UI is received, and attributes of the first virtual object are changed and displayed according to the user input.

[0182] According to one embodiment, in response to receiving a gesture input that changes the arrangement of the first virtual object, the first virtual object can be repositioned and displayed according to the gesture input.

[0183] FIG. 8a is an example of a wearable device sharing a real object according to one embodiment of the present disclosure.

[0184] According to one embodiment, wearable devices (e.g., the wearable device (103) of FIG. 4, the first wearable device (103a) and the second wearable device (103b) of FIG. 5) may be located in different spaces (801, 802).

[0185] For example, a first wearable device (e.g., the first wearable device (103a) of FIG. 5) may be located in a first space (801), and a second wearable device (e.g., the second wearable device (103a) of FIG. 5) may be located in a second space (802).

[0186] The first wearable device (103a) can acquire a scene (810) corresponding to a user's field of view (FOV) within a first space (801) by using the cameras of the first wearable device (103a) (e.g., the camera (260-4) of FIG. 2b or the cameras (260-11, 260-12) of FIG. 3b). The first wearable device (103a) can share a first object (811) within the first scene with a second wearable device (103b). The first wearable device (103a) can initiate sharing of a real object / real background with the second wearable device (103b) according to user input to share the first object (811) with the second wearable device (103b). The first wearable device (103a) can select a first object within a first scene (810) according to user input. The first wearable device (103a) can obtain various information from images obtained through cameras for capturing the first object (811) to share the first object (811) and can generate three-dimensional object information (831) for the first object (811). The first wearable device (103a) can transmit the three-dimensional object information (831) to the second wearable device (103b).

[0187] The second wearable device (103b) receives three-dimensional object information (831) regarding the first object (811) from the first wearable device (103a) and can create a first virtual object (821) based on the three-dimensional object information (831). The second wearable device (103b) can display the first virtual object (821) by superimposing it on a real scene (820) corresponding to the user's field of view within the second space (802) obtained using the cameras of the second wearable device (103b) (e.g., the camera (260-4) of FIG. 2b or the cameras (260-11, 260-12) of FIG. 3b).

[0188] FIG. 8b is an example of a wearable device sharing a real-world background according to one embodiment of the present disclosure.

[0189] According to one embodiment, the second wearable device (103b) may start sharing a real object / real background with the first wearable device (103a) in response to receiving three-dimensional object information (831) about the first object (811) from the first wearable device (103a).

[0190] The second wearable device (103b) can share a real scene (820) corresponding to the user's field of view of the second space (802), obtained using cameras for shooting (e.g., the camera (260-4) of FIG. 2b or the cameras (260-11, 260-12) of FIG. 3b), with the first wearable device (103a) as a real background. The second wearable device (103b) can obtain information about the real scene (820) using SLAM and generate three-dimensional background information (841). The second wearable device (103b) can transmit the three-dimensional background information (841) to the first wearable device (103a).

[0191] The first wearable device (103a) receives three-dimensional background information (841) and can display a virtual space (813) corresponding to the three-dimensional background information (841) by superimposing it on a first scene (810) of reality within the first space (801). The first wearable device (103a) can display the virtual space (813) in an area excluding the first object (811) shared with the second wearable device (103b) within the first scene (810). To distinguish and display the first scene (810) and the virtual space (813), a glasses-shaped display (103a) worn by a user was displayed.

[0192] FIGS. 8a and 8b show that the first wearable device (103a) first shares a real-world object with the second wearable device (103b), but in another embodiment, the wearable device may first share a real-world background with another wearable device.

[0193] FIG. 9a is an example of a wearable device sharing a real-world background according to one embodiment of the present disclosure.

[0194] According to one embodiment, a wearable device (e.g., the wearable device (103) of FIG. 4) can share a real background from an external electronic device.

[0195] In the first screen (910), the wearable device (103) can display the user's field of view (911) within the real space (910) acquired through a camera via a display (915). The wearable device (103) can receive a sharing of the real background from an external electronic device. The real background may be a real scene in which the external electronic device is located, acquired by the external electronic device through the external electronic device's camera. The real background may be received in the form of three-dimensional background information. The three-dimensional background information may include one or more objects within a certain area. For example, three-dimensional background information for a part of an indoor space may be received.

[0196] In the second screen (910), the wearable device (103) can create a virtual space (921) based on three-dimensional background information received from an external electronic device. The wearable device (103a) can display the virtual space by superimposing it on the first scene (910) obtained through a camera. The wearable device (103) can display the immerse level by adjusting the transparency. In FIG. 9a, the transparency of the virtual space (921) superimposed on the real space (920) is 0, illustrating that only the virtual space (921) is displayed on the display (915).

[0197] While sharing the real-world background of the external electronic device, the user is exposed to the real-world background of the external electronic device, but the wearable device (103) can use an image analysis AI model to recommend to the user that the object be placed in the virtual space (921) if it discovers an object suitable for the virtual space (921) in the real-world scene obtained through the camera.

[0198] FIG. 9b is an example of a wearable device receiving a real-world object according to one embodiment of the present disclosure.

[0199] According to one embodiment, a wearable device (e.g., the wearable device (103) of FIG. 4) can share a real object from an external electronic device.

[0200] In the third screen (930), the wearable device (103) can display the user's field of view (931) within the real space (930) obtained through the camera via the display (935). The wearable device (103) can receive a share of a real object from an external electronic device. The real object is included in the real scene where the external electronic device is located, through the camera of the external electronic device.

[0201] In the fourth screen (940), the wearable device (103) can generate a virtual object (904) based on three-dimensional object information received from an external electronic device. The wearable device (103) can display the virtual object by overlaying it on the fourth scene (940) acquired through the camera. The wearable device (103) can recommend the placement of the virtual object (904) within the real scene (940) acquired through the camera using an image analysis AI model.

[0202] FIG. 10 is an example of a wearable device recommending a shared object according to one embodiment of the present disclosure.

[0203] According to one embodiment, a wearable device (e.g., the wearable device (103) of FIG. 4) can select recommended objects for sharing in a real scene that changes through a camera as the user moves, while sharing real objects and a real background with an external electronic device.

[0204] In the first screen (1010), the wearable device (103) can display a real-world background shared by an external electronic device through a display (1015) by superimposing it on a real-world scene (1010) obtained through a camera. The wearable device (103) can select a recommended object (1011) that fits the real-world background in the real-world scene (1010) and suggest to the user that the recommended object (1011) be placed in the real-world background. The wearable device (103) can select a recommended object according to the characteristics of the virtual space using an image analysis AI model. For example, if the virtual space is an indoor space, the recommended object can be any one of the items that can be placed in the indoor space. For example, when the wearable device (103) recommends an electronic device, it can check the home IoT information of the external electronic device that shares the real-world background and select and recommend an electronic device that can be installed on the external electronic device.

[0205] When the wearable device (103) decides to place the recommended object (1011) in a real background by user input, it can share the recommended object (1011) with an external electronic device that shares the real background.

[0206] In the second screen (1020), the external electronic device can superimpose and display the recommended object (1021) shared from the wearable device (103) within the real scene (1020) obtained through the camera.

[0207] FIG. 11a is an example of manually adjusting the placement of a shared object of a wearable device according to one embodiment of the present disclosure.

[0208] According to one embodiment, a wearable device (e.g., the wearable device (103) of FIG. 4) can change the placement of an object shared from an external electronic device according to user input. For example, touch and drag input can change the position of the object.

[0209] In the first screen (1110), the wearable device (103) can display a first object shared from an external electronic device at a first location (1111). In the first screen (1110), the wearable device (103) can receive a first user input (1101) that touches the first object. In the second screen (1120), the wearable device (103) can receive a second user input (1102) that drags following the first user input (1101). In the third screen (1130), the wearable device (103) can move and display the first object at a second location (1131) where the second user input (1102) ends.

[0210] Depending on the type of gesture for the shared object, various layout changes such as moving, rotating, and resizing may be possible. For example, touch and turn input can rotate the object.

[0211] FIG. 11b is an example of automatically adjusting the placement of shared objects of a wearable device according to one embodiment of the present disclosure.

[0212] According to one embodiment, a wearable device (e.g., the wearable device (103) of FIG. 4) can change the arrangement of an object shared from an external electronic device using an image analysis AI model.

[0213] In the first screen (1110), similar to FIG. 11a, the wearable device (103) can display a first object shared from an external electronic device at a first location (1111). The wearable device (103) can select the optimal placement of the first object from images of a real scene (1110) obtained through a camera using an image analysis AI model. The wearable device (103) can recommend a new placement of the first object using an image analysis AI model. For example, the wearable device (103) can provide a message to the user recommending a new placement, such as, "I think the armchair would look better facing the sofa. Would you like to try placing it?"

[0214] In the fifth screen (1150), if the wearable device (103) receives user input accepting a new placement proposal by the AI ​​model, it can move the first object to the second position (1151) and change the placement to face another object in the real scene (1150).

[0215] In one embodiment, when the wearable device (103) accepts a recommendation for a new placement, it may share the new placement information with an external electronic device that shares a real object and a real background.

[0216] FIG. 12 is an example of a display screen that displays information of an object shared by a wearable device according to one embodiment of the present disclosure.

[0217] According to one embodiment, a wearable device (e.g., the wearable device (103) of FIG. 4) may display information about a shared object on a display screen. The information about the shared object may include a product name, model name, manufacturer, product specifications, color attributes, material attributes, price, place of purchase, or sales link.

[0218] In the first screen (1210), the wearable device (103) can display a shared object (1201) and information about the object (1211) together with a real scene acquired through a camera. The shared object (1201) is a virtual object and can be placed with its size or position adjusted to fit the real scene. The information about the object (1211) may include information necessary for the situation in which the object is shared. For example, if the object is shared as a shopping target, the information about the object may include product information, sales information, etc. If the object is shared at a tourist destination while traveling, the information about the object may include descriptions of the history, culture, art, and nature of the tourist resources.

[0219] FIG. 13 is an example of a display screen in which a wearable device changes and displays the attributes of an object received by a wearable device according to one embodiment of the present disclosure.

[0220] According to one embodiment, a wearable device (e.g., the wearable device (103) of FIG. 4) can change an object based on information about a shared object.

[0221] In the first screen (1310), the wearable device (103) may display the shared object (1311) superimposed on the real scene and display an attribute selection UI (1312) for the shared object (1311). The attribute selection UI (1312) may display the attributes displayed on the current object (1311) as selected.

[0222] In the second screen (1320), when the wearable device (103) receives user input (1301) selecting one of the attributes in the attribute selection UI (1321), it can display the shared object (1321) as the selected attribute.

[0223] The wearable device (103) may have different types of editable attributes depending on the characteristics of the shared object. The wearable device (103) can reflect and display attribute information on the shared object in real time according to user input.

[0224] FIG. 14 is an example of a wearable device editing a display screen displaying a real-world scene according to one embodiment of the present disclosure.

[0225] According to one embodiment, a wearable device (e.g., the wearable device (103) of FIG. 4) displays a real-world scene acquired through a camera on a display and can edit the displayed screen according to user input regarding the real-world scene displayed on the display.

[0226] In the first screen (1410), the wearable device (103) can display a real-world scene (1410) acquired through a camera on the display screen. The wearable device (103) can receive a first gesture input (1401) for a first object (1411) on the display screen. For example, the first gesture input (1401) can be matched to a function that deletes an object.

[0227] In the second screen (1420), the wearable device (103) can delete and display the first object (1411) on the display screen in response to receiving the first gesture input (1401).

[0228] In the third screen (1430), the wearable device (103) can display an object (1431) shared from an external electronic device by placing it in the area where the first object (1411) was deleted. The wearable device (103) can edit a display screen projecting a real-world scene to place the object shared from the external electronic device. The wearable device (103) can receive a user gesture input to remove the shared object (1431).

[0229] The embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as “coupled” or “connected” to another (e.g., 2nd) component, with or without the terms “functionally” or “communicationly,” it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0230] The term “module” as used in the embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0231] One embodiment of the present document may be implemented as software (e.g., program (140)) comprising one or more instructions stored in a storage medium (e.g., internal memory (136) or external memory (138)) readable by a machine (e.g., electronic device (101)). For example, a processor (e.g., processor (120)) of the machine (e.g., electronic device (101)) may call at least one of the one or more instructions stored in the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0232] According to one embodiment, the method according to the embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0233] According to one embodiment, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to one embodiment, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to one embodiment, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In a head-mounted display (HMD) device, At least one camera; display; Communication circuit; and At least one processor including a processing circuit; and The memory includes at least one storage medium for storing instructions, wherein the instructions, when executed individually or collectively by the at least one processor, cause the HMD device: Receiving user input requesting sharing of a first object within a first scene acquired through at least one camera, and Based on one or more images of the first scene, three-dimensional object information corresponding to the first object is generated, and Through the above communication circuit, the three-dimensional object information is shared with an external electronic device, and Through the communication circuit above, three-dimensional background information is received from the external electronic device in response to sharing the three-dimensional object information, and Generate a 3D virtual space based on the above 3D background information, and An HMD device that displays the three-dimensional virtual space in an area excluding the first object in the first scene above, causing it to be output through the display.

2. In Paragraph 1, The above user input includes at least one of user voice, user gesture, or gaze information, and When the above instructions are executed individually or collectively by the at least one processor, the HMD device: An HMD device that causes the identification of the first object within the first scene based on the above user input.

3. In Paragraph 1 or 2, The above external electronic device is an HMD device that causes the device to be identified by the user input or designated in advance in correspondence with a shared function.

4. In any one of paragraphs 1 through 3, When the above instructions are executed individually or collectively by the at least one processor, the HMD device: Displaying the three-dimensional virtual space semi-transparently by overlaying it on the first scene above, and An HMD device that causes the transparency of the above-mentioned three-dimensional virtual space to be adjusted according to the degree of immersion by user input.

5. In any one of paragraphs 1 through 4, When the above instructions are executed individually or collectively by the at least one processor, the HMD device: Based on an image analysis AI model, a second object suitable for the three-dimensional virtual space is selected from among one or more objects included in images acquired through at least one camera, and An HMD device that causes the second object to be placed in the above three-dimensional virtual space.

6. In any one of paragraphs 1 through 5, The above second object is an HMD device that causes to be included in the same or similar category as the above first object.

7. In any one of paragraphs 1 through 6, When the above instructions are executed individually or collectively by the at least one processor, the HMD device: An HMD device that causes the second object to be placed in the above three-dimensional virtual space by replacing the first object.

8. In any one of paragraphs 1 through 7, The above three-dimensional object information further includes additional information regarding the first object, and An HMD device that causes the above additional information to include data obtained through a search for the first object or data extracted from one or more images of the first scene.

9. In a head-mounted display (HMD) device, At least one camera; display; Communication circuit; and At least one processor including a processing circuit; and The memory includes at least one storage medium for storing instructions, wherein the instructions, when executed individually or collectively by the at least one processor, cause the HMD device: Through the above communication circuit, three-dimensional object information sharing is received from an external electronic device, and Three-dimensional background information is generated from images acquired for a first scene through at least one camera, and Through the above communication circuit, the three-dimensional background information is shared with the external electronic device, and An HMD device that causes a first virtual object corresponding to the three-dimensional object information to be placed in the first scene and output through the display.

10. In Paragraph 9, When the above instructions are executed individually or collectively by the at least one processor, the HMD device: Based on images acquired through the above at least one camera, spatial recognition of the user's field of view within the first scene is performed, and An HMD device that causes the above three-dimensional background information to include one or more object information recognized in the field of view.

11. In Paragraph 9 or 10, When the above instructions are executed individually or collectively by the at least one processor, the HMD device: Identify an empty area within the first scene above, and An HMD device that causes the first virtual object to be placed within the above empty area.

12. In any one of paragraphs 9 through 11, When the above instructions are executed individually or collectively by the at least one processor, the HMD device: An HMD device that causes to determine the placement of the first virtual object within the first scene based on an image analysis AI model.

13. In any one of paragraphs 9 through 12, When the above instructions are executed individually or collectively by the at least one processor, the HMD device: In response to a change in the scene acquired through the at least one camera due to movement of the HMD device, 3D background information according to the scene change is updated, and An HMD device that causes the above-mentioned updated three-dimensional background information to be shared with the above-mentioned external electronic device.

14. In any one of paragraphs 9 through 14, When the above instructions are executed individually or collectively by the at least one processor, the HMD device: Through the communication circuit above, additional information related to the three-dimensional object is further received from an external electronic device, and Output additional information about the first virtual object through the display, and An HMD device that causes the above additional information to include data obtained through a search for the first object or data extracted from one or more images of the first scene.

15. In a head-mounted display (HMD) device, At least one camera; display; Communication circuit; and At least one processor including a processing circuit; and The memory includes at least one storage medium for storing instructions, wherein the instructions, when executed individually or collectively by the at least one processor, cause the HMD device: Through the above communication circuit, first virtual object information or first virtual space information sharing is received from an external electronic device, and In response to receiving the first virtual object information, the first virtual object is placed in a first scene obtained through the at least one camera and output through the display, and An HMD device that, in response to receiving the first virtual space information, causes the first virtual space to be superimposed on a second scene obtained through the at least one camera and output through the display.

Images