Method and device for displaying target object on basis of source object

Abstract

According to the present invention, an electronic device comprises: a display; at least one processor including processing circuitry; and a memory including one or more storage media storing instructions. The instructions, when executed by the at least one processor, may cause the electronic device to: acquire a user input for selecting a portion of a source object disposed in a space; determine, on the basis of the selected portion of the source object, a target function among one or more candidate functions of the source object; acquire a target object corresponding to the target function; determine the location of the acquired target object within the space on the basis of the target function; and display the target object at the determined location.

Description

Method and device for displaying a target object based on a source object

[0001] A technique for displaying a target object based on a source object is disclosed below.

[0002] Recently, virtual reality, augmented reality, and mixed reality technologies utilizing computer graphics technology have been developing. In this context, virtual reality technology refers to a technology that uses a computer to construct a virtual space that does not exist in the real world and makes that virtual space feel like reality, while augmented reality or mixed reality technology refers to a technology that displays information generated by a computer overlaid on the real world—that is, a technology that enables real-time interaction with the user by combining the real world and the virtual world.

[0003] Among these, augmented reality and mixed reality technologies are being utilized by integrating them with technologies in various fields (e.g., broadcasting, medical, and gaming). Representative examples of augmented reality technology being applied in the broadcasting field include the weather map in front of a weathercaster changing naturally during a TV forecast, or the insertion of advertising images that are not actually present in the stadium into the screen during sports broadcasts to make them appear as if they are there.

[0004] The Metaverse is a representative service that provides augmented reality or mixed reality to users. The term Metaverse is a compound word formed from "Meta," meaning fabrication or abstraction, and "Universe," meaning the real world, referring to a three-dimensional virtual world. As a concept more advanced than the existing term "virtual reality environment," the Metaverse provides an augmented reality environment where the virtual worlds of the Web and the Internet are absorbed into the real world.

[0005] The information described above may be provided as related art for the purpose of aiding understanding of this document. None of the foregoing is to be claimed as prior art related to this document, nor is it to be used to determine prior art.

[0006] An electronic device comprises a memory including at least one processor including a display and a processing circuit, and one or more storage media for storing instructions. When the instructions are executed by the at least one processor, the electronic device may obtain a user input selecting a portion of a source object placed in a space, determine a target function based on the portion of the selected source object among at least one candidate function of the source object, obtain a target object corresponding to the target function, determine the location of the obtained target object within the space based on the target function, and display the target object at the determined location.

[0007] A method performed by an electronic device may include: acquiring user input selecting a portion of a source object placed in space; determining a target function based on the portion of the selected source object among at least one candidate function of the source object; acquiring a target object corresponding to the target function and determining the location of the acquired target object within the space based on the target function; and displaying the target object at the determined location.

[0008] FIG. 1 is a block diagram illustrating an exemplary configuration of an electronic device according to various embodiments.

[0009] FIG. 2 illustrates examples of optical see-through devices according to various embodiments.

[0010] FIG. 3 illustrates examples of optical systems relating to an eye-tracking camera, a transparent member, and a display according to various embodiments.

[0011] FIGS. 4a and FIGS. 4b are drawings showing examples of the front and rear of an electronic device according to various embodiments.

[0012] FIG. 5 illustrates examples of the construction of a space, input from a user within the space, and output to the user according to various embodiments.

[0013] FIG. 6 is a diagram illustrating an example of an operation in which an electronic device according to various embodiments provides space to a user.

[0014] FIG. 7 is a diagram illustrating an example of an operation to place a target object based on a source object according to various embodiments.

[0015] FIGS. 8a to 8d are drawings illustrating examples of user input for selecting an object according to various embodiments.

[0016] FIG. 9 is a flowchart illustrating an example of an operation in which an electronic device according to various embodiments displays a target object based on a source object.

[0017] FIG. 10 is a drawing illustrating a target object based on a source object according to various embodiments.

[0018] FIG. 11 is a diagram illustrating an example of an operation in which an electronic device creates a target object according to various embodiments.

[0019] FIG. 12 is a diagram illustrating an example of an operation in which an electronic device according to various embodiments determines the position of a target object.

[0020] FIG. 13 is a diagram illustrating an example of an operation in which an electronic device according to various embodiments determines the location of a target object based on another object.

[0021] FIG. 14 is a diagram illustrating an example of an operation in which an electronic device according to various embodiments determines one different object when there are multiple candidate input objects.

[0022] FIG. 15 is a drawing illustrating an example of an operation in which an electronic device according to various embodiments displays a target object for an output function.

[0023] Hereinafter, embodiments will be described in detail with reference to the attached drawings. In the description with reference to the attached drawings, identical components are given the same reference numeral regardless of the drawing number, and redundant descriptions thereof will be omitted.

[0024] FIG. 1 is a block diagram illustrating an exemplary configuration of an electronic device according to various embodiments.

[0025] FIG. 1 is a block diagram of an electronic device (101) in a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or may communicate with at least one of an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through a server (108).

[0026] According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display (160), audio module (170), sensor (176), interface (177), connection terminal (178), haptic module (179), camera (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor (176), camera (180), or antenna module (197)) may be integrated into a single component (e.g., display (160)).

[0027] The processor (120) may be implemented as one or more IC (integrated circuit (or circuitry)) chips and may perform various data processing operations. The processor (120) may include at least one electrical circuit and may process instructions (or programs (140), and / or data) stored in memory (130) individually or collectively in a distributed manner. The processor (120) may include a processor assembly comprising one or more processing circuits. The processor (120) may include any processing circuit that is operative to control the performance and operation of one or more components of the electronic device (101) (e.g., memory (130), display (160), camera (180), communication circuit (190), and / or sensor (176)).

[0028] The processor (120) can control at least one other component (e.g., hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., program (140)), for example, and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., sensor (176) or communication module (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., central processing unit or application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., graphics processing unit, neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use lower power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.

[0029] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display (160), sensor (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera (180) or communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0030] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor (176)). The data may include, for example, input data or output data for software (e.g., program (140)) and related instructions. The memory (130) may include volatile memory (132) or non-volatile memory (134).

[0031] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0032] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0033] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0034] A display (160) (e.g., a display) can visually provide information to an external (e.g., a user) outside of the electronic device (101). The display (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0035] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (101).

[0036] The sensor (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor. For example, the sensor (176) may include an inertial measurement unit (IMU).

[0037] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0038] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0039] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that can be perceived by the user through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0040] The camera (180) can capture still images and video. According to one embodiment, the camera (180) may include one or more lenses, one or more image sensors, one or more image signal processors, or one or more flashes.

[0041] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0042] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0043] A communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication circuits. The communication module (190) may include one or more communication processors (CP) that operate independently of a processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., LAN (local region network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct or IrDA (infrared data relation)) or a second network (199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips).The wireless communication module (192) can identify or authenticate the electronic device (101) within a communication network, such as a first network (198) or a second network (199), using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).

[0044] The wireless communication module (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), external electronic device (e.g., electronic device (104)), or network system (e.g., second network (199)). According to one embodiment, the wireless communication module (192) may support a Peak data rate (e.g., 20 Gbps or more) for eMBB realization, loss coverage (e.g., 164 dB or less) for mMTC realization, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for URLLC realization.

[0045] An antenna module (197) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197).

[0046] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0047] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0048] According to one embodiment, commands or data may be transmitted or received between an electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199).

[0049] Each of the external electronic devices (102, 104) and the server (108) may be the same or different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104) or the server (108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the request may perform at least part of the requested function or service, or additional functions or services related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result as is or additionally processed as at least part of the response to the request.

[0050] According to one embodiment, the electronic device (101) is an augmented reality device (e.g., the electronic device (201) of FIG. 2, the electronic device (301) of FIG. 3, the electronic device (401) of FIG. 4a and FIG. 4b), and can output a result of executing additional functions or services related to a space (e.g., virtual space) and space.

[0051] The electronic device (101) can acquire spatial information and object information. Spatial information is information about space and may include vertex coordinates, textures, or colors that define the space. Object information is information about an object in space and may include vertex coordinates, textures, or colors that define the appearance of the object. Although described in more detail later in FIG. 5, the object in space may include a physical object, a virtual object, or an avatar object.

[0052] For example, the electronic device (101) can acquire spatial information and object information while running an application. For example, the electronic device (101) can receive spatial information and object information from an external electronic device (e.g., electronic device (102), electronic device (104)) or a server (108).

[0053] The processor (120) may generate at least one of visual information, auditory information, or haptic information of a space and an object within the space. For example, as visual information, the processor (120) may generate rendering data (e.g., visual rendering data) that renders the appearance of the space (e.g., shape, size, color, or texture) and the appearance of an object located within the space (e.g., shape, size, color, or texture). Additionally, the processor (120) may generate rendering data that renders the interaction between objects within the space or changes based on at least one of user inputs to the object (e.g., changes in the object's appearance, sound generation, or haptic generation). The communication module (190) may establish communication between an electronic device (e.g., electronic device (101)) and another user's electronic device (e.g., electronic device (102)). The communication module (190) may transmit at least one of the aforementioned visual information, haptic information, or auditory information to another user's electronic device. For example, the communication module (190) can transmit rendering data.

[0054] When the electronic device (101) detects the user's movement through the sensor (176), the processor (120) of the electronic device (101) can correct the rendering data based on the movement information and output it to the display (160).

[0055] However, the electronic device (101) is not limited to generating and / or outputting rendering data from spatial information and object information, and at least a portion of the generation of rendering data may be performed by an external device of the electronic device (101) (e.g., electronic device (102), electronic device (104), server (108)). Below, examples of cases where additional functions or services related to space and space are provided by the server (108) are mainly described.

[0056] The server (108) can transmit the result of executing additional functions or services related to the space and space to the electronic device (101).

[0057] The server (108) may include a processor (181), a communication module (182), and a memory (183). The processor (181), the communication module (182), and the memory (183) may be configured similarly to the processor (120), the communication module (190), and the memory (130) of the electronic device (101). The processor (181) may provide interaction between users in a space and within a space by executing instructions stored in the memory (183).

[0058] The processor (181) can generate at least one of visual information, auditory information, or tactile information (or, also expressed as 'haptic information') of a space and an object within the space. The communication module (182) can establish communication with a user's first electronic device (e.g., electronic device (101)) and another user's second electronic device (e.g., electronic device (102)). The communication module (182) can transmit at least one of the aforementioned visual information, haptic information, or auditory information to the first electronic device and the second electronic device. For example, the communication module (182) can transmit rendering data.

[0059] The server (108) renders content data executed in the application and transmits it to the electronic device (101), and the electronic device (101) that receives the data can output the content data to the display (160). When the electronic device (101) detects the movement of a user through a sensor, the processor (120) of the electronic device (101) can correct the rendering data received from the external electronic device (102) based on the movement information and output it to the display (160). Alternatively, the rendering can be requested by transmitting the movement information to the server (108) so that the screen data is updated accordingly. However, it is not limited to this, and the aforementioned rendering can be performed by various types of external electronic devices (102, 104), such as a case device capable of storing and charging a smartphone or the electronic device (101). Rendering data corresponding to the aforementioned space generated by the external electronic device (102, 104) can be provided to the electronic device (101).

[0060] FIG. 2 illustrates examples of optical see-through devices according to various embodiments.

[0061] The electronic device (201) may include at least one of a display (e.g., the display (160) of FIG. 1), a vision sensor, a light source (230a, 230b), an optical element, or a substrate. According to one embodiment, the display of the electronic device (201) is transparent and may provide an image through the transparent display. According to one embodiment, the electronic device (201) may include a transparent member and a display connected to the transparent member. A user may look at an object placed in physical space (e.g., an object in the real world) through the transparent member. An electronic device (201) that allows light reflected from an object placed in physical space to pass through a transparent configuration (e.g., a transparent display, a transparent member separate from the display) and provides an image through the display may be referred to as an optical see-through device (OST device).

[0062] The display may include, for example, a liquid crystal display (LCD), a digital mirror device (DMD), a liquid crystal on silicon (LCoS), an organic light emitting diode (OLED), or a micro light emitting diode (micro LED).

[0063] In one embodiment, where the display is composed of a liquid crystal display, a digital mirror display, or a silicon liquid crystal display, the electronic device (201) may include a light source (230a, 230b) that irradiates light onto a screen output area of ​​the display (e.g., a screen display portion (215a, 215b)). In another embodiment, where the display can generate light on its own, for example, where it is composed of an organic light-emitting diode or a micro LED, the electronic device (201) may provide a virtual image of good quality to the user without including a separate light source (230a, 230b). In one embodiment, if the display is implemented as an organic light-emitting diode or a micro LED, the light source (230a, 230b) is unnecessary, so the electronic device (201) can be made lighter.

[0064] Referring to FIG. 2, the electronic device (201) may include a display, a first transparent member (225a) and / or a second transparent member (225b), and the user may use the electronic device (201) while wearing it on their face. The first transparent member (225a) and / or the second transparent member (225b) may be formed from a glass plate, a plastic plate, or a polymer, and may be made transparent or translucent. According to one embodiment, the first transparent member (225a) may be positioned facing the user's right eye, and the second transparent member (225b) may be positioned facing the user's left eye. The display may include a first display (205) that outputs a first image (e.g., right image) corresponding to the first transparent member (225a) and a second display (210) that outputs a second image (e.g., left image) corresponding to the second transparent member (225b). According to one embodiment, when each display is transparent, each display and the transparent member may be positioned to face the user's eyes to form a screen display unit (215a, 215b).

[0065] In one embodiment, light emitted from a display (205, 210) may be guided along a light path to a waveguide through an input optical member (220a, 220b). Light traveling inside the waveguide may be guided toward the user's eye through an output optical member (e.g., output optical member (340) of FIG. 3). Screen displays (215a, 215b) may be determined based on the light emitted toward the user's eye.

[0066] For example, light emitted from the display (205, 210) can be reflected by the grating region of the waveguide formed in the input optical member (220a, 220b) and the screen display portions (215a, 215b) and transmitted to the user's eye.

[0067] The optical element may include at least one of a lens or an optical waveguide.

[0068] The lens can adjust the focus so that the screen output to the display can be seen by the user's eyes. The lens may include, for example, at least one of a Fresnel lens, a pancake lens, or a multichannel lens.

[0069] An optical waveguide can transmit image rays generated from a display to the user's eye. For example, the image rays may represent light emitted by a light source (230a, 230b) passing through a screen output area of ​​the display. The optical waveguide may be made of glass, plastic, or polymer. The optical waveguide may include a nano-pattern formed on some internal or external surface, for example, a grating structure of a polygonal or curved shape. An exemplary structure of the optical waveguide is described later in FIG. 3 below.

[0070] The vision sensor may include at least one of a camera sensor or a depth sensor.

[0071] The first camera (265a, 265b) is a recognition camera and may be used for 3DoF and 6DoF head tracking, hand detection, hand tracking, and spatial recognition. The first camera (265a, 265b) may primarily include a GS (global shutter) camera. Since stereo cameras are required for head tracking and spatial recognition, the first camera (265a, 265b) may include two or more GS cameras. A GS camera may have superior performance compared to a RS (rolling shutter) camera in terms of detecting fast hand movements and fine movements such as fingers, and tracking movements. For example, a GS camera may have low image blur. The first camera (265a, 265b) can capture image data used for 6DoF spatial recognition and SLAM functions through depth capture. In addition, a user gesture recognition function can be performed based on image data captured by the first camera (265a, 265b).

[0072] The second camera (270a, 270b) is an ET (eye tracking) camera and can be used to capture image data for detecting and tracking the user's pupils. The second camera (270a, 270b) is described later in FIG. 3 below.

[0073] The third camera (245) may be a camera for shooting. The third camera (245) may include a high-resolution camera for capturing images of HR (high resolution) or PV (photo video). The third camera (245) may include a color camera equipped with functions for obtaining high-quality images, such as AF function and optical image stabilization (OIS). The third camera (245) may be a GS camera or an RS camera.

[0074] The fourth camera (e.g., the face recognition camera (325, 326) of FIGS. 4a and 4b below) is a face recognition camera, and the FT (face tracking) camera can be used to detect and track the user's facial expressions.

[0075] A depth sensor (not shown) may represent a sensor that senses information for determining the distance to an object, such as Time of Flight (TOF). TOF is a technology that measures the distance to an object using a signal (e.g., near-infrared, ultrasound, or laser). A depth sensor based on TOF technology emits a signal from a transmitter and measures the signal at a receiver, and can measure the flight time of the signal.

[0076] A light source (230a, 230b) (e.g., an illumination module) may include a device (e.g., a light emitting diode) that emits light of various wavelengths. The illumination module may be attached in various locations depending on the application. In one use case, a first illumination module (e.g., an LED device) attached around the frame of an augmented reality glasses device may emit light to assist in gaze detection when tracking eye movements with an ET camera. The first illumination module may, for example, include an IR LED of infrared wavelength. In another use case, a second illumination module (e.g., an LED device) may be attached adjacent to a camera mounted around a hinge (240a, 240b) connecting the frame and the temple, or around a bridge connecting the frame. The second illumination module may emit light to supplement ambient brightness when the camera is taking pictures. If subject detection is not easy in a dark environment, the second illumination module may emit light.

[0077] A substrate (235a, 235b) (e.g., a printed circuit board (PCB)) can support the aforementioned components.

[0078] A printed circuit board (PCB) may be placed on the temple of the glasses. The FPCB can transmit electrical signals to each module (e.g., camera, display, audio module, sensor) and other printed circuit boards. According to one embodiment, at least one printed circuit board may be in the form of a first board, a second board, and an interposer disposed between the first board and the second board. Electrical signals may be transmitted to each module and other printed circuit boards.

[0079] Other components may include at least one of a plurality of microphones (e.g., a first microphone (250a), a second microphone (250b), a third microphone (250c)), a plurality of speakers (e.g., a first speaker (255a), a second speaker (255b)), a battery (260), an antenna, or a sensor (e.g., an accelerometer, a gyroscope, or a touch sensor).

[0080] FIG. 3 illustrates examples of optical systems relating to an eye-tracking camera, a transparent member, and a display according to various embodiments.

[0081] FIG. 3 is a diagram illustrating the operation of an eye-tracking camera included in an electronic device according to one embodiment. Referring to FIG. 3, the process of tracking a user's eye (309), that is, the user's gaze, using light (e.g., infrared light) output from a display (320) (e.g., the first eye-tracking camera (270a) and the second eye-tracking camera (270b) of FIG. 2) of an electronic device (301) according to one embodiment is illustrated.

[0082] The second camera (e.g., the second camera (270a, 270b) of FIG. 2) may be an eye-tracking camera (310) that collects information to position the center of the virtual image projected onto the electronic device (301) according to the direction in which the wearer's eye of the electronic device (301) gazes. The second camera may also include a GS camera to detect the pupil and track rapid eye movements. An ET camera may also be installed for the left eye and the right eye, respectively, and the same camera performance and specifications may be used for each. The eye-tracking camera (310) may include a gaze tracking sensor (315). The gaze tracking sensor (315) may be included inside the eye-tracking camera (310). Infrared light output from the display (320) may be transmitted to the user's eye (309) as infrared reflected light (303) by a half mirror. The eye tracking sensor (315) can detect infrared transmitted light (305) reflected from the user's eye (309) by infrared reflected light (303). The eye tracking camera (310) can track the user's eye (309), that is, the user's gaze, based on the detection result of the eye tracking sensor (315).

[0083] The display (320) may include a plurality of visible light pixels and a plurality of infrared pixels. The visible light pixels may include R, G, and B pixels. The visible light pixels may output visible light corresponding to a virtual object image. The infrared pixels may output infrared light. The display (320) may include, for example, micro light emitting diodes (micro LEDs) or organic light emitting diodes (OLEDs).

[0084] The display optical waveguide (350) and the eye-tracking camera optical waveguide (360) may be contained within a transparent member (370) (e.g., the first transparent member (225a) and the second transparent member (225b) of FIG. 2). The transparent member (370) may be formed from a glass plate, a plastic plate, or a polymer, and may be made transparent or translucent. The transparent member (370) may be positioned facing the user's eye. At this time, the distance between the transparent member (370) and the user's eye (309) may be referred to as the 'eye relief' (380).

[0085] The transparent member (370) may include optical waveguides (350, 360). The transparent member (370) may include an input optical member (330) and an output optical member (340). Additionally, the transparent member (370) may include an eye-tracking splitter (375) that splits the input light into multiple waveguides.

[0086] According to one embodiment, light incident on one end of the display optical waveguide (350) can be propagated within the display optical waveguide (350) by a nano pattern and provided to the user. Additionally, the display optical waveguide (350), composed of a free-form prism, can provide image light to the user through a reflective mirror after the incident light. The display optical waveguide (350) may include at least one diffractive element (e.g., DOE (diffractive optical element), HOE (holographic optical element)) or a reflective element (e.g., a reflective mirror). The display optical waveguide (350) can guide display light (e.g., image light) emitted from a light source to the user's eye using at least one diffractive element or reflective element included in the display optical waveguide (350). For reference, in FIG. 3, the output optical member (340) is depicted as being separated from the eye-tracking optical waveguide (360), but the output optical member (340) may be included inside the eye-tracking optical waveguide (360).

[0087] According to various embodiments, the diffraction element may include an input optical member (330) and an output optical member (340). For example, the input optical member (330) may refer to an input grating region. The output optical member (340) may refer to an output grating region. The input grating region may serve as an input terminal that diffracts (or reflects) light output from (e.g., Micro LED) to transmit light to a transparent member (e.g., first transparent member, second transparent member) of a screen display. The output grating region may serve as an output terminal that diffracts (or reflects) light transmitted to a transparent member (e.g., first transparent member, second transparent member) of a waveguide to the user's eye.

[0088] According to various embodiments, the reflection element may include a total internal reflection optical element or a total internal reflection waveguide for total internal reflection (TIR). For example, total internal reflection is a method of inducing light, which may mean creating an angle of incidence such that light (e.g., a virtual image) entering through an input grating area is 100% reflected from one side (e.g., a specific side) of the waveguide and is transmitted 100% to an output grating area.

[0089] In one embodiment, light emitted from the display (320) may be guided along a light path to a waveguide through an input optical member (330). Light traveling inside the waveguide may be guided toward the user's eye through an output optical member (340). The screen display may be determined based on the light emitted toward the eye.

[0090] FIGS. 4a and 4b are drawings illustrating examples of the front and rear views of an electronic device according to various embodiments. FIG. 4a may be the external view of the electronic device (401) viewed from a first direction (①), and FIG. 4b may be the external view of the electronic device (401) viewed from a second direction (②). When a user wears the electronic device (401), the external view seen by the user's eyes may be FIG. 4b.

[0091] Referring to FIG. 4a, according to various embodiments, an electronic device (401) (e.g., the electronic device (101) of FIG. 1, the electronic device (201) of FIG. 2, and the electronic device (301) of FIG. 3) may provide a service that provides an extended reality (XR) experience to a user. For example, XR or XR service may be defined as a service that collectively refers to virtual reality (VR), augmented reality (AR), and / or mixed reality (MR).

[0092] According to one embodiment, the electronic device (401) may refer to a head-mounted device or a head-mounted display worn on the user's head, but may be configured in the form of at least one of glasses, goggles, a helmet, or a hat. The electronic device (401) may include an OST (optical see-through) type configured to allow external light to reach the user's eyes through the glass when worn, or a VST (video see-through) type configured to block external light so that light emitted from the display reaches the user's eyes when worn, but external light does not reach the user's eyes.

[0093] According to one embodiment, an electronic device (401) may be worn on the head of a user to provide the user with images related to an extended reality (XR) service. For example, the electronic device (401) may provide XR content (hereinafter referred to as XR content images) that outputs at least one virtual object superimposed on an area determined to be a display area or the user's field of view (FoV). According to one embodiment, XR content may refer to images related to real space acquired through a camera (e.g., a camera for shooting) or images or videos that appear to have at least one virtual object superimposed on a virtual space. According to one embodiment, the electronic device (401) may provide XR content based on a function being performed on the electronic device (401) and / or a function being performed on one or more external electronic devices (e.g., the electronic devices (102, 104) of FIG. 1, the server (108) of FIG. 1).

[0094] According to one embodiment, the electronic device (401) is at least partially controlled by an external electronic device (e.g., the electronic device (102, 104) of FIG. 1), and at least one function may be performed under the control of the external electronic device, but at least one function may also be performed independently.

[0095] Referring to FIG. 4a, a vision sensor may be disposed on a first surface of the housing of the main body (410) of the electronic device (401). The vision sensor may include cameras (e.g., cameras for a second function (411, 412), cameras for a first function (415)) and / or a depth sensor (417) for acquiring information related to the surrounding environment of the electronic device (401).

[0096] In one embodiment, the second function cameras (411, 412) can acquire images related to the surrounding environment of the electronic device (401). The first function cameras (415) can acquire images while the wearable electronic device is worn by a user. The first function cameras (415) can be used for hand detection, tracking, and user gesture (e.g., hand movements) recognition. The first function cameras (415) can be used for 3DoF, 6DoF head tracking, location (space, environment) recognition, and / or movement recognition. In one embodiment, the second function cameras (411, 412) may be used for hand detection and tracking, and user gestures.

[0097] In one embodiment, the depth sensor (417) may be configured to transmit a signal and receive a signal reflected from the subject, and may be used for determining the distance to the object, such as time of flight (TOF). Instead of or in addition to the depth sensor (417), cameras (411, 412, 415, 416) may determine the distance to the object.

[0098] Referring to FIG. 4b, a face recognition camera (425, 426) and / or a display (421) (and / or a lens) may be disposed on the second surface (420) of the main body (410) housing.

[0099] In one embodiment, a face recognition camera (425, 426) adjacent to the display may be used to recognize the user's face or to recognize and / or track both of the user's eyes.

[0100] In one embodiment, the display (421) (and / or lens) may be disposed on a second surface (420) of the electronic device (401). In one embodiment, the electronic device (401) may not include some of the plurality of cameras (415). Although not illustrated in FIGS. 4a and 4b, the electronic device (401) may further include at least one of the configurations illustrated in FIG. 2.

[0101] According to one embodiment, the electronic device (401) may include a main body (410) that implements at least some of the components of FIG. 1, a display (421) (e.g., the display (160) of FIG. 1) disposed in a first direction (①) of the main body (410), a first function camera (e.g., a recognition camera) (415) disposed in a second direction (②) of the main body (410), a second function camera (e.g., a shooting camera) (411, 412) disposed in a second direction (②), a third function camera (e.g., a gaze tracking camera) (428) disposed in a first direction (①), a fourth function camera (e.g., a face recognition camera) (425, 426) disposed in a first direction (①), a depth sensor (417) disposed in a second direction (②), and a touch sensor (413) disposed in a second direction (②). Although not shown in the drawing, the main body (410) may include a memory (e.g., memory (130) of FIG. 1) and a processor (e.g., processor (120) of FIG. 1) inside, and may further include other components shown in FIG. 1.

[0102] According to one embodiment, the display (421) may include a liquid crystal display (LCD), a digital mirror device (DMD), a liquid crystal on silicon (LCoS), an organic light emitting diode (OLED), or a micro light emitting diode (micro LED).

[0103] In one embodiment, if the display (421) is one of a liquid crystal display, a digital mirror display, or a silicon liquid crystal display, the electronic device (401) may include a light source that irradiates light onto the screen output area of ​​the display (421). In another embodiment, if the display (421) can generate light itself, for example, if the electronic device (401) is one of an organic light-emitting diode or a micro LED, the electronic device (401) can provide a user with high-quality XR content images without including a separate light source. In one embodiment, if the display (421) is implemented as an organic light-emitting diode or a micro LED, a light source is unnecessary, so the electronic device (401) can be made lighter.

[0104] According to one embodiment, the electronic device (401) may include a plurality of cameras. For example, the cameras may include a first functional camera (e.g., a recognition camera) (415) positioned in the second direction (②) of the main body (410), a second functional camera (e.g., a shooting camera) (411, 412) positioned in the second direction (②), a third functional camera (e.g., a gaze tracking camera) (428) positioned in the first direction (①) and / or a fourth functional camera (e.g., a face recognition camera) (425, 426) positioned in the first direction (①), but may further include cameras of other functions not illustrated.

[0105] The first functional camera (e.g., recognition camera) (415) may be used for detecting user movement or user gesture recognition functions. The first functional camera (415) may support at least one of head tracking, hand detection and hand tracking, and spatial recognition. For example, the first functional camera (415) may primarily use a GS (global shutter) camera, which has superior performance compared to an RS (rolling shutter) camera, to detect hand movements and fine finger movements and to track movements, and may be composed of a stereo camera including two or more GS cameras for head tracking and spatial recognition. The first functional camera (415) may perform SLAM (simultaneous localization and mapping) functions to recognize information related to the surrounding space (e.g., location and / or orientation) through spatial recognition for 6DoF and depth capture.

[0106] The second function camera (e.g., a camera for shooting) (411, 412) can be used to capture the outside and generate an image or video corresponding to the outside and transmit it to a processor (e.g., the processor (120) of FIG. 1). The processor can display the image provided by the second function camera (411, 412) on a display (421). The second function camera (411, 412) may be referred to as HR (high resolution) or PV (photo video) and may include a high-resolution camera. For example, the second function camera (411, 412) may include a color camera equipped with functions for obtaining high-quality images, such as AF (auto focus) and shake correction (OIS (optical image stabilizer)), but is not limited thereto, and the second function camera (411, 412) may also include a GS camera or an RS camera.

[0107] A third-function camera (e.g., eye-tracking camera) (428) may be placed in the display (421) (or inside the main body) such that the camera lens faces the user's eyes when the user is equipped with the electronic device (401). The third-function camera (428) may be used for detecting and tracking the pupils (ET: eye tracking). A processor may determine the direction of gaze by tracking the movement of the user's left and right eyes in the image received from the third-function camera (428). By tracking the position of the pupils in the image, the processor may position the center of the XR content image displayed in the display area according to the direction the pupils are gazing. As an example, a GS camera may be used for the third-function camera (428) to detect the pupils and track pupil movements. The third-function camera (428) may be installed for the left and right eyes respectively, and the same camera performance and specifications may be used for each.

[0108] A fourth functional camera (e.g., a face recognition camera) (425, 426) can be used to detect and track (FT: face tracking) the user's facial expression when the user is wearing the electronic device (401).

[0109] According to one embodiment, the electronic device (401) may include a lighting unit (e.g., LED) (not shown) as an auxiliary means for the cameras. For example, the third function camera (425) may use lighting included in the display so that emitted light (e.g., IR LED of infrared wavelength) is directed toward both eyes of the user as an auxiliary means to facilitate gaze detection when tracking eye movements. As another example, the second function cameras (411, 412) may further include a lighting unit (e.g., flash) as an auxiliary means to supplement ambient brightness when shooting outdoors.

[0110] According to one embodiment, a depth sensor (or depth camera) (417) may be used to determine the distance to an object (e.g., object) using time of flight (TOF). Time of flight (TOF) is a technique for measuring the distance to an object using a signal (e.g., near-infrared, ultrasound, or laser), in which a transmitter transmits a signal, a receiver measures the signal, and the distance to the object can be measured based on the flight time of the signal.

[0111] According to one embodiment, the touch sensor (413) may be positioned in the second direction (②) of the main body (410). For example, when a user wears the electronic device (401), the user's eyes may look toward the first direction (①) of the main body. The touch sensor (413) may be implemented as a single type or a type separated into left and right sides depending on the shape of the main body (410), but is not limited thereto. For example, if the touch sensor (413) is implemented as a type separated into left and right sides as shown in FIG. 4a, when a user wears the electronic device (401), the first touch sensor (413a) may be positioned at the user's left eye position as in the fourth direction (④), and the second touch sensor (413b) may be positioned at the user's right eye position as in the third direction (③).

[0112] The touch sensor (413) can recognize touch input in at least one of, for example, capacitive, pressure-sensitive, infrared, or ultrasonic methods. For example, the capacitive touch sensor (413) may be capable of recognizing physical touch (or contact) input or hovering input (or proximity) of an external object. According to some embodiments, the electronic device (401) may use a proximity sensor (not shown) to enable proximity recognition of an external object.

[0113] According to one embodiment, the touch sensor (413) has a two-dimensional surface and can transmit touch data (e.g., touch coordinates) of an external object (e.g., user finger) that contacts the touch sensor (413) to a processor (e.g., processor (120) of FIG. 1). The touch sensor (413) can detect a hovering input for an external object (e.g., user finger) that approaches within a first distance from the touch sensor (413), or detect a touch input that touches the touch sensor (413).

[0114] According to one embodiment, when an external object touches the touch sensor (413), the touch sensor (413) may provide two-dimensional information about the contact point to the processor (120) as "touch data." The touch data may be described as "touch mode." When an external object is located within a first distance from the touch sensor (413) (or is in close proximity, hovering above the touch sensor), the touch sensor (413) may provide hovering data to the processor (120) regarding the time or location of hovering around the touch sensor (413). The hovering data may be described as "hovering mode / closeness mode."

[0115] According to one embodiment, the electronic device (401) can acquire hovering data using at least one of the touch sensor (413), a proximity sensor (not shown) and / or a depth sensor (417) to generate information regarding the distance, location, or time between the touch sensor (413) and an external object.

[0116] According to one embodiment, the interior of the main body (410) may include a processor (e.g., the processor (120) of FIG. 1) and a memory (e.g., the memory (130) of FIG. 1).

[0117] Memory can store various instructions that can be executed by the processor. Instructions may include arithmetic and logical operations, data movement, or control instructions such as input / output that can be recognized by the processor. Memory may include volatile memory (e.g., volatile memory (132) of FIG. 1) and non-volatile memory (e.g., non-volatile memory (134) of FIG. 1) and may store various data temporarily or permanently.

[0118] The processor may be configured to be operatively, functionally, and / or electrically connected to each component of the electronic device (401) to perform operations or data processing regarding the control and / or communication of each component. The operations performed by the processor may be executed by instructions that are stored in memory and, at execution, cause the processor to operate.

[0119] Hereinafter, although there are no limitations on the computation and data processing functions that the processor can implement on the electronic device (401), a series of operations related to XR content service functions will be described. The operations of the processor described below can be performed by executing instructions stored in memory.

[0120] According to one embodiment, the processor can create a virtual object based on virtual information based on image information. The processor can output a virtual object related to an XR service along with background space information through a display (421). For example, the processor can acquire image information by capturing an image related to a real space corresponding to the field of view of a user wearing an electronic device (401) through a second function camera (411, 412), or can create a virtual space for a virtual environment. For example, the processor can control the display (421) to display XR content (hereinafter referred to as the XR content screen) such that at least one virtual object is superimposed on an area determined to be a field of view or a user's field of view (FoV).

[0121] According to one embodiment, the electronic device (401) may have a form factor for being worn on a user's head. The electronic device (401) may further include a strap and / or a wearing member for being secured on a part of the user's body. The electronic device (401) may provide a user experience based on augmented reality, virtual reality, and / or mixed reality while being worn on the user's head.

[0122] FIG. 5 illustrates examples of the construction of a space, input from a user within the space, and output to the user according to various embodiments.

[0123] An electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (201) of FIG. 2, the electronic device (301) of FIG. 3, and the electronic device (401) of FIG. 4a and FIG. 4b) can acquire physical space information about the physical space where the sensor is located by using a sensor. The physical space information may include the geographical location of the physical space where the sensor is located, the size of the physical space, the appearance of the physical space, the location of a physical object (551) placed within the space, the size of the physical object (551), the appearance of the physical object (551), and illuminant information. The appearance of the space and the physical object (551) may include at least one of the shape, texture, or color of the space and the physical object (551). The illuminant information is information regarding a light source emitting light acting within the physical space, and may include at least one of the intensity, direction, or color of the illuminant. The aforementioned sensor may collect information for providing augmented reality. For example, referring to the augmented reality device illustrated in FIGS. 2, FIGS. 3, FIGS. 4a, and FIGS. 4b, the sensor may include a camera and a depth sensor. However, it is not limited thereto, and the sensor may further include at least one of an infrared sensor, a depth sensor (e.g., a LiDAR sensor, a radar sensor, or a stereo camera), a gyroscope, an accelerometer, or a geomagnetic sensor.

[0124] The electronic device (501) can collect physical space information over multiple time frames. For example, in each time frame, the electronic device (501) can collect information regarding the physical space of a part of the scene within the sensor's sensing range (e.g., field of view (FOV)) at the location of the electronic device (501) in physical space. By analyzing the physical space information over multiple time frames, the electronic device (501) can track changes in an object over time (e.g., movement of position or change of state). By integrally analyzing the physical space information collected through multiple sensors, the electronic device (501) can also obtain integrated physical space information regarding the integrated sensing range of multiple sensors (e.g., an image of scenes around the electronic device (501) spatially stitched in physical space).

[0125] An electronic device (501) according to one embodiment can analyze physical space into three-dimensional information by utilizing various input signals of a sensor (e.g., sensing data of an RGB camera, infrared sensor, depth sensor, or stereo camera). For example, the electronic device (501) can analyze at least one of the shape, size, location of physical space, and the shape, size, or location of a physical object (551).

[0126] For example, the electronic device (501) can detect an object captured within a scene corresponding to the camera's field of view by using the camera's sensing data (e.g., a captured image). The electronic device (501) can determine the label of a physical object (551) (e.g., information indicating the classification of the object, including a value indicating a chair, a monitor, or a plant) and the area occupied by the physical object (551) within the two-dimensional scene (e.g., a bounding box) from the camera's two-dimensional scene image. Thus, the electronic device (501) can obtain two-dimensional scene information at the position where the user (590) is looking. Additionally, the electronic device (501) can also calculate the position of the electronic device (501) within physical space based on the camera's sensing data.

[0127] The electronic device (501) can obtain location information of the user (590) and depth information of the physical space in the direction of view using sensing data (e.g., depth data) from a depth sensor. The depth information is information indicating the distance from the depth sensor to each point and can be expressed in the shape of a depth map. The electronic device (501) can analyze the distance in pixels at the 3D location that the user (590) is looking at.

[0128] The electronic device (501) can obtain information including a three-dimensional point cloud and a mesh using various sensing data. The electronic device (501) can obtain a face, a mesh, or a three-dimensional coordinate point cluster that constitutes the space by analyzing the physical space. The electronic device (501) can obtain a three-dimensional point cloud representing physical objects based on the information obtained as described above.

[0129] The electronic device (501) can analyze the physical space to obtain information including at least one of the three-dimensional position coordinates, three-dimensional shape, or three-dimensional size (e.g., three-dimensional bounding box) of physical objects placed within the physical space.

[0130] Accordingly, the electronic device (501) can acquire physical object information detected within a three-dimensional space and semantic segmentation information for the three-dimensional space. The physical object information may include at least one of the location, appearance (e.g., shape, texture, and color) or size of the physical object (551) within the three-dimensional space. The semantic segmentation information is information that semantically divides the three-dimensional space into sub-spaces, and may include, for example, information indicating that the three-dimensional space is divided into objects and background, and information indicating that the background is divided into walls, floors, and ceilings. As described above, the electronic device (501) can acquire and store three-dimensional information (e.g., physical space information) regarding the physical object (551) and the physical space. The electronic device (501) can store the user's (590) three-dimensional location information within the space along with the physical space information.

[0131] An electronic device (501) according to one embodiment can construct a space (500) based on the physical location of the electronic device (501) and / or the user (590). The electronic device (501) can create the space (500) by referring to the aforementioned physical space information. Based on the physical space information, the electronic device (501) can create a space (500) of the same scale as the physical space and place objects within the created space (500). The electronic device (501) can provide complete virtual reality to the user (590) by outputting an image that represents the entire physical space. The electronic device (501) can provide mixed reality (MR) or augmented reality (AR) by outputting an image that represents a part of the physical space. However, while the construction of a space (500) based on physical space information obtained by the analysis of the physical space described above is explained, the electronic device (501) may also construct the space (500) regardless of the physical location of the user (590). As will be described in more detail later in FIG. 6, the space (500) may include a virtual space constructed based on a physical space, a virtual space constructed independently of a physical space, or a space combined with a physical space. In this specification, the space (500) may represent a space corresponding to augmented reality or virtual reality.

[0132] For example, the electronic device (501) may provide a virtual graphic representation that replaces at least some space in the physical space. The electronic device (501) based on an optical see-through may output a virtual graphic representation by overlaying it on a screen area corresponding to at least some space in the screen display unit. The electronic device (501) based on a video see-through may output an image generated by replacing an image area corresponding to at least some space in a spatial image corresponding to the physical space rendered based on spatial information with a virtual graphic representation. The electronic device (501) may replace at least some part of the background in the physical space with a virtual graphic representation, but is not limited thereto. The electronic device (501) may only perform additional placement of a virtual object (552) within the space (500) based on spatial information without changing the background.

[0133] The electronic device (501) can place and output a virtual object (552) within a space (500). The electronic device (501) can set an operation area for the virtual object (552) in the space occupied by the virtual object (552) (e.g., a volume corresponding to the external shape of the virtual object (552)). The operation area may represent an area where operations on the virtual object (552) occur. Additionally, the electronic device (501) can replace a physical object (551) with a virtual object (552) and output it. The virtual object (552) corresponding to the physical object (551) may have the same or similar shape as the physical object (551). However, it is not limited to this, and the electronic device (501) may set only an operation area in the space occupied by the physical object (551) or at a location corresponding to the physical object (551) without outputting a virtual object (552) that replaces the physical object (551). In other words, the electronic device (501) transmits visual information representing the physical object (551) (e.g., light reflected from the physical object (551) or an image captured of the physical object (551)) to the user (590) without alteration and can set an operation area on the physical object (551). The operation area may be set with the same shape and volume as the space occupied by the virtual object (552) or the physical object (551), but is not limited thereto. The electronic device (501) may also set an operation area smaller than the space occupied by the virtual object (552) or the space occupied by the physical object (551).

[0134] According to one embodiment, an electronic device (501) may place a virtual object (e.g., an avatar object) representing a user (590) within a space (500). When the avatar object is provided in a first-person view, the electronic device (501) may visualize a graphic representation corresponding to a part of the avatar object (e.g., hands, torso, or legs) to the user (590) through the aforementioned display (e.g., an optical see-through display or a video see-through display). However, this is not limited thereto, and when the avatar object is provided in a third-person view, the electronic device (501) may visualize a graphic representation corresponding to the entire shape of the avatar object (e.g., back view) to the user (590) through the aforementioned display. The electronic device (501) may provide the user (590) with an experience integrated with the avatar object.

[0135] Additionally, the electronic device (501) may provide an avatar object of another user that has entered the same space (500). The electronic device (501) may receive feedback information that is identical or similar to the feedback information provided to another electronic device that has entered the same space (500) (e.g., information based on at least one of visual, auditory, or haptic). For example, if an object is placed in any space (500) and multiple users access the space (500), the electronic devices (501) of the multiple users may receive feedback information (e.g., graphic representation, sound signal, or haptic feedback) of the same object placed in the space (500) and provide it to each user (590).

[0136] The electronic device (501) can detect inputs to avatar objects of other electronic devices and can also receive feedback information from avatar objects of other electronic devices. The exchange of inputs and feedback per space (500) can be performed by a server (e.g., the server (108) of FIG. 1). For example, a server (e.g., a server providing a metaverse space) can transmit inputs and feedback between the avatar objects of a user (590) and other users between the users (590). However, it is not limited to this, and without going through a server, the electronic device (501) can establish direct communication with other electronic devices to provide inputs based on avatar objects or receive feedback.

[0137] For example, the electronic device (501) may determine that a physical object (551) corresponding to the selected operation area is selected by the user (590) based on detecting input from a user selecting an operation area. The input from the user (590) may include at least one of gesture input using a part of the body (e.g., hand, eye), input using a separate virtual reality accessory device, or voice input from the user.

[0138] The gesture input is an input corresponding to a gesture identified based on tracking the body part (510) of the user (590), for example, the gesture input may include an input that directs or selects an object. The gesture input may include at least one of a gesture in which a part of the body (e.g., hand) is directed toward an object for a predetermined amount of time, a gesture in which a part of the body (e.g., finger, eye, head) points toward an object, or a gesture in which a part of the body and an object are in spatial contact. A gesture of pointing toward an object with the eyes may be identified based on eye tracking. A gesture of pointing toward an object with the head may be identified based on head tracking.

[0139] Tracking of the body part (510) of the user (590) may be performed primarily based on a camera of the electronic device (501), but is not limited thereto. The electronic device (501) may also track the body part (510) based on the cooperation of sensing data from a vision sensor (e.g., image data from a camera and depth data from a depth sensor) and information collected by an accessory device described later (e.g., controller tracking, finger tracking within the controller). Finger tracking may be performed by sensing the distance or contact between an individual finger and the controller based on a sensor (e.g., an infrared sensor) embedded in the controller.

[0140] The accessory device for virtual reality may include a rideable device, a wearable device, a controller device (520), or other sensor-based devices. The rideable device is a device operated by a user (590) while riding on it, and may include, for example, at least one of a treadmill-type device or a chair-type device. The wearable device is an operating device worn on at least part of the user's (590) body, and may include, for example, at least one of a full-body and half-body suit-type controller, a vest-type controller, a shoe-type controller, a bag-type controller, a glove-type controller (e.g., a haptic glove), or a face mask-type controller. The controller device (520) may include an input device (e.g., a stick-type controller, or a firearm) operated by, for example, a hand, foot, toe, or other body part (510).

[0141] The electronic device (501) may establish direct communication with the accessory device to track at least one of the location or motion of the accessory device, but is not limited thereto. The electronic device (501) may also communicate with the accessory device via a base station for virtual reality.

[0142] For example, the electronic device (501) may determine that the virtual object (552) has been selected based on detecting the act of gazing at the virtual object (552) for a predetermined amount of time or longer through the aforementioned eye gaze tracking technology. As another example, the electronic device (501) may recognize a gesture pointing to the virtual object (552) through hand tracking technology. The electronic device (501) may determine that the virtual object (552) has been selected based on the direction in which the tracked hand points to the virtual object (552) for a predetermined amount of time or longer, or based on the user's (590) hand touching or entering the area occupied by the virtual object (552) within the space (500).

[0143] The user's voice input is an input corresponding to the user's voice obtained by the electronic device (501), and may include voice data that is sensed by, for example, an input module (e.g., a microphone) of the electronic device (501) or received from an external electronic device of the electronic device (501). The electronic device (501) may determine that a physical object (551) or a virtual object (552) has been selected by analyzing the user's voice input. For example, the electronic device (501) may determine that at least one of the physical object (551) or the virtual object (552) has been selected based on detecting a keyword indicating at least one of the physical object (551) or the virtual object (552) from the user's voice input.

[0144] The electronic device (501) can provide feedback described below as a response to the input of the aforementioned user (590).

[0145] Feedback may include visual feedback, auditory feedback, haptic feedback, olfactory feedback, or gustatory feedback. As described above in FIG. 1, the feedback may be rendered by a server (e.g., server (108) of FIG. 1), an electronic device (e.g., electronic device (101) of FIG. 1), or an external electronic device (e.g., electronic device (102, 104) of FIG. 1). Visual feedback may include the action of outputting an image through a display (e.g., transparent display, opaque display) of the electronic device (501). Auditory feedback may include the action of outputting sound through a speaker of the electronic device (501).

[0146] Haptic feedback may include force feedback that simulates weight, shape, texture, dimensions, and dynamics. For example, a haptic glove may include haptic elements (e.g., electrical muscles) that can simulate tactile sensations by tensing and relaxing the user's (590) body. Haptic elements inside the haptic glove may act as tendons. The haptic glove may provide haptic feedback across the user's (590) entire hand. The electronic device (501) may provide feedback indicating the shape, size, and stiffness of an object through the haptic glove. For example, the haptic glove may generate a force that mimics the shape, size, and stiffness of an object. The exoskeleton of the haptic glove (or suit-type device) may include sensors and finger movement measuring devices and may transmit haptic information to the body by transmitting a force (e.g., electromagnetic, DC motor, or pneumatic-based force) that pulls a cable to the user's (590) finger. Hardware providing haptic feedback may include sensors, actuators, power, and wireless transmission circuits. Haptic gloves may operate by inflating and deflating inflatable air pockets on the surface of the gloves.

[0147] The electronic device (501) can provide feedback to the user (590) based on the selection of an object within the space (500). For example, the electronic device (501) can output a graphic representation indicating the selected object (e.g., a representation highlighting the selected object) through a display. As another example, the electronic device (501) can output a sound (e.g., voice) guiding the selected object through a speaker. As yet another example, the electronic device (501) can provide the user (590) with haptic movements simulating the touch of the object by transmitting an electrical signal to a haptic-assisted accessory device (e.g., a haptic glove).

[0148] FIG. 6 is a diagram illustrating an example of an operation in which an electronic device according to various embodiments provides space to a user.

[0149] An electronic device according to one embodiment (e.g., electronic device (101) of FIG. 1, electronic device (201) of FIG. 2, electronic device (301) of FIG. 3, electronic device (401) of FIG. 4a and FIG. 4b, electronic device (501) of FIG. 5) can display an image rendering objects (621, 622, 623, 624, 625, 626) placed in a space (e.g., space (500) of FIG. 5). A space according to one embodiment may include at least one of a physical space or a virtual space.

[0150] For example, the space may include a virtual space constructed based on a physical space. For instance, an electronic device may acquire spatial information about the physical space surrounding the electronic device and construct and provide a virtual space based on the attributes (e.g., scale) of the acquired physical space. In various embodiments of the present disclosure, the space including a virtual space constructed based on the physical space surrounding the electronic device may also be represented as a video see-through space (VST space).

[0151] For example, the space may include a virtual space constructed independently of the physical space surrounding the electronic device. For example, the space may be a virtual space constructed based on a physical space different from the physical space surrounding the electronic device. For example, the space may be a virtual space constructed by a server (e.g., the server (108) of FIG. 1). In various embodiments of the present disclosure, the space including a virtual space constructed independently of the physical space surrounding the electronic device may also be expressed as a VR space (virtual reality space).

[0152] For example, the space may include both the physical space surrounding the electronic device and the virtual space in which virtual objects are placed. The virtual space in which virtual objects are placed may be constructed based on the physical space surrounding the electronic device. In other words, the space may include a combined space of the physical space surrounding the electronic device and the virtual space in which virtual objects are placed. For example, if the electronic device is an OST (optical see-through) based electronic device (e.g., the electronic device (201) of FIG. 2), at least a portion of the physical space surrounding the electronic device may be perceived directly by the user (610) through a transparent member, and at least a portion of the virtual space may be provided to the user (610) by displaying elements of the virtual space (e.g., visual effects, virtual objects) on a display. In other words, light from the outside (e.g., the background of the physical space surrounding the electronic device or physical objects) reaches the user's (610) eyes through the transparent member, so the user (610) perceives the physical space surrounding the user (610), and the user (610) may perceive the virtual space overlaid on the physical space surrounding the user (610) by displaying elements of the virtual space on a display. For example, the electronic device may provide the user (610) with a space in which elements of a virtual space are additionally placed against the background of the physical space around the user (610). In various embodiments of the present disclosure, the space including the physical space and the virtual space around the electronic device may also be expressed as an optical see-through space (OST space) or an augmented reality space (AR space).

[0153] Objects placed in space may include physical objects and virtual objects.

[0154] For example, a virtual object may include a virtual object for running an application (e.g., an application icon), a virtual object that provides information (e.g., a virtual object displaying weather information, a virtual object displaying notes), and a virtual object containing an application execution screen (e.g., a window displaying an application execution screen). A virtual object may include, for example, a widget, an icon, or an application execution screen (or a window displaying an execution screen).

[0155] For example, a virtual object may be related to a physical object. A virtual object based on a physical object may include a virtual object that replaces the physical object and / or a virtual object related to the physical object (e.g., a virtual controller for controlling the physical object).

[0156] Referring to FIG. 6, the electronic device can display a screen (620) of the space when a user (610) wearing the electronic device enters the space. The electronic device determines a subspace to be displayed to the user (610) from the space based on the user's (610) line of sight, and can display the screen (620) based on rendering data that renders objects (621, 622, 623, 624, 625, 626) placed in the subspace. In FIG. 6, the objects (621, 622, 623, 624, 625, 626) in the screen (620) are shown as being placed in a curved area, but are not limited thereto and may be placed on a surface other than the curved area.

[0157] According to one embodiment, the electronic device can obtain information about the viewpoint (e.g., the user’s perspective) of a user (610) wearing the electronic device. Based on the information about the user’s (610) viewpoint, the electronic device can display a screen (620) of a space corresponding to the viewpoint.

[0158] In the remaining part of the screen (620) of FIG. 6, excluding the objects (621, 622, 623, 624, 625, 626), at least a portion of the background of the space may be displayed. For example, if the space is a VST space, the remaining part excluding the objects (621, 622, 623, 624, 625, 626) may display a virtual space in which the physical space around the electronic device is reconstructed. For example, if the space is an AR space, the remaining part excluding the objects (621, 622, 623, 624, 625, 626) may display the physical space around the electronic device. For example, if the space is a VR space, the remaining part excluding the objects (621, 622, 623, 624, 625, 626) may display a virtual space (e.g., a virtual space constructed independently of the surroundings of the electronic device).

[0159] FIG. 7 is a diagram illustrating an example of an operation to place a target object based on a source object according to various embodiments.

[0160] An electronic device according to one embodiment (e.g., electronic device (101) of FIG. 1, electronic device (201) of FIG. 2, electronic device (301) of FIG. 3, electronic device (401) of FIG. 4a and FIG. 4b, electronic device (501) of FIG. 5) may provide a space (e.g., space (500) of FIG. 5) to a user. The space may be, for example, at least one of a VR space, an AR space, or a VST space.

[0161] In operation (701), the electronic device may obtain user input selecting a source object (711). The user may select the source object (711) using a part of the user's body (e.g., hand, eye) and / or an external device (e.g., pointer device). The electronic device may obtain (e.g., detect, receive) user input selecting the source object (711). In FIG. 7, the user may point to the source object (711) placed in the area (720) using the user's finger. The electronic device may obtain user input selecting the source object (711) by analyzing the position of the user's finger using a camera (e.g., the first camera (265a, 265b) of FIG. 2).

[0162] Examples of user inputs for selecting an object (e.g., source object (711)) are described in more detail later in FIGS. 8a through 8d.

[0163] In operation (702), the electronic device may obtain a movement input (e.g., a closer movement input) that pulls the source object (711) following a user input that selects the source object (711). After selecting the source object (711), the user may perform a gesture that directs the movement of the selected source object (711). In FIG. 7, for example, the gesture directing the movement (e.g., a closer movement) of the source object (711) may include a pinch gesture of closing and / or spreading two fingers. The electronic device may obtain a movement input that pulls the source object (711) based on detecting the pinch gesture.

[0164] In operation (703), the electronic device may create a target object (712) based on a source object (711). According to one embodiment, the source object (711) may be a physical object, and the electronic device may create a target object (712) associated with the source object (711) and move the target object (712) instead of moving the source object (711). The target object (712) may include at least one of a virtual object that emulates the source object (711), or a virtual object (e.g., a virtual controller) to support the function of the source object (711).

[0165] In various embodiments of the present disclosure, the source object (711) is primarily described as a physical object, but is not limited thereto. For example, the source object (711) may be a virtual object placed in space. An electronic device may acquire (or determine) the source object (711) as a target object (712) if the source object (711) is a virtual object capable of changing its position.

[0166] In operation (704), the electronic device can determine the location of the target object (712) and place the target object (712) at the determined location. The electronic device can place the target object (712) at a location closer to the electronic device than the location of the source object (711). The electronic device can place the target object (712) in a line of sight area (730). The line of sight area (730) may refer to an area in space that has a depth (or depth range) that the user frequently looks at.

[0167] However, the electronic device according to the various embodiments of the present disclosure is not limited to determining the position of the target object (712) based on the line of sight area (730). According to one embodiment, the electronic device may determine the position of the target object (712) based on the position of another object placed in space. The operation of determining the position of the target object (712) is described in more detail later in FIGS. 8 to 15.

[0168] FIGS. 8a to 8d are drawings illustrating examples of user input for selecting an object according to various embodiments.

[0169] According to one embodiment, an electronic device (e.g., electronic device (101) of FIG. 1, electronic device (201) of FIG. 2, electronic device (301) of FIG. 3, electronic device (401) of FIG. 4a and FIG. 4b, electronic device (501) of FIG. 5) can select an object using at least one of a part of the user's body (e.g., hand, eye) and / or an external device (e.g., pointer device).

[0170] Referring to FIG. 8a, a user can touch an object with a finger while looking at it with their eyes. For example, an electronic device can detect candidate objects corresponding to the user's gaze by tracking the user's eyes. The electronic device can obtain user input for selecting an object in response to detecting a tap gesture made by the user's finger touching a candidate object corresponding to the user's gaze or a location corresponding to a candidate object.

[0171] Referring to FIG. 8b, a user can perform an object selection gesture with their finger while looking at an object. For example, an electronic device can detect candidate objects corresponding to the user's gaze by tracking the user's eyes. The object selection gesture is a pre-set gesture for the object selection function and may include, for example, a pinch gesture. Based on detecting the object selection gesture, the electronic device can obtain user input selecting an object corresponding to a part of the user's body (e.g., a point where two fingers meet).

[0172] Referring to FIG. 8c, the user can select an object using an external device (e.g., a pointer device). For example, the electronic device can select an object that the external device points to by using information received from the external device and / or information about the external device obtained by the electronic device (e.g., image information).

[0173] According to one embodiment, a pointer device may transmit information instructing an object selection to an electronic device based on acquiring user input instructing an object selection. For example, the pointer device may transmit information regarding user input to an electronic device in response to acquiring user input through an input acquisition unit (e.g., a physical button, a touch sensor). The electronic device may acquire user input selecting an object pointed to by the pointer device at the time the user input occurs.

[0174] According to one embodiment, an electronic device tracks an object pointed to by a pointer device, and when the state in which the pointer device points to a specific object for a certain period of time or longer is maintained, the device can obtain user input for selecting a specific object.

[0175] Referring to FIG. 8d, the user can select an object using their eyes and / or gaze. The electronic device can track the user's eyes using a camera (e.g., the camera for the third function (428) of FIG. 4). For example, in response to acquiring user input indicating object selection, the electronic device can acquire user input selecting an object pointed to by the user's eyes at the time the user input occurs. For example, the electronic device can track an object pointed to by the user's eyes and acquire user input selecting a specific object when the state in which the user's eyes point to a specific object is maintained for a certain length of time or longer.

[0176] In various embodiments of the present disclosure, user input for selecting an object acquired by an electronic device is not limited to the examples described in FIGS. 8a through 8d. A user may select an object using two or more of one or more body parts (e.g., eyes, hands), gestures, or external devices (e.g., pointer devices).

[0177] FIG. 9 is a flowchart illustrating an example of an operation in which an electronic device according to various embodiments displays a target object based on a source object.

[0178] According to one embodiment, an electronic device (e.g., electronic device (101) of FIG. 1, electronic device (201) of FIG. 2, electronic device (301) of FIG. 3, electronic device (401) of FIG. 4a and FIG. 4b, electronic device (501) of FIG. 5) may acquire (e.g., create) a target object (e.g., target object (712) of FIG. 7) based on a source object (e.g., source object (711) of FIG. 7) and place the acquired target object in a space (e.g., space (500) of FIG. 5).

[0179] In operation (910), the electronic device may receive user input selecting a portion of a source object placed in space. The source object may be a real object placed in space. The source object may be another electronic device electrically or functionally connected to the electronic device. For example, the electronic device may include a communication circuit and may be connected wired and / or wirelessly to another electronic device that is the source object through the communication circuit.

[0180] In operation (920), the electronic device can determine a target function based on a selected portion of the source object among at least one candidate function of the source object.

[0181] A candidate function may refer to a function provided by a source object. A source object may include multiple parts that provide multiple candidate functions. Each candidate function may be associated with a part of the source object. For example, a hardware component within the source object that provides a candidate function may be associated with the candidate function.

[0182] According to one embodiment, a source object may include a display that outputs the state of the source object and an input acquisition unit for acquiring an input that controls the operation of the source object. One or more candidate functions of the source object may include an output function that outputs the state of the source object and an input function that acquires an input that controls the operation of the source object.

[0183] For example, an electronic device may receive user input selecting a display among source objects. The electronic device may determine an output function as a target function among candidate functions of the source objects. For example, the electronic device may receive user input selecting an input acquisition unit among source objects. The electronic device may determine an input function as a target function among candidate functions of the source objects.

[0184] In various embodiments of the present disclosure, examples of candidate functions of a source object are not limited to input functions and output functions. For example, the source object may provide more granular input functions and / or output functions.

[0185] According to one embodiment, the output function of a source object may include an output function for each attribute when the state of the source object includes a plurality of attributes. For example, when the state of the source object includes attribute values ​​of a plurality of attributes including current temperature, current humidity, operating mode, target temperature, or fine dust concentration, the output function may include a current temperature output function, a current humidity output function, an operating mode output function, a target temperature output function, or a fine dust concentration output function.

[0186] According to one embodiment, the input function of a source object may include an input function that obtains an input based on each attribute when controlling the operation of the source object based on a plurality of attributes. For example, when the control input regarding the operation of the source object includes at least one of the attribute values ​​of a plurality of attributes including a target temperature or an operation mode, the input function may include a target temperature input function or an operation mode input function.

[0187] If a source object provides one or more input functions and / or one or more output functions, among the candidate functions, the input functions may be classified as input types, and among the candidate functions, the output functions may be classified as output types.

[0188] In operation (930), the electronic device may acquire a target object corresponding to the target function. The target object may mean a virtual object for providing the target function.

[0189] According to one embodiment, an electronic device may generate a target object for providing a target function by applying an object generation model to information regarding a part of a source object. An object generation model may refer to a model that is generated and / or trained to output output data corresponding to a virtual target object from input data corresponding to information regarding a part of a source object. An object generation model may be implemented, for example, based on a machine learning model. An object generation model may include a neural network (e.g., a convolutional neural network (CNN)) and / or a large language model (LLM). An object generation model may generate a target object based on input data comprising at least one of information regarding a source object, information regarding a target function, or information regarding a part of a source object.

[0190] According to one embodiment, the electronic device may load a stored target object. For example, the electronic device may map information regarding the created target object to the source object, the part of the source object, or the target function based on the creation of a target object for a part of a source object and / or a target function. The electronic device may store the target object and metadata of the target object (e.g., source object, part of the source object, or target function) in internal memory or an external database. Based on obtaining user input selecting a part of the source object, the electronic device may determine whether a target object corresponding to the target function has already been created and stored in internal memory or an external database. If the target object has already been created and stored, the electronic device may obtain the target object by loading the stored target object.

[0191] In operation (940), the electronic device can determine the location of the target object acquired in space based on the target function.

[0192] According to one embodiment, the electronic device can determine the location of a target object based on another object providing a target function placed in space. The operation of determining the location of the target object is described in more detail later in FIG. 12.

[0193] In operation (950), the electronic device can display a target object at a determined location. The electronic device can place the target object at a determined location. The electronic device can obtain a visual representation of the target object by rendering the target object. The electronic device can display the visual representation based on the determined location.

[0194] Although not explicitly illustrated in FIG. 9, according to various embodiments of the present disclosure, an electronic device may acquire a movement input that drags a selected portion following a user input that selects a source object. Based on acquiring the movement input, the electronic device may perform at least one of determining a target function, acquiring a target object, determining the location of a target object, or displaying a target object.

[0195] FIG. 10 is a drawing illustrating a target object based on a source object according to various embodiments.

[0196] According to one embodiment, the source object (1010) may be another electronic device electrically or functionally connected to an electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (201) of FIG. 2, the electronic device (301) of FIG. 3, the electronic device (401) of FIG. 4a and 4b, the electronic device (501) of FIG. 5). The source object (1010) may include a first part (1011), a second part (1012), and a third part (1013). Each of the first part (1011), the second part (1012), and the third part (1013) may provide the function of the corresponding source object (1010).

[0197] For example, referring to FIG. 10, the source object (1010) may be a microwave. The source object (1010) may provide a first output function, a second output function, and an input function. The first output function may mean a function of outputting information about food (e.g., the appearance of the food). The second output function may mean a function of outputting the remaining cooking time. The input function may mean a function of obtaining a control input to control the operation of the source object (1010).

[0198] For example, the electronic device may obtain user input selecting a first part (1011). Based on the first part (1011), the electronic device may determine a first output function of a source object as a target function. The electronic device may obtain a first object (1021) as a target object for providing the first output function. The first object (1021) may include a virtual object containing a visual representation corresponding to food being cooked inside a source object (1010) (e.g., a microwave oven).

[0199] For example, the electronic device may obtain user input selecting a second part (1012). Based on the second part (1012), the electronic device may determine a second output function of the source object (1010) as a target function. The electronic device may obtain a second object (1022) as a target object for providing the second output function. The second object (1022) may include a virtual object including a timer object corresponding to the remaining cooking time.

[0200] For example, the electronic device may obtain user input selecting a third part (1013). Based on the third part (1013), the electronic device may determine the input function of the source object (1010) as the target function. The electronic device may obtain a third object (1023) for providing the input function as the target object. The third object (1023) may include a virtual object comprising buttons for controlling the operation of the introduction object (1010) (e.g., microwave oven). Based on obtaining an input selecting at least one of the buttons of the third object (1023), the electronic device may transmit a control command corresponding to the selected button to the source object (1010) (e.g., microwave oven). The source object (1010) (e.g., microwave oven) may operate based on the control command received from the electronic device.

[0201] FIG. 11 is a diagram illustrating an example of an operation in which an electronic device creates a target object according to various embodiments.

[0202] In operation (1110), an electronic device (e.g., electronic device (101) of FIG. 1, electronic device (201) of FIG. 2, electronic device (301) of FIG. 3, electronic device (401) of FIG. 4a and 4b, electronic device (501) of FIG. 5) can obtain user input. For example, the electronic device can obtain user input based on a hand-based gesture (also referred to as a 'hand gesture' in various embodiments of the present disclosure) (e.g., pinch gesture, tap gesture). For example, the electronic device can obtain user input based on an external device (e.g., a controller or a pointer device). For example, the electronic device can obtain user input based on a gaze pointing at a specific object or a specific area with the eye.

[0203] In operation (1120), the electronic device can determine a portion of a selected source object based on user input. For example, the electronic device can determine a portion of a source object based on the results of hand tracking. For example, the electronic device can determine a portion of a source object based on the results of eye tracking. For example, the electronic device can determine a portion of a source object by combining the results of hand tracking and eye tracking (represented as multimodal in FIG. 11).

[0204] In operation (1130), the electronic device can create a target object using an object creation model. According to one embodiment, the object creation model may include vision AI, generative AI, or depth information.

[0205] Vision AI can be used to analyze the category of objects within images acquired through an external camera. For example, if another controllable electronic device connected to the electronic device is selected as the source object, the electronic device can transmit a request to the generative AI to create a target object to perform an input function that acquires control inputs.

[0206] An electronic device can determine the location to display a target object generated using vision AI. For example, if the electronic device determines that interaction between the source object and / or target object and the user is required, the target object can be displayed at a location controllable by the user's body part (e.g., hand).

[0207] Generative AI can generate a new virtual object as a target object based on at least a portion of a source object in an image captured by a camera. An electronic device can generate the target object using data (e.g., images) stored in the device's internal memory and / or on a networked server. For example, if the electronic device has access to additional images captured of the source object, it can generate the target object using the additional images. The additional images are of higher quality than the image of the source object captured by the electronic device, and may include, for example, images captured by a high-resolution camera and / or images captured at a closer distance than the camera of the electronic device. The electronic device can generate a target object that outputs at least a portion of the source object using the additional images, with higher quality than the image corresponding to the source object.

[0208] Depth information can be used to determine the position of a target object after the target object is created. For example, an electronic device can adjust the size of a target object based on the depth information of a source object and the depth information of the determined position of the target object. As a result, the electronic device can provide a sense of realism to a user interacting with the target object.

[0209] According to one embodiment, the object creation model may be stored inside an electronic device (e.g., on-device). However, in various embodiments of the present disclosure, the object creation model is not limited to being stored inside an electronic device, and the electronic device may receive information regarding a target object created by a server using an object creation model stored externally (e.g., on-cloud).

[0210] In operation (1140), the electronic device can output (e.g., display) a target object. The electronic device can determine the location of the target object and display the target object at the determined location. The determination of the location of the target object is described in more detail later in FIGS. 12 to 15.

[0211] FIG. 12 is a diagram illustrating an example of an operation in which an electronic device according to various embodiments determines the position of a target object.

[0212] In operation (1210), the electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (201) of FIG. 2, the electronic device (301) of FIG. 3, the electronic device (401) of FIG. 4a and FIG. 4b, the electronic device (501) of FIG. 5) can determine whether there is another object providing a target function within the display area of ​​the display.

[0213] The display area of ​​a display may refer to an area within space that is displayed to the user through the display. The display area may be interpreted as substantially corresponding to the user's field of view (FOV).

[0214] Other objects may refer to physical objects that provide the target function. For example, if the target function is a text input function, other objects may be keyboard devices that provide text input functions. For example, if the target function is a drawing input function, other objects may include an electronic pen (or digital pen). However, other objects are not limited to being external electronic devices. For example, other objects may be determined to be a pen or pencil that does not include electronic circuits where the target function is a drawing input function.

[0215] In operation (1220), the electronic device can detect another object providing a target function within the display area. The electronic device can determine the position of the target object by aligning it with the other object.

[0216] According to one embodiment, the electronic device can determine the location of a target object based on the distance between the electronic device and another object. For example, the electronic device can determine the location of a target object as a position aligned with another object based on the fact that the distance between a source object and the electronic device is less than or equal to the distance between another object and the electronic device.

[0217] According to one embodiment, if the target function is an input type, the location of the target object can be determined based on the input object. Based on the fact that the target function is an input type, the electronic device can check whether an input object for obtaining user input exists within the display area of ​​the display. Based on the fact that an input object exists within the display area, the electronic device can determine the location of the target object to a position aligned with the input object.

[0218] In operation (1230), the electronic device may change the position of the target object while maintaining a relative position relationship between the other object and the target object in response to a change in the position of the other object. The relative position relationship between the first object and the second object may mean a position difference (e.g., a change in position) of the position of the second object relative to the position of the first object relative to the first object.

[0219] According to one embodiment, the electronic device may determine a change in position for the position of another object. The electronic device may determine the result of applying the determined change in position to the position of another object in a world coordinate system as the position of a target object. The world coordinate system may refer to a fixed coordinate system in space. The world coordinate system may refer to a coordinate system that remains constant independently of changes in the position and / or orientation of the electronic device. The change in position may include a change in position according to each of the axes of the world coordinate system (e.g., x-axis, y-axis, z-axis).

[0220] In various embodiments of the present disclosure, the position of a target object is not limited to being determined based on another object. According to one embodiment, an electronic device may determine the orientation of a target object based on another object. For example, the electronic device may determine the orientation of a target object based on the orientation of another object. For example, the electronic device may determine a change in orientation with respect to the orientation of another object and determine the orientation of the target object as a result of applying the orientation of the target object in a world coordinate system. The change in orientation may include a rotation angle (e.g., pitch, yaw, roll) along each of the axes (e.g., x-axis, y-axis, z-axis) of the world coordinate system.

[0221] The electronic device can change the orientation of a target object in response to a change in the orientation of another object, while maintaining a relative orientation relationship between the other object and the target object. The relative positional relationship between the first object and the second object may refer to the difference in orientation (e.g., amount of change in orientation) of the direction of the second object with respect to the direction of the first object, with respect to the first object.

[0222] In various embodiments of the present disclosure, the term 'pose' of an object may be used as a term including the position and orientation of an object. For example, the position and / or orientation of a target object may be expressed as the pose of the target object. Relative positional relationships and / or relative orientational relationships may be expressed as relative poseal relationships. A change in position and / or a change in orientation may be expressed as a change in pose.

[0223] In operation (1240), the electronic device can determine the location of the target object based on the location of the electronic device, based on the fact that there is no other object providing the target function within the display area of ​​the display.

[0224] An electronic device can determine a change in position for the position of the electronic device. The electronic device can determine the result of applying the determined change in position for the position of the electronic device in the world coordinate system as the position of the target object. A device coordinate system may refer to a coordinate system defined based on the position and orientation of the device. A device coordinate system may refer to a coordinate system that changes in response to the position and / or orientation of the device.

[0225] According to one embodiment, the electronic device may induce another object to enter the display area when the other object is not present within the display area. For example, the electronic device may output a notification that induces the movement of the electronic device to position the other object within the display area, based on the fact that the other object providing the target function is present in space but not in the display area. Outputting the notification may, for example, include playing a sound describing the direction in which the other object is placed and / or displaying an arrow object pointing to the direction in which the other object is placed.

[0226] According to one embodiment, the electronic device may change the position of a target object based on another object after another object is not present within the display area and another object enters the display area. For example, the electronic device may determine the position of a target object based on the position of the electronic device (e.g., in the device coordinate system) based on the fact that another object providing the target function is not present within the display area of ​​the display. After determining the position of the target object based on the position of the electronic device, the electronic device may change the position of the target object to a position aligned with another object based on the fact that another object providing the target function appears within the display area of ​​the display as the electronic device moves.

[0227] Although not explicitly illustrated in FIG. 12, the electronic device can determine the location of a target object based on gaze history information regarding the user's gaze. For example, the electronic device can determine the location of a target object having a representative depth determined for the user based on the gaze history information.

[0228] According to one embodiment, the electronic device may determine the depth at which the user gazes for the longest time (e.g., depth range) as a representative depth for the user based on gaze history information. The representative depth may be determined independently for each user (e.g., customized).

[0229] According to one embodiment, the electronic device may determine, based on gaze history information, the depth to which the user gazed before selecting a part of a source object as the representative depth for the user. For example, the electronic device may obtain user input selecting a source object within a threshold time from the point of interaction with the reference object after interacting with a reference object (e.g., a screen object of an application). The electronic device may determine the representative depth based on the position (e.g., depth) of the reference object.

[0230] According to one embodiment, an electronic device can determine the location of a target object based on the gaze direction among gaze history information. For example, the electronic device can determine the representative direction that the user looks at for the longest time based on the gaze history information. The electronic device can determine the location of the representative direction determined by the electronic device as the location of the target object.

[0231] According to one embodiment, the electronic device can determine a representative depth and a representative direction based on line-of-sight history information. The electronic device can determine a location situated in the representative direction with the representative depth as the location of a target object. The depth from the electronic device to the location of the target object can be determined based on the representative depth (e.g., determined to a depth equal to or similar to the representative depth). The direction from the electronic device to the location of the target object can be determined based on the representative direction (e.g., determined to a direction equal to or similar to the representative direction).

[0232] Although not explicitly illustrated in FIG. 12, the electronic device may determine the position of a target object based on additional user input. The additional user input may mean a user input that adjusts the position and / or orientation of the target object as a user input that follows the user input for selecting a source object.

[0233] According to one embodiment, the electronic device may acquire a user input for selecting a source object and a movement input for pulling the source object. The electronic device may attempt to determine the location of a target object based on at least one of another object, an electronic device, or gaze history information.

[0234] An electronic device can determine the position of a target object independently of the amount of position change indicated by a motion input that pulls the source object, based on the success of determining the position of the target object based on at least one of another object, the electronic device, or gaze history information. For example, a user may generate a motion input that pulls the source object slightly. The motion input obtained by the electronic device may represent a position change less than a threshold amount of position change. The electronic device can determine the position of the target object to have a position change greater than or equal to the amount of position change from the source object based on at least one of another object, the electronic device, or gaze history information. Even if the position change indicated by the motion input is less than the threshold amount of position change, the electronic device can place the target object at a determined target position greater than or equal to the threshold amount of position change. For example, even if a user generates a motion input that pulls the source object slightly, the target object can be placed at the determined target position immediately.

[0235] The electronic device may determine the position of a target object using additional user input (e.g., a motion input that pulls the source object) based on failure to determine the position of the target object based on at least one of other objects, the electronic device, or gaze history information. For example, the electronic device may display the target object at a default position and / or default orientation. The electronic device may adjust the position and / or orientation of the target object based on additional user input. The electronic device may determine the position of the target object based on the amount of position change indicated by the motion input. For example, if the user generates a motion input to the source object, the electronic device may change the position of the target object by the amount of position change indicated by the motion input. If the user generates a motion input that slightly pulls the source object (e.g., indicating a position change less than a threshold amount), the target object may be placed at a position slightly away from the position of the source object (e.g., a position where the position change indicated by the motion input is applied to the position of the source object). If the user generates a movement input that pulls the source object significantly (e.g., indicating a position change greater than a threshold change amount), the target object may be placed at a location far from the source object's location.

[0236] FIG. 13 is a diagram illustrating an example of an operation in which an electronic device according to various embodiments determines the location of a target object based on another object.

[0237] An electronic device according to one embodiment (e.g., the electronic device (101) of FIG. 1, the electronic device (201) of FIG. 2, the electronic device (301) of FIG. 3, the electronic device (401) of FIG. 4a and FIG. 4b, the electronic device (501) of FIG. 5) can determine whether an input object exists within a display area based on the fact that the target function is an input type.

[0238] Referring to FIG. 13, the electronic device may determine a text memo object (1311) as a source object. On the screen (1301), the electronic device may determine a function (hereinafter also referred to as a ‘text memo creation function’) that determines (e.g., modifies) text stored in the text memo object (1311) of the source object as a target function. Based on the fact that the text memo creation function is an input type, the electronic device may determine that a laptop (1312) (laptop) providing an input function exists within the display area.

[0239] The electronic device can determine the position of the target object as a position aligned with the input object. For example, on the screen (1302), the electronic device can place the text memo object (1311) at a position aligned with the laptop (1312).

[0240] An electronic device according to one embodiment can determine the position of a target object by aligning it with an input object, and then obtain a control input for a source object through the input object. The electronic device can control the source object based on the control input by transmitting the control input to the source object.

[0241] Referring to FIG. 13, the electronic device can obtain text input using a component (e.g., keyboard) of a laptop (1312). The electronic device can change (e.g., add) the text stored in the text memo object (1311) by transmitting information regarding the text input to the text memo object (1311).

[0242] FIG. 13 mainly illustrates an example where the source object is a virtual object, but as described above in FIG. 7 to 12, the source object may be an external electronic device. When an external electronic device is selected as the source object, information regarding user input obtained through an input device may be transmitted from the electronic device to the source object. Based on receiving information regarding user input, the source object may control the operation of the source object according to the user input.

[0243] FIG. 14 is a diagram illustrating an example of an operation in which an electronic device according to various embodiments determines one different object when there are multiple candidate input objects.

[0244] An electronic device according to one embodiment (e.g., the electronic device (101) of FIG. 1, the electronic device (201) of FIG. 2, the electronic device (301) of FIG. 3, the electronic device (401) of FIG. 4a and FIG. 4b, the electronic device (501) of FIG. 5) can determine whether an input object exists within a display area based on the fact that the target function is an input type. The electronic device can detect a plurality of candidate input objects existing within the display area.

[0245] Referring to FIG. 14, the electronic device can determine a text memo object (1410) as a source object. The electronic device can detect a laptop (1421), a mouse (1423), and an electronic pen (1425) as a plurality of candidate input objects that provide input functions within a display area.

[0246] Based on the existence of multiple candidate input objects within a display area of ​​a display, the electronic device can select an input object based on at least one of the relevance of the multiple candidate input objects to a target function, the location of each candidate input object, or the usage history information of each candidate input object. The electronic device can determine the location of a target object as a position aligned with the selected input object.

[0247] The relevance of a candidate input object to an input function may indicate the relationship and / or similarity between the input types supported by the input function (e.g., text type, drawing type, selection type) and the input types supported by the candidate input object. The higher the relevance of a candidate input object, the higher the likelihood that it will be selected as the input object.

[0248] The closer the location of a candidate input object is to an electronic device (or a part of the user's body), the higher the probability that the candidate input object will be selected as an input object.

[0249] The usage history information of a candidate input object may include the usage frequency, last usage time, or usage history information regarding the target function (or target object) of the candidate input object used by the user. The more frequently a candidate input object is used by the user, the higher the probability that the candidate input object will be selected as an input object. The more recent the last usage time of the candidate input object, the higher the probability that the candidate input object will be selected as an input object.

[0250] Referring to FIG. 14, the electronic device can determine the laptop (1421) as the input object among the laptop (1421), mouse (1423), and electronic pen (1425), based on the fact that the target function is a text memo writing function. The electronic device can determine the position of the text memo object (1410) to be aligned with the laptop (1421).

[0251] FIG. 15 is a drawing illustrating an example of an operation in which an electronic device according to various embodiments displays a target object for an output function.

[0252] An electronic device according to one embodiment (e.g., the electronic device (101) of FIG. 1, the electronic device (201) of FIG. 2, the electronic device (301) of FIG. 3, the electronic device (401) of FIG. 4a and FIG. 4b, the electronic device (501) of FIG. 5) can create a target object that emulates a part of a source object using a source object or a part of a source object, based on the fact that the target function is an output type.

[0253] According to one embodiment, the electronic device can determine the dimension of the target object as either three dimensions or two dimensions using a source object or a part, based on the fact that the target function is an output type. The electronic device can determine the dimension of the target object based on the attributes of the source object or a part of the source object.

[0254] For example, an electronic device may determine the dimension of a target object to be 2 dimensions when a user can perceive most of the information output from the source object while viewing the source object from one viewpoint direction. The electronic device may determine the dimension of a target object to be 2 dimensions based on the fact that the source object is at least substantially a 2-dimensional object (e.g., an image, a picture frame). The electronic device may determine the dimension of a target object to be 3 dimensions when a user can perceive only a portion of the information output from the source object while viewing the source object from one viewpoint direction, and it is necessary to view from another viewpoint direction to perceive other information.

[0255] For example, the electronic device may determine the dimensions of the target object to be two dimensions when a picture frame, image, display, clock, or paper is selected as the source object (or part of the source object). The electronic device may determine the dimensions of the target object to be three dimensions when a doll, vase (1520), ornament, sculpture, or model is selected as the source object (or part of the source object).

[0256] If the electronic device determines the dimensions of the target object to be three dimensions, it can create a three-dimensional object (e.g., a solid object) as the target object that occupies a three-dimensional space. If the electronic device determines the dimensions of the target object to be two dimensions, it can create a two-dimensional object (e.g., a planar object) as the target object that occupies a two-dimensional planar area.

[0257] The electronic device can create a target object that mimics a part of a determined target object with the dimensions of the target object. If the target object is a 3D object, the electronic device can change the orientation (e.g., orientation) of the target object based on a user's rotation input that changes the direction of the target object. As a result, the user can observe the target object created as a 3D object from various viewpoint directions.

[0258] According to one embodiment, an electronic device may generate a target object based on the result of supplementing a portion of a source object. For example, based on the fact that the target function is an output type, if the portion includes an occluded area, the electronic device may obtain a target object that includes the result of supplementing the occluded area. The electronic device may obtain additional data if at least a portion of the source object or a portion of the source object is occluded. The additional data may mean data containing information about the source object or a portion of the source object. For example, the additional data may include an image in which the occluded area appears in the display area. The electronic device may retrieve the additional data from the internal memory of the electronic device or an external database. Based on obtaining the additional data, the electronic device may generate a target object that mimics even the occluded area using the additional data.

[0259] In FIG. 15, on the screen (1501), the electronic device may select a calendar (1510) placed in space as a source object. The electronic device may determine the target function as a calendar output function. As shown in FIG. 15, a part of the calendar (1510) may be obscured by a vase (1520). The electronic device may create a virtual calendar object (1530) that mimics the calendar (1510) as a target object, and may supplement the obscured part of the virtual calendar object (1530) using additional data. On the screen (1502), the electronic device may display the virtual calendar object (1530).

[0260] An electronic device (101; 201; 301; 401; 501) includes a display; at least one processor including a processing circuit; and a memory including one or more storage media for storing instructions. When the instructions are executed by the at least one processor, the electronic device (101; 201; 301; 401; 501) may obtain a user input selecting a portion of a source object (711) placed in a space (500), determine a target function based on the portion of the selected source object (711) among at least one candidate function of the source object (711), obtain a target object (712) corresponding to the target function, determine the location of the obtained target object (712) within the space (500) based on the target function, and display the target object (712) at the determined location.

[0261] When the above instructions are executed by the at least one processor, the electronic device (101; 201; 301; 401; 501) may be made to create the target object (712) for providing the target function by applying an object creation model to information regarding the portion of the source object (711).

[0262] When the above instructions are executed by the at least one processor, the electronic device (101; 201; 301; 401; 501) may be made to detect another object providing the target function within the display area of ​​the display, determine the position of the target object (712) at a position aligned with the other object based on the fact that the distance between the source object (711) and the electronic device (101; 201; 301; 401; 501) is less than or equal to the distance between the other object and the electronic device (101; 201; 301; 401; 501), and change the position of the target object (712) while maintaining a relative position relationship between the other object and the target object (712) in response to a change in the position of the other object.

[0263] When the above instructions are executed by the at least one processor, the electronic device (101; 201; 301; 401; 501) may determine a position change amount for the position of the other object based on detecting the other object providing the target function within the display area of ​​the display, and determine the result of applying the determined position change amount to the position of the other object in the world coordinate system as the position of the target object (712).

[0264] When the above instructions are executed by the at least one processor, the electronic device (101; 201; 301; 401; 501) may be made to output a notification that induces the movement of the electronic device (101; 201; 301; 401; 501) so that the other object providing the target function is located within the display area based on the fact that the other object is located within the space (500) and not within the display area.

[0265] The above instructions, when executed by the at least one processor, may cause the electronic device (101; 201; 301; 401; 501) to check whether there is an input object for obtaining user input within the display area of ​​the display based on the fact that the target function is an input type, and to determine the position of the target object (712) at a position aligned with the input object based on the fact that the input object exists within the display area.

[0266] When the above instructions are executed by the at least one processor, the electronic device (101; 201; 301; 401; 501) may select an input object based on at least one of the correlations of the plurality of candidate input objects to the target function, the location of each candidate input object, or the usage history information of each candidate input object, based on the existence of a plurality of candidate input objects within the display area of ​​the display, and determine the location of the target object (712) at a location aligned with the selected input object.

[0267] When the above instructions are executed by the at least one processor, the electronic device (101; 201; 301; 401; 501) can determine the position of the target object (712) aligned with the input object based on the fact that the target function is an input type, obtain a control input for the source object (711) through the input object, and transmit the control input to the source object (711), thereby enabling the source object (711) to be controlled based on the control input.

[0268] The above instructions, when executed by the at least one processor, may cause the electronic device (101; 201; 301; 401; 501) to determine the location of the target object (712) based on the location of the electronic device (101; 201; 301; 401; 501) based on the fact that there is no other object providing the target function within the display area of ​​the display.

[0269] The above instructions, when executed by the at least one processor, may cause the electronic device (101; 201; 301; 401; 501) to determine a position change amount for the position of the electronic device (101; 201; 301; 401; 501) based on the fact that there is no other object providing the target function within the display area of ​​the display, and to determine the result of applying the determined position change amount for the position of the electronic device (101; 201; 301; 401; 501) in the device coordinate system as the position of the target object (712).

[0270] The above instructions, when executed by the at least one processor, may cause the electronic device (101; 201; 301; 401; 501) to determine the position of the target object (712) based on the position of the electronic device (101; 201; 301; 401; 501) based on the fact that there is no other object providing the target function within the display area of ​​the display, and after determining the position of the target object (712) based on the position of the electronic device (101; 201; 301; 401; 501), change the position of the target object (712) to a position aligned with the other object based on the fact that another object providing the target function appears in the display area of ​​the display according to the movement of the electronic device (101; 201; 301; 401; 501).

[0271] The above instructions, when executed by the at least one processor, may cause the electronic device (101; 201; 301; 401; 501) to obtain the target object (712) including the result of supplementing the occluded area, where the part includes the occluded area, based on the fact that the target function is an output type.

[0272] The above instructions, when executed by the at least one processor, may cause the electronic device (101; 201; 301; 401; 501) to determine the dimension of the target object (712) as either three dimensions or two dimensions using the source object (711) or the part, based on the fact that the target function is an output type, and to create the target object (712) that emulates the part with the determined dimension of the target object (712).

[0273] The above instructions, when executed by the at least one processor, may cause the electronic device (101; 201; 301; 401; 501) to perform at least one of determining the target function, obtaining the target object (712), determining the location of the target object (712), or displaying the target object (712) based on obtaining a movement input that pulls the selected part following the user input.

[0274] When the above instructions are executed by the at least one processor, the electronic device (101; 201; 301; 401; 501) may determine the location of the target object (712) having a representative depth determined for the user based on gaze history information regarding the user's gaze.

[0275] When the above instructions are executed by the at least one processor, the electronic device (101; 201; 301; 401; 501) may determine the depth at which the user gazes for the longest time as the representative depth for the user based on gaze history information regarding the user's gaze.

[0276] When the above instructions are executed by the at least one processor, the electronic device (101; 201; 301; 401; 501) may determine the depth that the user gazed at before selecting a part of the source object (711) as the representative depth for the user, based on gaze history information regarding the user's gaze.

[0277] When the above instructions are executed by the at least one processor, the electronic device (101; 201; 301; 401; 501) may determine the location of the target object (712) based on additional user input.

[0278] A method performed by an electronic device (101; 201; 301; 401; 501) may include: acquiring a user input selecting a portion of a source object (711) placed in a space (500); determining a target function based on the selected portion of the source object (711) among at least one candidate function of the source object (711); acquiring a target object (712) corresponding to the target function and determining the location of the acquired target object (712) within the space (500) based on the target function; and displaying the target object (712) at the determined location.

[0279] The electronic device according to the various embodiments disclosed in this document may be of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. The electronic device according to the embodiments of this document is not limited to the devices described above.

[0280] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as "coupled" or "connected" to another (e.g., 2nd) component, with or without the terms "functionally" or "communicationly," it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0281] The term “module” as used in the various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0282] Various embodiments of the present document may be implemented as software (e.g., program (140)) comprising one or more instructions stored in a storage medium (e.g., internal memory (136) or external memory (138)) readable by a machine (e.g., electronic device (101)). For example, a processor (e.g., processor (120)) of the machine (e.g., electronic device (101)) may call at least one of the one or more instructions stored in the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0283] According to one embodiment, the method according to the various embodiments disclosed herein may be provided as included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0284] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0285] The embodiments described above may be implemented as hardware components, software components, and / or combinations of hardware and software components. For example, the devices, methods, and components described in the embodiments may be implemented using a general-purpose computer or a special-purpose computer, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. In addition, other processing configurations, such as parallel processors, are also possible.

[0286] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or command the processing unit independently or collectively. Software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, or computer storage medium or device so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and may be stored or executed in a distributed manner. Software and data may be stored on computer-readable recording media.

[0287] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and may be recorded on a computer-readable medium. The computer-readable medium may contain program instructions, data files, or data structures alone or in combination, and the program instructions recorded on the medium may be those specifically designed and configured for the embodiment or may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, or flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter.

[0288] The hardware device described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

Claims

1. In an electronic device (101; 201; 301; 401; 501), Display(160; 205; 210; 320; 421); At least one processor (120) including a processing circuit; and It includes a memory (130) comprising one or more storage media for storing instructions, and When the above instructions are executed by the at least one processor (120), the electronic device (101; 201; 301; 401; 501) is made to, Obtaining user input to select a part of a source object (711) placed in space (500), Among at least one candidate function of the source object (711), a target function is determined based on the portion of the selected source object (711), and A target object (712) corresponding to the above target function is obtained, Based on the above target function, the location of the acquired target object (712) within the space (500) is determined, and Display the target object (712) at the determined location above. making, Electronic device (101; 201; 301; 401; 501).

2. In Paragraph 1, When the above instructions are executed by the at least one processor (120), the electronic device (101; 201; 301; 401; 501) is made to, By applying an object creation model to the information regarding the above part of the source object (711), the target object (712) for providing the target function is created. making, Electronic device (101; 201; 301; 401; 501).

3. In any one of paragraphs 1 to 2, When the above instructions are executed by the at least one processor (120), the electronic device (101; 201; 301; 401; 501) is made to, Detecting another object providing the target function within the display area of ​​the above display (160; 205; 210; 320; 421), and Based on the fact that the distance between the source object (711) and the electronic device (101; 201; 301; 401; 501) is less than or equal to the distance between the other object and the electronic device (101; 201; 301; 401; 501), the position of the target object (712) is determined to be aligned with the other object, and In response to a change in the position of the other object, change the position of the target object (712) while maintaining a relative position relationship between the other object and the target object (712). making, Electronic device (101; 201; 301; 401; 501).

4. In any one of paragraphs 1 through 3, When the above instructions are executed by the at least one processor (120), the electronic device (101; 201; 301; 401; 501) is made to, Based on detecting another object providing the target function within the display area of ​​the above display (160; 205; 210; 320; 421), the amount of position change for the position of the other object is determined, and The result of applying the determined position change amount to the position of the other object in the world coordinate system is determined as the position of the target object (712). making, Electronic device (101; 201; 301; 401; 501).

5. In any one of paragraphs 1 through 4, When the above instructions are executed by the at least one processor (120), the electronic device (101; 201; 301; 401; 501) is made to, Based on the fact that another object providing the target function exists within the space (500) and not within the display area, output a notification that induces movement of the electronic device (101; 201; 301; 401; 501) so that the other object is located within the display area. making, Electronic device (101; 201; 301; 401; 501).

6. In any one of paragraphs 1 through 5, When the above instructions are executed by the at least one processor (120), the electronic device (101; 201; 301; 401; 501) is enabled, Based on the fact that the above target function is an input type, check whether there exists an input object for obtaining user input within the display area of ​​the above display (160; 205; 210; 320; 421), and Based on the existence of the input object within the display area, the position of the target object (712) is determined to be aligned with the input object. making, Electronic device (101; 201; 301; 401; 501).

7. In any one of paragraphs 1 through 6, When the above instructions are executed by the at least one processor (120), the electronic device (101; 201; 301; 401; 501) is enabled, Based on the existence of a plurality of candidate input objects within the display area of ​​the above display (160; 205; 210; 320; 421), an input object is selected based on at least one of the relevance of the plurality of candidate input objects to the target function, the location of each candidate input object, or the usage history information of each candidate input object. Determine the position of the target object (712) at a position aligned with the selected input object. making, Electronic device (101; 201; 301; 401; 501).

8. In any one of paragraphs 1 through 7, When the above instructions are executed by the at least one processor (120), the electronic device (101; 201; 301; 401; 501) is made to, Based on the fact that the above target function is an input type, the position of the target object (712) is determined as a position aligned with the input object, and then a control input for the source object (711) is obtained through the input object, and By transmitting the above control input to the source object (711), the source object (711) is controlled based on the control input. making, Electronic device (101; 201; 301; 401; 501).

9. In any one of paragraphs 1 through 8, When the above instructions are executed by the at least one processor (120), the electronic device (101; 201; 301; 401; 501) is enabled, Based on the fact that no other object providing the target function exists within the display area of ​​the display (160; 205; 210; 320; 421), the position of the target object (712) is determined based on the position of the electronic device (101; 201; 301; 401; 501). making, Electronic device (101; 201; 301; 401; 501).

10. In any one of paragraphs 1 through 9, When the above instructions are executed by the at least one processor (120), the electronic device (101; 201; 301; 401; 501) is enabled, Based on the fact that there is no other object providing the target function within the display area of ​​the above display (160; 205; 210; 320; 421), a change in position of the electronic device (101; 201; 301; 401; 501) is determined, and The result of applying the determined position change amount to the position of the electronic device (101; 201; 301; 401; 501) in the device coordinate system is determined as the position of the target object (712). making, Electronic device (101; 201; 301; 401; 501).

11. In any one of paragraphs 1 through 10, When the above instructions are executed by the at least one processor (120), the electronic device (101; 201; 301; 401; 501) is enabled, Based on the fact that there is no other object providing the target function within the display area of ​​the above display (160; 205; 210; 320; 421), the location of the target object (712) is determined based on the location of the electronic device (101; 201; 301; 401; 501), and After determining the position of the target object (712) based on the position of the electronic device (101; 201; 301; 401; 501), the position of the target object (712) is changed to a position aligned with the other object based on the fact that another object providing the target function appears in the display area of ​​the display (160; 205; 210; 320; 421) according to the movement of the electronic device (101; 201; 301; 401; 501). making, Electronic device (101; 201; 301; 401; 501).

12. In any one of paragraphs 1 through 11, When the above instructions are executed by the at least one processor (120), the electronic device (101; 201; 301; 401; 501) is enabled, Based on the fact that the above target function is an output type and the above part includes an occluded area, the target object (712) including the result of supplementing the occluded area is obtained. making, Electronic device (101; 201; 301; 401; 501).

13. In any one of paragraphs 1 through 12, When the above instructions are executed by the at least one processor (120), the electronic device (101; 201; 301; 401; 501) is enabled, Based on the fact that the above target function is an output type, the dimension of the target object (712) is determined to be either 3 dimensions or 2 dimensions using the source object (711) or the above part, and Create the target object (712) that emulates the part having the dimensions of the determined target object (712) above. making, Electronic device (101; 201; 301; 401; 501).

14. A method performed by an electronic device (101; 201; 301; 401; 501), An action of obtaining user input to select a part of a source object (711) placed in space (500); Among at least one candidate function of the source object (711), an operation of determining a target function based on the portion of the selected source object (711); The operation of acquiring a target object (712) corresponding to the above target function and determining the position of the acquired target object (712) within the space (500) based on the above target function; and The operation of displaying the target object (712) at the determined position above, method.

15. A computer-readable recording medium storing one or more computer programs containing instructions for performing the method of paragraph 14.

Images