Electronic device and operation method thereof

Abstract

The present disclosure provides a method performed by an HMD device. The method comprises the steps of: identifying a first control target device included in an external image of the HMD device obtained by means of a second camera of the HMD device on the basis of gaze information about a user of the HMD device obtained by means of a first camera of the HMD device; displaying a first graphic element for executing a control function of the first control target device at a position associated with the first control target device; displaying a second graphic element indicating that the first control target device is selected on the basis of selection of the first graphic element, and a third graphic element for executing a control function of a second control target device; displaying a fourth graphic element indicating that the second control target device is selected on the basis of selection of the third graphic element; and displaying a fifth graphic element for executing an integrated control function of the first control target device and the second control target device.

Description

Electronic device and method of operation thereof

[0001] The present disclosure relates to an electronic device and a method of operation.

[0002] With the advancement of digital technology, electronic devices are being provided in various forms, such as smartphones, tablet PCs, or PDAs. Electronic devices are also being developed in wearable forms to enhance portability and user accessibility.

[0003] Electronic devices developed in a form that users can wear are being developed in the form of wearable electronic devices such as AR glasses (augmented reality glasses), VST (video see-through) devices, and HMD (head-mounted display) devices to provide virtual spaces in virtual environments, and the various services and additional functions provided by wearable electronic devices are gradually increasing. In order to enhance the utility value of these electronic devices and satisfy the needs of diverse users, telecommunications service providers or electronic device manufacturers are competitively developing electronic devices to provide various functions and differentiate themselves from other companies. Accordingly, the various functions provided through wearable electronic devices are also becoming increasingly sophisticated.

[0004] AR glasses or VST devices can provide a realistic experience to the user by displaying virtual images while worn on the user's body. AR glasses or VST devices can replace the usability of smartphones in various fields such as game entertainment, education, and social networking services (SNS). Users can receive content similar to reality through AR glasses or VST devices and feel as if they are staying in a virtual world through interaction.

[0005] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art related to the present disclosure.

[0006] According to one embodiment, an HMD device may be provided. The HMD device may include at least one processor comprising a camera; a display; a communication circuit; and a processing circuit; and a memory comprising at least one storage medium for storing instructions. The instructions may cause the HMD device to perform at least one operation when executed individually or collectively by the at least one processor. The at least one operation may include an operation of identifying a first control target device included in an external image of the HMD device obtained through a second camera of the HMD device based on gaze information of the user of the HMD device obtained through a first camera of the HMD device. The at least one operation may include an operation of displaying a first graphic element for executing a control function of the first control target device at a location associated with the first control target device. The at least one operation may include an operation of displaying a second graphic element indicating that the first control target device is selected and a third graphic element for executing a control function of a second control target device different from the first control target device, based on the selection of the first graphic element. The at least one operation may include an operation of displaying a fourth graphic element indicating that the second control target device is selected based on the selection of the third graphic element. The at least one operation may include an operation of displaying a fifth graphic element for executing an integrated control function of the first control target device and the second control target device.

[0007] According to one embodiment, a method of operation of an HMD device may be provided. The method of operation of the HMD device may include at least one operation. The at least one operation may include an operation of identifying a first control target device included in an external image of the HMD device obtained through a second camera of the HMD device, based on gaze information of the user of the HMD device obtained through a first camera of the HMD device. The at least one operation may include an operation of displaying a first graphic element for executing a control function of the first control target device at a location associated with the first control target device. The at least one operation may include an operation of displaying a second graphic element indicating that the first control target device is selected and a third graphic element for executing a control function of a second control target device different from the first control target device, based on the selection of the first graphic element. The at least one operation may include an operation of displaying a fourth graphic element indicating that the second control target device is selected based on the selection of the third graphic element. The at least one operation may include an operation of displaying a fifth graphic element for executing an integrated control function of the first control target device and the second control target device.

[0008] According to one embodiment, a storage medium storing at least one instruction readable by a computer may be provided. The at least one instruction may cause the HMD device to perform at least one operation when executed by at least a part of at least one processor of the HMD device. The at least one operation may include an operation of identifying a first control target device included in an external image of the HMD device obtained through a second camera of the HMD device based on gaze information of the user of the HMD device obtained through a first camera of the HMD device. The at least one operation may include an operation of displaying a first graphic element for executing a control function of the first control target device at a location associated with the first control target device. The at least one operation may include an operation of displaying a second graphic element indicating that the first control target device is selected and a third graphic element for executing a control function of a second control target device different from the first control target device, based on the selection of the first graphic element. The at least one operation may include an operation of displaying a fourth graphic element indicating that the second control target device is selected based on the selection of the third graphic element. The at least one operation may include an operation of displaying a fifth graphic element for executing an integrated control function of the first control target device and the second control target device.

[0009] In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components.

[0010] FIG. 1 is a block diagram of an electronic device in a network environment according to various embodiments of the present disclosure.

[0011] FIG. 2 is a drawing showing the configuration of a wearable electronic device according to one embodiment of the present disclosure.

[0012] FIGS. 3a to 3c are drawings showing the front and rear views of a wearable electronic device according to one embodiment of the present disclosure.

[0013] FIG. 4 is a flowchart illustrating a method for an HMD device to register an external electronic device according to one embodiment of the present disclosure.

[0014] FIG. 5 is a flowchart illustrating a method for an HMD device to control a device to be controlled, according to one embodiment of the present disclosure.

[0015] FIG. 6 is a diagram illustrating the operation of an HMD device providing graphic elements for controlling a device to be controlled, according to one embodiment of the present disclosure.

[0016] FIG. 7 is a diagram illustrating a method for an HMD device to register a controlled device according to one embodiment of the present disclosure.

[0017] FIGS. 8a to 8c are drawings for explaining a method for an HMD device to register a control target device using received signal information according to one embodiment of the present disclosure.

[0018] FIG. 9 is a diagram illustrating a method for an HMD device to store spatial map information along with the registration of a controlled device, according to one embodiment of the present disclosure.

[0019] FIG. 10 is a drawing for explaining a method for an HMD device to synthesize spatial map information according to one embodiment of the present disclosure.

[0020] FIG. 11 is a diagram illustrating the operation of an HMD device providing graphic elements for controlling a device to be controlled, according to one embodiment of the present disclosure.

[0021] FIG. 12 is a diagram illustrating an operation in which an HMD device provides graphic elements for controlling a device to be controlled using usage pattern information, according to one embodiment of the present disclosure.

[0022] FIGS. 13a and 13b are drawings for explaining the operation of an HMD device providing graphic elements for controlling a target device using spatial map information according to one embodiment of the present disclosure.

[0023] FIGS. 14a to 14c are drawings for explaining the operation of an HMD device providing graphic elements for integrated control of a plurality of control target devices according to one embodiment of the present disclosure.

[0024] FIG. 15 is a diagram illustrating the operation of an HMD device providing graphic elements for integrated control of a plurality of control target devices according to one embodiment of the present disclosure.

[0025] FIGS. 16a and 16b are drawings for illustrating an operation in which an HMD device switches modes of controlling a device to be controlled, according to one embodiment of the present disclosure.

[0026] FIGS. 17a and 17b are drawings for explaining the operation of an HMD device changing a controlled target device according to one embodiment of the present disclosure.

[0027] FIGS. 18a to 18c are drawings for explaining a method in which an HMD device controls a device to be controlled, according to one embodiment of the present disclosure.

[0028] FIG. 19 is a flowchart illustrating a method for an HMD device to control a device to be controlled, according to one embodiment of the present disclosure.

[0029] Hereinafter, embodiments of the present disclosure are described in detail with reference to the drawings so that those skilled in the art can easily practice them. However, the present disclosure may be embodied in various different forms and is not limited to the embodiments described herein. In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and brevity.

[0030] FIG. 1 is a block diagram of an electronic device in a network environment according to various embodiments of the present disclosure.

[0031] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or with an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through a server (108). According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (160), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)).

[0032] The processor (120) can control at least one other component (e.g., hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., program (140)), for example, and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., sensor module (176) or communication module (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., central processing unit or application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., graphics processing unit, neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use lower power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.

[0033] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence is performed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0034] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, input data or output data for software (e.g., program (140)) and related commands. The memory (130) may include volatile memory (132) or non-volatile memory (134).

[0035] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0036] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0037] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0038] The display module (160) can visually provide information to an external (e.g., user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0039] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (101).

[0040] The sensor module (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0041] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0042] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0043] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0044] The camera module (180) can capture still images and video. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0045] The power management module (188) can manage the power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0046] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0047] The communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a communication module (192) (e.g., cellular communication module, short-range communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The communication module (192) can identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).

[0048] The communication module (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The communication module (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The communication module (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The communication module (192) can support various requirements specified in the electronic device (101), external electronic device (e.g., electronic device (104)), or network system (e.g., second network (199)). According to one embodiment, the communication module (192) can support a Peak data rate (e.g., 20 Gbps or more) for eMBB realization, loss coverage (e.g., 164 dB or less) for mMTC realization, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for URLLC realization.

[0049] An antenna module (197) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197).

[0050] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0051] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface) and exchange signals (e.g., commands or data) with each other.

[0052] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In one embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (104) or the server (108) may be included within a second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0053] The number of processors (120) may be one or more. For example, the processor (120) may have the structure of a multi-core processor such as a dual core, a quad core, or a hexa core.

[0054] The processor (120) can control the operations of the electronic device (101) by executing instructions stored in memory (130). For example, the processor (120) may correspond to a plurality of processors that divide and collectively perform a plurality of operations among the processors.

[0055] FIG. 2 is a drawing showing the configuration of a wearable electronic device according to one embodiment of the present disclosure.

[0056] Referring to FIG. 2, according to one embodiment, a wearable electronic device (200) (e.g., the electronic device (101) of FIG. 1) may include a light output module (211), a display member (201), a camera module (250), and / or a speaker (261). According to one embodiment, the wearable electronic device (200) may include at least one additional configuration other than the configuration shown. For example, the wearable electronic device (200) may include all or part of the configuration included in the electronic device (101) of FIG. 1.

[0057] According to one embodiment, a light output module (211) (e.g., a display module (160) of FIG. 1) may include a light source capable of outputting an image and a lens that guides the image to a display member (201). The light output module (211) may include, for example, a liquid crystal display, a digital mirror device, a liquid crystal on silicon display, an organic light emitting diode and / or a micro light emitting diode (micro LED).

[0058] According to one embodiment, a display member (201) (e.g., a display module (160) of FIG. 1) may include an optical waveguide (e.g., a waveguide). According to one embodiment, an output image of an optical output module (211) incident on one end of the optical waveguide may propagate within the optical waveguide and be provided to a user. According to one embodiment, the optical waveguide may include at least one diffractive element (e.g., a diffractive optical element (DOE), a holographic optical element (HOE)) and / or a reflective element (e.g., a reflective mirror). For example, the optical waveguide may guide the output image of the optical output module (211) to the user's eye using at least one diffractive element or reflective element.

[0059] According to one embodiment, a camera module (250) (e.g., camera module (180) of FIG. 1) can capture images (e.g., still images and / or video). According to one embodiment, the camera module (250) may be placed within a lens frame and around a display member (201). In the present disclosure, images may be interpreted to include video as well as still images.

[0060] According to one embodiment, the first camera module (251) can capture and / or recognize the trajectory of the user's eye (e.g., pupil, iris) or gaze. According to one embodiment, the first camera module (251) can periodically or non-periodically transmit information related to the trajectory of the user's eye or gaze (e.g., trajectory information) to a processor (e.g., processor (120) of FIG. 1).

[0061] According to one embodiment, the second camera module (253) can capture an external image. For example, the second camera module (253) can capture an image of the external environment in the front direction of the wearable electronic device (200).

[0062] According to one embodiment, the third camera module (255) may be used for hand detection and tracking and user gesture (e.g., hand movements) recognition. According to one embodiment, the third camera module (255) may be used for 3 degrees of freedom (3DoF) and 6DoF head tracking, location (space, environment) recognition, and / or movement recognition. According to one embodiment, the second camera module (253) may be used for hand detection and tracking and user gesture recognition. According to one embodiment, at least one of the first camera module (251) to the third camera module (255) may be replaced with a sensor module (e.g., LiDAR sensor). For example, the sensor module may include at least one of a vertical cavity surface emitting laser (VCSEL), a diode, an infrared sensor, an infrared diode, and / or a photodiode.

[0063] According to one embodiment, a speaker (261) (e.g., the acoustic output module (155) of FIG. 1) can output an acoustic signal (e.g., sound and / or virtual vibration sound). Although the speaker (261) is described as being configured in a member that is mounted on the user's ear when the wearable electronic device (200) is worn in FIG. 2, it is not limited thereto and may be configured in other locations depending on the implementation of the wearable electronic device (200).

[0064] FIGS. 3a to 3c are drawings showing the front and rear views of a wearable electronic device according to one embodiment of the present disclosure.

[0065] Referring to FIGS. 3a through 3c, according to one embodiment, at least one first camera module (311, 312) and at least one second camera module (313, 314, 315, 316), a depth sensor (317), and / or a second display (350) may be disposed on a first surface (310) of a housing for acquiring information related to the surrounding environment of a wearable electronic device (300) (e.g., the electronic device (101) of FIG. 1). According to one embodiment, the wearable electronic device (300) may include at least one additional configuration other than the configuration shown. For example, the wearable electronic device (300) may include all or part of the configuration included in the electronic device (101) of FIG. 1.

[0066] According to one embodiment, at least one first camera module (311, 312) can capture an image of the outside of the wearable electronic device (300). For example, the first camera module (311, 312) can capture an image of the external environment in the front direction of the wearable electronic device (300).

[0067] According to one embodiment, at least one second camera module (313, 314, 315, 316) can acquire images while the wearable electronic device (300) is worn by a user. The second camera module (313, 314, 315, 316) may be used for hand detection, tracking, and user gesture (e.g., hand movements) recognition. The second camera module (313, 314, 315, 316) may be used for 3DoF, 6DoF head tracking, location (space, environment) recognition, and / or movement recognition. According to one embodiment, a first camera module (311, 312) may be used for hand detection and tracking and user gestures.

[0068] According to one embodiment, the depth sensor (317) may be configured to transmit a signal and receive a signal reflected from a subject, and may be used for purposes such as time of flight (TOF) to determine the distance to an object. Alternatively, or additionally, a second camera module (313, 314, 315, 316) may determine the distance to an object.

[0069] According to one embodiment, the second display (350) (and / or lens) may be placed on the first surface (310) of the wearable electronic device (300). According to one embodiment, the second display (350) may provide visual information to the outside of the wearable electronic device (300). For example, the second display (350) may be used to provide an alternative notification indicating the operating status of the first camera module (311, 312) in place of the light emitter (340).

[0070] According to one embodiment, camera modules (325, 326) for face recognition and / or a first display (321) (and / or a lens) may be disposed on the second surface (320) of the housing.

[0071] According to one embodiment, camera modules (325, 326) for face recognition adjacent to the first display (321) may be used to recognize the user's face or to recognize and / or track both of the user's eyes.

[0072] According to one embodiment, the first display (321) (and / or lens) may be disposed on the second surface (320) of the wearable electronic device (300). According to one embodiment, the wearable electronic device (300) may not include camera modules (315, 316) among a plurality of second camera modules (313, 314, 315, 316). Although not illustrated in FIG. 3a and 3b, the wearable electronic device (300) may further include at least one of the configurations illustrated in FIG. 2.

[0073] Referring to FIG. 3c, according to one embodiment, the wearable electronic device (300) may have a form factor (e.g., a head-mounted display (HMD)) for being worn on a user's head. The wearable electronic device (300) may further include a strap and / or a wearing member for being secured on a part of the user's body. The wearable electronic device (300) may include a volume button (331), a vent (333), a status indicator (335), and a power button (e.g., including a fingerprint recognition sensor) (337), and such configurations may be identically included in the wearable electronic device (300) illustrated in FIG. 3a and FIG. 3b. When worn on a user's head, it may provide a user experience based on augmented reality, virtual reality, and / or extended reality (or mixed reality). The wearable electronic device (300) configured in the form of an HMD may include configurations identical or similar to the components of FIG. 3a and FIG. 3b described above.

[0074] According to one embodiment, a speaker (318) (e.g., the acoustic output module (155) of FIG. 1 or the speaker (261) of FIG. 2) may output an acoustic signal (e.g., sound and / or virtual vibration sound). Although the speaker (318) has been described as being configured in a location adjacent to the vent (333) in FIG. 3a through 3c as an example, it is not limited thereto and may be configured in other locations depending on the implementation of the wearable electronic device (200).

[0075] According to one embodiment of the present disclosure, an HMD device (e.g., the electronic device (101) of FIG. 1, the wearable electronic device (200) of FIG. 2, or the wearable electronic device (300) of FIG. 3a to 3c) may provide a function for using an external electronic device (e.g., a home appliance such as an air conditioner or an air purifier). The function for using an external electronic device may include, for example, a function for registering an external electronic device and / or a function for controlling an external electronic device, but is not limited thereto. By providing a function for using an external electronic device through such an HMD device, a user can easily and conveniently use various external electronic devices.

[0076] FIG. 4 is a flowchart illustrating a method for an HMD device to register an external electronic device according to one embodiment of the present disclosure.

[0077] According to one embodiment, an HMD device (e.g., the electronic device (101) of FIG. 1, the wearable electronic device (200) of FIG. 2, or the wearable electronic device (300) of FIG. 3a to 3c) may register an external electronic device (e.g., the electronic device (101), the electronic device (102), or the electronic device (104) of FIG. 1) based on gaze information of a user wearing the HMD device, image information outside the HMD device (e.g., image information in the front direction of the HMD device), and / or received signal information. Registration of an external electronic device may mean, for example, registering the device to a service linked to a user account (e.g., Samsung’s SmartThings service). The registered external electronic device may be managed and controlled through an application associated with the said service. In the present disclosure, the external electronic device may be referred to as a device to be controlled.

[0078] Referring to FIG. 4, in operation 410, an HMD device (e.g., electronic device (101) of FIG. 1, wearable electronic device (200) of FIG. 2, or wearable electronic device (300) of FIG. 3a to 3c) can recognize a device to be registered (e.g., a device to be controlled) based on gaze information of a user wearing the HMD device. According to one embodiment, the HMD device can perform operation 410 in response to the activation of a device registration mode (device registration mode).

[0079] According to one embodiment, the HMD device can recognize a control target device to be registered through object analysis based on gaze information obtained through a first camera (e.g., the first camera module (251) of FIG. 2 or the face recognition camera modules (325, 326) of FIG. 3a), image information obtained through a second camera (e.g., the second camera module (253) of FIG. 2 or the first camera module (311, 312) of FIG. 3a) and / or received signal information obtained through a communication circuit (e.g., the communication module (190) of FIG. 1). Object analysis may be performed using a learned model (e.g., an AI model), but is not limited thereto.

[0080] According to one embodiment, gaze information is information about the gaze of a user wearing an HMD device, and may include, for example, information about a point where the user's gaze rests, information about the direction of the user's gaze, information about the time the user's gaze rests at a specific point, and / or information about the movement of the user's gaze. Image information may include, for example, information about an image in the front direction of the HMD device. Image information may be used for vision recognition. Received signal information may include, for example, information on the reception strength of a signal received by the HMD device (hereinafter referred to as the received signal) (e.g., RSSI (received signal strength indicator) value, RSRP (reference signal received power) value), information on the reception angle of the received signal, and / or identification information of a control target device included in the received signal. The received signal may be, for example, a Wi-Fi signal transmitted from a control target device.

[0081] According to one embodiment, the HMD device can use gaze information and image information to acquire an image portion of a device located in the direction of the user's gaze, and determine whether the device is a registrable device through analysis of the image portion (e.g., object analysis using vision recognition). According to one embodiment, the HMD device can use gaze information and received signal information to identify a device located in the direction of the user's gaze but not visible in the user's field of vision (e.g., a device not recognized through vision recognition based on image information), and determine whether the device is a registrable device through analysis of the identified device (e.g., object analysis). If the device is identified as a registrable device, the HMD device can recognize the device as a control target device to be registered.

[0082] In operation 420, the HMD device can identify whether network connection information for the device to be controlled exists. According to one embodiment, the network connection information may include information used for network settings for the registration of the device to be controlled. For example, the network connection information may include access point (AP) information (e.g., Wi-Fi network and password). According to one embodiment, the HMD device can identify whether network connection information for the device to be controlled exists through communication with a server providing a service for device registration (e.g., server (108) of FIG. 1) (e.g., a server providing Samsung’s SmartThings service). If it is identified that network connection information for the device to be controlled exists, operation 430 may be performed. If it is identified that network connection information for the device to be controlled does not exist, operation 440 may be performed.

[0083] In operation 430, the HMD device may display a graphic element (hereinafter referred to as the first registration graphic element) for registering a control target device through a display (e.g., the display member (201) of FIG. 2, the first display (321) of FIG. 3a to 3c). The first registration graphic element may include, for example, a virtualization button (registration button) for registering a control target device. In response to the reception of a user input selecting the first registration graphic element (e.g., gesture input (e.g., touch gesture input, grab gesture input, pinch-in gesture input)), the HMD device may register the control target device immediately without a network setup procedure. Thus, if network connection information for the control target device already exists, a separate network setup procedure for registering the control target device is unnecessary, and the HMD device may display the first registration graphic element immediately. In the present disclosure, the graphic element may be defined and referred to as a UI element or a graphic UI element.

[0084] In operation 440, the HMD device may display a graphic element (hereinafter referred to as the second registration graphic element) for providing manual information for setting network connection information for a controlled device. The second registration graphic element may include, for example, a virtualized button (manual provision button) for providing manual information for setting network connection information. In response to the reception of user input selecting the second registration graphic element (e.g., gesture input (e.g., touch gesture input, grab gesture input, pinch-in gesture input)), the HMD device may register the controlled device after performing a network setting procedure through the provision of manual information. Thus, since a network setting procedure for registering the controlled device needs to be performed first when network connection information for the controlled device does not exist, the HMD device may display the second registration graphic element. Through the provision of such manual information, the user can immediately check the method for network setting (or activation) without running a separate application.

[0085] According to one embodiment, the HMD device may register a control target device based on user input through a first registered graphic element or a second registered graphic element (e.g., a gesture input selecting the first registered graphic element or the second registered graphic element). The control target device thus registered may subsequently be controlled by the HMD device.

[0086] According to one embodiment, when registering a device to be controlled, the HMD device may generate spatial map information regarding the space where the HMD device, the user, or the device to be controlled is located. For example, the HMD device may generate spatial map information regarding the space based on image information. The HMD device may generate composite spatial map information by merging the generated spatial map information with at least one other spatial map information. The HMD device may store the generated spatial map information or the composite spatial map information. The spatial map information or the composite spatial map information may be stored in the HMD device or in a server where the device to be controlled is registered. The spatial map information thus stored may be used by the HMD device to control the device to be controlled. For example, the spatial map information may be used to identify and control a device to be controlled that is not visible to the user.

[0087] FIG. 5 is a flowchart illustrating a method for an HMD device to control a device to be controlled, according to one embodiment of the present disclosure.

[0088] FIG. 6 is a diagram illustrating the operation of an HMD device providing graphic elements for controlling a device to be controlled, according to one embodiment of the present disclosure.

[0089] According to one embodiment, the control operation of the device to be controlled by the HMD device (e.g., the electronic device (101) of FIG. 1, the wearable electronic device (200) of FIG. 2, or the wearable electronic device (300) of FIG. 3a to 3c) may be performed after the device to be controlled is registered, for example, through the method of FIG. 4.

[0090] Referring to FIGS. 5 and 6, in operation 510, the HMD device can identify (or recognize) a control target device based on external image information of the HMD device (e.g., image information including an image of the front direction of the HMD device). The external image information may be obtained, for example, through a second camera of the HMD device (e.g., the second camera module (253) of FIG. 2 or the first camera module (311, 312) of FIG. 3a). According to one embodiment, the operation of identifying a control target device may include an operation of selecting a control target device and / or an operation of performing image analysis (e.g., vision analysis) on the control target device.

[0091] According to one embodiment, the operation of selecting a control target device may be performed based on gaze information or user input (e.g., gesture input, voice input). For example, as illustrated in the first scene (601) of FIG. 6, the HMD device may use the user's gaze information to select an object (e.g., a stand-type air conditioner) located in the direction of the user's gaze (601) as a control target device (OB). For example, as illustrated in the first scene (601) of FIG. 6, the HMD device may use the user's gesture input (e.g., a gesture input of drawing a circle) to select an object associated with the gesture input (e.g., an object contained within a circle) as a control target device (OB). The user's gaze information may be obtained, for example, through a first camera (e.g., the first camera module (251) of FIG. 2 or the face recognition camera modules (325, 326) of FIG. 3a).

[0092] According to one embodiment, the operation of performing image analysis on a control target device may be performed based on an image corresponding to a selected control target device within the external image information of the HMD device. For example, as illustrated in FIG. 6, the HMD device may perform image analysis using vision recognition (602) on an image corresponding to a selected control target device. Vision recognition (602) corresponds to a technology that recognizes and understands objects based on images, and can recognize patterns, objects, scenes, and / or actions by processing and analyzing images using a specified algorithm and / or technique (e.g., CNN, Vision Transformer). Through this vision recognition (602), a control target device (e.g., a stand-type air conditioner) may be recognized, and identification information of the recognized control target device may be obtained. The identification information of the control target device may include information such as a model name and / or product ID, such as "AW06C7155GWAZ". Based on the identification information of the control target device, the HMD device may determine whether the control target device is a device that has already been registered. For example, the HMD device can confirm that the controlled device is a registered device if the identification information of the controlled device matches the identification information registered on the server.

[0093] In operation 520, the HMD device may display at least one graphic element on the display for executing a control function of the control target device based on the control target device being identified (or recognized). For example, as illustrated in the second scene (S2) of FIG. 6, the HMD device may display a graphic element (e.g., a graphic element for a remote control function of a stand-type air conditioner) (603) for executing a control function of the control target device (OB) (e.g., a stand-type air conditioner). According to one embodiment, the graphic element (e.g., a virtual remote control) may be displayed at a location adjacent to the control target device. Through the display of such graphic elements, a virtual controller (or remote control) for controlling the control target device may be provided to the user. In this case, the user can easily control the control target device using the virtual controller provided through the display of the HMD device without using a separate physical controller device. According to one embodiment, operation 520 may be performed only when it is confirmed that the control target device is a registered device.

[0094] FIG. 7 is a diagram illustrating a method for an HMD device to register a controlled device according to one embodiment of the present disclosure.

[0095] According to one embodiment, the description of the method for registering a controlled device may include all of the description of FIG. 4.

[0096] Referring to FIG. 7, an HMD device (e.g., the electronic device (101) of FIG. 1, the wearable electronic device (200) of FIG. 2, or the wearable electronic device (300) of FIG. 3a to 3c) can activate a device registration mode (or, device registration mode) (700). According to one embodiment, the HMD device can activate the device registration mode based on user input (e.g., voice input or gesture input).

[0097] According to one embodiment, the HMD device can select a control target device to be registered after the device registration mode is activated. For example, as illustrated in the first scene (S1), the HMD device can select a control target device (OB) (e.g., a stand-type air conditioner) corresponding to the user's gaze (710) based on gaze information acquired after the device registration mode is activated. For example, as illustrated in the first scene (S1), the HMD device can select a control target device (OB) corresponding to a gesture input based on a gesture input (e.g., a circle-drawing gesture) acquired after the device registration mode is activated.

[0098] According to one embodiment, the HMD device can perform an analysis of a selected target device for control. For example, as illustrated in FIG. 7, the HMD device can perform object analysis (713) using a learned AI model based on information (712) obtained through vision recognition of received signal information (711) (e.g., ID or network information of the target device (OB), such as "Airconditioner_0970978") and / or image information (e.g., model name of the target device (OB), such as "AW06C7155GWAZ"). Through this analysis, the HMD device can identify whether the target device for control is a registrable device. If the HMD device identifies that the target device for control is a registrable device, it can confirm that the recognition of the device to be registered is complete.

[0099] According to one embodiment, the HMD device can identify whether network connection information for the controlled device exists based on the identification that the controlled device is a registrable device. For example, in response to the identification that the controlled device is a registrable device, the HMD device can identify whether AP information (e.g., AP information for Wi-Fi communication) for the controlled device exists.

[0100] According to one embodiment, if it is identified that the control target device is a registrable device and that network connection information for the control target device exists, the HMD device may display a graphic element for registering the control target device (hereinafter, the first registration graphic element) through the display. For example, as illustrated in the second scene (S2) of FIG. 7, the HMD device may display a virtual registration button (720) through the display at a location adjacent to the control target device (OB). The HMD device may register the control target device (OB) in response to the reception of user input (e.g., gesture input or voice input) selecting the registration button (720). Thus, if network connection information for the control target device already exists, a separate network setup procedure for registering the control target device is unnecessary, so the HMD device may immediately display the first registration graphic element for registration. For a description of the first registration graphic element, refer to the description of operation 430 of FIG. 4.

[0101] According to one embodiment, if it is identified that the control target device is a registrable device but that network identification information for the control target device does not exist, the HMD device may display a graphic element (hereinafter referred to as the second registration graphic element) for providing manual information for setting network connection information for the control target device. For example, as illustrated in the third scene (S3) of FIG. 7, the HMD device may display a virtual AP activation method manual (730) via a display at a location adjacent to the control target device (OB). In response to receiving user input (e.g., gesture input or voice input) for selecting the AP activation method manual (730), the HMD device may provide a manual for activating the network for the control target device (OB) in stages. Thus, when network connection information for the control target device does not exist, since a network setting procedure for registering the control target device needs to be performed first, the HMD device may display the second registration graphic element for providing a manual for network setting. For a description of the second registration graphic element, refer to the description of operation 440 of FIG. 4.

[0102] FIGS. 8a to 8c are drawings for explaining a method for an HMD device to register a control target device using received signal information according to one embodiment of the present disclosure.

[0103] According to one embodiment, the description of the method for registering a controlled device may include both the description of FIG. 4 and the description of FIG. 7.

[0104] According to one embodiment, an HMD device (e.g., the electronic device (101) of FIG. 1, the wearable electronic device (200) of FIG. 2, or the wearable electronic device (300) of FIG. 3a to 3c) may use received signal information for the registration of a device to be controlled. As described above, the received signal information may include, for example, information on the reception intensity (e.g., RSSI value, RSRP value) of a signal received by the HMD device (hereinafter, received signal), information on the reception angle of the received signal, and / or identification information (e.g., ID) of the device to be controlled included in the received signal. The received signal may be, for example, a Wi-Fi signal transmitted by the device to be controlled. Using this received signal information, the HMD device may obtain information regarding the ID of the device to be controlled that transmitted the received signal, network connection information, location, direction, and / or the distance between the device to be controlled that transmitted the received signal and the HMD device.

[0105] According to one embodiment, the HMD device can use the user's gaze information and received signal information together to accurately identify the control target device to be registered and to check whether network connection information for the control target device exists. For example, as illustrated in FIG. 8a, the HMD device can accurately identify a first control target device (OB1) corresponding to the user's gaze (800a) based on the gaze information and received signal information, and if it is identified that network connection information (e.g., AP information) for the identified first control target device (OB1) exists, it can display a virtual registration button (810a) at a location adjacent to the first control target device (OB1). For example, as illustrated in FIG. 8b, the HMD device can accurately identify a second control target device (OB2) corresponding to the user's gaze (800a) based on gaze information and received signal information, and if it is identified that there is no network connection information (e.g., AP information) for the identified second control target device (OB2), virtual manual information (810b) can be displayed at a location adjacent to the second control target device (OB2).

[0106] According to one embodiment, the HMD device can register a control target device that is not visible in the user's field of vision by using received signal information. For example, as illustrated in FIG. 8c, the HMD device can identify a third control target device (OB3) that is not visible in the user's line of sight (800c) by using received signal information. The identification of the third control target device (OB3) may include, for example, the type of the third control target device (OB3) (e.g., model name, product name), identification information (e.g., ID), network connection information, location and / or direction. In this case, the HMD device can virtually display the shape of the identified third control target device (OB3) and a registration button (810c) through a display at a location adjacent to the third control target device (OB3). The HMD device can register the control target device that is not visible in the user's field of vision in response to the acquisition of user input (e.g., gesture input, voice input) selecting the registration button (810c).

[0107] FIG. 9 is a diagram illustrating a method for an HMD device to store spatial map information along with the registration of a controlled device, according to one embodiment of the present disclosure.

[0108] FIG. 10 is a drawing for explaining a method for an HMD device to synthesize spatial map information according to one embodiment of the present disclosure.

[0109] According to one embodiment, the description of a method for registering a control target device may include all of the descriptions of FIG. 4, FIG. 7, and FIG. 8a to 8c.

[0110] Referring to FIG. 9, an HMD device (e.g., the electronic device (101) of FIG. 1, the wearable electronic device (200) of FIG. 2, or the wearable electronic device (300) of FIG. 3a to 3c) can activate a device registration mode (or, device registration mode) (900). According to one embodiment, the HMD device can activate the device registration mode based on user input (e.g., voice input or gesture input).

[0111] According to one embodiment, the HMD device can select a control target device to be registered after the device registration mode is activated. For example, as illustrated in the first scene (S1), the HMD device can select a control target device (OB) (e.g., a stand-type air conditioner) corresponding to the user's gaze (910) based on gaze information acquired after the device registration mode is activated.

[0112] According to one embodiment, the HMD device can perform an analysis of a selected target device for control. For example, as illustrated in FIG. 9, the HMD device can perform object analysis based on information (712) obtained through vision recognition of received signal information (711) (e.g., ID of the target device for control (OB), such as "Airconditioner_0970978") and / or image information (e.g., model name of the target device for control (OB), such as "AW06C7155GWAZ"). Through this analysis, the HMD device can identify whether the target device for control is a registrable device. If the HMD device identifies that the target device for control is a registrable device, it can confirm that the recognition of the device to be registered is complete.

[0113] According to one embodiment, the HMD device can register the control target device (920) when it is identified that the control target device is a registrable device. For the registration (920) of the control target device, the first registration graphic element (e.g., the registration button (720) of FIG. 7) and the second registration graphic element (e.g., the manual information (730) of FIG. 7) described above may be provided.

[0114] According to one embodiment, when a control target device is registered, the HMD device may generate a spatial map of the space where the HMD device is located and store information about the generated spatial map. For example, as illustrated in FIG. 9, when a control target device (OB) is registered, the HMD device may generate a spatial map (940) by performing spatial recognition (930) of the space where the HMD device is located based on external image information captured through a camera. The spatial map (940) may include an object registered as a control target device. The HMD device may store information about the generated spatial map (940). Information about the spatial map (940) may be stored in the HMD device or in a server where the control target device is registered. The spatial map information thus stored may be used by the HMD device to control the control target device.

[0115] According to one embodiment, when a control target device is registered, the HMD device may synthesize a plurality of spatial maps to generate a composite spatial map and store information about the generated composite spatial map. For example, when a first control target device (OB1) is registered as exemplified in the first scene (S1) of FIG. 10, and / or when a second control target device (OB2) is registered as exemplified in the second scene (S2) of FIG. 10, the HMD device may generate a third spatial map (1030), which is a composite spatial map, by synthesizing a first spatial map (1010) for a first space where the first control target device (OB1) is located and a second spatial map (1020) for a second space where the second control target device (OB2) is located. The HMD device may store information about the generated third spatial map (1030). According to one embodiment, the HMD device can generate a synthetic space map in which each space map is synthesized without spatial sensing (e.g., scanning) of the entire space corresponding to the synthetic space map by performing analysis and recombination of the space maps based on spatial information of each space map and movement information of the HMD device (e.g., movement information including information on the distance moved by the HMD device (or the user wearing the HMD device) and / or the angle rotated by the HMD device). The generation of the synthetic space map can be performed using a trained AI model. According to one embodiment, the space map (940) may include at least one major surrounding object along with an object registered as a control target device. The major surrounding object may include, for example, an object suitable for expressing the characteristics of the space (e.g., a bed, a sofa). For example, as exemplified in the second space map (1020) or the third space map (1030), it may further include an object (e.g., a sofa) suitable for expressing the characteristics of the space (e.g., a living room space) where the second control target device (OB2) is located, along with the second control target device (OB2) being registered.Information regarding the synthetic spatial map stored in this way can be used by the HMD device to control target devices located in a wide range of space. For example, information regarding the synthetic spatial map can be used by the HMD device to identify and control target devices located in space that is not visible to the user's field of view.

[0116] FIG. 11 is a diagram illustrating the operation of an HMD device providing graphic elements for controlling a device to be controlled, according to one embodiment of the present disclosure.

[0117] According to one embodiment, the description of a method for controlling a device to be controlled may include both the descriptions of FIG. 5 and FIG. 6.

[0118] According to one embodiment, the control operation of the device to be controlled by the HMD device (e.g., the electronic device (101) of FIG. 1, the wearable electronic device (200) of FIG. 2, or the wearable electronic device (300) of FIG. 3a to 3c) may be performed after the device to be controlled is registered, for example, through the method of FIG. 4.

[0119] Referring to FIG. 11, the HMD device can select a control target device (OB) to be controlled. For example, as illustrated in the first scene (S1), the HMD device can select a control target device (OB) corresponding to the user's gaze (1101). For example, the HMD device can select an object located at a point where the user's gaze (1101) stays for a specified time (e.g., 3 seconds) or longer as the control target device (OB). The HMD device can recognize the control target device (OB) by performing vision recognition (1102) on the selected control target device (OB). Information about the control target device (OB) can be obtained through vision recognition (1102).

[0120] According to one embodiment, based on the recognition of a control target device (OB), the HMD device may display a first graphic element (1103) for executing a control function of the control target device (OB). For example, in response to the recognition that the control target device (OB) is a registered device, the HMD device may display a first graphic element (1103) for executing a control function of the control target device (OB) at a location associated with the control target device (OB). The first graphic element (1103) may be used to indicate that the control target device (OB) is selectable for control. For example, as illustrated in the third scene (S3), the first graphic element (1103) may include an image portion corresponding to the shape of the control target device (OB).

[0121] According to one embodiment, based on the selection of a first graphic element (1103), the HMD device may display a second graphic element (1104) (e.g., a virtual remote control) for executing a control function of a control target device (OB). For example, in response to the acquisition of user input selecting the first graphic element (1103) (e.g., a gesture input such as a long press, or a voice input), the HMD device may display a second graphic element (1104) for executing a control function of the control target device (OB) at a location associated with the control target device (OB). The second graphic element (1104) may be used to directly control the control target device (OB). The control operation of the control target device (OB) may include the operation of transmitting a control signal to the control target device (OB) for controlling the control target device (OB). For example, as illustrated in the fourth scene (S4), the second graphic element (1104) may include at least one graphic object (e.g., selection buttons) corresponding to a controller (e.g., physical controller device) of the control target device (OB). When a graphic object included in the second graphic element (1104) is selected, the HMD device may perform a control function corresponding to that graphic object. For example, when a graphic object corresponding to wind speed included in the second graphic element (1104) is selected, the HMD device may transmit a control signal to the control target device (OB) to adjust the wind speed to that corresponding graphic object. In this case, the control target device (OB) may adjust the wind speed based on the received control signal.

[0122] FIG. 12 is a diagram illustrating an operation in which an HMD device provides graphic elements for controlling a device to be controlled using usage pattern information, according to one embodiment of the present disclosure.

[0123] According to one embodiment, the description of a method for controlling a device to be controlled may include all of the descriptions in FIG. 5, FIG. 6, and FIG. 11.

[0124] Referring to the first scene (S1) of FIG. 12, a basic graphic element (1210) corresponding to a basic controller for a controlled device (OB) may be displayed. The basic graphic element (1210) may correspond, for example, to the second graphic element (1104) of FIG. 11.

[0125] According to one embodiment, an HMD device (e.g., the electronic device (101) of FIG. 1, the wearable electronic device (200) of FIG. 2, or the wearable electronic device (300) of FIG. 3a to 3c) can generate personalized graphic elements (1220) based on usage pattern information regarding the user's control target device (OB) of the HMD device. The HMD device can obtain usage pattern information for each registered control target device by analyzing (or learning) the user's usage pattern using an AI model. By providing personalized graphic elements (1220) based on such usage pattern information, graphic elements reflecting information about functions and states frequently used by the user can be provided.

[0126] Referring to the second scene (S2) of FIG. 12, a personalized graphic element (1220) corresponding to a personalized controller for a controlled device (OB) may be displayed. The personalized graphic element (1220) may include at least one personalized graphic object (1221) together with graphic objects (basic graphic objects) corresponding to, for example, the second graphic element (1104) of FIG. 11. For example, as illustrated in FIG. 12, the personalized graphic element (1220) may include a personalized graphic object (1221) associated with a wind door opening function frequently used by the user. By providing the personalized graphic object (1221), a function frequently used by the user (e.g., wind door opening function), which would otherwise require several inputs when providing the basic graphic element (1210), can be used immediately with a single input. Through this, the user can easily and quickly use the frequently used function. According to one embodiment, a personalized graphic object (1221) may be displayed prior to basic graphic objects. For example, on a personalized graphic element (1220), the personalized graphic object (1221) may be displayed at the very top. This allows the user to immediately recognize frequently used functions and use them easily and quickly.

[0127] FIGS. 13a and 13b are drawings for explaining the operation of an HMD device providing graphic elements for controlling a target device using spatial map information according to one embodiment of the present disclosure.

[0128] According to one embodiment, the description of a method for controlling a device to be controlled may include all of the descriptions in FIG. 5, FIG. 6, FIG. 11, and FIG. 12.

[0129] Referring to FIGS. 13a and 13b, in the first scene (S1), an HMD device (e.g., the electronic device (101) of FIG. 1, the wearable electronic device (200) of FIG. 2, or the wearable electronic device (300) of FIGS. 3a to 3c) may be located in the first space. The HMD device may acquire user input (e.g., gesture input, voice input) that calls the space map (1310).

[0130] According to one embodiment, in a second scene (S2), the HMD device may acquire spatial map information in response to acquiring user input calling a spatial map, and may display a spatial map (MAP) based on the acquired spatial map information. The spatial map information may be, for example, information about the spatial map (940) of FIG. 9, information about the first spatial map (1010) of FIG. 10, information about the second spatial map (1020), and / or information about the third spatial map (1030). The spatial map (MAP) may provide at least one controlled target device (OB1, OB2) and / or state information (ST1, ST2) for each controlled target device. For example, as illustrated in the second scene (S2), the space map (MAP) can display the first control target device (OB1) (e.g., a stand-type air conditioner) and the status information for the first control target device (OB1) (e.g., power OFF state) and / or the second control target device (OB2) (e.g., an air purifier) ​​and the status information for the second control target device (OB1) (e.g., PM 40 air quality state).

[0131] According to one embodiment, the HMD device can obtain user input for selecting one or more of at least one control target device (OB1, OB2) included in a spatial map (MAP). For example, as illustrated in the second scene (S1), the HMD device can obtain user input (1320) (e.g., gaze input) for selecting a first control target device (OB1) included in the spatial map (MAP).

[0132] According to one embodiment, the HMD device may display a first graphic element (1330) (e.g., a first virtual remote control) for executing a control function of a selected control target device based on the acquisition of user input selecting a control target device included in a spatial map (MAP). For example, as illustrated in the third scene (S3), the HMD device may display a first graphic element (1330) for executing a control function of a first control target device (OB1) at a location adjacent to the first control target device (OB1) in response to the acquisition of user input selecting a first control target device (OB1) included in a spatial map (MAP). The first graphic element (1330) may include a graphic object corresponding to a power button and may display the current power status of the first control target device (OB1) (e.g., a power-off state).

[0133] According to one embodiment, the HMD device may display a second graphic element (1340) (e.g., a second virtual remote control) for executing a control function corresponding to a selected graphic object based on the acquisition of a user input (1331) for selecting a graphic object included in a first graphic element (1330). For example, the HMD device may display the second graphic element (1340) in response to the acquisition of a user input (1331) for selecting a graphic object corresponding to a power button included in the first graphic element (1330). The second graphic element (1340) may include at least one basic graphic object and / or a personalized graphic object (1341) for controlling a selected first control target device (OB1) while the power is on.

[0134] According to one embodiment, the HMD device can reflect information about the state of the controlled device, which has been changed according to user input for the first graphic element (1330) and / or the second graphic element (1340), in a spatial map (MAP). For example, as illustrated in the fourth scene (S4), the HMD device can change the state information for the first controlled device (OB1) on the spatial map (MAP) from a first state (ST1) (e.g., power off state) to a second state (ST1') (e.g., power on state or display of the current temperature (e.g., 25 degrees) in the power on state).

[0135] FIGS. 14a to 14c are drawings for explaining the operation of an HMD device providing graphic elements for integrated control of a plurality of control target devices according to one embodiment of the present disclosure.

[0136] According to one embodiment, the description of a method for controlling a device to be controlled may include all of the descriptions in FIG. 5, FIG. 6, and FIG. 11 to 13b.

[0137] Referring to FIG. 14a, as illustrated in the first scene (S1), an HMD device (e.g., electronic device (101) of FIG. 1, wearable electronic device (200) of FIG. 2, or wearable electronic device (300) of FIG. 3a to 3c) can recognize (or select) a first control target device (OB1) corresponding to a user's gaze (1410a). As illustrated in the second scene (S2), the HMD device can display a first graphic element (1420a) (e.g., a first indicator) for executing a control function of the first control target device (OB1), and can obtain a first user input (I1) (e.g., a long press input) for selecting the first graphic element (1420a). As exemplified in the third scene (S3), the HMD device may display a first-1 graphic element (1430a) (e.g., a first-1 indicator) indicating that the first control target device (OB1) is selected based on the selection of the first graphic element (1420a), and may display a second graphic element (1432a) (e.g., a second indicator) for executing the control function of the second control target device (OB2). The HMD device may display a third graphic element (1433a) (e.g., a selection completion button) for completing the selection of the control target device(s) together with the second graphic element (1432a). The HMD device may obtain a second user input (I2) for selecting the second graphic element (1432a). As exemplified in the fourth scene (S4), the HMD device may display a second-1 graphic element (1440a) (e.g., a second-1 indicator) indicating that a second control target device (OB2) is selected, based on the selection of the second graphic element (1432a), in place of the second graphic element (1432a). Subsequently, the HMD device may obtain user input to select a third graphic element (1433a) for the completion of selection. Through this process, multiple control target devices (OB1, OB2) visible in the user's field of view may be selected for control.A control target device visible in the user's field of view may be, for example, a control target device included in an external image (e.g., an image in the front direction) acquired through the camera of the HMD device.

[0138] Referring to FIG. 14b, as illustrated in the first scene (S1), the HMD device can recognize (or select) a first control target device (OB1) corresponding to the user's gaze (1410b). As illustrated in the second scene (S2), the HMD device can display a first graphic element (1420b) (e.g., a first indicator) for executing a control function of the first control target device (OB1), and can obtain a first user input (I1) (e.g., a long press input) for selecting the first graphic element (1420a). As illustrated in the third scene (S3), based on the selection of the first graphic element (1420a), the HMD device can display a first-1 graphic element (1430a) (e.g., a first-1 indicator) indicating that the first control target device (OB1) is selected, replacing the first graphic element (1420a). As exemplified in the fourth scene (S4), the HMD device can use spatial map information to recognize a second control target device (OB2) (e.g., wall-mounted air conditioner) that can be controlled together with a first control target device (OB1) (e.g., stand-type air conditioner) even though it is not visible in the user's field of view. As exemplified in the fifth scene (S5), based on the recognition of the second control target device (OB2), the HMD device can display a graphic element corresponding to the shape (or form) of the second target device (OB2) along with a second graphic element (e.g., the second graphic element (1432a) of FIG. 14a) (e.g., second indicator) for executing the control function of the second control target device (OB2) and a third graphic element (e.g., the third graphic element (1433a) of FIG. 14a) (e.g., selection completion button). Based on acquiring a second user input (e.g., the second user input (I2) of FIG. 14a) that selects a second graphic element (1450a), the HMD device may display a second-1 graphic element (1450a) (e.g., a second-1 indicator) indicating that a second control target device (OB2) is selected, in place of the second graphic element.Subsequently, the HMD device can acquire user input for selecting a third graphic element to complete the selection. Through this process, a control target device (OB1) visible in the user's field of view, along with a control target device (OB2) not visible in the user's field of view, can be selected. A control target device not visible in the user's field of view may be, for example, a control target device that is not included in an external image (e.g., an image in the front direction) acquired through the camera of the HMD device.

[0139] Referring to FIG. 14c, the HMD device may display at least one graphic element (1410c, 1420c, 1430c) for individual control and / or integrated control. For example, the HMD device may display a first individual control graphic element (1410c) for individually controlling a first control target device (OB1) (e.g., a stand-type air conditioner) at a location adjacent to the first control target device (OB1). The first individual control graphic element (1410c) may include a personalized graphic object (1421c). For example, the HMD device may display a second individual control graphic element (1420c) for individually controlling a second control target device (OB2) (e.g., a wall-mounted air conditioner) at a location adjacent to the second control target device (OB2). For example, the HMD device may include an integrated control graphic element (1430c) for integrated control of the first control target device (OB1) and the second control target device (OB2). The integrated control graphic element (1430c) may include a graphic object (1431c) for mode selection, at least one basic graphic object for integrated control, and / or at least one personalized graphic object. The graphic object for mode selection (1431c) may be displayed along with information indicating the state of the current mode (e.g., integrated control mode). Based on user input for selecting the graphic object for mode selection (1431c), the HMD device may display the graphic object for mode selection in place of the graphic object (1431c). The selectable modes may be, for example, an integrated control mode, a first individual control mode (e.g., a wall-mounted air conditioner control mode), or a second individual control mode (e.g., a stand-type air conditioner control mode).

[0140] FIG. 15 is a diagram illustrating the operation of an HMD device providing graphic elements for integrated control of a plurality of control target devices according to one embodiment of the present disclosure.

[0141] According to one embodiment, the description of a method for controlling a device to be controlled may include all of the descriptions in FIG. 5, FIG. 6, FIG. 11 to FIG. 14c.

[0142] In the embodiment of FIG. 15, unlike the embodiments of FIG. 14a to 14c, an HMD device (e.g., the electronic device (101) of FIG. 1, the wearable electronic device (200) of FIG. 2, or the wearable electronic device (300) of FIG. 3a to 3c) may provide graphic elements for integrated control of a plurality of control target devices through voice input. The HMD device may obtain voice input (1500) for integrated control of a plurality of control target devices through the speaker of the HMD device. The voice input may be an input corresponding to a designated voice command, such as, for example, "make me a remote control."

[0143] Referring to FIG. 15, as exemplified in the first scene (S1), the HMD device may display first graphic elements (1511, 1512, 1513) (e.g., first indicators) for executing control functions of a plurality of control target devices (OB1, OB2, OB3) in response to the acquisition of voice input (1500). The HMD device may also display graphic elements for the completion of selection. The plurality of control target devices (OB1, OB2, OB3) are control target devices located in the space viewed by the user, and may include not only control target devices (OB1, OB2) visible in the user's field of vision, but also control target device (OB3) not visible in the user's field of vision. As illustrated in the second scene (S2), the HMD device may each display a second graphic element (1521, 1522) (e.g., a second indicator) indicating that at least one of the multiple control target devices (OB1, OB2, OB3) is selected based on receiving a user input selecting at least one of the multiple control target devices (OB1, OB2, OB3).

[0144] According to one embodiment, the HMD device may display a third graphic element (1530) (e.g., a virtual remote control) for integrated control. For example, the HMD device may display a third graphic element (1530) for integrated control of a selected first control target device (OB1) (e.g., a stand-type air conditioner) and a second target control device (OB2) (e.g., a wall-mounted air conditioner). The third graphic element (1530) for integrated control may be identical to, for example, the graphic element (1430c) of FIG. 14c.

[0145] FIGS. 16a and 16b are drawings for illustrating an operation in which an HMD device switches modes of controlling a device to be controlled, according to one embodiment of the present disclosure.

[0146] According to one embodiment, the description of a method for controlling a device to be controlled may include all of the descriptions in FIG. 5, FIG. 6, and FIG. 11 to FIG. 15.

[0147] According to one embodiment, as described above, the mode for controlling the device to be controlled may be, for example, an integrated control mode, a first individual control mode (e.g., a wall-mounted air conditioner control mode), or a second individual control mode (e.g., a stand-type air conditioner control mode).

[0148] As illustrated in FIGS. 16a and 16b, and as illustrated in the first scene (S1), an HMD device (e.g., electronic device (101) of FIG. 1, wearable electronic device (200) of FIG. 2, or wearable electronic device (300) of FIGS. 3a to 3c) may display a graphic element (R1) (e.g., a first virtual remote control) for integrated control of a first control target device (OB1) and a second control target device (OB2). In the present disclosure, the graphic element (R1) for integrated control may be referred to as a multi-remote control graphic element (R1). The multi-remote control graphic element (R1) may include at least one graphic object for executing an integrated control function (e.g., a graphic object for temperature control of a plurality of control target devices) and a graphic object (1610) for mode selection (hereinafter, a first mode selection graphic element). For example, as exemplified in the first scene (S1), when a first user input (I1) is obtained for selecting a graphic object to control the temperature increase, the HMD device can raise the temperature of all of the plurality of control target devices (OB1, OB2) by the temperature corresponding to the first user input (I1).

[0149] According to one embodiment, as illustrated in the second scene (S2), when a second user input (I2) is obtained for selecting a first mode selection graphic object (1610), the HMD device may display a second mode selection graphic object (1611) that provides mode selection information in place of the first mode selection graphic object (1610). The second mode selection graphic object (1611) may provide information about selectable modes (e.g., integrated control mode, wall-mounted air conditioner control mode, stand-type air conditioner control mode). When a third user input (I3) is obtained for selecting one of the selectable modes through the second mode selection graphic object (1611) (e.g., user input for selecting a wall-mounted air conditioner control mode), the HMD device may display an individual remote control graphic element (R2) corresponding to the selected mode (e.g., an individual remote control graphic element for the control function of the wall-mounted air conditioner control mode) (e.g., a second virtual remote control). An individual remote control graphic element (R2) may include at least one graphic object for executing an individual control function (e.g., a graphic object for temperature control of a wall-mounted air conditioner) and a graphic object (1620) that provides information about the type of the device to be controlled (e.g., a wall-mounted air conditioner). For example, when a fourth user input (I4) is obtained for selecting a graphic object for controlling the temperature increase, the HMD device may raise only the temperature of the device to be controlled (OB2) by the temperature corresponding to the fourth user input (I4). As illustrated in the second scene (S2), when the device to be controlled (e.g., a wall-mounted air conditioner) is selected through the second mode selection graphic object (1611), the HMD device may display a graphic object (1621) indicating that the device to be controlled has been selected.

[0150] FIGS. 17a and 17b are drawings for explaining the operation of an HMD device changing a controlled target device according to one embodiment of the present disclosure.

[0151] According to one embodiment, the description of a method for controlling a device to be controlled may include all of the descriptions in FIG. 5, FIG. 6, FIG. 11 to FIG. 16b.

[0152] Referring to FIG. 17a, as illustrated in the first scene (S1), an HMD device (e.g., the electronic device (101) of FIG. 1, the wearable electronic device (200) of FIG. 2, or the wearable electronic device (300) of FIG. 3a to 3c) can display a first remote control graphic element (R1) (e.g., a first virtual remote control) for executing a control function of a first control target device (OB1) (e.g., a stand-type air conditioner). As illustrated in the second scene (S2), with the first remote control graphic element (R1) displayed, the HMD device can obtain user input (e.g., a gaze input corresponding to the user gaze (1700)) for selecting a second control target device (OB2) (e.g., a wall-mounted air conditioner) as the control target device. As exemplified in the third scene (S3), the HMD device may display a graphic element (1730) (e.g., an indicator) for executing a control function of the second control target device (OB2) based on user input for selecting the second control target device (OB2) as the control target device. The HMD device may display a second remote control graphic element (R2) (e.g., a second virtual remote control) for executing a control function of the second control target device (OB2) based on user input for selecting the graphic element (1730). The second remote control graphic element (R2) may be displayed in place of the first remote control graphic element (R1).

[0153] Referring to FIG. 17b, the first remote control graphic element (R1) may include at least one graphic object for executing an individual control function of the first control target device (OB1), and the second remote control graphic element (R2) may include at least one graphic object for executing an individual control function of the second control target device (OB2). The first remote control graphic element (R1) may be identical to the second remote control graphic element (R2), but is not limited thereto.

[0154] FIGS. 18a to 18c are drawings for explaining a method in which an HMD device controls a device to be controlled, according to one embodiment of the present disclosure.

[0155] According to one embodiment, the description of a method for controlling a device to be controlled may include all of the descriptions in FIG. 5, FIG. 6, FIG. 11 to FIG. 17b.

[0156] Referring to FIGS. 18a through 18c, an HMD device (e.g., electronic device (101) of FIG. 1, wearable electronic device (200) of FIG. 2, or wearable electronic device (300) of FIGS. 3a through 3c) may enter a first space (e.g., room 1). As illustrated in the first scene (S1), the first space in which the HMD device (or a user wearing the HMD device) is located may contain a plurality of control target devices (e.g., monitor 1, light 1 through light 4).

[0157] According to one embodiment, as illustrated in the second scene (S2), the HMD device may display the first graphic element (1810) based on the acquisition of a voice input (e.g., a voice input calling "SmartThings" or "remote control") for displaying a virtual first graphic element (1810) (e.g., a first virtual remote control) for controlling a device to be controlled that exists in the first space. The first graphic element (1810) may include, for example, graphic objects for individually controlling all or part of the devices to be controlled that exist in the first space (e.g., monitor 1, lights 1 to 4).

[0158] According to one embodiment, as illustrated in the third scene (S3), the HMD device may, in response to acquiring a first user input (I1) for selecting at least two of the control target devices present in the first space (e.g., a gesture input drawing an area (1801) including light 2 (OB2) to light 4 (OB4)), activate an integrated control function for the selected at least two control target devices and display a virtual second graphic element (1820) (e.g., a second virtual remote control) for controlling the selected at least two control target devices in place of the first graphic element (1810). The HMD device may display an indicator (ID2, ID3, ID4) corresponding to each selected control target device (light 2, light 3, light 4) at a location adjacent to each control target device (light 2, light 3, light 4). The indicator (ID2, ID3, ID4) may include, for example, an image and / or identification ID of the corresponding control target device (lighting 2, lighting 3, lighting 4). The second graphic element (1820) may display a graphic object for the function of multi-device control (or integrated control) (hereinafter, integrated control graphic object) together with graphic object(s) (hereinafter, individual control graphic object(s)) for individually controlling control target devices (e.g., monitor 1, lighting 1 to lighting 4) existing in the first space. The HMD device may identify integrated controllable function(s) using a learned AI model based on the type and / or provided function of the selected control target devices. The HMD device may include object(s) corresponding to the identified function(s) as multi-device control graphic objects within the second graphic element (1820) to provide control functions through the selected control target devices. Unlike the first graphic element (1810), in the second graphic element (1820), graphic objects corresponding to the selected control target device (lighting 2 to lighting 4) may be displayed ahead of graphic objects corresponding to the unselected control target device (monitor 1, lighting 1).Through this, the user can quickly and easily recognize the selected control target device.

[0159] According to one embodiment, as illustrated in the fourth scene (S4), the HMD device can turn on the power of all selected control target devices (lights 2 to 4) based on obtaining a second user input (I2) for turning on the power to the multi-device through an integrated control graphic object. After the integrated control function is activated, the HMD device can perform a control function for the brightness by determining a value corresponding to the level of each control target device (lights 2 to 4) corresponding to a level (e.g., brightness level) adjusted based on the user input, in response to obtaining a user input for a level change function included in the integrated control graphic object.

[0160] According to one embodiment, as illustrated in the fifth scene (S5), the HMD device may identify a control target device (OB2) corresponding to a graphic object based on acquiring a third user input (I3) for selecting a graphic object to control an individual control target device (e.g., light 2) while the integrated control function is activated, and display an indicator (ID2) indicating that the control target device (OB2) is selected at a location adjacent to the control target device (OB2).

[0161] FIG. 19 is a flowchart illustrating a method for an HMD device to control a device to be controlled, according to one embodiment of the present disclosure.

[0162] According to one embodiment, the description of a method for controlling a device to be controlled may include all of the descriptions in FIG. 5, FIG. 6, FIG. 11 to FIG. 18c.

[0163] Referring to FIG. 19, in operation 1910, an HMD device (e.g., electronic device (101) of FIG. 1, wearable electronic device (200) of FIG. 2, or wearable electronic device (300) of FIG. 3a to 3c) can identify a first control target device included in an external image of the HMD device obtained through a second camera based on the gaze information of the user of the HMD device obtained through a first camera.

[0164] In operation 1920, the HMD device can display a first graphic element for executing a control function of the first control target device at a location associated with the first control target device through a display.

[0165] In operation 1930, the HMD device may display, through a display, a second graphic element indicating that a first control target device is selected based on the selection of a first graphic element, and a third graphic element for executing a control function of a second control target device.

[0166] In operation 1940, the HMD device may display, through the display, a fourth graphic element indicating that a second control target device is selected based on the selection of a third graphic element.

[0167] In operation 1950, the HMD device can display a fifth graphic element through a display to execute an integrated control function of the first control target device and the second control target device.

[0168] According to one embodiment, the second control target device may correspond to a control target device not included in the external image. Based on the selection of the first graphic element, the HMD device can identify the second control target device based on spatial map information about the space where the user is located and context analysis associated with the spatial map information.

[0169] According to one embodiment, context analysis can be performed using an artificial intelligence model learned based on at least one of gaze information, registration device information, or received signal information and spatial map information.

[0170] According to one embodiment, the received signal information may include at least two of the following: reception strength information of a received signal received from an HMD device, reception angle information of a received signal, or identification information of a control target device included in the received signal.

[0171] According to one embodiment, the fifth graphic element may include a graphic object for executing a personalized control function generated based on user usage pattern information.

[0172] According to one embodiment, the fifth graphic element may include a graphic object for setting a control mode for a first control target device and a second control target device. The control mode may be set to one of a first control mode for integrated control of the first control target device and the second control target device, a second control mode for individual control of the first control target device, or a third control mode for individual control of the second control target device.

[0173] According to one embodiment, the HMD device further includes a communication circuit and can register a control target device based on gaze information, image information obtained through a second camera, and received signal information obtained through the communication circuit.

[0174] According to one embodiment, the operation of registering the control target device may include, based on line of sight information, image information and received signal information, an operation of recognizing the control target device to be registered; an operation of identifying whether network connection information for the control target device exists; an operation of displaying a sixth graphic element for registering the control target device when it is identified that network connection information for the control target device exists; and / or an operation of displaying a seventh graphic element for providing manual information for setting network connection information for the control target device when it is identified that network connection information for the control target device does not exist.

[0175] According to one embodiment, the operation of registering a controlled device may include the operation of generating spatial map information for the space where the controlled device is located, based on image information.

[0176] According to one embodiment, the operation of registering a control target device may include the operation of generating synthesized spatial map information by merging spatial map information with at least one other spatial map information based on movement information of the HMD device, and the operation of storing the synthesized spatial map information. The movement information of the HMD device may include information regarding the movement distance and rotation angle of the HMD device.

[0177] According to one embodiment of the present disclosure, a head-mounted display (HMD) device may include a first camera; a second camera; a display; at least one processor including a processing circuit; and a memory including at least one storage medium for storing instructions. When the above commands are executed individually or collectively by the at least one processor, the HMD device may be caused to: identify a first control target device included in an external image of the HMD device acquired through the second camera based on the gaze information of the user of the HMD device acquired through the first camera; display a first graphic element for executing a control function of the first control target device at a location associated with the first control target device through the display; display a second graphic element indicating that the first control target device is selected and a third graphic element for executing a control function of the second control target device through the display based on the selection of the first graphic element; display a fourth graphic element indicating that the second control target device is selected through the display based on the selection of the third graphic element; and display a fifth graphic element for executing an integrated control function of the first control target device and the second control target device through the display.

[0178] According to one embodiment, the second control target device corresponds to a control target device not included in the external image, and when the instructions are executed individually or collectively by the at least one processor, the HMD device may be caused to identify the second control target device based on spatial map information about the space where the user is located and context analysis associated with said spatial map information based on the selection of the first graphic element.

[0179] According to one embodiment, the context analysis may be performed using an artificial intelligence model learned based on at least one of the gaze information, registration device information, or received signal information and the spatial map information.

[0180] According to one embodiment, the received signal information may include at least two of the following: reception strength information of the received signal received from the HMD device, reception angle information of the received signal, or identification information of the control target device included in the received signal.

[0181] According to one embodiment, the fifth graphic element may include a graphic object for executing a personalized control function generated based on the user's usage pattern information.

[0182] According to one embodiment, the fifth graphic element includes a graphic object for setting a control mode for the first control target device and the second control target device, and the control mode may be set to one of a first control mode for integrated control of the first control target device and the second control target device, a second control mode for individual control of the first control target device, or a third control mode for individual control of the second control target device.

[0183] According to one embodiment, the communication circuit is further included, and when the instructions are executed individually or collectively by the at least one processor, the HMD device may cause the device to register a control target device based on the gaze information, image information obtained through the second camera, and received signal information obtained through the communication circuit.

[0184] According to one embodiment, the operation of registering the control target device may include: an operation of recognizing the control target device to be registered based on the line of sight information, the image information, and the received signal information; an operation of identifying whether network connection information for the control target device exists; an operation of displaying a sixth graphic element for registering the control target device when it is identified that network connection information for the control target device exists; and an operation of displaying a seventh graphic element for providing manual information for setting network connection information for the control target device when it is identified that network connection information for the control target device does not exist.

[0185] According to one embodiment, the operation of registering the control target device may include: generating spatial map information for the space where the control target device is located based on the image information.

[0186] According to one embodiment, the operation of registering the control target device includes: an operation of generating synthesized spatial map information by merging the spatial map information with at least one other spatial map information based on the movement information of the HMD device; and an operation of storing the synthesized spatial map information, wherein the movement information of the HMD device may include information regarding the movement distance and rotation angle of the HMD device.

[0187] The embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" each may include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as “coupled” or “connected” to another (e.g., 2nd) component, with or without the terms “functionally” or “communicationly,” it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0188] The term “module” as used in the embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0189] One embodiment of the present document may be implemented as software (e.g., program (140) of FIG. 1) comprising one or more instructions stored in a storage medium (e.g., internal memory (136) of FIG. 1 or external memory (138) of FIG. 1) that is readable by a machine (e.g., electronic device (101) of FIG. 1). For example, a processor (e.g., processor (120) of FIG. 1) of the machine (e.g., electronic device (101) of FIG. 1) may call at least one of the one or more instructions stored from the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0190] According to one embodiment, the method according to the embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0191] According to one embodiment, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to one embodiment, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to one embodiment, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In a head-mounted display (HMD) device, First camera; Second camera; display; At least one processor including a processing circuit; and The device includes a memory comprising at least one storage medium for storing instructions, wherein the instructions, when executed individually or collectively by the at least one processor, cause the HMD device: Based on the user's gaze information of the HMD device obtained through the first camera, a first control target device included in the external image of the HMD device obtained through the second camera is identified, and A first graphic element for executing a control function of the first control target device at a location associated with the first control target device is displayed through the display, and Based on the selection of the first graphic element, a second graphic element indicating that the first control target device is selected and a third graphic element for executing a control function of the second control target device different from the first control target device are displayed through the display. Based on the selection of the third graphic element, a fourth graphic element indicating that the second control target device is selected is displayed through the display, and An HMD device that causes a fifth graphic element to be displayed through the display for executing an integrated control function of the first control target device and the second control target device.

2. In Paragraph 1, The second control target device above corresponds to a control target device not included in the external image, and When the above instructions are executed individually or collectively by the at least one processor, the HMD device: An HMD device that causes to identify the second control target device based on spatial map information about the space where the user is located and context analysis associated with the spatial map information, based on the selection of the first graphic element.

3. In Paragraph 2, The above context analysis is, An HMD device that performs using an artificial intelligence model learned based on at least one of the above-mentioned gaze information, registration device information, or received signal information and the above-mentioned spatial map information.

4. In Paragraph 3, The HMD device, wherein the received signal information comprises at least two of the following: reception strength information of a received signal received by the HMD device, reception angle information of the received signal, or identification information of the control target device included in the received signal.

5. In any one of paragraphs 1 through 4, The above-described fifth graphic element is an HMD device comprising a graphic object for executing a personalized control function generated based on the user's usage pattern information.

6. In any one of paragraphs 1 through 5, The above-mentioned fifth graphic element includes a graphic object for setting a control mode for the first control target device and the second control target device, and An HMD device, wherein the above control mode is set as one of a first control mode for integrated control of the first control target device and the second control target device, a second control mode for individual control of the first control target device, or a third control mode for individual control of the second control target device.

7. In any one of paragraphs 1 through 6, It further includes a communication circuit, When the above instructions are executed individually or collectively by the at least one processor, the HMD device: An HMD device that causes a control target device to be registered based on the above-mentioned gaze information, image information obtained through the above-mentioned second camera, and received signal information obtained through the above-mentioned communication circuit.

8. In Paragraph 7, The operation of registering the above-mentioned control target device is: An operation to recognize a control target device to be registered based on the above gaze information, the above image information, and the above received signal information; An operation to identify whether network connection information for the above-mentioned control target device exists; An operation of displaying a sixth graphic element for registering the control target device when identifying that network connection information for the control target device exists; and An HMD device comprising an operation to display a seventh graphic element for providing manual information for setting network connection information for the control target device when it is identified that network connection information for the control target device does not exist.

9. In Paragraph 7 or 8, The operation of registering the above-mentioned control target device is: An HMD device comprising an operation to generate spatial map information for the space where the control target device is located, based on the above image information.

10. In Paragraph 9, The operation of registering the above-mentioned control target device is: An operation to generate synthesized spatial map information by merging the spatial map information with at least one other spatial map information based on the movement information of the above HMD device; and It includes the operation of storing the above-mentioned synthesized spatial map information, The movement information of the HMD device includes information regarding the movement distance and rotation angle of the HMD device.

11. In a method for an HMD (head mounted display) device, An operation to identify a first control target device included in an external image of the HMD device obtained through a second camera of the HMD device, based on gaze information of the user of the HMD device obtained through a first camera of the HMD device; An operation of displaying a first graphic element for executing a control function of the first control target device at a location associated with the first control target device; An operation of displaying a second graphic element indicating that the first control target device is selected based on the selection of the first graphic element, and a third graphic element for executing a control function of the second control target device different from the first control target device; An operation of displaying a fourth graphic element indicating that the second control target device is selected based on the selection of the third graphic element; A method comprising an operation of displaying a fifth graphic element for executing an integrated control function of the first control target device and the second control target device.

12. In Paragraph 11, The above second control target device corresponds to a control target device not included in the above external image, and the method is: A method comprising identifying the second control target device based on spatial map information about the space where the user is located and context analysis associated with the spatial map information, based on the selection of the first graphic element.

13. In Paragraph 12, The above context analysis is, A method performed using an artificial intelligence model learned based on at least one of the above-mentioned gaze information, registration device information, or received signal information and the above-mentioned spatial map information.

14. In Paragraph 13, A method comprising at least two of the following: reception signal information, wherein the reception signal information comprises reception strength information of a reception signal received from the HMD device, reception angle information of the reception signal, or identification information of the control target device included in the reception signal.

15. In any one of paragraphs 11 through 14, A method in which the fifth graphic element comprises a graphic object for executing a personalized control function generated based on the user's usage pattern information.

Images