Electronic device and operation method thereof

Abstract

The present disclosure provides a method of an HMD device. This method may comprise the operations of: displaying a first graphic UI element; identifying, on the basis of a first gesture input for the first graphic UI element, whether a first condition for activating, in a wearable electronic device, a function for supporting the HMD device is satisfied; displaying, on the basis of identifying that the first condition is satisfied, an activation area associated with the wearable electronic device; identifying, on the basis of a second gesture input for the first graphic UI element and the activation area, whether a second condition for activating, in the wearable electronic device, the function for supporting the HMD device is satisfied; and, on the basis of identifying that the second condition is satisfied, transmitting, to the wearable electronic device, a signal for activating the function for supporting the HMD device.

Description

Electronic device and method of operation thereof

[0001] The present disclosure relates to an electronic device and a method of operation.

[0002] With the advancement of digital technology, electronic devices are being provided in various forms, such as smartphones, tablet PCs, or PDAs. Electronic devices are also being developed in wearable forms to enhance portability and user accessibility.

[0003] Electronic devices developed in a form that users can wear are being developed in the form of wearable electronic devices such as AR glasses (augmented reality glasses), VST (video see-through) devices, and HMD (head-mounted display) devices to provide virtual spaces in virtual environments, and the various services and additional functions provided by wearable electronic devices are gradually increasing. To enhance the utility value of these electronic devices and satisfy the needs of diverse users, telecommunications service providers or electronic device manufacturers are competitively developing electronic devices to provide various functions and differentiate themselves from other companies. Accordingly, the various functions provided through wearable electronic devices are also becoming increasingly sophisticated.

[0004] AR glasses or VST devices can provide a realistic experience to the user by displaying virtual images while worn on the user's body. AR glasses or VST devices can replace the usability of smartphones in various fields such as game entertainment, education, and social networking services (SNS). Users can receive content similar to reality through AR glasses or VST devices and feel as if they are staying in a virtual world through interaction.

[0005] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art related to the present disclosure.

[0006] According to one embodiment, an HMD device may be provided. The HMD device may include at least one processor comprising a camera; a display; a communication circuit; and a processing circuit; and a memory comprising at least one storage medium for storing instructions. The instructions may cause the HMD device to perform at least one operation when executed individually or collectively by the at least one processor. The at least one operation may include an operation of displaying a first graphic UI (user interface) element. The at least one operation may include an operation of identifying whether a first condition for activating a function to support the HMD device in a wearable electronic device is satisfied based on a first gesture input to the first graphic UI element. The function to support the HMD device may be associated with the first graphic UI element. The at least one operation may include an operation of displaying an activation area associated with the wearable electronic device based on identifying that the first condition is satisfied. The at least one operation may include an operation of identifying whether a second condition for activating a function to support the HMD device in the wearable electronic device is satisfied based on a second gesture input to the first graphic UI element and the activation area. The at least one operation may include an operation of transmitting a signal to the wearable electronic device to activate a function to support the HMD device based on identifying that the second condition is satisfied. The wearable electronic device may be configured to activate a function to support the HMD device based on the signal and to display a second graphic element corresponding to the first graphic UI element at a location associated with the wearable electronic device.

[0007] According to one embodiment, a method of operating an HMD device may be provided. The method of operating an HMD device may include at least one operation. The at least one operation may include an operation of displaying a first graphic UI (user interface) element. The at least one operation may include an operation of identifying whether a first condition for activating a function to support the HMD device in a wearable electronic device is satisfied based on a first gesture input to the first graphic UI element. The function to support the HMD device may be associated with the first graphic UI element. The at least one operation may include an operation of displaying an activation area associated with the wearable electronic device based on identifying that the first condition is satisfied. The at least one operation may include an operation of identifying whether a second condition for activating a function to support the HMD device in the wearable electronic device is satisfied based on a second gesture input to the first graphic UI element and the activation area. The at least one operation may include an operation of transmitting a signal to the wearable electronic device to activate a function to support the HMD device based on identifying that the second condition is satisfied. The wearable electronic device may be configured to enable a function to support the HMD device based on the signal and to display a second graphic element corresponding to the first graphic UI element at a location associated with the wearable electronic device.

[0008] According to one embodiment, a storage medium storing at least one instruction readable by a computer may be provided. The at least one instruction may cause the HMD device to perform at least one action when executed by at least a part of at least one processor of the HMD device. The at least one action may include an action of displaying a first graphic UI (user interface) element. The at least one action may include an action of identifying whether a first condition for activating a function to support the HMD device in a wearable electronic device is satisfied based on a first gesture input to the first graphic UI element. The function to support the HMD device may be associated with the first graphic UI element. The at least one action may include an action of displaying an activation area associated with the wearable electronic device based on identifying that the first condition is satisfied. The at least one action may include an action of identifying whether a second condition for activating a function to support the HMD device in the wearable electronic device is satisfied based on a second gesture input to the first graphic UI element and the activation area. The above at least one operation may include an operation of transmitting a signal to activate a function to support the HMD device to the wearable electronic device based on identifying that the second condition is satisfied. The wearable electronic device may be configured to activate a function to support the HMD device based on the signal and to display a second graphic element corresponding to the first graphic UI element at a location associated with the wearable electronic device.

[0009] In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components.

[0010] FIG. 1 is a block diagram of an electronic device in a network environment according to various embodiments of the present disclosure.

[0011] FIG. 2 is a drawing showing the configuration of a wearable electronic device according to one embodiment of the present disclosure.

[0012] FIGS. 3a to 3c are drawings showing the front and rear views of a wearable electronic device according to one embodiment of the present disclosure.

[0013] FIG. 4 is a drawing showing the configuration of a wearable electronic device according to one embodiment of the present disclosure.

[0014] FIG. 5 is a flowchart illustrating a method for an HMD device to enable an HMD device support function of a wearable electronic device according to one embodiment of the present disclosure.

[0015] FIG. 6 is a diagram illustrating a method for an HMD device to enable an HMD device support function of a wearable electronic device according to one embodiment of the present disclosure.

[0016] FIG. 7 is a diagram illustrating the operation of a wearable electronic device switching to an HMD support mode according to one embodiment of the present disclosure.

[0017] FIG. 8 is a diagram illustrating the mapping relationship between a physical UI of a wearable electronic device and a physical UI of an HMD device according to one embodiment of the present disclosure.

[0018] FIG. 9 is a diagram illustrating the correspondence between a graphic UI element of a wearable electronic device and a graphic UI element of an HMD device according to one embodiment of the present disclosure.

[0019] FIG. 10 is a diagram illustrating the operation of an HMD device and a wearable electronic device performing the same function through graphic UI elements according to one embodiment of the present disclosure.

[0020] FIGS. 11a to 11d are drawings for explaining the operation of providing a graphic UI of an HMD device through a wearable electronic device according to one embodiment of the present disclosure.

[0021] FIGS. 12 and 13 are drawings for explaining the operation of providing graphic UI elements of an HMD device through a wearable electronic device using a dynamic layout function according to one embodiment of the present disclosure.

[0022] FIGS. 14a and 14b are drawings for explaining the operation of providing a graphic UI of an HMD device through a wearable electronic device using a dynamic layout function according to one embodiment of the present disclosure.

[0023] FIGS. 15a and 15b are drawings for explaining the operation of providing a graphic UI of an HMD device through a wearable electronic device using a dynamic layout function according to one embodiment of the present disclosure.

[0024] FIGS. 16a and 16b are drawings for explaining the operation of setting graphic UI elements according to the state of an HMD device according to one embodiment of the present disclosure.

[0025] FIG. 17 is a diagram illustrating the operation of an HMD device and a wearable electronic device performing the same function through graphic UI elements according to one embodiment of the present disclosure.

[0026] FIGS. 18a to 18c are drawings for explaining an operation of providing a notification according to one embodiment of the present disclosure.

[0027] FIG. 19 is a drawing illustrating a method for an external electronic device to provide an HMD device support function according to one embodiment of the present disclosure.

[0028] Hereinafter, embodiments of the present disclosure are described in detail with reference to the drawings so that those skilled in the art can easily practice them. However, the present disclosure may be embodied in various different forms and is not limited to the embodiments described herein. In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and brevity.

[0029] FIG. 1 is a block diagram of an electronic device in a network environment according to various embodiments of the present disclosure.

[0030] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or with an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through a server (108). According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (160), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)).

[0031] The processor (120) can control at least one other component (e.g., hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., program (140)), for example, and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., sensor module (176) or communication module (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., central processing unit or application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., graphics processing unit, neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use lower power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.

[0032] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence is performed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0033] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, input data or output data for software (e.g., program (140)) and related commands. The memory (130) may include volatile memory (132) or non-volatile memory (134).

[0034] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0035] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0036] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0037] The display module (160) can visually provide information to an external (e.g., user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0038] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (101).

[0039] The sensor module (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0040] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0041] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0042] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0043] The camera module (180) can capture still images and video. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0044] The power management module (188) can manage the power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0045] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0046] The communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a communication module (192) (e.g., cellular communication module, short-range communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The communication module (192) can identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).

[0047] The communication module (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The communication module (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The communication module (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The communication module (192) can support various requirements specified in the electronic device (101), external electronic device (e.g., electronic device (104)), or network system (e.g., second network (199)). According to one embodiment, the communication module (192) can support a Peak data rate (e.g., 20 Gbps or more) for eMBB realization, loss coverage (e.g., 164 dB or less) for mMTC realization, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for URLLC realization.

[0048] An antenna module (197) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197).

[0049] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0050] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface) and exchange signals (e.g., commands or data) with each other.

[0051] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In one embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (104) or the server (108) may be included within a second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0052] The number of processors (120) may be one or more. For example, the processor (120) may have the structure of a multi-core processor such as a dual core, a quad core, or a hexa core.

[0053] The processor (120) can control the operations of the electronic device (101) by executing instructions stored in memory (130). For example, the processor (120) may correspond to a plurality of processors that divide and collectively perform a plurality of operations among the processors.

[0054] FIG. 2 is a drawing showing the configuration of a wearable electronic device according to one embodiment of the present disclosure.

[0055] Referring to FIG. 2, according to one embodiment, a wearable electronic device (200) (e.g., the electronic device (101) of FIG. 1) may include a light output module (211), a display member (201), a camera module (250), and / or a speaker (261). According to one embodiment, the wearable electronic device (200) may include at least one additional configuration other than the configuration shown. For example, the wearable electronic device (200) may include all or part of the configuration included in the electronic device (101) of FIG. 1.

[0056] According to one embodiment, a light output module (211) (e.g., a display module (160) of FIG. 1) may include a light source capable of outputting an image and a lens that guides the image to a display member (201). The light output module (211) may include, for example, a liquid crystal display, a digital mirror device, a liquid crystal on silicon display, an organic light emitting diode and / or a micro light emitting diode (micro LED).

[0057] According to one embodiment, a display member (201) (e.g., a display module (160) of FIG. 1) may include an optical waveguide (e.g., a waveguide). According to one embodiment, an output image of an optical output module (211) incident on one end of the optical waveguide may propagate within the optical waveguide and be provided to a user. According to one embodiment, the optical waveguide may include at least one diffractive element (e.g., a diffractive optical element (DOE), a holographic optical element (HOE)) and / or a reflective element (e.g., a reflective mirror). For example, the optical waveguide may guide the output image of the optical output module (211) to the user's eye using at least one diffractive element or reflective element.

[0058] According to one embodiment, a camera module (250) (e.g., camera module (180) of FIG. 1) can capture images (e.g., still images and / or video). According to one embodiment, the camera module (250) may be placed within a lens frame and around a display member (201). In the present disclosure, images may be interpreted to include video as well as still images.

[0059] According to one embodiment, the first camera module (251) can capture and / or recognize the trajectory of the user's eye (e.g., pupil, iris) or gaze. According to one embodiment, the first camera module (251) can periodically or non-periodically transmit information related to the trajectory of the user's eye or gaze (e.g., trajectory information) to a processor (e.g., processor (120) of FIG. 1).

[0060] According to one embodiment, the second camera module (253) can capture an external image. For example, the second camera module (253) can capture an image of the external environment in the front direction of the wearable electronic device (200).

[0061] According to one embodiment, the third camera module (255) may be used for hand detection and tracking and user gesture (e.g., hand movements) recognition. According to one embodiment, the third camera module (255) may be used for 3 degrees of freedom (3DoF) and 6DoF head tracking, location (space, environment) recognition, and / or movement recognition. According to one embodiment, the second camera module (253) may be used for hand detection and tracking and user gesture recognition. According to one embodiment, at least one of the first camera module (251) to the third camera module (255) may be replaced with a sensor module (e.g., LiDAR sensor). For example, the sensor module may include at least one of a vertical cavity surface emitting laser (VCSEL), a diode, an infrared sensor, an infrared diode, and / or a photodiode.

[0062] According to one embodiment, a speaker (261) (e.g., the acoustic output module (155) of FIG. 1) can output an acoustic signal (e.g., sound and / or virtual vibration sound). Although the speaker (261) is described as being configured in a member that is mounted on the user's ear when the wearable electronic device (200) is worn in FIG. 2, it is not limited thereto and may be configured in other locations depending on the implementation of the wearable electronic device (200).

[0063] FIGS. 3a to 3c are drawings showing the front and rear views of a wearable electronic device according to one embodiment of the present disclosure.

[0064] Referring to FIGS. 3a through 3c, according to one embodiment, at least one first camera module (311, 312) and at least one second camera module (313, 314, 315, 316), a depth sensor (317), and / or a second display (350) may be disposed on a first surface (310) of a housing for acquiring information related to the surrounding environment of a wearable electronic device (300) (e.g., the electronic device (101) of FIG. 1). According to one embodiment, the wearable electronic device (300) may include at least one additional configuration other than the configuration shown. For example, the wearable electronic device (300) may include all or part of the configuration included in the electronic device (101) of FIG. 1.

[0065] According to one embodiment, at least one first camera module (311, 312) can capture an image of the outside of the wearable electronic device (300). For example, the first camera module (311, 312) can capture an image of the external environment in the front direction of the wearable electronic device (300).

[0066] According to one embodiment, at least one second camera module (313, 314, 315, 316) can acquire images while the wearable electronic device (300) is worn by a user. The second camera module (313, 314, 315, 316) may be used for hand detection, tracking, and user gesture (e.g., hand movements) recognition. The second camera module (313, 314, 315, 316) may be used for 3DoF, 6DoF head tracking, location (space, environment) recognition, and / or movement recognition. According to one embodiment, a first camera module (311, 312) may be used for hand detection and tracking and user gestures.

[0067] According to one embodiment, the depth sensor (317) may be configured to transmit a signal and receive a signal reflected from a subject, and may be used for purposes such as time of flight (TOF) to determine the distance to an object. Alternatively, or additionally, a second camera module (313, 314, 315, 316) may determine the distance to an object.

[0068] According to one embodiment, the second display (350) (and / or lens) may be placed on the first surface (310) of the wearable electronic device (300). According to one embodiment, the second display (350) may provide visual information to the outside of the wearable electronic device (300). For example, the second display (350) may be used to provide an alternative notification indicating the operating status of the first camera module (311, 312) in place of the light emitter (340).

[0069] According to one embodiment, camera modules (325, 326) for face recognition and / or a first display (321) (and / or a lens) may be disposed on the second surface (320) of the housing.

[0070] According to one embodiment, camera modules (325, 326) for face recognition adjacent to the first display (321) may be used to recognize the user's face or to recognize and / or track both of the user's eyes.

[0071] According to one embodiment, the first display (321) (and / or lens) may be disposed on the second surface (320) of the wearable electronic device (300). According to one embodiment, the wearable electronic device (300) may not include camera modules (315, 316) among a plurality of second camera modules (313, 314, 315, 316). Although not illustrated in FIG. 3a and 3b, the wearable electronic device (300) may further include at least one of the configurations illustrated in FIG. 2.

[0072] Referring to FIG. 3c, according to one embodiment, the wearable electronic device (300) may have a form factor (e.g., a head-mounted display (HMD)) for being worn on a user's head. The wearable electronic device (300) may further include a strap and / or a wearing member for being secured on a part of the user's body. The wearable electronic device (300) may include a volume button (331), a vent (333), a status indicator (335), and a power button (e.g., including a fingerprint recognition sensor) (337), and such configurations may be identically included in the wearable electronic device (300) illustrated in FIG. 3a and FIG. 3b. When worn on a user's head, it may provide a user experience based on augmented reality, virtual reality, and / or extended reality (or mixed reality). The wearable electronic device (300) configured in the form of an HMD may include configurations identical or similar to the components of FIG. 3a and FIG. 3b described above.

[0073] According to one embodiment, a speaker (318) (e.g., the acoustic output module (155) of FIG. 1 or the speaker (261) of FIG. 2) may output an acoustic signal (e.g., sound and / or virtual vibration sound). Although the speaker (318) has been described as being configured in a location adjacent to the vent (333) in FIG. 3a through 3c as an example, it is not limited thereto and may be configured in other locations depending on the implementation of the wearable electronic device (300).

[0074] FIG. 4 is a drawing showing the configuration of a wearable electronic device according to one embodiment of the present disclosure.

[0075] Referring to FIG. 4, the wearable electronic device (400) (e.g., the electronic device (101) of FIG. 1) may be a watch-type electronic device (e.g., a smart watch) that can be worn on a part of the user's body (e.g., the wrist). However, the wearable electronic device according to the various embodiments disclosed in this document is not limited to the illustrated embodiments.

[0076] According to one embodiment, the wearable electronic device (200) may include a display (410) and / or at least one physical UI (PUI) (420). According to one embodiment, the display (200) may be positioned on the upper part of the housing of the wearable electronic device (200), and at least one PUI (420) may be positioned on the side (e.g., right side) of the housing of the wearable electronic device (200). According to one embodiment, the wearable electronic device (400) may include at least one additional configuration other than the configuration shown. For example, the wearable electronic device (400) may include all or part of the configuration included in the electronic device (101) of FIG. 1.

[0077] In one embodiment, the display (410) (e.g., the display module (160) of FIG. 1) may be visually exposed in the front direction of the wearable electronic device (400). The display (410) may output information regarding the operation and / or state of the wearable electronic device (400) corresponding, for example, to at least one PUI (420).

[0078] According to one embodiment, at least one PUI (420) can receive user input (or, command) for implementing various functions of the wearable electronic device (400) by rotating and / or pressing by user operation. At least one PUI (420) may include a plurality of physical buttons (421, 422) disposed on the side of the housing (210). According to one embodiment, at least one of the plurality of physical buttons (421, 422) may be implemented as an electrode button capable of detecting a user's biosignal (e.g., electrocardiogram) upon contact by a part of the user's body (e.g., finger). At least one PUI (420) may further include a wheel key (423) disposed on the side of the housing (210) and configured to be rotatable. The wheel key (423) can receive user input (or, command) for implementing various functions of the wearable electronic device (400) by rotating by user operation.

[0079] According to one embodiment of the present disclosure, an HMD device (e.g., the electronic device (101) of FIG. 1, the wearable electronic device (200) of FIG. 2, or the wearable electronic device (300) of FIG. 3a to 3c) may perform processing that enables an external electronic device (e.g., the electronic device (101) of FIG. 1, the wearable electronic device (400) of FIG. 4, or the electronic device (1900) of FIG. 19) connected (e.g., paired) to the HMD device to perform a function that supports the HMD device (e.g., HMD support mode). The function that supports the HMD device may include, for example, a function of providing a graphic UI (e.g., a system UI such as a task bar) provided by the HMD device from the external electronic device, a function of providing a physical UI (e.g., a UI provided through a physical button) provided by the HMD device from the external electronic device, and / or a function of providing a notification (or information) provided by the HMD device from the external electronic device, but is not limited thereto. User convenience can be enhanced through the support of HMD devices by these external electronic devices.

[0080] For convenience of explanation, the following description uses as an example an external electronic device providing HMD device support functions that is a watch-type wearable device worn on the user's wrist (e.g., the wearable electronic device (400) of FIG. 4). However, the embodiments of the present disclosure may also be applied to various types of electronic devices (e.g., the electronic device (101) of FIG. 1 or the electronic device (1900) of FIG. 19) or various types of wearable electronic devices (e.g., a ring-type wearable device or a bracelet-type wearable device).

[0081] FIG. 5 is a flowchart illustrating a method for an HMD device to enable an HMD device support function of a wearable electronic device according to one embodiment of the present disclosure.

[0082] FIG. 6 is a diagram illustrating a method for an HMD device to enable an HMD device support function of a wearable electronic device according to one embodiment of the present disclosure.

[0083] Referring to FIGS. 5 and 6, in operation 510, an HMD device (e.g., electronic device (101) of FIG. 1, wearable electronic device (200) of FIG. 2, or wearable electronic device (300) of FIGS. 3a through 3c) may display a first graphic UI element (610) through a display. For example, as illustrated in the first scene (6100), the HMD device may display the first graphic UI element (610) together with at least one execution screen (6110). The first graphic UI element (610) and / or at least one execution screen (6110) may be virtual objects displayed on an environment (e.g., virtual space) provided by the HMD device, for example. The at least one screen (6110) may include a plurality of execution screens (6111, 6112, 6113) of the HMD device. Each execution screen may provide different content, but is not limited thereto.

[0084] According to one embodiment, the HMD device may display an external image of the HMD device (real image) obtained through a camera, along with an image (virtual image) corresponding to a virtual object, through a display. For example, the HMD device may display the real image by overlaying it onto the virtual image, but the opposite is also possible. The external image of the HMD device may include, for example, an image corresponding to the hand (6130) of a user wearing the HMD device and / or an image corresponding to a wearable electronic device (400) worn on the user's wrist. The image corresponding to the user's hand (6130) may include, for example, an image corresponding to a first hand (e.g., left hand) (6131) wearing the wearable electronic device (400) and a second hand (e.g., right hand) (6132) for gesture input.

[0085] According to one embodiment, the first graphic UI element (610) may include a system UI of the HMD device. The system UI may be an interface that supports interaction between the operating system and the user. This system UI may include various graphic objects or elements (e.g., screen elements and / or interface elements) that help the user use the device intuitively. For example, the system UI may include, but is not limited to, a task bar, a setting menu, a quick setting menu, a navigation bar, gesture navigation, a status bar, a notification, a launcher, a shortcut (e.g., an AI shortcut), an application list, and / or widgets.

[0086] According to one embodiment, the first graphic UI element (610) may be configured to be grab-capable and / or movable. According to one embodiment, the first graphic UI element (610) may be switched to a grab state based on a first gesture input to the first graphic UI element (610). For example, if the HDM device identifies a first gesture input corresponding to a gesture of grabbing the first graphic UI element (610) with the user's hand (6132) via a camera (grab gesture), the first graphic UI element (610) may be switched to a grab state. The grab gesture may be, for example, a pinch-in gesture of a portion of the display area of ​​the first graphic UI element (610) (e.g., a designated specific area), but is not limited thereto. For example, the grab gesture may be a pinch-in gesture that reduces the distance between the thumb and index finger (e.g., bringing the thumb and index finger together) while the thumb and index finger are in contact with a portion of the display area of ​​the first graphic UI element (610). The first graphic UI element (610) may be configured to be movable in the grab state. According to one embodiment, while the grab gesture for the first graphic UI element (610) is maintained (e.g., while the first graphic UI element (610) is in the grab state), the HMD device may track the movement of the hand (6132) that has grabbed the first graphic UI element (610) and move the first graphic UI element (610) to a position and / or direction corresponding to the movement of the hand (6132). For example, as illustrated in the second scene (6200), the first graphic UI element (610) may be moved in the grab state to a position adjacent to the wearable electronic device (400).Maintaining the grab gesture (or maintaining the grab state) may include, for example, maintaining the state in which the distance between the thumb and index finger is within a specified distance through a pinch-in gesture (e.g., the thumb and index finger are touching). While the grab gesture is maintained (or the grab state is maintained), the first graphic UI element is associated with the hand (6132) that grabbed the first graphic UI element (610) and may be moved according to the movement of the hand (6132).

[0087] According to one embodiment, in operation 520, the HMD device can identify whether a first condition for activating a function to support the HMD device (hereinafter, HMD device support function) in the wearable electronic device (400) is satisfied. For example, according to one embodiment, the HMD device can identify whether the first condition is satisfied based on a first gesture input to a first graphic UI element (610) (e.g., a gesture input corresponding to a grab gesture).

[0088] According to one embodiment, the HMD device support function may be associated with, for example, a first graphic element (610). For example, the HMD device support function may include a function to display a second graphic element (620) corresponding to the first graphic element (610) in an area associated with the wearable electronic device (400).

[0089] According to one embodiment, the HMD device can identify (e.g., recognize) a wearable electronic device (400) through a camera while the first graphic UI element (610) is in a grab state. The identified wearable electronic device (400) may be, for example, a wearable electronic device located within a specified distance from the first graphic UI element (610) among the wearable devices captured through the camera. In the example of FIG. 6, for convenience of explanation, the identification of a single wearable electronic device is described as an example, but is not limited thereto. For example, a plurality of wearable electronic devices may be identified.

[0090] According to one embodiment, the HMD device may identify whether a first condition is satisfied based on at least one of movement information of the wearable electronic device (400) or connection status information between the wearable electronic device (400) and the HMD device. The movement information of the wearable electronic device (400) (e.g., position and / or orientation information) may be obtained, for example, based on image data obtained through a camera of the HMD device, but is not limited thereto. The first condition may be associated, for example, with a readiness state for activating an HMD device support function. In the present disclosure, the first condition may be referred to as an activation readiness condition.

[0091] According to one embodiment, the operation of identifying whether the first condition is satisfied may include an operation of identifying whether the amount of change in movement (e.g., change in position and / or direction) of the wearable electronic device (400) based on movement information is maintained at or below a reference amount of change for a specified period (e.g., 2 seconds) (hereinafter, operation 1-1) and / or an operation of identifying whether the wearable electronic device (400) and the HMD device are connected based on connection status information (hereinafter, operation 1-2). According to one embodiment, the operation 1-2 may be performed after the operation 1-1. For example, the operation 2 may be performed in response to identifying that the amount of change in movement is maintained at or below a reference amount of change for a specified period through the operation 1-1. The HMD device may identify that the first condition is satisfied in response to identifying that the wearable electronic device and the HMD device are connected (e.g., pairing) through the operation 1-2. For example, the HMD device performs a first-2 operation in response to identifying that the amount of change in movement is maintained below a reference amount of change for a specified period through a first-1 operation, and can identify that the first condition is satisfied when it identifies that the wearable electronic device and the HMD device are connected through the first-2 operation.

[0092] According to one embodiment, in operation 530, the HMD device may display an active area (630) associated with the wearable electronic device (400) through a display based on identifying that a first condition is satisfied. The active area (630) may be displayed, for example, through a virtual graphic object (e.g., an object displaying a blue effect). The active area (630) may be displayed on at least a portion of the wearable electronic device (400) through the display of the HMD device. For example, the active area (630) may be virtually displayed on the display area of ​​the wearable electronic device (400). According to one embodiment, the HMD device may display the active area (630) by recognizing a plane corresponding to a display placed on the surface of the wearable electronic device (400) based on image data acquired through a camera, and by displaying a virtual graphic object displayed in a first color (e.g., blue) on the said plane. These activation areas (630) can be used to inform the user that they are ready to activate the HMD device support function, and the user can transmit user input (e.g., a second gesture input) intended to activate the HMD device support function through the activation areas (630) to the HMD device.

[0093] According to one embodiment, in operation 540, the HMD device can identify whether a second condition for activating an HMD device support function in a wearable electronic device is satisfied. For example, after the first condition is satisfied, the HMD device can identify whether the second condition is satisfied based on a second gesture input and / or an activation area (630) for a first graphic UI element (610). In the present disclosure, the second condition may be referred to as an activation condition.

[0094] According to one embodiment, the operation of identifying whether the second condition is satisfied may include an operation of identifying whether the second condition is satisfied based on information regarding an area (hereinafter, overlapping area) where the activation area (630) and the area corresponding to the first graphic UI element (610) (e.g., the display area of ​​the first graphic UI element (610)) overlap (e.g., overlay). For example, the operation of identifying whether the second condition is satisfied may include an operation of identifying whether the activation area (630) and the area corresponding to the first graphic UI element (610) overlap for a specified period of time or larger than a specified size based on information regarding the overlapping area (hereinafter, operation 2-1) and / or an operation of identifying whether the first graphic UI element (610) transitions from a grab state to an ungrab state (hereinafter, operation 2-2). According to one embodiment, operation 2-2 may be performed after operation 2-1. For example, the second-2 operation may be performed after identifying, through the second-1 operation, that the area corresponding to the activation area (630) and the first graphic UI element (610) overlaps by more than a specified size for a specified period. According to one embodiment, the HMD device may identify that the second condition is satisfied in response to identifying that the first graphic UI element (610) transitions to an ungrabbed state through the second-2 operation. For example, the HMD device may perform the second-2 operation in response to identifying, through the second-1 operation, that the area corresponding to the activation area (630) and the first graphic UI element (610) overlaps by more than a specified size for a specified period, and may identify that the second condition is satisfied when the first graphic UI element (610) transitions to an ungrabbed state through the second-2 operation.

[0095] According to one embodiment, the HMD device may change the display of the active area (630) when it is identified through the above-described 2-1 operation that the area corresponding to the active area (630) and the first graphic UI element (610) overlaps by more than a specified size for a specified period. The change in the display of the active area (630) may include, for example, a change in color, a change in size, and / or a change in shape of the active area (630). For example, the HMD device may change the color of the active area (630) from a first color (e.g., blue) to a second color (e.g., purple) when it is identified that the area corresponding to the active area (630) and the first graphic UI element (610) overlaps by more than a specified size for a specified period. Through this change in the display of the active area (630), the user can recognize when the grab of the first graphic UI element (610) is released.

[0096] According to one embodiment, the first graphic UI element (610) may be switched from a grab state to a grab release state. According to one embodiment, the HMD device may be switched to a grab release state based on a second gesture input for the first graphic UI element (610) obtained in the grab state. For example, if the HMD device identifies a second gesture input corresponding to a gesture (grab release gesture) that releases the user's hand (6132) that has grabbed the first graphic UI element (610) through a camera, the first graphic UI element (610) may be switched to a grab release state. The grab release gesture may be, for example, a gesture that switches a part of the display area of ​​the first graphic UI element (610) (e.g., a designated specific area) from a pinch-in state to a pinch-out state, but is not limited thereto. For example, the grab release gesture may be a pinch-out gesture in which the thumb and index finger are separated by more than a specified distance from a state where they are within a specified distance (e.g., the thumb and index finger are touching). According to one embodiment, the HMD device may switch the first graphic UI element (610) from a grab state to a grab state if a second gesture input corresponding to the grab release gesture is identified after the activation area (630) and the area corresponding to the first graphic UI element (610) overlap by more than a specified size for a specified period through the above-described 2-1 action, or after the display of the activation area (630) is changed.

[0097] According to one embodiment, in operation 550, the HMD device may transmit a signal (hereinafter, activation signal) to activate an HMD device support function to a wearable electronic device through a communication circuit based on identifying that a second condition is satisfied. The activation signal may include, for example, information for triggering the activation of the HMD device support function (hereinafter, trigger information), and / or additional information related to the activation of the HMD device support function (hereinafter, activation-related additional information). The activation-related additional information may include, for example, information about a first graphic UI element (610) and / or information about at least one physical UI (PUI) element set in the HMD device. According to one embodiment, the HMD device may stop displaying the first graphic UI element (610) after transmitting the activation signal, but is not limited thereto.

[0098] According to one embodiment, the wearable electronic device (400) can activate an HMD device support function based on a received activation signal. For example, the wearable electronic device (400) can enter a designated mode (e.g., HMD support mode) to activate the HMD device support function in response to the reception of an activation signal.

[0099] According to one embodiment, the wearable electronic device (400) may display a second graphic UI element (620) corresponding to a first graphic UI element (610) through the display of the wearable electronic device (400) in response to the activation of the HMD device support function (or in response to entering the HMD support mode), as exemplified in the third scene (6300), and / or provide at least one second physical UI element mapped to at least one first physical UI element of the HMD device. Meanwhile, according to an embodiment, instead of the second graphic UI element (620) being actually displayed through the display of the wearable electronic device (400), it may be virtually displayed through the display of the HMD device. For example, the HMD device may display a virtual object corresponding to the second graphic UI element (620) through the display of the HMD device on a display area of ​​the wearable electronic device (400) identified through a camera. By using these second graphic UI elements (620), the same functions of the HMD device can be performed without using the first graphic UI elements (610) of the HMD device, thereby increasing user convenience.

[0100] According to one embodiment, the second graphic UI element (620) may include at least some of one or more graphic objects (e.g., icons) included in the first graphic UI element (610). Each graphic object included in the first graphic UI element (610) may be used to perform a corresponding function. For example, an application icon included in the first graphic UI element (610) may be used to run the corresponding application.

[0101] According to one embodiment, the second graphic UI element (620) may have a different shape (or size) from the first graphic UI element (610). For example, the second graphic UI element (620) may have the same shape as the display of the wearable electronic device (400) on which the second graphic element (620) is displayed (e.g., circular), and the first graphic UI element (610) may have a different specified shape (e.g., the shape of a horizontal bar). By providing a second graphic UI element modified according to the shape of the part displayed in this way, a graphic UI element with a shape suitable for display on the wearable electronic device (400) can be provided to the user.

[0102] According to one embodiment, the wearable electronic device (400) can map at least one physical UI element of the HMD device (e.g., a physical button of the HMD device) to at least one physical UI element of the wearable electronic device (400). For example, the wearable electronic device (400) can map a first physical button of the HMD device to the first physical button of the wearable electronic device (400) and map a second physical button of the HMD device to the second physical button of the wearable electronic device (400). In this case, a function corresponding to the physical UI element of the HMD device can be performed on the HMD device through the physical UI element of the wearable electronic device (400). Through this, the user does not need to raise their hand above their head to perform a function corresponding to the physical UI element of the HMD device, thereby providing the user with convenience in controlling the function.

[0103] FIG. 7 is a diagram illustrating the operation of a wearable electronic device switching to an HMD support mode according to one embodiment of the present disclosure.

[0104] Referring to FIG. 7, according to one embodiment, a wearable electronic device (400) may operate in one of a plurality of modes. The plurality of modes may include, for example, a general mode that supports the functions of the wearable electronic device (400) and an HMD support mode that supports the functions of an HMD device (e.g., the electronic device (101) of FIG. 1, the wearable electronic device (200) of FIG. 2, or the wearable electronic device (300) of FIG. 3a to 3c). In the present disclosure, the general mode may be referred to as a first mode or a first operation mode, and the HMD support mode may be referred to as a second mode or a second operation mode.

[0105] According to one embodiment, the wearable electronic device (400) may switch modes based on a specified condition or a specified input signal. For example, the wearable electronic device (400) may switch the mode from a normal mode to an HMD support mode based on receiving an activation signal to activate an HMD support function (HMD device support function) from an HMD device in a normal mode. For example, the wearable electronic device (400) may switch the mode from an HMD support mode to a normal mode based on identifying that the connection between the HMD device and the wearable electronic device is disconnected in the HMD support mode.

[0106] According to one embodiment, the wearable electronic device (400) may provide a screen including a general graphic UI element (600) in a general mode, as exemplified in the first scene (710), and may provide a second screen including a second graphic UI element (620) in an HMD support mode, as exemplified in the second scene (720). The general graphic UI element (600) may include, for example, at least one graphic object (601) (e.g., an icon) associated with the performance of a function of the wearable electronic device (400). The second graphic UI element (620) may include, for example, at least one graphic object (621) (e.g., an icon) associated with the performance of an HMD device support function. The second graphic UI element (620) may correspond, for example, to the first graphic UI element (610) of FIG. 6. In this way, in HMD support mode, a second graphic UI element (620) associated with the HMD device support function is provided through the screen of the wearable electronic device (400), thereby enabling support of the HMD device through the wearable electronic device (400).

[0107] According to one embodiment, the wearable electronic device (400) can provide at least one function of the wearable electronic device through at least one physical UI element (e.g., PUI (420) of FIG. 4) in a normal mode, and can provide at least one function associated with an HMD device support function through at least one physical UI element (e.g., PUI (420) of FIG. 4) in an HMD support mode. For example, when user input is received through a first physical UI element (e.g., first PUI (421) of FIG. 4) in a normal mode, the wearable electronic device (400) can provide a designated function of the wearable electronic device corresponding to the first physical UI element. For example, when user input is received through a first physical UI element (e.g., the first PUI (421) of FIG. 4) in HMD support mode, the wearable electronic device (400) may provide a designated function of the HMD device corresponding to the first physical UI element of the HMD device mapped to the first physical UI element. Thus, even if user input is received through the same physical UI element, different functions may be provided depending on the operation mode.

[0108] FIG. 8 is a diagram illustrating the mapping relationship between a physical UI of a wearable electronic device and a physical UI of an HMD device according to one embodiment of the present disclosure.

[0109] Referring to FIG. 8, according to one embodiment, an HMD device (200, 300) and a wearable electronic device (400) may each provide at least one PUI. For example, the HMD device (200, 300) and the wearable electronic device (400) may each provide at least one PUI through a physical input device included in the respective device. The physical input device may include, for example, a physical button, a pressure sensor, a touch sensor, a slider and / or a dial, but is not limited thereto. Unlike a visual-oriented graphic UI, the PUI may provide intuitive and sensory interaction.

[0110] According to one embodiment, when the wearable electronic device (400) operates in an HMD support mode or when the HMD device support function is enabled, at least one PUI of the wearable electronic device (400) (e.g., PUI (420) of FIG. 4) may be mapped to at least one PUI of the HMD device (200, 300). For example, a first PUI of the wearable electronic device (400) (e.g., first PUI (421) of FIG. 4) may be mapped to the first PUI of the HMD device (200, 300), and a second PUI of the wearable electronic device (400) (e.g., second PUI (422) of FIG. 4) may be mapped to the second PUI of the HMD device (200, 300). In this case, the operation result resulting from user input through the first PUI of the wearable electronic device (400) may be the same as the operation result resulting from user input through the first PUI of the HMD device (200, 300) that is mapped to each other (hereinafter, the first operation result), and the second operation result resulting from user input through the second PUI of the wearable electronic device (400) (hereinafter, the second operation result) may be the same as the operation result resulting from user input through the second PUI of the HMD device (200, 300) that is mapped to each other. For example, the operation corresponding to user input of tapping the first PUI of the HMD device (200, 300) once, user input of long tapping, and user input of tapping four times may be an operation that executes the A1 function, A2 function, and A3 function of the HMD device (200, 300), respectively. This may be identical to the actions corresponding to a user input pressing the first PUI (e.g., the first physical button) of the wearable electronic device (400) once, a user input pressing it for a long time, and a user input pressing it four times, respectively.For example, actions corresponding to a user input that taps the second PUI of the HMD device (200, 300) once, a user input that long-taps, and a user input that taps four times may be actions that execute the B1 function, B2 function, and B3 function of the HMD device (200, 300), respectively. These may be identical to actions corresponding to a user input that presses the second PUI (e.g., the second physical button) of the wearable electronic device (400) once, a user input that long-presses, and a user input that presses four times, respectively. Through this PUI mapping, a function corresponding to the PUI of the HMD device (200, 300) can be provided through the PUI of the wearable electronic device (400). In this case, the user does not need to raise their hand above their head to perform a function corresponding to the PUI element of the HMD device (200, 300), thereby providing the user with convenience in controlling the function.

[0111] FIG. 9 is a diagram illustrating the correspondence between a graphic UI element of a wearable electronic device and a graphic UI element of an HMD device according to one embodiment of the present disclosure.

[0112] Referring to FIG. 9, according to one embodiment, an HMD device (e.g., the electronic device (101) of FIG. 1, the wearable electronic device (200) of FIG. 2, or the wearable electronic device (300) of FIG. 3a to 3c) and a wearable electronic device (e.g., the wearable electronic device (400) of FIG. 4) may provide corresponding graphic UI elements. For example, as illustrated in the first scene (9100) and the second scene (9200), when the wearable electronic device is operating in an HMD support mode or when the HMD device support function is activated, the wearable electronic device may display a second graphic UI element (620) corresponding to the first graphic UI element (610) of the HMD device. By using this second graphic UI element (620), at least some of the functions provided by the HMD device through the first graphic UI element (610) may be provided through the wearable electronic device.

[0113] According to one embodiment, the second graphic UI element (620) may include at least one second graphic object corresponding to one or more of the at least one first graphic object included in the first graphic UI element (610). The first graphic object and the second graphic object corresponding to the first graphic object may be used to perform the same function of the HMD device. For example, an action corresponding to a user input selecting the first-1 graphic object (611) may provide the same result of action (e.g., providing the same function to enter the settings menu of the HMD device) as an action corresponding to a user input selecting the second-1 graphic object (621) corresponding to the first-1 graphic object (611). For example, an action corresponding to a user input selecting the first-2 graphic object (612) may provide the same result of action (e.g., providing the same function to enter the detailed app list of the HMD device) as an action corresponding to a user input selecting the second-2 graphic object (622) corresponding to the first-2 graphic object (612).

[0114] According to one embodiment, the wearable electronic device may transmit a signal to the HMD device indicating that the second graphic object (621, 622) has been selected in response to receiving user input selecting the second graphic object (621, 622). In this case, the HMD device may perform the function of the HMD device associated with the first graphic object (611, 612) corresponding to the selected second graphic object. In this way, the same function of the HMD device may be performed through the corresponding graphic object.

[0115] According to one embodiment, the first graphic object (611, 612) and the second graphic object (621, 622) corresponding to the first graphic object (611, 612) may have the same shape. For example, the second graphic object (621, 622) corresponding to the first graphic object (611, 612) may have the same shape as the first graphic object (611, 612) and may have a reduced or enlarged size of the same shape.

[0116] According to one embodiment, the second graphic UI element (620) may be set based on status information of the HMD device. For example, the number, shape, and / or size of the second graphic objects included in the second graphic UI element (620) may be set differently depending on the status of the HMD device. An example of setting the second graphic UI element (620) according to the status information of the HMD device is described below with reference to FIGS. 16a and 16b.

[0117] According to one embodiment, the second graphic UI element (620) may be displayed even when the first graphic UI element (610) of the HMD device is hidden or not displayed. Thus, even when the first graphic UI element (610) of the HMD device is unavailable, the user can use at least one function (e.g., system settings function) associated with the first graphic UI element (610) through the second graphic UI element (620) of the wearable electronic device.

[0118] FIG. 10 is a diagram illustrating the operation of an HMD device and a wearable electronic device performing the same function through graphic UI elements according to one embodiment of the present disclosure.

[0119] In the embodiment of FIG. 10, it is assumed that a first graphic UI element (e.g., the first graphic element (610) of FIG. 6) is in a hidden state in an HMD device (e.g., the electronic device (101) of FIG. 1, the wearable electronic device (200) of FIG. 2, or the wearable electronic device (300) of FIG. 3a to 3c). The first graphic UI element may be switched to a hidden state, for example, after an activation signal is transmitted or after a second graphic UI element (e.g., the second graphic element (620) of FIG. 6) is displayed.

[0120] Referring to FIG. 10, as exemplified in the first scene (1010), a first graphic UI element (e.g., a task bar) may not be displayed through the display of the HMD device. In this state, the HMD device may display the first graphic UI element through the display, as exemplified in the second scene (1020), in response to receiving an input (101) for displaying the first graphic UI element. The HMD device may perform a function corresponding to the first graphic object (e.g., a function to display the settings menu of the HMD device), as exemplified in the third scene (1030), in response to receiving an input (102) for selecting a first graphic object included in the first graphic UI element (e.g., a graphic object related to the settings menu of the HMD device). Thus, in order for the HMD device to perform the function provided by the first graphic UI element while the first graphic UI element is hidden, two stages of user input are required.

[0121] According to one embodiment, as illustrated in the fourth scene (1040), the second graphic element can be displayed through the display of the wearable electronic device even when the first graphic element is not displayed. In response to the acquisition of an input (103) selecting a second graphic object included in the second graphic element (e.g., a graphic object related to the settings menu of the HMD device), the wearable electronic device can perform a function corresponding to the first graphic object (e.g., a function to display the settings menu of the HMD device), as illustrated in the third scene (1030). Thus, only one step of user input is required to perform the function provided by the corresponding first graphic UI element through the second graphic UI element of the wearable electronic device while the first graphic UI element is hidden. Therefore, the function of the HMD device can be provided more quickly and conveniently.

[0122] FIGS. 11a to 11d are drawings for explaining the operation of providing a graphic UI of an HMD device through a wearable electronic device according to one embodiment of the present disclosure.

[0123] According to one embodiment, an HMD device (e.g., the electronic device (101) of FIG. 1, the wearable electronic device (200) of FIG. 2, or the wearable electronic device (300) of FIG. 3a to 3c) may transmit an activation signal to a wearable electronic device (e.g., the wearable electronic device (400) of FIG. 4) to activate an HMD device support function based on identifying that a first condition (activation preparation condition) and a second condition (activation condition) are satisfied for activating an HMD device support function associated with a first graphic UI element (e.g., the first graphic UI element (610) of FIG. 6). The HMD device support function associated with the first graphic UI element may include, for example, a function to display a second graphic UI element (e.g., the first graphic UI element (620) of FIG. 6) corresponding to the first graphic UI element through the display of the wearable electronic device. The wearable electronic device can activate an HMD device support function based on a received signal and display a second graphic UI element corresponding to a first graphic UI element through the display of the wearable electronic device.

[0124] According to one embodiment, the second graphic UI element may be configured based on the first graphic UI element and / or the display settings of the wearable electronic device (e.g., display size, shape). As such, since the second graphic UI element is configured in consideration of the first graphic UI element and the display settings of the wearable electronic device, the second graphic UI element, which dynamically changes the first graphic UI element to match the characteristics of the wearable electronic device's display, may be displayed. Hereinafter, embodiments providing the first graphic UI element and the second graphic UI element are described, exemplified by the first graphic UI element and the second graphic UI element being graphic UI elements for providing various system UIs.

[0125] According to one embodiment, as illustrated in FIG. 11a, the first graphic UI element (1110a) may be a graphic UI element corresponding to a task bar and may include at least one first graphic object for task management and quick access. In this case, the wearable electronic device may display the first graphic UI element (1110a) of the task bar as a second graphic UI element (1120a) by changing the layout to suit the display characteristics of the wearable electronic device. In the example of FIG. 11a, all second graphic objects corresponding to all of the first graphic objects included in the first graphic UI element (1110a) may be displayed within one screen of the wearable electronic device.

[0126] According to one embodiment, as illustrated in FIG. 11b, the first graphic UI element (1110b) may be a graphic UI element corresponding to a settings menu (e.g., a quick settings menu) and may include at least one first graphic object for system settings. In this case, the wearable electronic device may display the first graphic UI element (1110b) of the settings menu as a second graphic UI element (1120b) by changing the layout to suit the display characteristics of the wearable electronic device. In the example of FIG. 11b, only the second graphic objects corresponding to some of the first graphic objects included in the first graphic UI element (1110b) may be displayed within one screen of the wearable electronic device.

[0127] According to one embodiment, as illustrated in FIG. 11c, the first graphic UI element (1110c) may be a graphic UI element corresponding to a navigation menu (e.g., a gesture navigation menu) and may include at least one first graphic object for navigation. In this case, the wearable electronic device may display the first graphic UI element (1110c) of the navigation menu as a second graphic UI element (1120c) by changing the layout to match the display characteristics of the wearable electronic device. In the example of FIG. 11c, a second graphic object having the same top-bottom-left-right arrangement as the first graphic objects within the first graphic UI element (1110c) may be displayed on one screen of the wearable electronic device.

[0128] According to one embodiment, as illustrated in FIG. 11d, the first graphic UI element (1110d) may be a graphic UI element corresponding to an application menu (e.g., an application group) and may include at least one first graphic object for executing an application. In this case, the wearable electronic device may display the first graphic UI element (1110d) of the application menu as a second graphic UI element (1120d) by changing the layout to suit the display characteristics of the wearable electronic device. In the example of FIG. 11d, only the second graphic objects corresponding to some of the first graphic objects included in the first graphic UI element (1110d) may be displayed on one screen.

[0129] FIGS. 12 and 13 are drawings for explaining the operation of providing graphic UI elements of an HMD device through a wearable electronic device using a dynamic layout function according to one embodiment of the present disclosure.

[0130] The embodiments of FIGS. 12 and 13 may be examples of the embodiments of FIGS. 5 and 6. The description of the embodiments of FIGS. 12 and 13 may include all descriptions of the embodiments of FIGS. 5 and 6, provided that they do not contradict each other.

[0131] According to one embodiment, the dynamic layout function may include a function to display an active area (630), a function to change the display of the active area (630) (e.g., display of the color of the active area), and / or a function to change the display of a first graphic UI element (610) (e.g., display of the size and / or shape of the first graphic element (610). In the present disclosure, the dynamic layout function may be referred to as a dynamic display function.

[0132] According to one embodiment, the function of displaying the activation area (630) may be performed when a first condition for activating the HMD device support function is satisfied. For an explanation of this, refer to the description related to the first condition in FIGS. 5 and FIGS. 6.

[0133] According to one embodiment, the function of changing the display of the active area (630) may be performed after the first condition is satisfied. For example, the function of changing the display of the active area (630) may be performed when a specified display change condition is satisfied after the first condition is satisfied. The display change condition may be satisfied, for example, when the area corresponding to the active area (630) and the first graphic UI element (610) overlaps for a specified period of time with a size greater than a specified size. For an explanation of this, refer to the description related to the second condition in FIGS. 5 and FIGS. 6.

[0134] According to one embodiment, the function of changing the display of the first graphic UI element (610) may be performed after the first condition is satisfied or after the second condition for activating the HMD device support function is satisfied. For example, the function of changing the display of the first graphic UI element (610) may be performed when a specified display change condition is satisfied after the first condition is satisfied, similar to the function of changing the display of the activation area (630). For example, the function of changing the display of the first graphic UI element (610) may be performed after the second condition is satisfied and before the second graphic UI element corresponding to the first graphic UI element (610) is displayed on the wearable electronic device. For an explanation of this, refer to the description related to the second condition in FIGS. 5 and FIGS. 6.

[0135] According to one embodiment, as illustrated in FIG. 12, the first graphic UI element (610) may be a graphic UI element corresponding to a task bar. As illustrated in the first scene (1210), the HMD device may virtually display an active area (630) on an area corresponding to the display of the wearable electronic device in response to the first condition being satisfied. The active area (630) may have a first color (e.g., blue). As illustrated in the second scene (1220), the HMD device may change the color of the active area (630) from the first color to a second color (e.g., purple) in response to the satisfaction of a specified display change condition after the first condition is satisfied. Through this, the changed active area (630a) may be displayed in place of the active area (630). The HMD device may change the display of the first graphic UI element (610) to be identical to the display of the second graphic UI element (620) to be displayed on the wearable electronic device in response to the satisfaction of a specified display change condition after the first condition is satisfied, or in response to the satisfaction of a second condition through the release (e.g., drop) of the first graphic UI element (610). Through this, the changed first graphic UI element (610a) may be displayed in place of the first graphic UI element (610). Subsequently, as exemplified in the third scene (1230), the HMD device may virtually display the second graphic UI element (620) on an area corresponding to the display of the wearable electronic device. Through this dynamic display change of the first graphic UI element (610), a visual effect may be provided in which the first graphic UI element (610) naturally changes into the second graphic UI element (620) and is displayed on the wearable electronic device.

[0136] According to one embodiment, as illustrated in FIG. 13, the first graphic UI element (610) may be a graphic UI element corresponding to a settings menu (e.g., quick settings menu). As illustrated in the first scene (1310), the HMD device may virtually display an active area (630) on an area corresponding to the display of the wearable electronic device in response to the first condition being satisfied. The active area (630) may have a first color (e.g., blue). As illustrated in the second scene (1320), the HMD device may change the color of the active area (630) from the first color to a second color (e.g., purple) in response to the satisfaction of a specified display change condition after the first condition is satisfied. Through this, the changed active area (630a) may be displayed in place of the active area (630). The HMD device may change the display of the first graphic UI element (610) to be identical to the display of the second graphic UI element (620) to be displayed on the wearable electronic device in response to the satisfaction of a specified display change condition after the first condition is satisfied, or in response to the satisfaction of a second condition through the release (e.g., drop) of the first graphic UI element (610). Through this, the changed first graphic UI element (610a) may be displayed in place of the first graphic UI element (610). Subsequently, as exemplified in the third scene (1330), the HMD device may virtually display the second graphic UI element (620) on an area corresponding to the display of the wearable electronic device. Through this dynamic display change of the first graphic UI element (610), a visual effect may be provided in which the first graphic UI element (610) naturally changes into the second graphic UI element (620) and is displayed on the wearable electronic device.

[0137] FIGS. 14a and 14b are drawings for explaining the operation of providing a graphic UI of an HMD device through a wearable electronic device using a dynamic layout function according to one embodiment of the present disclosure.

[0138] The embodiments of FIGS. 14a and 14b may be examples of the embodiments of FIGS. 5 and 6. The description of the embodiments of FIGS. 14a and 14b may include all descriptions of the embodiments of FIGS. 5 and 6, provided that they do not contradict each other.

[0139] According to one embodiment, the dynamic layout function may include a function to display an active area (630), a function to change the display of the active area (630) (e.g., display of the color of the active area), and / or a function to change the display of a first graphic UI element (610) (e.g., display of the size and / or shape of the first graphic element (610).

[0140] In the embodiments of FIGS. 14a and 14b, it is assumed that a first graphic UI element of an HMD device (e.g., the electronic device (101) of FIG. 1, the wearable electronic device (200) of FIG. 2, or the wearable electronic device (300) of FIGS. 3a to 3c) is a graphic UI element corresponding to a task bar, and that a grab gesture and a grab release gesture for the first graphic UI element are used to activate an HMD device support function associated with the first graphic UI element. However, the embodiments are not limited thereto, and various types of system UI and various types of input may be used.

[0141] According to one embodiment, as illustrated in the first scene (1410), the HMD device may display at least one execution screen (e.g., the execution screen (6110) of FIG. 6) and / or a first graphic UI element (e.g., the first graphic UI element (610) of FIG. 6) corresponding to a task bar in an HMD environment (e.g., a virtual environment). The HMD device may display a graphic element (1441) (e.g., an indicator) in a portion of the display area of ​​the first graphic UI element (610) to indicate that the first graphic UI element (610) is grabable. For example, a semicircle marked with a designated color (e.g., white) in the right area of ​​the display area of ​​the first graphic UI element (610) may be provided as the graphic element (1441). Through this graphic element (1441), the user may recognize that the first graphic UI element (610) is grabable.

[0142] According to one embodiment, as illustrated in the second scene (1420), the HMD device can identify (e.g., detect) a gesture input corresponding to a grab gesture that grabs the first graphic UI element (610). As described above, the grab gesture may correspond, for example, to a pinch-in gesture of at least a portion of the display area of ​​the first graphic UI element (e.g., a designated specific area). In response to the identification of a gesture input corresponding to the grab gesture, the HMD device may display a graphic element (1421) (e.g., an indicator) indicating that the first graphic UI element (610) has transitioned to a grab state on at least a portion of the display area of ​​the first graphic UI element (610). For example, a visual effect that displays the border of the display area of ​​the first graphic UI element (610) in a designated color (e.g., white) may be provided as the graphic element (1421). Through this graphic element (1421), the user can recognize that the first graphic UI element (610) is in a grab state. By displaying the entire border in a designated color in this way, when the first graphic UI element (610) is moved, the area (1431) corresponding to the part where the first graphic UI element (610) was moved remains empty, as exemplified in the third scene (1430), thereby providing a visual effect such as peeling off a sticker.

[0143] According to one embodiment, as illustrated in the third scene (1430), the HMD device may identify (e.g., detect) that a hand is moving while maintaining a grab gesture on the first graphic UI element (610), and may move the first graphic UI element (610) to a position and / or direction corresponding to the movement of the hand. In this case, the first graphic UI element (610) may maintain the same size and shape.

[0144] According to one embodiment, as illustrated in the fourth scene (1440), the HMD device may reduce the size of the first graphic UI element (610) from a first size to a second size in response to identifying (e.g., detecting) that the hand moves for a specified period or a specified distance while maintaining a grab gesture on the first graphic UI element (610). The first graphic UI element (610b) reduced to the second size (hereinafter, reduced first graphic UI element) may include all or part of the graphic objects corresponding to the first graphic objects included in the first graphic UI element (610). When using this reduced first graphic UI element (610b), the portion of the graphic UI element that is obscured on the screen while it is moving may be relatively reduced compared to when using the first graphic UI element (610). This prevents the user's field of vision from being obstructed.

[0145] According to one embodiment, as illustrated in the fifth scene (1450), the HMD device may display an activation area (630) on an area corresponding to the display of a wearable electronic device (e.g., the wearable electronic device (400) of FIG. 4) (hereinafter, display area) in response to identification that a first condition (activation readiness condition) for activating an HMD device support function (e.g., HMD support mode) is satisfied. Through the activation area (630), the user can recognize the readiness state for the HMD device support function.

[0146] According to one embodiment, as illustrated in the sixth scene (1460), the HMD device can identify whether a second condition (activation condition) for activating the HMD device support function is satisfied. In the embodiments of FIG. 14a and 14b, unlike the embodiment of FIG. 6, whether the second condition is satisfied can be identified based on a reduced first graphic UI element (610b) rather than the first graphic UI element (610). The description of the second condition determination operation for the first graphic UI element (610) described above can be applied in the same way to the description of the second condition determination operation for the reduced first graphic UI element (610b). For example, the HMD device may identify that the second condition is satisfied when, after at least a portion of the reduced first graphic UI element (610b) (e.g., an area larger than a specified size) is maintained in an overlaid state (e.g., an overlapping state) on the display area of ​​the wearable electronic device for a certain period of time, a gesture input is obtained to release the grab gesture for the reduced first graphic UI element (610b). As described above, the HMD device may change the color of the active area (630) from the first color (e.g., blue) to the second color (e.g., purple) in response to the satisfaction of the specified display change condition after the first condition is satisfied. Through this, the changed active area (630a) may be displayed in place of the active area (630). By confirming the display of the changed active area (630a), the user may release the grab (e.g., drop) the reduced first graphic UI element (610b).

[0147] According to one embodiment, as illustrated in the seventh scene (1470), the HMD device may transmit an activation signal containing trigger information for activating an HMD device support function to a wearable electronic device in response to identifying that the second condition is satisfied.

[0148] According to one embodiment, as illustrated in the eighth scene (1480), the wearable electronic device may activate an HMD device support function (e.g., HMD support mode) based on a received signal and display a second graphic UI element (620) corresponding to a first graphic UI element (610). The wearable electronic device may map at least one PUI of the HMD device to at least one PUI of the wearable electronic device.

[0149] FIGS. 15a and 15b are drawings for explaining the operation of providing a graphic UI of an HMD device through a wearable electronic device using a dynamic layout function according to one embodiment of the present disclosure.

[0150] The embodiments of FIGS. 15a and 15b may be examples of the embodiments of FIGS. 5 and 6. The description of the embodiments of FIGS. 15a and 15b may include all descriptions of the embodiments of FIGS. 5 and 6, provided that they do not contradict each other.

[0151] According to one embodiment, the dynamic layout function may include a function to display a first graphic UI element (610) (or a reduced first graphic UI element (610b)) based on gesture input, a function to display an active area (630), a function to change the display of the active area (630) (e.g., display of the color of the active area), and / or a function to change the display of the first graphic UI element (610) (e.g., display of the size and / or shape of the first graphic element (610).

[0152] In the embodiments of FIGS. 14a and 14b, it is assumed that a first graphic UI element of an HMD device (e.g., the electronic device (101) of FIG. 1, the wearable electronic device (200) of FIG. 2, or the wearable electronic device (300) of FIGS. 3a to 3c) is a graphic UI element corresponding to a task bar, and that a type of gesture other than a grab gesture and a grab release gesture is used to activate the HMD device support function associated with the first graphic UI element. However, the embodiments are not limited thereto, and various types of system UI and various types of input may be used.

[0153] According to one embodiment, as illustrated in the first scene (1510), the HMD device may acquire a first gesture input corresponding to a designated first gesture (1632a) (e.g., a gesture showing the back of the hand (or, palm)) while the first graphic UI element corresponding to the task bar (e.g., the first graphic UI element (610) of FIG. 6) is not displayed. In response to the acquisition of the first gesture input, the HMD device may display a first graphic object / element (1511) indicating that the first graphic UI element (e.g., the first graphic UI element (610) of FIG. 6) is available for display. The first graphic element (1511) may be displayed at a location adjacent to the hand associated with the first gesture (1632a).

[0154] According to one embodiment, as illustrated in the second scene (1520), the HMD device may acquire a second gesture input corresponding to a designated second gesture (1632b) (e.g., a hand-flipping gesture) while the first graphic element (1511) is displayed. In response to the acquisition of the second gesture input, the HMD device may display a reduced second graphic UI element (610b) (or the first graphic UI element) in place of the first graphic element (1511). The first graphic UI element (610b) may be displayed at the location where the first graphic element (1511) is displayed.

[0155] According to one embodiment, as illustrated in the third scene (1530), the HMD device can identify (e.g., detect) a third gesture (1632c) in which a hand associated therewith moves while a reduced second graphic UI element (610b) is displayed, and move the reduced second graphic UI element (610b) to a position and / or direction corresponding to the hand movement.

[0156] According to one embodiment, as illustrated in the fourth scene (1540) and the fifth scene (1550), the HMD device may display an activation area (630) on an area corresponding to the display of a wearable electronic device (e.g., the wearable electronic device (400) of FIG. 4) (hereinafter, display area) in response to identification that a first condition (activation readiness condition) for activating an HMD device support function (e.g., HMD support mode) is satisfied. Through the activation area (630), the user can recognize the readiness state for the HMD device support function.

[0157] According to one embodiment, as illustrated in the sixth scene (1560), the HMD device can identify whether a second condition (activation condition) for activating the HMD device support function is satisfied. In the embodiments of FIG. 15a and 15b, unlike the embodiment of FIG. 6, whether the second condition is satisfied can be identified based on a reduced first graphic UI element (610b) rather than the first graphic UI element (610b). The description of the second condition determination operation for the first graphic UI element described above can be applied in the same way to the description of the second condition determination operation for the reduced first graphic UI element (610b). For example, the HMD device may identify that the second condition is satisfied when, after at least a portion of the reduced first graphic UI element (610b) (e.g., an area larger than a specified size) is maintained in an overlaid state (e.g., an overlapping state) on the display area of ​​the wearable electronic device for a certain period of time, a gesture input corresponding to a gesture (1632d) (e.g., a pinch-in gesture) that releases the association between the reduced first graphic UI element (610b) and the hand is obtained. As described above, the HMD device may change the color of the active area (630) from the first color (e.g., blue) to the second color (e.g., purple) in response to the satisfaction of the specified display change condition after the first condition is satisfied. Through this, the changed active area (630a) may be displayed in place of the active area (630). According to one embodiment, the HMD device may transmit an activation signal to a wearable electronic device, including trigger information for activating an HMD device support function, in response to identifying that a second condition is satisfied.

[0158] According to one embodiment, as illustrated in the seventh scene (1570), the wearable electronic device may activate an HMD device support function (e.g., HMD support mode) based on a received signal and display a second graphic UI element (620) corresponding to a first graphic UI element (610). The wearable electronic device may map at least one PUI of the HMD device to at least one PUI of the wearable electronic device.

[0159] FIGS. 16a and 16b are drawings for explaining the operation of setting graphic UI elements according to the state of an HMD device according to one embodiment of the present disclosure.

[0160] According to one embodiment, the first graphic UI element (610) and / or the second graphic UI element (620) corresponding to the first graphic UI element (610) may be set differently depending on the state of the HMD device. For example, as illustrated in the first scene (1610) of FIG. 16a, when the HMD device is in a first state (state A), the first graphic UI element (610) and the second graphic UI element (620) set according to the first setting may be displayed. For example, as illustrated in the second scene (1620) of FIG. 16b, when the HMD device is in a second state (state B), the first graphic UI element (610c) and the second graphic UI element (620c) set according to a second setting different from the first setting may be displayed. According to one embodiment, the first state may be a state in which at least one execution screen is displayed on a first screen (e.g., full screen) of the HMD device, and the second state may be a state in which at least one execution screen is displayed on a part of the first screen (e.g., full screen) of the HMD device. In this case, for a graphic UI element (e.g., system UI) associated with the second state, it may be required to provide fewer pieces of information compared to a graphic UI element (e.g., system UI) associated with the first state.

[0161] According to one embodiment, the first graphic UI element (610c) and the second graphic UI element (620c) set according to the second setting may have different characteristics from the first graphic UI element (610) and the second graphic UI element (620) set according to the first setting. For example, the number of graphic objects (e.g., icons) included in the first graphic UI element (610c) and the second graphic UI element (620c) set according to the second setting (e.g., 7) may be different (e.g., fewer) from the number of graphic objects (e.g., icons) included in the first graphic UI element (610) and the second graphic UI element (620) set according to the first setting (e.g., 4).

[0162] FIG. 17 is a diagram illustrating the operation of an HMD device and a wearable electronic device performing the same function through graphic UI elements according to one embodiment of the present disclosure.

[0163] The embodiment of FIG. 17 may be, for example, an example of the embodiment of FIG. 10. The description of the embodiment of FIG. 17 may include all of the description of the embodiment of FIG. 10, provided that they are not contradictory.

[0164] In the embodiment of FIG. 17, it is assumed that a first graphic UI element (e.g., the first graphic element (610) of FIG. 6) is hidden in an HMD device (e.g., the electronic device (101) of FIG. 1, the wearable electronic device (200) of FIG. 2, or the wearable electronic device (300) of FIG. 3a to 3c). For example, when a specific application, such as a game application, is running, it may be necessary to keep the first graphic UI element (610) hidden and not display it on the screen of the HMD device so as not to distract the user.

[0165] Referring to FIG. 17, as exemplified in the first scene (1710), the first graphic UI element (e.g., a task bar) may not be displayed through the display of the HMD device. In this state, the HMD device may display the first graphic UI element (610) through the display, as exemplified in the second scene (1720), in response to receiving a first input for calling the first graphic UI element. The HMD device may perform a function corresponding to the first graphic object (e.g., a function to display the settings menu of the HMD device), as exemplified in the fourth scene (1740), in response to receiving a second input for selecting a first graphic object included in the first graphic UI element (610), such as a graphic object related to the settings menu of the HMD device.

[0166] According to one embodiment, as illustrated in the third scene (1730), a second graphic element (e.g., the second graphic element (620) of FIG. 6) can be displayed through the display of a wearable electronic device (e.g., the wearable electronic device (400) of FIG. 4) even when the first graphic element (610) is not displayed. In response to receiving an input selecting a second graphic object (e.g., a graphic object related to the settings menu of the HMD device) included in the second graphic element (620), the wearable electronic device can perform a function corresponding to the second graphic object (e.g., a function to display the settings menu of the HMD device) as illustrated in the fourth scene (1740). Thus, the corresponding function of the HMD device can be performed directly through the second graphic UI element (620) of the wearable electronic device even when the first graphic UI element (610) is hidden.

[0167] FIGS. 18a to 18c are drawings for explaining an operation of providing a notification according to one embodiment of the present disclosure.

[0168] According to one embodiment, the HMD device may provide a notification through the HMD device and / or a wearable electronic device connected to the HMD device. For example, the HMD device may select one of a plurality of notification provision methods and provide a notification through the HMD device and / or a wearable electronic device connected to the HMD device according to the selected notification provision method. The plurality of notification provision methods may include, for example, a first notification provision method that provides a notification to both the HMD device and the first wearable electronic device, and a second notification provision method that provides a notification only to the first wearable electronic device.

[0169] According to one embodiment, the HMD device may select a notification delivery method based on information regarding a concentration score (hereinafter, concentration score information) related to an application being used on the HMD device (hereinafter, application being used) and / or information regarding a importance score (hereinafter, importance score information) related to a notification. For example, as illustrated in the first scene (1810) of FIG. 18a, if the concentration score is less than a reference concentration score, the HMD device may select a first notification delivery method as a notification delivery method and provide a notification to both the HMD device and the wearable electronic device according to the first notification delivery method. According to one embodiment, if the concentration score is greater than or equal to the reference concentration score, the HMD device may determine whether the importance score is greater than or equal to the reference importance score. For example, as illustrated in the second scene (1820) of FIG. 18b, if the concentration score is greater than or equal to the reference concentration score and the importance score is less than the reference importance score, the HMD device may select the second notification delivery method as the notification delivery method and provide notifications only to the wearable electronic device according to the second notification delivery method. By doing so, the user's concentration can be maintained by not providing low-importance notifications through the HMD device while the user of the HMD device is using the application with concentration. For example, as illustrated in the third scene (1830) of FIG. 18c, if the concentration score is greater than or equal to the reference concentration score and the importance score is greater than or equal to the reference importance score, the HMD device may select the first notification delivery method as the notification delivery method and provide notifications only to the wearable electronic device according to the first notification delivery method. By doing so, the user of the HMD device can check important notifications without missing them even while using the application with concentration.

[0170] According to one embodiment, an HMD device can obtain an attention score through a model (e.g., an AI model) learned based on behavioral information (or body information) of a user wearing the HMD device and / or application information of the application being used. The user's behavioral information may include, for example, information regarding the movement of the user's pupils, the blinking of the user's pupils, and / or the degree of interaction with the UI. The application information may include, for example, information regarding the execution status of the application and / or the type (or category) of the application. The correlation between the degree of interaction with the UI and the attention score may vary depending on the type of application. For example, in the case of a video application, high attention may be perceived when there is no interaction, while in the case of a drawing application, high attention may be perceived when there is a lot of interaction. The attention score may be associated with the type of application. For example, in the case of immersive applications such as game applications or video applications, high attention may be perceived compared to applications such as internet search applications. The concentration score can be associated with the user's physical information (e.g., eye information). For example, if eye movements are slow and blinking is minimal, the level of concentration may be perceived as high. The concentration score can be associated with the application's execution state. For example, compared to an application running in a normal state, if it runs in a state such as within a specific space to enhance user immersion, the level of concentration may be perceived as relatively higher. The trained AI model can output a concentration score by using all or part of the information related to the aforementioned concentration score as input data. The trained AI model may be stored in the HMD device or on a server connected to the HMD device.

[0171] According to one embodiment, an HMD device may obtain a notification importance score using a learned model (e.g., an AI model) based on the content of the notification, the frequency of notification checks, and / or related additional information. The importance score may be associated, for example, with the results of analyzing the content of the notification. The importance score may be obtained based on the results of analyzing the content of the notification, for example, whether the content of the notification is spam; whether the notification was sent to a phone number in the user's contact information; whether the notification was sent to a phone number not in the user's contact information but from a person currently in contact via message; whether the notification was sent to a phone number set as important; and / or whether the notification is a notification or schedule set by the user. The importance score may be associated, for example, with the results of analyzing the frequency of notification checks. For example, if a specific notification is frequently checked even in a state of high concentration, the importance score of that notification may be set to a value higher than the reference importance score. The trained AI model can output an importance score by using all or part of the information related to the aforementioned importance score as input data. The trained AI model can be stored in the HMD device or in a server connected to the HMD device.

[0172] FIG. 19 is a drawing illustrating a method for an external electronic device to provide an HMD device support function according to one embodiment of the present disclosure.

[0173] The embodiment of FIG. 19 may be, for example, an example of the embodiment of FIG. 5 and FIG. 6. The description of the embodiment of FIG. 19 may include all descriptions of the embodiment of FIG. 5 and FIG. 6, provided they are not contradictory. For example, the descriptions of the operation for determining whether the first condition and the second condition for activating the HMD device support function are satisfied, and the operation related to gesture input, as described in FIG. 5 and FIG. 6, may be applied in the same way to the embodiment of FIG. 19.

[0174] In the embodiment of FIG. 19, unlike the embodiments of FIG. 5 and FIG. 6, it is assumed that the electronic device providing the HMD device support function is a different type of electronic device, rather than a wearable electronic device worn on a user's wrist. For example, a mobile electronic device having a larger display area compared to the wearable electronic device (400) of FIG. 4 (e.g., the electronic device (101) of FIG. 1) may be used as the electronic device providing the HMD device support function. In this case, a larger number of information related to the HMD device support function may be provided through the larger display area.

[0175] Referring to FIG. 19, the HMD device can identify whether a first condition (enablement readiness condition) and / or a second condition (enablement condition) is satisfied for enabling an HMD device support function associated with a first-1 graphic UI element (1910a) (e.g., a task bar) based on a first activation area (1930a). In response to identifying that the first condition and / or the second condition is satisfied, the HMD device transmits a first activation signal to a wearable electronic device, and the wearable electronic device can display a second-1 graphic UI element (1920a) corresponding to the first-1 graphic UI element (1910a) based on the first activation signal. With the second-1 graphic UI element (1920a) displayed, the HMD device can identify whether a first condition (enablement readiness condition) and / or a second condition (enablement condition) for enabling an HMD device support function associated with the first-2 graphic UI element (1910b) (e.g., navigation bar) is satisfied based on the second activation area (1930b). In response to identifying that the first condition and / or the second condition is satisfied, the HMD device transmits a second activation signal to a wearable electronic device, and the wearable electronic device can display a second-2 graphic UI element (1920b) corresponding to the first-2 graphic UI element (1910b) together with the second-1 graphic UI element (1910a) based on the second activation signal. Through this, multiple second graphic UI elements can be provided by an external electronic device, and a wider variety of options for HMD device support through the external electronic device can be provided to the user.

[0176] According to one embodiment of the present disclosure, a head-mounted display (HMD) device may include a camera; a display; at least one processor including a communication circuit; and a processing circuit; and a memory including at least one storage medium for storing instructions. When the above commands are executed individually or collectively by the at least one processor, the HMD device may be caused to: display a first graphic UI (user interface) element through the display; identify whether a first condition for activating a function to support the HMD device in the wearable electronic device based on a first gesture input to the first graphic UI element is satisfied; the function to support the HMD device is associated with the first graphic UI element, and based on identifying that the first condition is satisfied, display an activation area associated with the wearable electronic device through the display; identify whether a second condition for activating a function to support the HMD device in the wearable electronic device is satisfied based on a second gesture input to the first graphic UI element and the activation area; and based on identifying that the second condition is satisfied, transmit a signal to the wearable electronic device through the communication circuit to activate the function to support the HMD device. The wearable electronic device may be configured to enable a function to support the HMD device based on the signal and to display a second graphic UI element corresponding to the first graphic UI element at a location associated with the wearable electronic device.

[0177] According to one embodiment, the first gesture input includes a gesture input corresponding to a gesture of grabbing the first graphic UI element, and the first graphic UI element is configured to be movable in a grabbed state according to the first gesture input, and the commands, when executed individually or collectively by the at least one processor, may cause the HMD device to: identify the wearable electronic device through the camera while the first graphic UI element is in a grabbed state, and identify whether the first condition is satisfied based on at least one of movement information of the wearable electronic device or connection status information between the wearable electronic device and the HMD device.

[0178] According to one embodiment, the operation of identifying whether the first condition is satisfied may include: an operation of identifying that, based on the movement information, the amount of change in movement of the wearable electronic device is maintained at or below a reference amount of change for a specified period; an operation of identifying that the wearable electronic device and the HMD device are connected based on the connection status information in response to identifying that the amount of change in movement is maintained at or below the reference amount of change for the specified period; and an operation of identifying that the first condition is satisfied in response to identifying that the wearable electronic device and the HMD device are connected.

[0179] According to one embodiment, the activation area may be displayed on the display area of ​​the wearable electronic device identified through the camera.

[0180] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the HMD device may be caused to identify whether the second condition is satisfied based on information regarding an overlap area in which the activation area and the area corresponding to the first graphic UI element overlap.

[0181] According to one embodiment, the second gesture input includes a gesture input corresponding to a gesture that releases the grabbed state of the first graphic UI element, and the operation of identifying that the second condition is satisfied may include: an operation of identifying that the activation area and the area corresponding to the first graphic UI element overlap for a specified period of time with a size greater than a specified size based on information about the overlap area; an operation of identifying that the first graphic UI transitions from a grabbed state to an ungrabbed state according to the second gesture input after identifying that the activation area and the area corresponding to the first graphic UI element overlap for a specified period of time with a size greater than the specified size; and an operation of identifying that the second condition is satisfied in response to identifying that the first graphic UI element transitions to an ungrabbed state.

[0182] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the HMD device is caused to: select a notification delivery method based on information regarding an attention score associated with an application being used on the HMD device and information regarding an importance score of a notification, and, according to the selected notification delivery method, to provide the notification on at least one of the HMD device or the wearable electronic device, wherein the notification delivery method may be a first notification delivery method that provides the notification on both the HMD device and the first wearable electronic device, or a second notification delivery method that provides the notification only on the first wearable electronic device.

[0183] According to one embodiment, the operation of selecting the notification providing method may include: selecting the first notification providing method as the notification providing method when the concentration score is less than the reference concentration score; selecting the second notification providing method as the notification providing method when the concentration score is greater than or equal to the reference concentration score and the importance score is less than the reference importance score; and selecting the first notification providing method as the notification providing method when the concentration score is greater than or equal to the reference concentration score and the importance score is greater than or equal to the reference importance score.

[0184] According to one embodiment, the concentration score is obtained through a model learned based on behavioral information of a user wearing the HMD device and application information of the application, and the importance score can be obtained through a model learned based on information about the content of the notification and information about the frequency of checking the notification.

[0185] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the HMD device may cause the size of the first graphic UI element to decrease based on identifying that the first graphic UI element has moved more than a specified time or distance while in a grabbed state.

[0186] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the HMD device may cause the second graphic UI element to be virtually displayed on an area corresponding to a display placed on the surface of the wearable electronic device identified through the camera, based on identifying that the second condition is satisfied.

[0187] According to one embodiment, the wearable electronic device may be configured to map at least one first physical UI element of the HMD device to at least one second physical UI element of the wearable electronic device based on the signal.

[0188] According to one embodiment, the first graphic UI element may include the system UI of the HMD device.

[0189] According to one embodiment, the wearable electronic device may be a watch-type wearable electronic device worn on a user's wrist.

[0190] The embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" each may include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as “coupled” or “connected” to another (e.g., 2nd) component, with or without the terms “functionally” or “communicationly,” it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0191] The term “module” as used in the embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0192] One embodiment of the present document may be implemented as software (e.g., program (140) of FIG. 1) comprising one or more instructions stored in a storage medium (e.g., internal memory (136) of FIG. 1 or external memory (138) of FIG. 1) that is readable by a machine (e.g., electronic device (101) of FIG. 1). For example, a processor (e.g., processor (120) of FIG. 1) of the machine (e.g., electronic device (101) of FIG. 1) may call at least one of the one or more instructions stored from the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0193] According to one embodiment, the method according to the embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0194] According to one embodiment, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to one embodiment, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to one embodiment, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

In a head-mounted display (HMD) device, camera; display; Communication circuit; At least one processor including a processing circuit; and The device includes a memory comprising at least one storage medium for storing instructions, wherein the instructions, when executed individually or collectively by the at least one processor, cause the HMD device: A first graphic UI (user interface) element is displayed through the above display, and Identifying whether a first condition for activating a function to support the HMD device in a wearable electronic device based on a first gesture input to the first graphic UI element is satisfied, and the function to support the HMD device is associated with the first graphic UI element, Based on identifying that the above first condition is satisfied, an active area associated with the wearable electronic device is displayed through the display, and Based on the second gesture input for the first graphic UI element and the activation area, identify whether a second condition for activating a function to support the HMD device in the wearable electronic device is satisfied, and Based on identifying that the above second condition is satisfied, it causes a signal to be transmitted through the communication circuit to activate a function to support the HMD device to the wearable electronic device, and The wearable electronic device is configured to enable a function to support the HMD device based on the signal, and to display a second graphic UI element corresponding to the first graphic UI element at a location associated with the wearable electronic device. In paragraph 1, The first gesture input includes a gesture input corresponding to a gesture of grabbing the first graphic UI element, and the first graphic UI element is configured to be movable in a grabbed state according to the first gesture input. When the above instructions are executed individually or collectively by the at least one processor, the HMD device: An HMD device that, while the first graphic UI element is in a grabbed state, identifies the wearable electronic device through the camera and causes whether the first condition is satisfied based on at least one of movement information of the wearable electronic device or connection status information between the wearable electronic device and the HMD device. In paragraph 2, The operation of identifying whether the above first condition is satisfied is: An operation to identify, based on the above movement information, that the amount of change in movement of the wearable electronic device is maintained at or below a reference amount of change for a specified period; An operation to identify that the wearable electronic device and the HMD device are connected based on the connection status information in response to identifying that the above amount of change in movement is maintained below the reference amount of change during the above specified period; and An HMD device comprising an operation of identifying that the first condition is satisfied in response to identifying that the wearable electronic device and the HMD device are connected. In any one of paragraphs 1 through 3, The above-mentioned activation area is displayed on the display area of ​​the wearable electronic device identified through the camera, for an HMD device. In any one of paragraphs 1 through 4, When the above instructions are executed individually or collectively by the at least one processor, the HMD device: An HMD device that causes to identify whether the second condition is satisfied based on information regarding an overlapping area in which the activation area and the area corresponding to the first graphic UI element overlap. In any one of paragraphs 1 through 5, The second gesture input includes a gesture input corresponding to a gesture that releases the grabbed state of the first graphic UI element, and The action of identifying that the above second condition is satisfied is: An action of identifying, based on information regarding the overlap area, that the activation area and the area corresponding to the first graphic UI element overlap for a specified period of time with a size greater than or equal to a specified size; After identifying that the activation area and the area corresponding to the first graphic UI element overlap by more than the specified size during the specified period, an action of identifying that the first graphic UI is switched from a grabbed state to an ungrabbed state according to the second gesture input; and An HMD device comprising an action of identifying that the second condition is satisfied in response to identifying that the first graphic UI element is switched to a grabbed-ungrabbed state. In any one of paragraphs 1 through 6, When the above instructions are executed individually or collectively by the at least one processor, the HMD device: A notification delivery method is selected based on information regarding the concentration score associated with the application being used on the above HMD device and information regarding the importance score of the notification, and According to the selected notification provision method above, the notification is provided by at least one of the HMD device or the wearable electronic device, and The above notification providing method is a first notification providing method that provides the notification in both the HMD device and the first wearable electronic device, or a second notification providing method that provides the notification only in the first wearable electronic device. In Paragraph 7, The action of selecting the above notification provision method is: If the concentration score is less than the reference concentration score, the action of selecting the first notification provision method as the notification provision method; If the concentration score is greater than or equal to the reference concentration score and the importance score is less than the reference importance score, the operation of selecting the second notification providing method as the notification providing method; and An HMD device comprising an operation to select the first notification providing method as the notification providing method when the concentration score is greater than or equal to the reference concentration score and the importance score is greater than or equal to the reference importance score. In Article 7 or Article 8, The above concentration score is obtained through a model learned based on behavioral information of a user wearing the HMD device and application information of the application, and An HMD device, wherein the above importance score is obtained through a model learned based on information regarding the content of the above notification and information regarding the frequency of confirmation of the above notification. In any one of paragraphs 1 through 9, When the above instructions are executed individually or collectively by the at least one processor, the HMD device: An HMD device that causes the size of the first graphic UI element to decrease based on identifying that the first graphic UI element has moved beyond a specified time or distance while in a grabbed state. In any one of paragraphs 1 through 10, When the above instructions are executed individually or collectively by the at least one processor, the HMD device: An HMD device that causes the second graphic UI element to be virtually displayed on an area corresponding to a display placed on the surface of the wearable electronic device identified through the camera, based on identifying that the second condition is satisfied. In any one of paragraphs 1 through 11, The wearable electronic device is configured to map at least one first physical UI element of the HMD device to at least one second physical UI element of the wearable electronic device based on the signal. In any one of paragraphs 1 through 12, The above first graphic UI element includes the system UI of the HMD device, an HMD device. In any one of paragraphs 1 through 13, The above-mentioned wearable electronic device is an HMD device, which is a watch-type wearable electronic device worn on the user's wrist. In a method for an HMD (head mounted display) device, Action of displaying the first graphic UI (user interface) element; An operation to identify whether a first condition for activating a function to support the HMD device in a wearable electronic device based on a first gesture input to the first graphic UI element is satisfied, and the function to support the HMD device is associated with the first graphic UI element; An operation of indicating an active region associated with the wearable electronic device based on identifying that the above first condition is satisfied; An operation to identify whether a second condition for activating a function to support the HMD device in the wearable electronic device is satisfied based on a second gesture input to the first graphic UI element and the activation area; and Based on identifying that the above second condition is satisfied, the method includes the operation of transmitting a signal to activate a function to support the HMD device with the wearable electronic device, and A method in which the wearable electronic device is configured to enable a function to support the HMD device based on the signal, and to display a second graphic element corresponding to the first graphic UI element at a location associated with the wearable electronic device.

Images