Wearable device, electronic device connected to wearable device, and driving method thereof

Abstract

Embodiments of the present disclosure relate to a wearable device, an electronic device connected to the wearable device, and a driving method thereof. The wearable device comprises a processor, and a memory storing instructions, wherein when the instructions are executed by the processor, the wearable device is caused to: determine, by using at least one sensor, whether the state of the wearable device is changed; determine at least one task to be executed in at least one external electronic device; and cause the at least one external device to continuously execute at least a portion of the at least one determined task by transmitting information and a command related to the at least one task to the at least one external device.

Description

Wearable device, electronic device connected to the wearable device, and method of operating the same

[0001] Embodiments of the present disclosure relate to a wearable device, an electronic device connected to the wearable device, and a method of operating the same.

[0002] With recent technological advancements, electronic devices are moving away from uniform rectangular shapes and gradually transforming into various shapes. For example, electronic devices may include wearable devices that can be worn on a part of the body. Wearable devices may include head-mounted displays (HMDs) that can be worn on a user's head. Wearable devices may be referred to as head-mounted devices (HMDs), headgear electronic devices, glasses-type electronic devices, video see-through (VST) or visible see-through (VST) devices, extended reality (XR) devices, virtual reality (VR) devices, and / or augmented reality (AR) devices.

[0003] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art in relation to the present disclosure.

[0004] Because wearable devices are mobile devices, users may be restricted from continuing to use content depending on the remaining battery level built into the wearable device. For example, if a problem occurs where the battery level of the wearable device is low, it may be difficult for the user to continue using content through the wearable device.

[0005] Embodiments of the present disclosure may provide a wearable device that can enhance user convenience by enabling a user to continuously use content that was being used through the wearable device through another external device, an electronic device connected to the wearable device, and a method for operating the same.

[0006] The technical problems to be solved in this disclosure are not limited to those mentioned above, and other unmentioned technical problems will be clearly understood by those skilled in the art from the description below.

[0007] A wearable device according to one embodiment of the present disclosure includes a processor and a memory for storing instructions, wherein the instructions, when executed by the processor, allow the wearable device to determine whether there is a change in the state of the wearable device using at least one sensor, determine at least one task to be executed by at least one external electronic device, and transmit information and instructions related to the at least one task to the at least one external device, thereby enabling the at least one external electronic device to subsequently execute at least a part of the determined at least one task.

[0008] An electronic device according to one embodiment of the present disclosure includes a processor and a memory for storing instructions, wherein, when executed by the processor, the electronic device receives information and instructions related to at least one task being executed by the wearable device from the wearable device, displays at least some of the information related to the at least one task, selects one task among the at least one task in response to user input, executes the selected task, and displays a screen related to the executed task.

[0009] A driving method for a wearable device according to one embodiment of the present disclosure may include an operation of determining whether the state of the wearable device changes using at least one sensor, an operation of determining at least one task to be executed by at least one external electronic device, and an operation of transmitting information and commands related to the at least one task to the at least one external device so that the at least one external electronic device subsequently executes at least a part of the determined at least one task.

[0010] According to embodiments of the present disclosure, user convenience can be enhanced by enabling a user to continuously use content that was being used through a wearable device via another external device.

[0011] In addition, various effects that can be identified directly or indirectly through this document may be provided.

[0012] Other aspects, features, and advantages according to specific embodiments of the present disclosure will become more apparent from the accompanying drawings and description.

[0013] FIG. 1 is a block diagram of an electronic device in a network environment according to one embodiment.

[0014] FIG. 2a illustrates an example of a perspective view of a wearable device.

[0015] FIG. 2b illustrates an example of one or more hardware components placed within a wearable device.

[0016] FIGS. 3a and 3b illustrate an example of the appearance of a wearable device.

[0017] FIG. 4 illustrates an example of a block diagram of a wearable device.

[0018] Figure 5 shows an example of a block diagram of a wearable device for displaying an image in a virtual space.

[0019] Figure 6 illustrates an example of a structure of multiple layers.

[0020] FIG. 7 is a flowchart illustrating a method of operating a wearable device according to one embodiment.

[0021] FIG. 8 is a conceptual diagram illustrating a method of operating a wearable device according to one embodiment.

[0022] FIG. 9 is a flowchart illustrating a method of a wearable device delivering a task according to one embodiment.

[0023] FIG. 10 is an example drawing of an external electronic device displaying a pop-up notification according to one embodiment.

[0024] FIG. 11 is a flowchart illustrating a method of a wearable device delivering a task according to one embodiment.

[0025] FIG. 12 is an example diagram of a wearable device receiving user input according to one embodiment.

[0026] FIG. 13 is an example of a layout displaying a task received by an external electronic device according to one embodiment.

[0027] FIG. 14 is a flowchart illustrating the operation of a wearable device in a low battery state according to an embodiment.

[0028] FIG. 15 is a flowchart illustrating the operation of a wearable device according to one embodiment when executing a task that requires high performance.

[0029] Figure 16 is an example of a notification indicating the estimated time of work for a task requiring high performance.

[0030] FIG. 17 is an example of a mixed reality space including augmented reality or virtual reality provided by a wearable device according to one embodiment.

[0031] Figure 18 is an example of a notification indicating the execution status of a task.

[0032] FIG. 19 is an example illustrating the operation of a wearable device when the removal of a user's wearable device is detected.

[0033] FIG. 20 is a flowchart illustrating a method of operating a wearable device according to one embodiment.

[0034] FIG. 21 is a flowchart illustrating a method of driving an electronic device according to one embodiment.

[0035] Each of the embodiments described with reference to the drawings of the present disclosure may be configured independently as a single embodiment. For example, the embodiment of FIG. 1 and the embodiment of FIG. 2a may each be configured independently of each other. Each of the embodiments described with reference to the drawings of the present disclosure may operate independently as a single embodiment. For example, the embodiment of FIG. 1 and the embodiment of FIG. 2a may each operate independently of each other.

[0036] At least two of the embodiments described with reference to the drawings of the present disclosure may be combined. For example, at least a part of the embodiment of FIG. 1 and at least a part of the embodiment of FIG. 2a may be combined with each other. At least two of the embodiments described with reference to the drawings of the present disclosure may be combined and operated. For example, at least a part of the embodiment of FIG. 1 and at least a part of the embodiment of FIG. 2a may be combined and operated with each other.

[0037] When at least two of the embodiments described with reference to the drawings of the present disclosure are combined, at least some of the configurations and / or at least some of the operations included in each embodiment may be omitted. For example, when the embodiment of FIG. 1 and the embodiment of FIG. 2a are combined, at least some of the configurations and / or at least some of the operations included in the embodiment of FIG. 1 may be omitted, and at least some of the configurations and / or at least some of the operations included in the embodiment of FIG. 2a may be omitted.

[0038] FIG. 1 is a block diagram of an electronic device (101) in a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or may communicate with at least one of an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through a server (108). According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (160), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)).

[0039] The processor (120) can control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., a program (140)), and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use lower power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.

[0040] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0041] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, input data or output data for software (e.g., program (140)) and related commands. The memory (130) may include volatile memory (132) or non-volatile memory (134).

[0042] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0043] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0044] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0045] The display module (160) can visually provide information to an external (e.g., user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0046] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (101).

[0047] The sensor module (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0048] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0049] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0050] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that can be perceived by the user through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0051] The camera module (180) can capture still images and video. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0052] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0053] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0054] The communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).

[0055] The wireless communication module (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), external electronic device (e.g., electronic device (104)), or network system (e.g., second network (199)). According to one embodiment, the wireless communication module (192) may support a Peak data rate (e.g., 20 Gbps or more) for eMBB realization, loss coverage (e.g., 164 dB or less) for mMTC realization, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for URLLC realization.

[0056] An antenna module (197) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197).

[0057] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0058] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0059] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (104) or the server (108) may be included within a second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0060] The electronic device according to the various embodiments disclosed in this disclosure may be a device of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. The electronic device according to the embodiments of this disclosure is not limited to the devices described above.

[0061] The various embodiments of the present disclosure and the terms used therein are not intended to limit the technical features described in the present disclosure to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In the present disclosure, phrases such as “A or B,” “at least one of A and B,” “at least one of A or B,” “A, B or C,” “at least one of A, B and C,” and “at least one of A, B, or C” each may include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as “first,” “second,” or “first” or “second” may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as "coupled" or "connected" to another (e.g., 2nd) component, with or without the terms "functionally" or "communicationly," it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0062] The term “module” as used in various embodiments of the present disclosure may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0063] Various embodiments of the present disclosure may be implemented as software (e.g., program (140)) comprising one or more instructions stored in a storage medium (e.g., internal memory (136) or external memory (138)) readable by a machine (e.g., electronic device (101)). For example, a processor (e.g., processor (120)) of the machine (e.g., electronic device (101)) may call at least one of the one or more instructions stored in the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0064] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or an application store (e.g., Play Store). TM It can be distributed online (e.g., downloaded or uploaded) through ) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0065] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0066] In embodiments of the present disclosure, an electronic device for displaying an image in a virtual space (e.g., the electronic device (101) of FIG. 1) may be a wearable device. The wearable device (101) may include a head-mounted display (HMD) that is wearable on a user's head. The wearable device (101) may be referred to as a head-mount device (HMD), a headgear electronic device, a glasses-type electronic device, a video see-through (VST) or visible see-through (VST) device, an extended reality (XR) device, a virtual reality (VR) device, and / or an augmented reality (AR) device. Although the external appearance of the wearable device (101) having the form of glasses is illustrated, the embodiments are not limited thereto. An example of a hardware configuration included within the wearable device (101) is described exemplarily with reference to FIG. 4. An example of the structure of a wearable device (101) that can be worn on the head of a user (110) is described with reference to FIG. 2a, FIG. 2b, FIG. 3a and / or FIG. 3b. The wearable device (101) may be referred to as an electronic device. For example, the electronic device may be combined with an accessory (e.g., a strap) to be attached to the user's head to form an HMD.

[0067] According to one embodiment, a wearable device (101) can perform functions related to augmented reality (AR) and / or mixed reality (MR). For example, while a user (110) is wearing the wearable device (101), the wearable device (101) may include at least one lens positioned adjacent to the user's (110) eyes. The wearable device (101) may combine light emitted from a display of the wearable device (101) with ambient light passing through the lens. The display area of ​​the display may be formed within the lens through which the ambient light passes. Because the wearable device (101) combines the ambient light and the light emitted from the display, the user (110) can see a mixed image of a real object perceived by the ambient light and a virtual object formed by the light emitted from the display. The aforementioned augmented reality, mixed reality, and / or virtual reality may be referred to as extended reality (XR).

[0068] According to one embodiment, a wearable device (101) can perform functions related to VST (video see-through or visible see-through) and / or virtual reality (VR). For example, while a user (110) is wearing the wearable device (101), the wearable device (101) may include a housing that covers the user's (110) eyes. The wearable device (101) may include a display disposed on a first surface of the housing facing the eyes while in the state. The wearable device (101) may include a camera disposed on a second surface opposite to the first surface. Using the camera, the wearable device (101) may acquire an image and / or video representing ambient light. A wearable device (101) may output the image and / or video within a display placed on the first surface, thereby allowing a user (110) to perceive the ambient light through the display. A displaying area (or displaying region) (or active area or active region) of the display placed on the first surface may be formed by one or more pixels included in the display. The wearable device (101) may composite a virtual object with the image and / or video output through the display, thereby allowing the user (110) to perceive the virtual object along with the actual object perceived by the ambient light.

[0069] According to one embodiment, a wearable device (101) can identify or recognize the position or location and / or direction or orientation of the wearable device (101) based on an image (and / or video) obtained or acquired using a camera. The wearable device (101) can obtain information about the external space using one or more cameras and / or one or more sensors. The information may include a geographic location of the external space (e.g., GPS (global positioning system) coordinates) identified by one or more sensors. The information may include an image and / or video of the external space identified by one or more cameras. The wearable device (101) can perform object recognition on the image and / or video to identify external objects contained in the external space from the image and / or video.

[0070] Hereinafter, with reference to FIGS. 2a, FIGS. 2b, FIGS. 3a, FIGS. 3b, and FIGS. 4, an example of a hardware configuration of a wearable device (101) is described.

[0071] FIG. 2a illustrates an example of a perspective view of a wearable device. FIG. 2b illustrates an example of one or more hardware components disposed within the wearable device. According to one embodiment, the wearable device (101) may have the form of glasses that are wearable on a part of a user's body (e.g., head). The wearable device (101) of FIG. 2a and FIG. 2b may be an example of the wearable device (101) of FIG. 1. The wearable device (101) may include a head-mounted display (HMD). For example, the housing of the wearable device (101) may include a flexible material such as rubber and / or silicone that has a shape that adheres to a part of the user's head (e.g., a part of the face covering both eyes). For example, the housing of the wearable device (101) may include one or more straps that can be twined around the user's head, and / or one or more temples that can be attached to the ears of the head.

[0072] Referring to FIG. 2a, a wearable device (101) according to one embodiment may include at least one display (250) and a frame (200) supporting at least one display (250).

[0073] According to one embodiment, a wearable device (101) may be worn on a part of a user's body. The wearable device (101) may provide augmented reality (AR), virtual reality (VR), or mixed reality (MR) that combines augmented reality and virtual reality to a user wearing the wearable device (101). For example, the wearable device (101) may display a virtual reality image provided by at least one optical device (282, 284) of FIG. 2b on at least one display (250) in response to a specified gesture of the user obtained through the motion recognition camera (260-2, 260-3) of FIG. 2b.

[0074] According to one embodiment, at least one display (250) can provide visual information to a user. For example, at least one display (250) may include a transparent or translucent lens. At least one display (250) may include a first display (250-1) and / or a second display (250-2) spaced apart from the first display (250-1). For example, the first display (250-1) and the second display (250-2) may be positioned at locations corresponding to the user's left eye and right eye, respectively.

[0075] Referring to FIG. 2b, at least one display (250) may provide visual information transmitted from external light to a user through a lens included in at least one display (250) and other visual information distinct from said visual information. The lens may be formed based on at least one of a Fresnel lens, a pancake lens, or a multi-channel lens. For example, at least one display (250) may include a first surface (231) and a second surface (232) opposite to the first surface (231). A display area may be formed on the second surface (232) of at least one display (250). When a user wears the wearable device (101), external light may be transmitted to the user by being incident on the first surface (231) and transmitted through the second surface (232). As another example, at least one display (250) can display an augmented reality image combined with a virtual reality image provided by at least one optical device (282, 284) on a real image transmitted through external light in a display area formed on the second surface (232).

[0076] In one embodiment, at least one display (250) may include at least one waveguide (233, 234) that diffracts light emitted from at least one optical device (282, 284) and transmits it to a user. At least one waveguide (233, 234) may be formed based on at least one of glass, plastic, or polymer. A nano pattern may be formed on the exterior or at least a portion of the interior of at least one waveguide (233, 234). The nano pattern may be formed based on a polygonal and / or curved grating structure. Light incident on one end of at least one waveguide (233, 234) may be propagated to the other end of at least one waveguide (233, 234) by the nano pattern. At least one waveguide (233, 234) may include at least one diffractive element (e.g., DOE (diffractive optical element), HOE (holographic optical element)) and at least one reflective element (e.g., a reflective mirror). For example, at least one waveguide (233, 234) may be placed within a wearable device (101) to guide a screen displayed by at least one display (250) to the user's eye. For example, the screen may be transmitted to the user's eye based on total internal reflection (TIR) ​​occurring within at least one waveguide (233, 234).

[0077] A wearable device (101) can analyze an object included in a real-world image collected through a shooting camera (260-4), combine a virtual object corresponding to an object among the analyzed objects that is the target of augmented reality provision, and display it on at least one display (250). The virtual object may include at least one of text and an image regarding various information related to the object included in the real-world image. The wearable device (101) can analyze the object based on a multi-camera such as a stereo camera. For the object analysis, the wearable device (101) can perform spatial recognition (e.g., SLAM (simultaneous localization and mapping)) using a multi-camera and / or time-of-flight (ToF). A user wearing the wearable device (101) can view the image displayed on at least one display (250).

[0078] According to one embodiment, the frame (200) may be formed as a physical structure that allows the wearable device (101) to be worn on the user's body. According to one embodiment, the frame (200) may be configured so that when the user wears the wearable device (101), the first display (250-1) and the second display (250-2) can be positioned corresponding to the user's left and right eyes. The frame (200) may support at least one display (250). For example, the frame (200) may support the first display (250-1) and the second display (250-2) so that they are positioned corresponding to the user's left and right eyes.

[0079] Referring to FIG. 2a, the frame (200) may include an area (220) in which at least a portion of the frame contacts a part of the user's body when the user wears the wearable device (101). For example, the area (220) of the frame (200) in contact with a part of the user's body may include an area in contact with a part of the user's nose, a part of the user's ear, and a part of the side of the user's face that the wearable device (101) contacts. According to one embodiment, the frame (200) may include a nose pad (210) that contacts a part of the user's body. When the wearable device (101) is worn by the user, the nose pad (210) may contact a part of the user's nose. The frame (200) may include a first temple (204) and a second temple (205) that contact a different part of the user's body distinct from the part of the user's body.

[0080] For example, the frame (200) may include a first rim (201) covering at least a portion of a first display (250-1), a second rim (202) covering at least a portion of a second display (250-2), a bridge (203) positioned between the first rim (201) and the second rim (202), a first pad (211) positioned along a portion of the edge of the first rim (201) from one end of the bridge (203), a second pad (212) positioned along a portion of the edge of the second rim (202) from the other end of the bridge (203), a first temple (204) extending from the first rim (201) and fixed to a portion of the wearer's ear, and a second temple (205) extending from the second rim (202) and fixed to a portion of the ear opposite to the first. The first pad (211) and the second pad (212) may come into contact with a part of the user's nose, and the first temple (204) and the second temple (205) may come into contact with a part of the user's face and a part of the ear. The temples (204, 205) may be rotatably connected to the rim through the hinge units (206, 207) of FIG. 2B. The first temple (204) may be rotatably connected to the first rim (201) through a first hinge unit (206) positioned between the first rim (201) and the first temple (204). The second temple (205) may be rotatably connected to the second rim (202) through a second hinge unit (207) positioned between the second rim (202) and the second temple (205). According to one embodiment, a wearable device (101) can identify an external object touching the frame (200) (e.g., a user's fingertip) and / or a gesture performed by said external object by using a touch sensor, a grip sensor, and / or a proximity sensor formed on at least a portion of the surface of the frame (200).

[0081] According to one embodiment, the wearable device (101) may include hardware that performs various functions (e.g., hardware to be described later based on the block diagram of FIG. 4). For example, the hardware may include a battery module (270), an antenna module (275), at least one optical device (282, 284), speakers (e.g., speakers (255-1, 255-2)), a microphone (e.g., microphones (265-1, 265-2, 265-3)), a light-emitting module (not shown), and / or a PCB (printed circuit board) (290) (e.g., a printed circuit board). The various hardware may be placed within a frame (200).

[0082] According to one embodiment, a microphone (e.g., microphones (265-1, 265-2, 265-3)) of a wearable device (101) is positioned on at least a portion of a frame (200) to acquire a sound signal. A first microphone (265-1) positioned on a bridge (203), a second microphone (265-2) positioned on a second rim (202), and a third microphone (265-3) positioned on a first rim (201) are shown in FIG. 2b, but the number and position of the microphones (265) are not limited to the embodiment of FIG. 2b. If there are two or more microphones (265) included in the wearable device (101), the wearable device (101) can identify the direction of the sound signal by using a plurality of microphones positioned on different portions of the frame (200).

[0083] According to one embodiment, at least one optical device (282, 284) may project a virtual object onto at least one display (250) to provide various image information to a user. For example, at least one optical device (282, 284) may be a projector. At least one optical device (282, 284) may be disposed adjacent to at least one display (250) or included within at least one display (250) as part of at least one display (250). According to one embodiment, a wearable device (101) may include a first optical device (282) corresponding to a first display (250-1) and a second optical device (284) corresponding to a second display (250-2). For example, at least one optical device (282, 284) may include a first optical device (282) positioned at the edge of a first display (250-1) and a second optical device (284) positioned at the edge of a second display (250-2). The first optical device (282) may transmit light to a first waveguide (233) positioned on the first display (250-1), and the second optical device (284) may transmit light to a second waveguide (234) positioned on the second display (250-2).

[0084] In one embodiment, the camera (260) may include a shooting camera (260-4), an eye tracking camera (ET CAM) (260-1), and / or a motion recognition camera (260-2, 206-3). The shooting camera (260-4), the eye tracking camera (260-1), and the motion recognition camera (260-2, 260-3) may be positioned at different locations on the frame (200) and may perform different functions. The eye tracking camera (260-1) may output data indicating the position of the eyes or the gaze of a user wearing the wearable device (101). For example, the wearable device (101) may detect the gaze from an image containing the user's pupils obtained through the eye tracking camera (260-1). A wearable device (101) can identify an object focused by a user (e.g., a real object, and / or a virtual object) by using the user's gaze obtained through an eye-tracking camera (260-1). The wearable device (101), having identified the focused object, can perform a function (e.g., gaze interaction) for interaction between the user and the focused object. The wearable device (101) can represent a portion corresponding to the eyes of an avatar representing the user in a virtual space by using the user's gaze obtained through an eye-tracking camera (260-1). The wearable device (101) can render an image (or screen) displayed on at least one display (250) based on the position of the user's eyes. For example, the visual quality of a first area related to the gaze within the image and the visual quality of a second area distinguished from the first area (e.g., resolution, brightness, saturation, grayscale, PPI) may differ from each other. A wearable device (101) can obtain an image having a visual quality of a first area that matches the user's gaze and a visual quality of a second area by using foveated rendering.For example, if the wearable device (101) supports an iris recognition function, user authentication can be performed based on iris information obtained using an eye-tracking camera (260-1). An example in which the eye-tracking camera (260-1) is positioned toward the user's right eye is shown in FIG. 2b, but the embodiment is not limited thereto, and the eye-tracking camera (260-1) may be positioned solely toward the user's left eye or toward both eyes.

[0085] In one embodiment, the camera (260-4) can capture a real image or background to be matched with a virtual image in order to implement augmented reality or mixed reality content. The camera (260-4) can be used to acquire high-resolution images based on HR (high resolution) or PV (photo video). The camera (260-4) can capture an image of a specific object located at the position viewed by the user and provide the image to at least one display (250). The at least one display (250) can display a single image in which information regarding a real image or background including the image of the specific object acquired using the camera (260-4) and a virtual image provided through at least one optical device (282, 284) are superimposed. The wearable device (101) can compensate for depth information (e.g., the distance between the wearable device (101) and an external object acquired through a depth sensor) using the image acquired through the camera (260-4). The wearable device (101) can perform object recognition through an image acquired using a shooting camera (260-4). The wearable device (101) can perform a function of focusing on an object (or subject) in an image (e.g., auto focus) and / or an optical image stabilization (OIS) function (e.g., anti-shake function) using the shooting camera (260-4). The wearable device (101) can perform a pass-through function to superimpose an image acquired through the shooting camera (260-4) onto at least a portion of a screen representing a virtual space while displaying the screen representing a virtual space on at least one display (250). In one embodiment, the shooting camera (260-4) may be placed on a bridge (203) positioned between a first rim (201) and a second rim (202).

[0086] The eye tracking camera (260-1) can achieve more realistic augmented reality by tracking the gaze of a user wearing the wearable device (101), thereby matching the user's gaze with visual information provided to at least one display (250). For example, when the user looks straight ahead, the wearable device (101) can naturally display environmental information related to the user's front on at least one display (250) at the location where the user is situated. The eye tracking camera (260-1) may be configured to capture an image of the user's pupil to determine the user's gaze. For example, the eye tracking camera (260-1) may receive a gaze detection light reflected from the user's pupil and track the user's gaze based on the position and movement of the received gaze detection light. In one embodiment, the eye tracking camera (260-1) may be positioned at locations corresponding to the user's left and right eyes. For example, the eye-tracking camera (260-1) may be positioned within the first rim (201) and / or the second rim (202) to face the direction in which the user wearing the wearable device (101) is located.

[0087] The motion recognition camera (260-2, 260-3) can provide a specific event to a screen provided on at least one display (250) by recognizing the movement of the user's entire body or part thereof, such as the user's torso, hands, or face. The motion recognition camera (260-2, 260-3) can recognize the user's gesture, acquire a signal corresponding to the gesture, and provide a display corresponding to the signal to at least one display (250). The processor can identify the signal corresponding to the gesture and, based on the identification, perform a designated function. The motion recognition camera (260-2, 260-3) can be used to perform spatial recognition functions using SLAM and / or depth maps for a 6-degrees-of-freedom pose (6 dof pose). The processor can use the motion recognition camera (260-2, 260-3) to perform gesture recognition functions and / or object tracking functions. In one embodiment, a motion recognition camera (260-2, 260-3) may be placed on the first rim (201) and / or the second rim (202).

[0088] The camera (260) included in the wearable device (101) is not limited to the eye-tracking camera (260-1) and motion recognition camera (260-2, 260-3) described above. For example, the wearable device (101) can identify external objects included within the FoV by using a camera positioned toward the user's FoV. The identification of external objects by the wearable device (101) can be performed based on a sensor for identifying the distance between the wearable device (101) and the external object, such as a depth sensor and / or a time of flight (ToF) sensor. The camera (260) positioned toward the FoV may support an autofocus function and / or an optical image stabilization (OIS) function. For example, the wearable device (101) may include a camera (260) (e.g., a face tracking camera) positioned toward the face to acquire an image including the face of a user wearing the wearable device (101).

[0089] Although not illustrated, according to one embodiment, the wearable device (101) may further include a light source (e.g., LED) that emits light toward a subject (e.g., user's eye, face, and / or an object outside the FoV) being photographed using a camera (260). The light source may include an LED of infrared wavelength. The light source may be placed in at least one of the frame (200) and hinge units (206, 207).

[0090] According to one embodiment, the battery module (270) can supply power to the electronic components of the wearable device (101). In one embodiment, the battery module (270) may be placed within the first temple (204) and / or the second temple (205). For example, the battery module (270) may be a plurality of battery modules (270). The plurality of battery modules (270) may each be placed in the first temple (204) and the second temple (205). In one embodiment, the battery module (270) may be placed at the end of the first temple (204) and / or the second temple (205).

[0091] The antenna module (275) can transmit a signal or power to the outside of the wearable device (101) or receive a signal or power from the outside. In one embodiment, the antenna module (275) may be placed within the first temple (204) and / or the second temple (205). For example, the antenna module (275) may be placed close to one side of the first temple (204) and / or the second temple (205).

[0092] The speaker (255) can output an acoustic signal to the outside of the wearable device (101). The acoustic output module may be referred to as the speaker. In one embodiment, the speaker (255) may be placed within a first temple (204) and / or a second temple (205) to be positioned adjacent to the ear of a user wearing the wearable device (101). For example, the speaker (255) may include a second speaker (255-2) positioned adjacent to the user's left ear by being placed within the first temple (204), and a first speaker (255-1) positioned adjacent to the user's right ear by being placed within the second temple (205).

[0093] A light-emitting module (not shown) may include at least one light-emitting element. The light-emitting module may emit light of a color corresponding to a specific state or emit light with an action corresponding to a specific state in order to visually provide information regarding a specific state of the wearable device (101) to the user. For example, if the wearable device (101) requires charging, it may emit red light at a constant frequency. In one embodiment, the light-emitting module may be placed on the first rim (201) and / or the second rim (202).

[0094] Referring to FIG. 2b, a wearable device (101) according to one embodiment may include a printed circuit board (PCB) (290). The PCB (290) may be included in at least one of a first temple (204) or a second temple (205). The PCB (290) may include an interposer disposed between at least two sub-PCBs. One or more hardware components included in the wearable device (101) (e.g., hardware components illustrated by different blocks in FIG. 4) may be disposed on the PCB (290). The wearable device (101) may include a flexible PCB (FPCB) for interconnecting the hardware components.

[0095] According to one embodiment, a wearable device (101) may include at least one of a gyroscope sensor, a gravity sensor, and / or an acceleration sensor for detecting the posture of the wearable device (101) and / or the posture of a body part (e.g., head) of a user wearing the wearable device (101). Each of the gravity sensor and the acceleration sensor may measure gravitational acceleration and / or acceleration based on designated three-dimensional axes (e.g., x-axis, y-axis, and z-axis) that are perpendicular to each other. The gyroscope sensor may measure the angular velocity of each of the designated three-dimensional axes (e.g., x-axis, y-axis, and z-axis). At least one of the gravity sensor, the acceleration sensor, and the gyroscope sensor may be referred to as an inertial measurement unit (IMU). According to one embodiment, the wearable device (101) can identify a user's motion and / or gesture performed to execute or stop a specific function of the wearable device (101) based on an IMU.

[0096] FIGS. 3a and 3b illustrate an example of the appearance of a wearable device (e.g., a wearable device (101)). The wearable device (101) of FIGS. 3a and 3b may be an example of the wearable device (101) of FIG. 1. According to one embodiment, an example of the appearance of a first surface (310) of the housing of the wearable device (101) may be illustrated in FIG. 3a, and an example of the appearance of a second surface (320) opposite to the first surface (310) may be illustrated in FIG. 3b.

[0097] Referring to FIG. 3a, according to one embodiment, a first surface (310) of a wearable device (101) may have a shape that is attachable to a part of a user's body (e.g., the user's face). Although not illustrated, the wearable device (101) may further include a strap for fixing to a part of a user's body and / or one or more temples (e.g., a first temple (204) and / or a second temple (205) of FIG. 2a and FIG. 2b). A first display (250-1) for outputting an image to the left eye among the user's two eyes, and a second display (250-2) for outputting an image to the right eye among the two eyes may be disposed on the first surface (310). The wearable device (101) may further include rubber or silicone packing formed on the first surface (310) to prevent interference by light different from light emitted from the first display (250-1) and the second display (250-2) (e.g., ambient light).

[0098] According to one embodiment, a wearable device (101) may include cameras (260-1) for photographing and / or tracking both eyes of a user adjacent to each of the first display (250-1) and the second display (250-2). The cameras (260-1) may be referenced to the eye-tracking camera (260-1) of FIG. 2B. According to one embodiment, a wearable device (101) may include cameras (260-5, 260-6) for photographing and / or recognizing a user's face. The cameras (260-5, 260-6) may be referenced to FT cameras. The wearable device (101) may control an avatar representing the user in a virtual space based on the motion of the user's face identified using the cameras (260-5, 260-6). For example, the wearable device (101) can change the texture and / or shape of a part of an avatar (e.g., a part of an avatar representing a human face) by using information obtained by cameras (260-5, 260-6) (e.g., FT cameras) and representing the facial expression of a user wearing the wearable device (101).

[0099] Referring to FIG. 3b, on a second surface (320) opposite to the first surface (310) of FIG. 3a, a camera (e.g., cameras (260-7, 260-8, 260-9, 260-10, 260-11, 260-12)), and / or a sensor (e.g., a depth sensor (330)) may be placed to acquire information related to the external environment of the wearable device (101). For example, cameras (260-7, 260-8, 260-9, 260-10) may be placed on the second surface (320) to recognize external objects. The cameras (260-7, 260-8, 260-9, 260-10) may be referenced to the motion recognition cameras (260-2, 260-3) of FIG. 2b.

[0100] By using cameras (260-11, 260-12), the wearable device (101) can acquire images and / or videos to be transmitted to each of the user's two eyes. Camera (260-11) may be placed on the second surface (320) of the wearable device (101) to acquire an image to be displayed through a second display (250-2) corresponding to the right eye among the two eyes. Camera (260-12) may be placed on the second surface (320) of the wearable device (101) to acquire an image to be displayed through a first display (250-1) corresponding to the left eye among the two eyes. Cameras (260-11, 260-12) may be referenced to the shooting camera (260-4) of FIG. 2B.

[0101] According to one embodiment, a wearable device (101) may include a depth sensor (330) disposed on a second surface (320) to identify the distance between the wearable device (101) and an external object. Using the depth sensor (330), the wearable device (101) may obtain spatial information (e.g., a depth map) for at least a portion of the FoV of a user wearing the wearable device (101). Although not illustrated, a microphone may be disposed on the second surface (320) of the wearable device (101) to obtain sound output from an external object. The number of microphones may be one or more, depending on the embodiment.

[0102] Hereinafter, with reference to FIG. 4, the hardware or software configuration of the wearable device (101) will be described.

[0103] FIG. 4 illustrates an example of a block diagram of a wearable device (e.g., a wearable device (101)). The wearable device (101) of FIG. 4 may be an example of the electronic device (101) of FIG. 1 and the wearable device (101) of FIG. 2a to FIG. 3b.

[0104] Referring to FIG. 4, a wearable device (101) according to one embodiment may include a processor (410) (e.g., processor (120)), memory (415), a display (250) (e.g., a first display (250-1) and / or a second display (250-2) of FIG. 2a, FIG. 2b, FIG. 3a, and FIG. 3b), a sensor (420) (e.g., an image sensor (421) and / or a motion sensor (422)), and / or a communication circuit (430) (e.g., including at least a part of the communication module (190) of FIG. 1). The processor (410), memory (415), display (250), sensor (420), and / or communication circuit (430) may be electrically and / or operationally connected to each other by an electronic component such as a communication bus (402). In the present disclosure, the operational connection of electronic components may include a direct connection established between the electronic components and / or an indirect connection established between the electronic components such that a first electronic component among the electronic components is controlled by a second electronic component among the electronic components. The type and / or number of electronic components included in the wearable device (101) are not limited to those shown in FIG. 4. For example, the wearable device (101) may include only some of the electronic components shown in FIG. 4.

[0105] A processor (410) of a wearable device (101) according to one embodiment may include a circuit (e.g., a processing circuit) for processing data based on one or more instructions. The circuit for processing data may include, for example, an arithmetic and logic unit (ALU), a field programmable gate array (FPGA), a central processing unit (CPU), and / or an application processor (AP). In one embodiment, the wearable device (101) may include one or more processors. The processor (410) may have a structure of a multi-core processor such as a dual core, a quad core, a hexa core, and / or an octa core. The multi-core processor structure of the processor (410) may include a structure based on a plurality of core circuits (e.g., a big-little structure), distinguished by power consumption, clock, and / or computational power per unit time. In one embodiment comprising a processor (410) having a multi-core processor structure, the operations and / or functions of the present disclosure may be performed individually or collectively by one or more cores included in the processor (410).

[0106] A memory (415) of a wearable device (101) according to one embodiment may include electronic components for storing data and / or instructions that are input to or output from a processor (410). The memory (415) may include, for example, volatile memory such as random-access memory (RAM) and / or non-volatile memory such as read-only memory (ROM). Volatile memory may include, for example, at least one of dynamic RAM (DRAM), static RAM (SRAM), cache RAM, and pseudo SRAM (PSRAM). Non-volatile memory may include, for example, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, hard disk, compact disk, and embedded multi-media card (eMMC). In one embodiment, the memory (415) may be referred to as storage.

[0107] In one embodiment, a display (250) of a wearable device (101) can output visualized information to a user of the wearable device (101). A display (250) arranged in front of the eyes of a user wearing the wearable device (101) may be placed in at least a part of the housing of the wearable device (101) (e.g., a first display (250-1) and / or a second display (250-2) of FIG. 2a, FIG. 2b, FIG. 3a, and FIG. 3b). For example, the display (250) may be included within a display assembly. For example, the display (250) may be controlled by a processor (410) including circuits such as a CPU (411), a GPU (graphic processing unit) (412), and / or a DPU (display processing unit) (413) to output visualized information to a user. The display (250) may include a flexible display, a flat panel display (FPD), and / or electronic paper. The display (250) may include a liquid crystal display (LCD), a plasma display panel (PDP), and / or one or more light emitting diodes (LEDs). The LEDs may include organic LEDs (OLEDs). Embodiments are not limited thereto, for example, if the wearable device (101) includes a lens for transmitting external light (or ambient light), the display (250) may include a projector (or projection assembly) for projecting light onto the lens. In one embodiment, the display (250) may be referred to as a display panel and / or a display module.The pixels included in the display (250) may be positioned toward either of the user's two eyes when worn by the user of the wearable device (101). For example, the display (250) may include display areas (or active areas) corresponding to each of the user's two eyes.

[0108] In one embodiment, a sensor (420) of a wearable device (101) may generate electrical information that can be processed by a processor (410) and / or a memory (415) from non-electronic information associated with the wearable device (101). For example, the sensor (420) may include a global positioning system (GPS) sensor for detecting the geographic location of the wearable device (101). In addition to the GPS method, the sensor (420) may generate information indicating the geographic location of the wearable device (101) based on a global navigation satellite system (GNSS), such as Galileo or Beidou (compass). This information may be stored in the memory (415), processed by the processor (410), and / or transmitted to another electronic device distinct from the wearable device (101) via a communication circuit.

[0109] Referring to FIG. 4, an image sensor (421) and / or a motion sensor (422) are illustrated as examples of sensors (420) included in a wearable device (101). The sensor (420) may include one or more light sensors (e.g., a CCD (charged coupled device) sensor, a CMOS (complementary metal oxide semiconductor) sensor) that generate electrical signals representing the color and / or brightness of light. The image sensor (421) may be referred to as a camera. A plurality of light sensors included in the image sensor (421) may be arranged in the form of a two-dimensional grid (4-dimensional array). The image sensor (421) may acquire the electrical signals of each of the plurality of light sensors substantially simultaneously to generate two-dimensional frame data corresponding to the light reaching the light sensors of the two-dimensional grid. For example, photo data captured using the image sensor (421) may refer to one (a) two-dimensional frame data acquired from the image sensor (421). For example, video data captured using an image sensor (421) may refer to a sequence of multiple two-dimensional frame data obtained from the image sensor (421) along a frame rate. The image sensor (421) may further include a flash light for outputting light in the direction in which the image sensor (421) receives light.

[0110] According to one embodiment, the wearable device (101) may include a plurality of image sensors arranged toward different directions as an example of an image sensor (421). As described above with reference to FIGS. 2A, 2B, 3A, and 3B, the plurality of image sensors may include an eye-tracking camera (e.g., the eye-tracking camera (260-1) of FIGS. 2B and 3A) configured to be arranged toward the eyes of a user wearing the wearable device (101). The plurality of image sensors may include an outward camera. A processor (410) may identify the direction of the user's gaze using an image and / or video obtained from the eye-tracking camera. The eye-tracking camera may include an infrared (IR) sensor. The eye-tracking camera may be referred to as an eye sensor and / or an eye tracker.

[0111] An external camera may be positioned facing forward of a user wearing the wearable device (101) (e.g., a direction in which both eyes may face). The wearable device (101) may include a plurality of external cameras. The embodiments are not limited thereto, and the external camera may be positioned facing an external space. Using images and / or videos obtained from the external camera, the processor (410) may identify external objects. For example, the processor (410) may identify the position, shape, and / or gesture (e.g., hand gesture) of the user wearing the wearable device (101) based on images and / or videos obtained from the external camera. Using images and / or videos of the external environment obtained from the external camera, the processor (410) may recognize or track one or more objects within the external environment.

[0112] According to one embodiment, the motion sensor (422) may output an electrical signal representing gravitational accelerations, accelerations, and / or angular velocities of a plurality of axes (e.g., x-axis, y-axis, and z-axis) that are perpendicular to each other and are based on a designated origin within the wearable device (101) and / or the motion sensor (422). For example, the processor (410) may repeatedly receive or acquire sensor data from the motion sensor (422), including accelerations, angular velocities, and / or magnitudes of the magnetic field of a number of the plurality of axes, based on a designated period (e.g., 1 millisecond). In one embodiment, the motion sensor (422) may be referred to as an inertial measurement unit (IMU). The sensor (420) included in the wearable device (101) is not limited to the above and may include a grip sensor, a proximity sensor, a heart rate sensor, a fingerprint sensor, an ambient light sensor, and / or a ToF sensor. By using a motion sensor (422), the processor (410) can detect motion of the wearable device (101) (e.g., motion of the wearable device (101) caused by a user wearing the wearable device (101)).

[0113] In one embodiment, the communication circuit (430) of the wearable device (101) may include a hardware component for supporting the transmission and / or reception of a signal between the wearable device (101) and an external electronic device (e.g., electronic device (102), electronic device (104)). The communication circuit (430) may include, for example, at least one of a modem, an antenna, and an O / E (optic / electronic) converter. The communication circuit (430) may support the transmission and / or reception of an electrical signal based on various types of protocols such as Ethernet, LAN (local area network), WAN (wide area network), WiFi (wireless fidelity), Bluetooth, BLE (Bluetooth low energy), Zigbee, LTE (long term evolution), and 5G NR (new radio).

[0114] According to one embodiment, within the memory (415) of the wearable device (101), one or more instructions (or commands) representing data to be processed by the processor (410) of the wearable device (101), calculations to be performed, and / or operations may be stored. A set of one or more instructions may be referred to as a program, firmware, operating system, process, routine, sub-routine, and / or software application (hereinafter, application). For example, the wearable device (101), and / or processor (410) may perform at least one of the operations described below when a set of a plurality of instructions distributed in the form of an operating system, firmware, driver, program, and / or software application is executed. In the following, the statement that a software application is installed in the wearable device (101) means that one or more instructions provided in the form of a software application (or package) are stored in memory (415), and that the one or more applications are stored in an executable format (e.g., a file having an extension specified by the operating system of the wearable device (101)) by the processor (410). For example, the application may include a program and / or library related to a service provided to the user.

[0115] Referring to FIG. 4, programs installed on a wearable device (101) may be included in any one of different layers, including an application layer (440), a framework layer (450), and / or a hardware abstraction layer (HAL) (480), based on the target. For example, within the hardware abstraction layer (480), programs (e.g., modules, or drivers) designed to target the hardware of the wearable device (101) (e.g., a display (250), and / or a sensor (420)) may be included. The framework layer (450) may be referred to as an XR framework layer in that it includes one or more programs for providing XR (extended reality) services. For example, the layers illustrated in FIG. 4 are logically (or for convenience of explanation) separated, and may not imply that the address space of memory (415) is separated by said layers.

[0116] Within the framework layer (450), programs designed to target at least one of the hardware abstraction layer (480) and / or the application layer (440) (e.g., a location tracker (471), a spatial recognizer (472), a gesture tracker (473), and / or an eye tracker (474)) may be included. The programs included in the framework layer (450) may provide an application programming interface (API) that is executable (or invokeable) based on other programs.

[0117] Within the application layer (440), programs designed to target users of the wearable device (101) may be included. Examples of programs included in the application layer (440) include an XR (extended reality) system UI (user interface) (441) and / or an XR application (442), but embodiments are not limited thereto. For example, programs included in the application layer (440) (e.g., software applications) may call APIs to cause the execution of functions supported by programs included in the framework layer (450).

[0118] A wearable device (101) may display one or more visual objects on a display (250) to perform interaction with a user based on the execution of an XR system UI (441). A visual object may mean an object that can be placed on a screen for the transmission of information and / or interaction, such as text, images, icons, videos, buttons, checkboxes, radio buttons, text boxes, sliders, and / or tables. A visual object may be referred to as a visual guide, a virtual object, a visual element, a UI element, a view object, and / or a view element. A wearable device (101) may provide the user with functions available in a virtual space based on the execution of an XR system UI (441).

[0119] Referring to FIG. 4, a lightweight renderer (443) and / or an XR plugin (444) are depicted within the XR system UI (441), but are not limited thereto. For example, based on the XR system UI (441), the processor (410) may execute a lightweight renderer (443) and / or an XR plugin (444) within the framework layer (450).

[0120] A wearable device (101) may acquire resources (e.g., APIs, system processes and / or libraries) used to define, create, and / or execute a rendering pipeline, which is permitted to be partially modified, based on the execution of a lightweight renderer (443). The lightweight renderer (443) may be referred to as a lightweight render pipeline in terms of defining a rendering pipeline, which is permitted to be partially modified. The lightweight renderer (443) may include a renderer built prior to the execution of a software application (e.g., a pre-built renderer). For example, the wearable device (101) may acquire resources (e.g., APIs, system processes and / or libraries) used to define, create, and / or execute the entire rendering pipeline based on the execution of an XR plugin (444). The XR plugin (444) may be referred to as an open XR native client in terms of defining (or setting) the entire rendering pipeline.

[0121] The wearable device (101) may display a screen representing at least a portion of a virtual space on a display (250) based on the execution of an XR application (442). The XR plugin (444-1) included in the XR application (442) may include instructions that support functions similar to the XR plugin (444) of the XR system UI (441). Descriptions of the XR plugin (444-1) that overlap with descriptions of the XR plugin (444) may be omitted. The wearable device (101) may trigger the execution of a virtual space manager (451) based on the execution of the XR application (442).

[0122] A wearable device (101) can display an image on a display (250) in a virtual space based on the execution of an application (445). The application (445) may be configured to output image information for displaying a two-dimensional image. A wearable device (101) may trigger the execution of a virtual space manager (451) based on the execution of the application (445). A wearable device (101) may generate dual image information to display the two-dimensional image in a three-dimensional virtual space based on the execution of the application (445). Here, the dual image information may include a first image information for the left eye and a second image information for the right eye, taking into account binocular parallax. To display the two-dimensional image in a three-dimensional virtual space, a wearable device (101) may generate the dual image information based on the image information for displaying the two-dimensional image.

[0123] According to one embodiment, a wearable device (101) can provide a virtual space service based on the execution of a virtual space manager (451). For example, the virtual space manager (451) may include a platform for supporting the virtual space service. Based on the execution of the virtual space manager (451), the wearable device (101) can identify a virtual space formed based on the user's location indicated by data acquired through a sensor (420), and can display at least a portion of the virtual space on a display (250). The virtual space manager (451) may be referred to as a composition presentation manager (CPM).

[0124] The virtual space manager (451) may include a runtime service (452). For example, the runtime service (452) may be referred to as an OpenXR runtime module (or OpenXR runtime program). The wearable device (101) may execute at least one of a user pose prediction function, a frame timing function, and / or a spatial input function based on the execution of the runtime service (452). For example, the wearable device (101) may perform rendering for a virtual space service for the user based on the execution of the runtime service (452). For example, a virtual space-related function, executable by the application layer (440), may be supported based on the execution of the runtime service (452).

[0125] The virtual space manager (451) may include a pass-through manager (453). The wearable device (101) may display an image and / or video representing a real space acquired through an external camera superimposed on at least a portion of the screen while displaying a screen representing a virtual space on the display (250) based on the execution of the pass-through manager (453).

[0126] The virtual space manager (451) may include an input manager (454). The wearable device (101) may identify acquired data (e.g., sensor data) by executing one or more programs included within the recognition service layer (470) based on the execution of the input manager (454). The wearable device (101) may identify user inputs associated with the wearable device (101) using the acquired data. The user inputs may be associated with user motions (e.g., hand gestures), gaze, and / or speech identified by a sensor (420) (e.g., an image sensor (421) such as an external camera). The user inputs may be identified based on an external electronic device connected (or paired) via a communication circuit.

[0127] The perception abstract layer (460) can be used for data exchange between the virtual space manager (451) and the perception service layer (470). In terms of being used for data exchange between the virtual space manager (451) and the perception service layer (470), the perception abstract layer (460) can be referred to as an interface. For example, the perception abstract layer (460) can be referred to as OpenPX. The perception abstract layer (460) can be used for a perception client and a perception service.

[0128] According to one embodiment, the recognition service layer (470) may include one or more programs for processing data obtained from the sensor (420). The one or more programs may include at least one of a location tracker (471), a spatial recognizer (472), a gesture tracker (473), and / or an eye tracker (474). The type and / or number of the one or more programs included in the recognition service layer (470) are not limited to those shown in FIG. 4.

[0129] The wearable device (101) can identify the posture of the wearable device (101) using a sensor (420) based on the operation of a position tracker (471). The wearable device (101) can identify the 6 degrees of freedom pose (6 dof pose) of the wearable device (101) using data acquired using an external camera (e.g., image sensor (421)) and / or an IMU (e.g., motion sensor (422) including a gyroscope, accelerometer, and / or geomagnetic sensor) based on the operation of the position tracker (471). The position tracker (471) may be referred to as a head tracking (HeT) module (or head tracker, head tracking program).

[0130] A wearable device (101) can acquire information to provide a three-dimensional virtual space corresponding to the surrounding environment (e.g., external space) of the wearable device (101) (or the user of the wearable device (101)) based on the execution of a spatial recognition device (472). Based on the execution of the spatial recognition device (472), the wearable device (101) can reproduce the surrounding environment of the wearable device (101) in three dimensions using data acquired using an external camera (e.g., image sensor (421)). Based on the surrounding environment of the wearable device (101) reproduced in three dimensions based on the execution of the spatial recognition device (472), the wearable device (101) can identify at least one of a plane, an incline, and a staircase. The spatial recognition device (472) may be referred to as a scene understanding (SU) module (or scene understanding program).

[0131] The wearable device (101) can identify (or recognize) the pose and / or gesture of the user's hand of the wearable device (101) based on the execution of the gesture tracker (473). For example, the wearable device (101) can identify the pose and / or gesture of the user's hand using data acquired from an external camera (e.g., image sensor (421)) based on the execution of the gesture tracker (473). For example, the wearable device (101) can identify the pose and / or gesture of the user's hand based on data (or images) acquired using an external camera based on the execution of the gesture tracker (473). The gesture tracker (473) may be referred to as a hand tracking (HaT) module (or hand tracking program) and / or a gesture tracking module.

[0132] The wearable device (101) can identify (or track) the movement of the user's eyes of the wearable device (101) based on the execution of the eye tracker (474). For example, the wearable device (101) can identify the movement of the user's eyes using data obtained from an eye-tracking camera (e.g., image sensor (421)) based on the execution of the eye tracker (474). The eye tracker (474) may be referred to as an eye tracking (ET) module (or eye tracking program) and / or a gaze tracking module.

[0133] The recognition service layer (470) of the wearable device (101) may further include a face tracker (475) for tracking the user's face. For example, the wearable device (101) may identify (or track) the movement of the user's face and / or the user's facial expression based on the execution of the face tracker (475). The wearable device (101) may estimate the user's facial expression based on the movement of the user's face based on the execution of the face tracker (475). For example, the wearable device (101) may identify the movement of the user's face and / or the user's facial expression based on data (e.g., images and / or videos) obtained using a camera (425) (e.g., a camera facing at least a part of the user's face) based on the execution of the face tracker (475).

[0134] Referring to FIG. 4, an example of a processor (410) is shown as a CPU (411), a GPU (graphic processing unit) (412), and / or a DPU (display processing unit) (413). A renderer (490) may include instructions for rendering images in a three-dimensional virtual space. A processor (410) (e.g., DPU (413)) that executes the renderer (490) may acquire at least one image to be displayed at least partially in a display area of ​​a display (250) in a software application (e.g., a software application executed by the CPU (411) and / or GPU (412)). For example, a processor (410) that executes the renderer (490) may determine the location of an area where an application (e.g., an XR application (442), an application (445)) will be rendered. The processor (410) that executed the renderer (490) can generate an image of the application to be displayed on the display (250). The renderer (490) can synthesize images to generate a composite image to be displayed on the display (250).

[0135] A processor (410) that executes a renderer (490) can divide the display area of ​​a display (250) into a foveated portion (or may be referred to as a foveated area) and a peripheral portion (or may be referred to as a residual area) using a gaze position calculated using a position tracker (471) and / or a gaze tracker (474). For example, a processor (410) that detects coordinate values ​​of the gaze position can determine the portion of the display area containing said coordinate values ​​as the foveated area. A DPU (413) that executes a renderer (490) can acquire at least one image corresponding to each of said foveated area and said residual area, having a size smaller than the size of the entire display area of ​​the display (250) or having a resolution less than the resolution of the display area.

[0136] A processor (410) that executes a renderer (490) can obtain or generate a composite image to be displayed on a display (250) by synthesizing an image corresponding to a foveated area and an image corresponding to a surrounding area. For example, the processor (410) can perform upscaling to enlarge the image corresponding to the surrounding area to the size of the entire display area of ​​the display (250). On the enlarged image, the processor (410) can combine the image corresponding to the foveated area to generate a composite image to be displayed on the display (250). Along the boundary line of the image corresponding to the foveated area, the processor (410) can mix the enlarged image and the image corresponding to the foveated area by applying a visual effect such as blur.

[0137] FIG. 5 illustrates an example of a block diagram of a wearable device (e.g., the electronic device (101) of FIG. 1) for displaying an image in a virtual space. FIG. 5 describes an example in which multiple programs / instructions for displaying an image in a virtual space are executed. The multiple programs / instructions may all be executed on a single processor (e.g., AP) or may be executed by multiple processors (e.g., AP, GPU (graphic processing unit), NPU (neural processing unit)). The meaning of being able to be executed by multiple processors is that some programs / instructions may be executed by a first processor and other programs / instructions may be executed by a second processor different from the first processor.

[0138] Referring to FIG. 5, a wearable device (101) may execute a virtual space manager (550) (e.g., virtual space manager (451) of FIG. 4, CPM) to render an image in a virtual space. For the virtual space manager (550), at least some of the descriptions of the virtual space manager (451) of FIG. 4 may be referenced. The virtual space manager (550) may include a platform for supporting virtual space services. The virtual space manager (550) may include a runtime service (551) (e.g., OpenXR Runtime), a panel rendering (552) (e.g., 2D Panel Render), and an XR composite unit (553) (XR Compositor). Based on the execution of the runtime service (551), the wearable device (101) may execute at least one of a user pose prediction function, a frame timing function, and / or a spatial input function. For the runtime service (551), at least some of the descriptions of the runtime service (452) of FIG. 4 may be referenced. The wearable device (101) may display at least one image (video) on a panel (e.g., a 2D panel) to enable the implementation of a virtual space through a display, based on the execution of panel rendering (552). For example, the wearable device (101) may display a rendering image corresponding to RGB information (566) for a panel from the spatialization manager (540) described later through a display (e.g., a display (250)). The wearable device (101) may composite an image of a real area (hereinafter, a pass-through image) captured through a camera in a virtual space with an image of a virtual area, based on the execution of an XR composite unit (553) (XR Compositor). For example, the wearable device (101) can generate a composite image by merging the pass-through image and the virtual region image based on the execution of the XR synthesis unit (553).The wearable device (101) can transmit the generated composite image to a display buffer so that the composite image is displayed. The wearable device (101) can identify a virtual space through a virtual space manager (550) and display at least a portion of the virtual space on a display (250). The virtual space manager (550) may be referred to as CPM. The wearable device (101) can execute the virtual space manager (550) to render an image corresponding to at least a portion of the virtual space.

[0139] According to one embodiment, a wearable device (101) can execute a spatialization manager (540). The spatialization manager (540) can perform processing for displaying an image in a three-dimensional virtual space. The wearable device (101) can perform preprocessing based on the execution of the spatialization manager (540) so that an image can be rendered in a three-dimensional virtual space through a virtual space manager (550). For example, the wearable device (101) can perform at least some of the functions of the renderer (490) of FIG. 4 based on the execution of the spatialization manager (540). The wearable device (101) can process image information provided by an application (e.g., an XR application (510), a non-XR general 2D screen application (520), or a system UI (530) based on the execution of the spatialization manager (540). A spatialization manager (540) (e.g., Space Flinger) may include a system screen manager (541) (e.g., System scene), an input manager (542) (e.g., Input Routing), and a lightweight rendering engine (543) (e.g., Impress Engine). The system screen manager (541) may be executed to display the system UI (530). System UI-related information (564) may be transmitted to the system screen manager (541) from a program (e.g., API) that provides the system UI (530). System UI-related information (564) may be obtained through a spatializer API and / or a Same-process private API. The spatialization manager (540) may determine the layout (e.g., position, display order) of the system UI (530) screen in three-dimensional space through pre-allocated resources. The system screen manager (541) can transmit image information (567) for rendering the screen of the system UI (530) according to the above layout to the virtual space manager (550).The input manager (542) may be configured to process user input (e.g., user input on a system screen or an app screen). The lightweight rendering engine (543) may be a renderer for image generation (e.g., lightweight renderer (443)). For example, the lightweight rendering engine (543) may be used to display the system UI (530). According to one embodiment, the spatialization manager (540) may include a lightweight rendering engine (543) for rendering the system UI. According to one embodiment, if the lightweight rendering engine (543) does not have sufficient resources to render an avatar used in an HMD, at least one external rendering engine may be used. In this case, to resolve compatibility issues with external rendering (e.g., a third-party engine), an external rendering engine support module may be added within the spatialization manager (540).

[0140] According to one embodiment, an electronic device may execute an application. For example, in response to the execution of an XR application (510) (e.g., an XR application (442), a 3D game, an XR map, or other immersive application), a virtual space manager (550) may be executed. The wearable device (101) may provide dual image information (561) provided from the XR application (510) to the virtual space manager (550). To display images in three-dimensional space, the dual image information (561) may include two image information that account for binocular parallax. For example, the dual image information (561) may include a first image information for the user's left eye and a second image information for the user's right eye to render in three-dimensional virtual space. Hereinafter, the term dual image information is used in the present disclosure to refer to image information for displaying images for both eyes in three-dimensional space. In addition to the dual image information, the above dual image information may utilize binocular image information, dual image information, dual image data, dual image, binocular image data, stereoscopic image information, 3D image information, spatial image information, spatial image data, 2D-3D conversion data, dimension conversion image data, binocular parallax image data, and / or equivalent technical terms. The wearable device (101) can generate a composite image by merging image layers through a virtual space manager (550). The wearable device (101) can transmit the generated composite image to a display buffer. The composite image can be displayed on the display (250) of the wearable device (101).

[0141] According to one embodiment, the electronic device may execute at least one application among an XR application (510) and other applications (520) (e.g., a first application (520-1), a second application (520-2), and a Nth application (520-N)). According to one embodiment, the application (520) may be configured to output image information for displaying a two-dimensional image. In other words, the application (520) may provide a two-dimensional image. For example, the application (520) may be a video application, a schedule application, or an internet browser application. Let us assume that, in response to the execution of the application (520), image information (562) provided by the application (520) is provided to the virtual space manager (550). Since the image information (562) has only x-coordinates and y-coordinates within a two-dimensional plane, it may be difficult to consider the sequential relationship between other applications centered on the user (i.e., distance from the user). The wearable device (101) may execute a spatialization manager (540) to provide dual image information to a virtual space manager (550), even when displaying an application (520) that provides a general 2D screen. For example, based on the execution of the spatialization manager (540), the wearable device (101) may receive application-related information (563) from a first application (520-1). For example, the application-related information (563) may include image information representing a 2D image of the first application (520-1) (e.g., information including RGB per pixel) and / or content information in the first application (520-1) (e.g., characteristics of the content executed in the first application, type of content). The application-related information (563) may be obtained through a spatializer API.Based on the execution of the spatialization manager (540), the wearable device (101) can identify information regarding the location of the area to be rendered and the size of the area to be rendered (hereinafter, location information). Based on the execution of the spatialization manager (540), the wearable device (101) can generate dual image information (565, e.g., RGBx2) that takes into account the user's binocular parallax through the image information and the location information. Based on the execution of the spatialization manager (540), the wearable device (101) can provide the dual image information (565) to the virtual space manager (550). By converting a simple two-dimensional image into dual image information (565), the problem caused by the image information (562) being directly transmitted to the virtual space manager (550) can be resolved. Additionally, as at least some of the functions for displaying images in virtual space are performed by the spatialization manager (540) instead of the virtual space manager (550), the burden on the virtual space manager (550) may be reduced. However, as image information from the application (520) is not transmitted directly to the virtual space manager (550) but is transmitted through the spatialization manager (540), the quality of the image ultimately output to the user may be lowered. For example, in the first application (520-1), an image is rendered at a resolution of approximately 2756 x 1846, but the image may be downsampled during the process of being transmitted to the virtual space manager (550) through the spatialization manager (540) (e.g., downsampled from a resolution of approximately 2756 x 1846 to a resolution of approximately 1160 x 680). Afterwards, the virtual space manager (550) can upsample the downsampled image (e.g., upsample from a resolution of about 1160 x 680 to a resolution of about 1625 x 1070) and pass the upsampled image to the display buffer.As such, in the process of an image being transmitted from an application (520) to a spatialization manager (540) and from a spatialization manager (540) to a virtual space manager (550), a resolution mismatch may occur, or an aliasing problem or a degradation of image quality may occur during the upsampling process. To resolve the above-mentioned problems, the present disclosure describes techniques for controlling the resolution of an area to be displayed in an application and performing foveation rendering based on the system structure illustrated in FIG. 5.

[0142] Figure 6 illustrates an example of a structure of multiple layers.

[0143] Referring to FIG. 6, programs installed on a wearable device (101) can be classified into one of a platform layer (610), a recognition service layer (620) (e.g., the recognition service layer (470) of FIG. 4), and a sensor service layer (630). For example, the wearable device (101) can operate based on the platform layer (610), the recognition service layer (620), and the sensor service layer (630).

[0144] According to one embodiment, the platform layer (610) may be configured for XR services. For example, the platform layer (610) may include a platform (e.g., an Android platform) for supporting XR services. For example, the platform layer (610) may include the virtual space manager (550) of FIG. 5. The platform layer (610) may include a runtime service (611). For the runtime service (611), the descriptions of the runtime service (551) of FIG. 5 and the descriptions of the runtime service (452) of FIG. 4 may be referenced. For example, the runtime service (611) may be referenced as an OpenXR runtime module. The runtime service (611) may be used to provide at least one of a user pose prediction function, a frame timing function, and / or a spatial input function through the wearable device (101). For example, the runtime service (611) may be used to perform rendering for XR services for the user. For example, based on the runtime service (611), an application (e.g., a Unity or OpenXR native application) can be implemented.

[0145] The perception abstraction layer (612) can be used for data exchange between the platform layer (610) and the perception service layer (620). For the perception abstraction layer (612), the descriptions of the perception abstraction layer (460) of FIG. 4 may be referenced. For example, the perception abstraction layer (612) may be referenced as OpenPX. The perception abstraction layer (612) can be used for a perception client and a perception service.

[0146] According to one embodiment, the recognition service layer (620) may include a service module (621), a recognition plugin layer (622), a sensor management module (623), a playback module (624), and / or an external data management module (625). For example, the recognition service layer (620) may include at least one of the service module (621), the recognition plugin layer (622), the sensor management module (623), the playback module (624), and / or an external data management module (625). For example, at least some of the service module (621), the recognition plugin layer (622), the sensor management module (623), the playback module (624), and the external data management module (625) may be omitted.

[0147] The service module (621) can manage input data of the wearable device (101). The service module (621) can be used to manage data (e.g., gesture information) obtained from a plurality of recognition modules included in the recognition plugin layer (622). As an example, the service module (621) may be referred to as SxrDataService.

[0148] The service module (621) can perform interfacing with an upper layer (e.g., platform layer (610) or runtime service (611)). The service module (621) can exchange data with the upper layer (e.g., platform layer (610) or runtime service (611)) through the recognition abstraction layer (612). For example, the recognition abstraction layer (612) may be referred to as OpenPX. According to an embodiment, the service module (621) may support OpenXR Extension as well as OpenPX. The service module (621) may be used to exchange data (e.g., gesture information) between a plurality of recognition modules. The service module (621) may be configured to manage data processed by the recognition service layer (620). The service module (621) may select data among the data to be recognized as input to the wearable device (101). The above data may include data obtained from a plurality of recognition modules and data obtained through an external data management module (625). The service module (621) may manage data to be used in the recognition abstraction layer (612). The service module (621) may select data to be recognized as an input to the wearable device (101) from the above data and provide it to the recognition abstraction layer (612).

[0149] The recognition plugin layer (622) may include a plurality of recognition modules. The plurality of recognition modules may be referred to as a plurality of perception solutions.

[0150] For example, a plurality of recognition modules may include at least one of a head tracking (HeT) module (622-1), a scene understanding (SU) module (622-2), a hand tracking (HaT) module (622-3), an eye tracking (ET) module (622-4), and a face tracking (FT) module (622-5). Each of the plurality of recognition modules included in the recognition plugin layer (622) may include a common interface for connection (or interaction) with a sensor management module (623). Each of the plurality of recognition modules may include a common interface for connection (or interaction) with a sensor management module (623).

[0151] The head tracking module (622-1) can identify the posture of the wearable device (101) using at least one sensor of the wearable device (101). For example, the head tracking module (622-1) can identify the 6 degrees of freedom pose (6 dof pose) of the wearable device (101) based on data acquired using a camera (e.g., image sensor (421) of FIG. 4) and an IMU.

[0152] The environment recognition module (622-2) can be used to construct the surrounding environment of the wearable device (101) (or the user of the wearable device (101)) into a three-dimensional virtual space. The environment recognition module (622-2) can be used to reconstruct the surrounding environment of the wearable device (101) in three dimensions based on data acquired using a camera (e.g., the image sensor (421) of FIG. 4). Based on the surrounding environment of the wearable device (101) reconstructed in three dimensions, the environment recognition module (622-2) can identify at least one of a plane, an incline, and a staircase.

[0153] A hand tracking module (622-3) may be used to identify (or recognize) the pose and / or gesture of the user's hand of the wearable device (101). For example, the hand tracking module (622-3) may identify the pose and / or gesture of the user's hand based on data acquired from at least one sensor. For example, the hand tracking module (622-3) may identify the pose and / or gesture of the user's hand based on data (e.g., an image) acquired using a camera.

[0154] An eye tracking module (622-4) may be used to identify (or track) the movement of the user's eyes of the wearable device (101). For example, the eye tracking module (622-4) may identify the user's eye movement based on data obtained from at least one sensor. For example, the eye tracking module (622-4) may identify the user's eye movement based on data obtained using a camera (e.g., the eye tracking camera (260-1) of FIG. 2b and FIG. 3a)) and / or an IR LED (infrared light emitting diode).

[0155] The face tracking module (622-5) can be used to identify (or track) the movement of the user's face and / or the user's facial expression. The face tracking module (622-5) can estimate the user's facial expression based on the movement of the user's face. For example, the face tracking module (622-5) can identify the movement of the user's face and / or the user's facial expression based on data (e.g., images) acquired using a camera (e.g., camera (260) in FIG. 2a and FIG. 2b).

[0156] For example, a plurality of recognition modules included in the recognition plugin layer (622) may be configured in a plugin structure. For example, some of the plurality of recognition modules may be replaced with other modules regardless of the sensor service layer (630) and platform layer (610), which are lower layers of the recognition service layer (620).

[0157] According to one embodiment, the sensor management module (623) may be used to provide (or transmit) data to each of the plurality of recognition modules through a common interface. For example, the sensor management module (623) may be used to separate (or eliminate) the dependency between the lower layer, the sensor service layer (630), and the upper layer, the recognition plugin layer (622). For example, the sensor management module (623) may be referred to as SxrSensorSeviceManger.

[0158] The sensor management module (623) can support various modules (or sensor services) of the sensor service layer (630). Multiple recognition modules may not directly interface with the sensor service layer (630). Multiple recognition modules may receive data (e.g., sensor data) through the sensor management module (623). Therefore, even if a module of the sensor service layer (630) is changed, it may not affect the multiple recognition modules.

[0159] The sensor management module (623) may further include a load balancing module. The load balancing module can identify data provided from the sensor service layer (630). Based on the data provided from the sensor service layer (630), the load balancing module can identify at least some of the recognition modules among the plurality of recognition modules. The load balancing module can provide data to the identified at least some of the recognition modules. For example, the load balancing module can distribute data to the plurality of recognition modules based on the status of the plurality of recognition modules and / or the status of the wearable device (101). For example, the load balancing module can filter the data provided to the plurality of recognition modules based on the status of the plurality of recognition modules and / or the status of the wearable device (101). According to an embodiment, the load balancing module may be configured independently of the sensor management module (623). The load balancing module may be referred to as SxrPerceptionLoadBalancer.

[0160] The playback module (624) may be used to provide a stored dataset to at least one of a plurality of recognition modules in real time via playback. For example, the dataset may be stored through the playback module (624) based on specified specifications. The dataset may include first data obtained from the sensor service layer (630) as well as second data obtained based on the first data obtained from the sensor service layer (630) (e.g., virtual object data or synthetic data). For example, the first data may be referenced as sensor data. The second data may be referenced as virtual data.

[0161] According to an embodiment, the wearable device (101) may receive data from an external electronic device. For example, the data received from the external electronic device may include first data obtained from a service layer included in the external electronic device and / or second data obtained based on the first data. The wearable device (101) may perform playback (or a playback function) using the data received from the external electronic device. The wearable device (101) may transmit the result of the playback (or playback function) to the external electronic device. For example, the wearable device (101) may be used to process the data obtained from the external electronic device on its behalf. The wearable device (101) may receive data obtained from at least one sensor of the external electronic device. Based on the received data, the wearable device (101) may obtain information (e.g., information about a 6-degree-of-freedom posture) obtained through a playback module (624) (or a plurality of recognition modules). The wearable device (101) can transmit the acquired information to an external electronic device. The external electronic device can provide XR services based on the acquired information.

[0162] The playback module (624) can perform playback (or a playback function) based on at least one of the first data and the second data. According to an embodiment, the playback module (624) can perform playback by combining (or mixing) real-time data (e.g., runtime data) and pre-stored data.

[0163] For example, playback may refer to a function that utilizes stored data (or gesture information) according to the operation of the wearable device (101). For example, playback may refer to a function that identifies a value regarding the performance of the XR service through a comparison between gesture information obtained based on a specified operation regarding the XR service and reference gesture information according to said specified operation.

[0164] For example, playback may refer to a function for obtaining information on the performance of an XR service provided to a user of a wearable device (101). The playback module (624) may identify information (e.g., gesture information) regarding a user who has performed a designated action (e.g., mission) regarding the XR service. The playback module (624) may identify reference information regarding the designated action. Reference information may refer to information for determining the completion of the performance of the designated action. The playback module (624) may identify the similarity between the information regarding the user who has performed the designated action and the reference information. Based on the similarity, the playback module (624) may identify whether the performance of the action designated by the user has been completed.

[0165] According to an embodiment, the playback module (624) may be included in the sensor management module (623). For example, the playback module (624) can perform playback through the sensor management module (623) without changing the plurality of recognition modules.

[0166] The external data management module (625) can be used to manage data obtained through an external electronic device (e.g., a smart watch, a smartphone, or a tablet PC) (or at least one sensor of the external electronic device) connected to the wearable device (101). For example, the external data management module (625) can improve the accuracy of a plurality of recognition modules using data obtained from the external electronic device. For example, the external data management module (625) can correct data (or gesture information) obtained from a plurality of recognition modules using data obtained from the external electronic device. According to an embodiment, the external data management module (625) may not be included in the recognition service layer (620).

[0167] The sensor service layer (630) may be used to control at least one sensor (e.g., camera, IMU, TOF (time of flight) sensor). For example, the sensor service layer (630) may be used to provide a service for accessing at least one sensor. For example, the sensor service layer (630) may include at least one of a module for VR services (e.g., QVRservice), a module for XR services (e.g., SxrSensorService), a sensor API (e.g., android sensor API), and a sensor hardware abstraction layer (sensor HAL).

[0168] According to one embodiment, the sensor management module (623) can provide sensor data to the recognition plugin layer (622) through a common interface. For example, the sensor management module (623) can provide sensor data to each of a plurality of recognition modules through the same interface. For example, the sensor management module (623) can provide sensor data according to the operation of the recognition module to the recognition module without changing the configuration information of the recognition plugin layer (622) based on changing (or modifying) the configuration information (e.g., configuration file) regarding the sensor management module (623).

[0169] The sensor management module (623) can identify sensor data for at least one recognition module based on the operation of at least one recognition module among a plurality of recognition modules. The sensor management module (623) can provide the identified sensor data to the at least one recognition module.

[0170] According to one embodiment, when the head tracking module (622-1) is driven, the sensor management module (623) can acquire camera data and IMU data through at least one of a module for VR services, a module for XR services, a sensor API, and a sensor hardware abstraction layer in the sensor service layer (630). The sensor management module (623) can provide the camera data and IMU data to the head tracking module (622-1). According to an embodiment, the camera data and IMU data may be acquired through different modules.

[0171] According to one embodiment, when the environment recognition module (622-2) is operated in playback mode, the sensor management module (623) can identify stored camera data and stored posture data. The sensor management module (623) can provide the camera data and posture data to the environment recognition module (622-2).

[0172] According to one embodiment, the service module (621) may be configured to eliminate dependency on the upper layer of the recognition plugin layer (622). For example, the upper layer of the recognition plugin layer (622) may include a platform layer (610) (e.g., Android XR) and / or an application layer (e.g., the application layer (440) of FIG. 4).

[0173] The service module (621) can manage input data of the wearable device (101). The service module (621) can be configured to integrate and manage information (e.g., gesture information or tracking data) obtained from multiple recognition modules. The service module (621) can convert the information (e.g., gesture information or tracking data) according to the requirements of the upper layer without changing the multiple recognition modules, and then provide the converted information to the upper layer.

[0174] For example, a service module (621) can obtain information about a 6-degree-of-freedom attitude from a head tracking module (622-1). The information about the 6-degree-of-freedom attitude obtained from the head tracking module (622-1) can be configured in a quaternion format. On the other hand, an upper layer (e.g., platform layer (610)) can request information about the 6-degree-of-freedom attitude configured in an axis-angle representation format. The service module (621) can change (or convert) the information about the 6-degree-of-freedom attitude configured in a quaternion format into information about the 6-degree-of-freedom attitude configured in an axis-angle representation format. The service module (621) can provide information about the 6-degree-of-freedom attitude configured in an axis-angle representation format to the upper layer (e.g., platform layer (610)). However, it is not limited thereto. For example, the service module (621) can change (or convert) information about a 6-degree-of-freedom attitude configured in an axis-angle representation format into information about a 6-degree-of-freedom attitude configured in a quaternion format and provide it to the upper layer.

[0175] For example, a service module (621) can obtain information about the movement of the hand from a hand tracking module (622-3). Information about the movement of the hand can be obtained based on the movement of a first number of joints. On the other hand, an upper layer (e.g., platform layer (610)) can request information about the movement of the hand obtained based on the movement of a second number of joints. The service module (621) can perform either a joint interpolation procedure or a simplification procedure. Based on performing either a joint interpolation procedure or a simplification procedure, the service module (621) can support the structure of the joints required by the upper layer.

[0176] FIG. 7 is a flowchart illustrating a method of operating a wearable device (101) according to one embodiment.

[0177] The operations illustrated in FIG. 7 can be performed by instructions stored in memory (e.g., memory (130) of FIG. 1). For example, when the instructions are executed by a processor (e.g., processor (120) of FIG. 1), the wearable device (101) (e.g., electronic device (101) of FIG. 1) can perform the operations illustrated in FIG. 7.

[0178] At least some of the operations shown in FIG. 7 may be omitted. At least some operations mentioned in the present disclosure with reference to other drawings may be additionally inserted before or after at least some of the operations shown in FIG. 7.

[0179] According to one embodiment, at least some of the operations illustrated in FIG. 7 may be performed sequentially. According to one embodiment, at least some of the operations illustrated in FIG. 7 may be performed in parallel (simultaneously). According to one embodiment, at least some of the operations illustrated in FIG. 7 may be performed with their order changed.

[0180] Hereinafter, a method of operating a wearable device (101) according to one embodiment will be described with reference to FIG. 7. In FIG. 7, the wearable device (101) may be named as a first electronic device (101) and may be a head-mount device (HMD), a headgear electronic device, a glasses-type electronic device, a video see-through or visible see-through (VST) device, an extended reality (XR) device, a virtual reality (VR) device, and / or an augmented reality (AR) device. In FIG. 7, the external electronic device may be named as a second electronic device (e.g., the second electronic device (820) of FIG. 8). The second electronic device (820) may be a mobile phone, a tablet PC, a desktop PC, a laptop device, or a TV.

[0181] In operation 710, a second electronic device (820) may be communicatively connected to a first electronic device (101) according to one embodiment. According to one embodiment, the first electronic device (101) and the second electronic device (820) may be devices registered to a server based on a user account. For example, the first electronic device (101) and the second electronic device (820) may be devices registered to a server with the same user account. According to one embodiment, the first electronic device (101) and the second electronic device (820) may transmit data (e.g., tasks, and / or commands) to each other through a server. According to one embodiment, the first electronic device (101) and the second electronic device (820) may communicate directly with each other via short-range communication. For example, the first electronic device (101) may transmit data (e.g., tasks, and / or commands) directly to the second electronic device (820) using short-range communication such as Bluetooth or WiFi.

[0182] In operation 720, software or content may be executed on a first electronic device (101) according to one embodiment. According to one embodiment, the first electronic device (101) may execute one or more applications, software, or content based on user input. For example, the first electronic device (101) may execute an application related to document work, an application related to editing sound or video, or an application for playing content based on user input.

[0183] In operation 730, the first electronic device (101) according to one embodiment can detect that it is detached from the user. According to one embodiment, the first electronic device (101) can detect that the user is detaching the first electronic device (101) by using at least one sensor.

[0184] In operation 740, according to one embodiment, a first electronic device (101) can transmit a task currently being executed to a second electronic device (820) that is already connected. According to one embodiment, when the first electronic device (101) detects detachment, it can transmit a task related to an application or content that was being executed and a command related to the task to the second electronic device (820). According to one embodiment, the first electronic device (101) can transmit the task and command to the second electronic device (820) via a server. According to one embodiment, the first electronic device (101) can communicate directly with the second electronic device (820) and transmit the task and command to the second electronic device (820) through direct communication. According to one embodiment, if there is no second electronic device (820) that is already connected, the first electronic device (101) can transmit the task and command to another electronic device that has a history of connection or is capable of connection. For example, if the first electronic device (101) does not have a connected second electronic device (820), it can establish a connection with a second electronic device (820) that can be connected. According to one embodiment, the first electronic device (101) may use a cloud or an account viewing location synchronization service as a method of transmitting tasks and commands to the second electronic device (820).

[0185] According to one embodiment, the task information transmitted by the first electronic device (101) may include information on an application currently running, a software file, or settings and data for operations such as simulation.

[0186] In operation 750, the first electronic device (101) according to one embodiment may turn off the power or change to sleep mode. According to one embodiment, the first electronic device (101) may turn off the power or enter sleep mode when it has completed transmitting tasks and commands to the second electronic device (820). According to one embodiment, sleep mode may mean a low-power mode, and may mean, for example, a state in which the performance and power consumption of the components included in the first electronic device (101) are lower than in normal mode.

[0187] In operation 760, a task received from a second electronic device (820) according to one embodiment may be displayed in the form of a pop-up notification according to a preset priority. According to one embodiment, the second electronic device (820) may display a task received from a first electronic device (101) in the form of a pop-up notification according to priority. For example, the second electronic device (820) may receive n tasks from the first electronic device (101) and may sort a pop-up list of n tasks according to priority. According to one embodiment, the priority of the second electronic device (820) in displaying tasks may vary based on the order in which they were received from the first electronic device (101), the frequency of use in the second electronic device (820), the order in which they were recently saved or worked on in the first electronic device (101), or whether a program related to the task is installed in the second electronic device (820). According to one embodiment, when the second electronic device (820) receives a task from the first electronic device (101), if software for executing the transmitted task is not installed, the second electronic device (820) may download and install software from a download link or a server. According to one embodiment, if software is not installed, the second electronic device (820) may receive software directly from the first electronic device (101) and may display a pop-up screen for this purpose.

[0188] In operation 770, software or content can be used continuously after selecting the corresponding task in the second electronic device (820) according to one embodiment. According to one embodiment, the second electronic device (820) can continuously execute the task that was executed in the first electronic device (101) based on user input selecting the task. According to one embodiment, the second electronic device (820) can display a screen corresponding to the result of executing the task. Accordingly, the user can continuously use the software or content that was used through the first electronic device (101) through the second electronic device (820) after the first electronic device (101) is detached.

[0189] FIG. 8 is a conceptual diagram illustrating a method of operating a wearable device (101) according to one embodiment.

[0190] In FIG. 8, reference numeral 810 represents a mixed reality space (810) including augmented reality or virtual reality provided by a wearable device (101). According to one embodiment, the wearable device (101) may display an execution screen of at least one application in the mixed reality space (810). For example, the wearable device (101) may display an execution screen (811) of a first application, an execution screen (812) of a second application, and an execution screen (813) of a third application in the mixed reality space (810), but the present invention is not limited thereto.

[0191] In FIG. 8, the wearable device (101) may be named "first electronic device (101)".

[0192] According to one embodiment, as illustrated in the example, the wearable device (101) can display a first application execution screen (811), a second application execution screen (812), and a third application execution screen (813) in a mixed reality space (810) based on user input. According to one embodiment, the wearable device (101) can detect that the user is taking off the wearable device (101) while displaying the first application execution screen (811), the second application execution screen (812), and the third application execution screen (813), as described in operations 730 and 740 of FIG. 7.

[0193] According to one embodiment, the wearable device (101) can transmit a task to at least one external electronic device (820, 830) in response to detecting removal. For example, the wearable device (101) can transmit tasks and commands related to the first application, the second application, and the third application to the second electronic device (820) and the third electronic device (830).

[0194] According to one embodiment, the second electronic device (820) and the third electronic device (830) receive a task from the wearable device (101) and can display the received task in the form of a pop-up notification. For example, the second electronic device (820) can display a first pop-up notification (821) related to a first application, a second pop-up notification (822) related to a second application, a third pop-up notification (823) related to a third application, and a pop-up notification (824) related to the detection of removal of the wearable device (101). For example, the third electronic device (830) can display a first pop-up notification (831) related to a first application, a second pop-up notification (832) related to a second application, a third pop-up notification (833) related to a third application, and a pop-up notification (834) related to the detection of removal of the wearable device (101).

[0195] In the illustrated example, the second electronic device (820) is a laptop device and the third electronic device (830) is a mobile phone, but the invention is not limited thereto. Each of the second electronic device (820) and the third electronic device (830) may be a mobile phone, a tablet PC, a desktop PC, a laptop device, or a TV.

[0196] FIG. 9 is a flowchart illustrating a method by which a wearable device (101) according to one embodiment delivers a task.

[0197] The operations illustrated in FIG. 9 can be performed by instructions stored in memory (e.g., memory (130) of FIG. 1). For example, when the instructions are executed by a processor (e.g., processor (120) of FIG. 1), the wearable device (101) (e.g., electronic device (101) of FIG. 1) can perform the operations illustrated in FIG. 9.

[0198] At least some of the operations shown in FIG. 9 may be omitted. At least some operations mentioned in the present disclosure with reference to other drawings may be additionally inserted before or after at least some of the operations shown in FIG. 9.

[0199] According to one embodiment, at least some of the operations illustrated in FIG. 9 may be performed sequentially. According to one embodiment, at least some of the operations illustrated in FIG. 9 may be performed in parallel (simultaneously). According to one embodiment, at least some of the operations illustrated in FIG. 9 may be performed with their order changed.

[0200] Hereinafter, with reference to FIG. 9, a method of transmitting a task by a wearable device (101) according to one embodiment is described. In FIG. 9, the wearable device (101) may be named as a first electronic device (101) and may be a head-mount device (HMD), a headgear electronic device, a glasses-type electronic device, a video see-through or visible see-through (VST) device, an extended reality (XR) device, a virtual reality (VR) device, and / or an augmented reality (AR) device. In FIG. 9, the external electronic device may be named as a second electronic device (e.g., the second electronic device (820) of FIG. 8) and / or a third electronic device (e.g., the third electronic device (830) of FIG. 8). Each of the second electronic device (820) and the third electronic device (830) may be a mobile phone, a tablet PC, a desktop PC, a laptop device, or a TV.

[0201] In operation 911, a second electronic device (820) and a third electronic device (830) may be communicated to a first electronic device (101) according to one embodiment. The first electronic device (101) and the second electronic device (820) and / or the third electronic device (830) may be devices registered to a server based on a user account. For example, the first electronic device (101) and the second electronic device (820) and / or the third electronic device (830) may be devices registered to a server with the same user account. According to one embodiment, the first electronic device (101) and / or the third electronic device (830) may transmit data (e.g., tasks, and / or commands) to each other through a server. According to one embodiment, the first electronic device (101) and / or the third electronic device (830) may communicate directly with each other via short-range communication. For example, data (e.g., tasks, and / or commands) can be directly transmitted from the first electronic device (101) to the third electronic device (830) using short-range communication such as Bluetooth. Operation 911 may be at least partially similar to operation 710 described with reference to FIG. 7.

[0202] In operation 913, software or content may be executed on the first electronic device (101) according to one embodiment. Operation 913 may be at least partially similar to operation 720 described with reference to FIG. 7. According to one embodiment, the first electronic device (101) may display an execution screen (811) of a first application, an execution screen (812) of a second application, and an execution screen (813) of a third application in a mixed reality space (810) based on user input, as illustrated in the example in FIG. 8.

[0203] In operation 915, the first electronic device (101) according to one embodiment may be detected to be detached from the user. Operation 915 may be at least partially similar to operation 730 described with reference to FIG. 7.

[0204] In operation 917, according to one embodiment, a first electronic device (101) may transmit a task currently being executed to a connected second electronic device (820) and a third electronic device (830). Operation 917 may be at least partially similar to operation 740 described with reference to FIG. 7. According to one embodiment, the first electronic device (101) may transmit a task to at least one external electronic device (820, 830) in response to detecting the user's removal. For example, the first electronic device (101) may transmit tasks and commands related to the first application, the second application, and the third application to the second electronic device (820) and the third electronic device (830).

[0205] In operation 919, the first electronic device (101) according to one embodiment may turn off the power or change to a sleep mode (or low power mode). Operation 919 may be at least partially similar to operation 750 described with reference to FIG. 7.

[0206] In operation 921, a task received according to a preset priority in the second electronic device (820) according to one embodiment may be displayed in the form of a pop-up notification. Operation 921 may be at least partially similar to operation 760 described with reference to FIG. 7. For example, as shown in the example illustrated in FIG. 8, the second electronic device (820) may display a first pop-up notification related to a first application, a second pop-up notification related to a second application, a third pop-up notification related to a third application, and a pop-up notification related to the detection of detachment of the first electronic device (101).

[0207] In operation 923, software or content may be used sequentially after selecting the corresponding task in the second electronic device (820) according to one embodiment. For example, the second electronic device (820) may select tasks to be executed sequentially based on user input. Operation 923 may be at least partially similar to operation 770 described with reference to FIG. 7.

[0208] In operation 925, a task received according to a preset priority in a third electronic device (830) according to one embodiment may be displayed in the form of a pop-up notification. Operation 925 may be at least partially similar to operation 740 described with reference to FIG. 7. For example, as shown in the example illustrated in FIG. 8, the third electronic device (830) may display a first pop-up notification related to a first application, a second pop-up notification related to a second application, a third pop-up notification related to a third application, and a pop-up notification related to the detection of detachment of the first electronic device (101).

[0209] In operation 927, in a third electronic device (830) according to one embodiment, a task selected by the second electronic device (820) can be removed from a popup notification. According to one embodiment, the list of tasks received by the second electronic device (820) from the first electronic device (101) may be the same as the list of tasks received by the third electronic device (830) from the first electronic device (101). According to one embodiment, when the second electronic device (820) selects a specific task based on user input, the third electronic device (830) can remove the selected task from the popup list displayed on the third electronic device (830). For example, when a user selects a task related to the second application through the second electronic device (820), the third electronic device (830) may receive information related to the user's selection through the first electronic device (101), the second electronic device (820), or the server, and may delete a popup notification related to the second application from the popup notification list based on the received information.

[0210] FIG. 10 is an example drawing of an external electronic device displaying a pop-up notification according to one embodiment. For example, FIG. 10 illustrates a second electronic device (820) and a third electronic device (830) as external electronic devices. The second electronic device (820) and the third electronic device (830) illustrated in FIG. 10 may be substantially the same as the second electronic device (820) and the third electronic device (830) described with reference to FIG. 8.

[0211] Referring to FIG. 10, a second electronic device (820) and a third electronic device (830) according to one embodiment can display a list of tasks received from a first electronic device (101) (e.g., a wearable device (101)) according to operations 917, 921, and 925 described with reference to FIG. 9.

[0212] According to one embodiment, the second electronic device (820) may display a first pop-up notification (821) related to a first application, a second pop-up notification (822) related to a second application, a third pop-up notification (823) related to a third application, and a pop-up notification (824) related to the detection of removal of the wearable device (101).

[0213] According to one embodiment, the third electronic device (830) may display a first pop-up notification (831) related to a first application, a second pop-up notification (832) related to a second application, a third pop-up notification (833) related to a third application, and a pop-up notification (834) related to the detection of removal of the wearable device (101).

[0214] FIG. 11 is a flowchart illustrating a method by which a wearable device (101) according to one embodiment delivers a task.

[0215] The operations illustrated in FIG. 11 can be performed by instructions stored in memory (e.g., memory (130) of FIG. 1). For example, when the instructions are executed by a processor (e.g., processor (120) of FIG. 1), the wearable device (101) (e.g., electronic device (101) of FIG. 1) can perform the operations illustrated in FIG. 11.

[0216] At least some of the operations shown in FIG. 11 may be omitted. At least some operations mentioned in the present disclosure with reference to other drawings may be additionally inserted before or after at least some of the operations shown in FIG. 11.

[0217] According to one embodiment, at least some of the operations illustrated in FIG. 11 may be performed sequentially. According to one embodiment, at least some of the operations illustrated in FIG. 11 may be performed in parallel (simultaneously). According to one embodiment, at least some of the operations illustrated in FIG. 11 may be performed with their order changed.

[0218] Hereinafter, with reference to FIG. 11, a method of transmitting a task by a wearable device (101) according to one embodiment is described. In FIG. 11, the wearable device (101) may be named as a first electronic device (101) and may be a head-mount device (HMD), a headgear electronic device, a glasses-type electronic device, a video see-through or visible see-through (VST) device, an extended reality (XR) device, a virtual reality (VR) device, and / or an augmented reality (AR) device. In FIG. 11, the external electronic device may be named as a second electronic device (820). The second electronic device (820) may be a mobile phone, a tablet PC, a desktop PC, a laptop device, or a TV.

[0219] In operation 1110, a second electronic device (820) may be communicated to a first electronic device (101) according to one embodiment. Operation 1110 may be at least partially similar to operation 710 described with reference to FIG. 7.

[0220] In operation 1120, software or content may be executed on the first electronic device (101) according to one embodiment. Operation 1120 may be similar in at least part to operation 720 described with reference to FIG. 7.

[0221] In operation 1130, a program continuation (or application continuation) or target device can be set through a menu within the first electronic device (101) according to one embodiment. According to one embodiment, when the first electronic device (101) transmits a task to the second electronic device (820), it may receive input from the user selecting the task to be transmitted directly. According to one embodiment, when the first electronic device (101) transmits a task to the second electronic device (820), it may determine the task to be transmitted based on detecting user gestures or the user's gaze. According to one embodiment, when the first electronic device (101) determines the task to be transmitted to the second electronic device (820), it may provide a continuation icon or a move button within the mixed reality space (810). According to one embodiment, when the first electronic device (101) receives user input selecting the continuation icon or the move button, it may transmit the corresponding task to the second electronic device (820). According to one embodiment, the first electronic device (101) can transfer a task to the second electronic device (820) based on user input selecting the second electronic device (820) visible within the mixed reality space (810) in see-through mode. According to one embodiment, the first electronic device (101) can select a task by eye tracking and transfer the selected task to the second electronic device (820). According to one embodiment, the first electronic device (101) can directly set the target device through program and application-specific icons or set the target device through environment settings. According to one embodiment, the first electronic device (101) can automatically transfer a task to the second electronic device (820) when it detects that the user has removed the first electronic device (101) based on a preset for a continue icon.

[0222] In operation 1140, a first electronic device (101) according to one embodiment may transfer a task currently being executed to a second electronic device (820) that is already connected. Operation 1140 may be at least partially similar to operation 740 described with reference to FIG. 7.

[0223] In operation 1150, the first electronic device (101) according to one embodiment detects that it has been detached from the user and may turn off the power or change to sleep mode. Operation 1150 may be at least partially similar to operation 750 described with reference to FIG. 7.

[0224] In operation 1160, a second electronic device (820) according to one embodiment may continuously execute the selected program. Operation 1160 may be at least partially similar to operation 770 described with reference to FIG. 7.

[0225] FIG. 12 is an example diagram showing a wearable device (101) receiving user input according to one embodiment. In FIG. 12, the wearable device (101) may be referred to as "first electronic device (101)".

[0226] Referring to FIG. 12, a wearable device (101) according to one embodiment may display a mixed reality space (810) including augmented reality or virtual reality. According to one embodiment, the wearable device (101) may run one or more applications, software, or content in the mixed reality space (810). For example, the wearable device (101) may run an application related to document work, an application related to editing sound or video, or an application for playing content based on user input.

[0227] According to one embodiment, as described with reference to operation 1130 of FIG. 11, the wearable device (101) can set a continuing application or target device through a menu. According to one embodiment, when the wearable device (101) transmits a task to the second electronic device (820), it can receive input from the user to select the task to be transmitted directly. According to one embodiment, when the wearable device (101) transmits a task to the second electronic device (820), it can determine the task to be transmitted based on detecting a user gesture or the user's gaze. For example, the wearable device (101) can detect a user gesture in which a part of the application execution screen (1210) in the mixed reality space (810) is set as the starting point (1201) and the second electronic device (820) visible within the mixed reality space (810) in see-through mode is set as the ending point (1202). According to one embodiment, the wearable device (101) can transmit a task to a second electronic device (820) in response to a detected user gesture.

[0228] According to one embodiment, the wearable device (101) may provide a continue icon (1211) or a move button within the mixed reality space (810) when determining a task to be transmitted to the second electronic device (820). For example, the wearable device (101) may display a continue icon (1211) or a move button on a part of the application execution screen. According to one embodiment, when the wearable device (101) receives user input (1203) selecting a continue icon or a move button, it may transmit the corresponding task to the second electronic device (820). According to one embodiment, if there are multiple external electronic devices connected to the wearable device (101) through communication, the wearable device (101) may display multiple continue icons (1211) or move buttons.

[0229] FIG. 13 is an example of a layout in which an external electronic device displays a task received according to one embodiment. In FIG. 13, 1301 indicates a state in which a wearable device (101) displays a mixed reality space (810). In FIG. 13, 1302 indicates a state in which at least one external electronic device (820, 830), e.g., a second electronic device (820) and a third electronic device (830), executes a task received from the wearable device (101) and displays an execution screen. In FIG. 13, the wearable device (101) may be referred to as "the first electronic device (101)".

[0230] Referring to FIG. 13, in state 1301, the wearable device (101) displays a mixed reality space (810) and can run one or more applications, software, or content within the mixed reality space (810). For example, the wearable device (101) can run an application related to document work, an application related to editing sound or video, or an application for playing content based on user input.

[0231] According to one embodiment, the wearable device (101) can display an execution screen (811) of a first application, an execution screen (812) of a second application, and an execution screen (813) of a third application in a mixed reality space (810) based on user input. According to one embodiment, the wearable device (101) can detect that the user is taking off the wearable device (101) while displaying the execution screen (811) of the first application, the execution screen (812) of the second application, and the execution screen (813) of the third application, as described in operations 730 and 740 of FIG. 7. According to one embodiment, when the wearable device (101) detects taking off, it can transmit tasks regarding the first application, the second application, and the third application to the second electronic device (820) and the third electronic device (830).

[0232] According to one embodiment, when the wearable device (101) transmits a task to the second electronic device (820) and the third electronic device (830), the arrangement of execution screens that the wearable device (101) displays in the mixed reality space (810) can be made similar in the second electronic device (820) and the third electronic device (830). For example, in state 1301, the wearable device (101) can arrange the execution screen of the first application (811), the execution screen of the second application (812), and the execution screen of the third application (813) in order from the left. According to one embodiment, the wearable device (101) can transmit tasks and commands to the second electronic device (820) and the third electronic device (830) so that the second electronic device (820) and the third electronic device (830) arrange the execution screen of the first application (1310), the execution screen of the second application (1320), and the execution screen of the third application (1330) in order from left to right.

[0233] According to one embodiment, the wearable device (101) can determine the relative arrangement of the second electronic device (820) and the third electronic device (830) as seen within the mixed reality space (810) in see-through mode. For example, the wearable device (101) can determine that the second electronic device (820) as seen within the mixed reality space (810) in see-through mode is located to the left of the third electronic device (830). According to one embodiment, the wearable device (101) can transmit tasks and commands based on confirming that the second electronic device (820) shown in the mixed reality space (810) is located to the left of the third electronic device (830), so that the second electronic device (820) displays the execution screen (1310) of the first application and the execution screen (1320) of the second application from the left, and the third electronic device (830) positioned to the right of the second electronic device (820) displays the execution screen (1330) of the third application.

[0234] FIG. 14 is a flowchart illustrating the operation of a wearable device (101) according to an embodiment when it is in a low battery state.

[0235] The operations illustrated in FIG. 14 can be performed by instructions stored in memory (e.g., memory (130) of FIG. 1). For example, when the instructions are executed by a processor (e.g., processor (120) of FIG. 1), the wearable device (101) (e.g., electronic device (101) of FIG. 1) can perform the operations illustrated in FIG. 14. In FIG. 14, the wearable device (101) may be referred to as the "first electronic device (101)".

[0236] At least some of the operations shown in FIG. 14 may be omitted. At least some operations mentioned in the present disclosure with reference to other drawings may be additionally inserted before or after at least some of the operations shown in FIG. 14.

[0237] According to one embodiment, at least some of the operations illustrated in FIG. 14 may be performed sequentially. According to one embodiment, at least some of the operations illustrated in FIG. 14 may be performed in parallel (simultaneously). According to one embodiment, at least some of the operations illustrated in FIG. 14 may be performed with their order changed.

[0238] Hereinafter, with reference to FIG. 14, the operation of a wearable device (101) according to one embodiment in a case where it is not easy to transmit a task to an external electronic device is described. In FIG. 14, the wearable device (101) may be named as a first electronic device (101) and may be a head-mount device (HMD), a headgear electronic device, a glasses-type electronic device, a video see-through or visible see-through (VST) device, an extended reality (XR) device, a virtual reality (VR) device, and / or an augmented reality (AR) device. In FIG. 14, the external electronic device may be named as a second electronic device (820). The second electronic device (820) may be a mobile phone, a tablet PC, a desktop PC, a laptop device, or a TV.

[0239] In operation 1410, a second electronic device (820) may be communicated to a first electronic device (101) according to one embodiment. Operation 1410 may be at least partially similar to operation 710 described with reference to FIG. 7.

[0240] In operation 1420, software or content may be executed in the first electronic device (101) according to one embodiment. Operation 1410 may be at least partially similar to operation 720 described with reference to FIG. 7.

[0241] In operation 1430, a first electronic device (101) according to one embodiment may detect a low battery state. According to one embodiment, a low battery state may be a state in which the battery level (or SOC) of the battery included in the first electronic device (101) is lower than a specified threshold. According to one embodiment, if the first electronic device (101) is in a low battery state, it may consider it difficult to continuously perform a task and suggest to the user that the task be transferred to a second electronic device (820). According to one embodiment, in determining whether there is a low battery state, the first electronic device (101) may consider factors such as the expected usage time according to the running application, the current battery level (or SOC), or the temperature of the device.

[0242] In operation 1440, according to one embodiment, the first electronic device (101) may check whether to transfer a task to the second electronic device (820). According to one embodiment, the first electronic device (101) may display a notification regarding a low battery status and receive confirmation from the user regarding whether to transfer a task to the second electronic device (820). Operation 1440 may be at least partially similar to operation 1130 described with reference to FIG. 11.

[0243] In operation 1450, a first electronic device (101) according to one embodiment may transfer a task currently being executed to a second electronic device (820) that is already connected. Operation 1450 may be at least partially similar to operation 740 described with reference to FIG. 7.

[0244] In operation 1460, the first electronic device (101) according to one embodiment may switch to a mirroring mode. According to one embodiment, the first electronic device (101) may receive a screen displayed by the second electronic device (820) in the mirroring mode and display the received screen.

[0245] In operation 1470, the first electronic device (101) according to one embodiment may switch to a see-through mode. According to one embodiment, the first electronic device (101) may display a see-through image including real space in the see-through mode. According to one embodiment, the first electronic device (101) may display, in the see-through mode, a screen of a task that is received from the first electronic device (101) and executed by a second electronic device (820) placed in real space as a see-through image.

[0246] In operation 1480, an application program can be executed sequentially in the second electronic device (820) according to one embodiment. Operation 1480 may be similar in at least part to operation 770 described with reference to FIG. 7. According to one embodiment, when the second electronic device (820) executes a task to generate a screen, the first electronic device (101) receives the screen displayed by the second electronic device (820) in mirroring mode and can display the transmitted screen.

[0247] In operation 1490, an application program can be executed sequentially in a second electronic device (820) according to one embodiment. Operation 1490 may be similar in at least part to operation 770 described with reference to FIG. 7. According to one embodiment, when the second electronic device (820) executes a task and generates a screen, the first electronic device (101) can display the screen of the task received from the first electronic device (101) and executed by the second electronic device (820), which is placed in real space in see-through mode, as a see-through image.

[0248] FIG. 15 is a flowchart illustrating the operation of a wearable device (101) according to one embodiment when executing a task that requires high performance.

[0249] The operations illustrated in FIG. 15 may be performed by instructions stored in memory (e.g., memory (130) of FIG. 1). For example, instructions may cause a wearable device (101) (e.g., electronic device (101) of FIG. 1) to perform the operations illustrated in FIG. 15 when executed by a processor (e.g., processor (120) of FIG. 1). At least some of the operations illustrated in FIG. 15 may be omitted. At least some operations mentioned in the present disclosure with reference to other drawings may be additionally inserted before or after at least some of the operations illustrated in FIG. 15.

[0250] According to one embodiment, at least some of the operations illustrated in FIG. 15 may be performed sequentially. According to one embodiment, at least some of the operations illustrated in FIG. 15 may be performed in parallel (simultaneously). According to one embodiment, at least some of the operations illustrated in FIG. 15 may be performed with their order changed.

[0251] Hereinafter, with reference to FIG. 15, the operation of a wearable device (101) according to one embodiment when executing a task requiring high performance is described. In FIG. 15, the wearable device (101) may be named as a first electronic device (101) and may be a head-mount device (HMD), a headgear electronic device, a glasses-type electronic device, a video see-through or visible see-through (VST) device, an extended reality (XR) device, a virtual reality (VR) device, and / or an augmented reality (AR) device. In FIG. 15, the external electronic device may be named as a second electronic device (820). The second electronic device (820) may be a mobile phone, a tablet PC, a desktop PC, a laptop device, or a TV.

[0252] In operation 1510, a second electronic device (820) may be communicated to a first electronic device (101) according to one embodiment. Operation 1510 may be at least partially similar to operation 710 described with reference to FIG. 7.

[0253] In operation 1520, software or content may be executed on the first electronic device (101) according to one embodiment. Operation 1520 may be similar in at least part to operation 720 described with reference to FIG. 7.

[0254] In operation 1530, a high-performance computing task can be executed in the first electronic device (101) according to one embodiment. According to one embodiment, a high-performance computing task may mean a task that requires a large amount of computation beyond a specified threshold and is expected to take longer than a specified time to complete processing.

[0255] In operation 1540, when a high-performance computing task is performed on a second electronic device (820) connected to a first electronic device (101) according to one embodiment, the estimated time can be compared and output. According to one embodiment, when the first electronic device (101) performs a task including a high-performance computing task based on user input, it can calculate the estimated time when the task is performed on an external electronic device. For example, the second electronic device (820), which is an external electronic device, may be a high-performance desktop PC or a server. According to one embodiment, when the first electronic device (101) performs a high-performance computing task, it may display a guide for switching to the second electronic device (820), for example, the estimated simulation completion time upon switching. According to one embodiment, the first electronic device (101) may store capability-related information in advance, including the hardware specifications, battery level, or wireless LAN speed of the second electronic device (820). According to one embodiment, the first electronic device (101) can calculate the estimated time for a task when the second electronic device (820) performs a high-performance computing task based on pre-stored capabilities information of the second electronic device (820).

[0256] In operation 1550, a first electronic device (101) according to one embodiment can transfer a task to a second electronic device (820) and switch to a mirroring mode. According to one embodiment, the first electronic device (101) can continue to run the software including the high-performance computing task even after transferring the task to the second electronic device (820). According to one embodiment, when the second electronic device completes the high-performance computing task, the first electronic device (101) can receive the completed result data from the second electronic device (820) and update the task and screen of the software based on the received result data.

[0257] FIG. 16 is an example of a notification indicating the estimated time of work for a task requiring high performance. In FIG. 16, the wearable device (101) may be referred to as the "first electronic device (101)".

[0258] Referring to FIG. 16, a wearable device (101) according to one embodiment can output an estimated time when a high-performance computing task is performed on a second electronic device (820) connected to the wearable device (101), as described in operation 1540 of FIG. 15. For example, the wearable device (101) may display together a first notification (1610) indicating when the high-performance computing task is performed by the second electronic device (820) instead, and a second notification (1620) indicating when the high-performance computing task is performed directly by the wearable device (101). According to one embodiment, the wearable device (101) may include the first estimated time (1611) when the task is performed by the second electronic device (820) instead in the first notification (1610). According to one embodiment, the wearable device (101) may include a second estimated time (1621) in the second notification (1620) when the wearable device (101) directly performs a high-performance computing task.

[0259] FIG. 17 is an example of a mixed reality space (810) including augmented reality or virtual reality provided by a wearable device (101) according to one embodiment. FIG. 18 is an example of a notification indicating the execution status of a task. In FIG. 17 and FIG. 18, the wearable device (101) may be referred to as "the first electronic device (101)".

[0260] In FIG. 17, reference numeral 810 represents a mixed reality space (810) including augmented reality or virtual reality provided by a wearable device (101). According to one embodiment, the wearable device (101) may display an execution screen of at least one application in the mixed reality space (810). For example, the wearable device (101) may display an execution screen (811) of a first application, an execution screen (812) of a second application, and an execution screen (813) of a third application in the mixed reality space (810), but the present invention is not limited thereto.

[0261] Referring to FIG. 17, a wearable device (101) according to one embodiment may display a notification indicating the execution status of a task. For example, the wearable device (101) may apply a specified effect to the outline of a window representing the screen of the application when the task of the application is performed in an on-device manner, or when the screen of the application being performed on a second electronic device (820) is displayed in a mirrored manner, or when at least a part of the task of the application is processed by a second external device instead. For example, in the example illustrated in FIG. 17, the wearable device (101) is shown applying a dotted line effect to the outline of a window representing the execution screen (813) of a third application. According to one embodiment, the effect applied to the outline of the window may include various colors, dotted lines, or brightness.

[0262] Referring to FIG. 18, the wearable device (101) may display a notification indicating the execution status of a task in the form of an icon. For example, when the wearable device (101) displays the screen of an application being executed on a second electronic device (820) in a mirrored form, as in state 1801 of FIG. 18, the wearable device (101) may display a first icon (1811) on at least a part of the execution screen or window (1810). For example, when the task of the application is being executed in an on-device form, as in state 1802 of FIG. 18, the wearable device (101) may display a second icon (1812) on at least a part of the execution screen or window (1810). For example, the wearable device (101) may display a third icon (1813) on at least a part of the execution screen or window (1810) when at least a part of the task of the application is processed by a second external device instead, as in state 1803 of FIG. 18.

[0263] FIG. 19 is an example illustrating the operation of a wearable device (101) when the wearable device (101) of a user is detected to be removed. In FIG. 19, state 1901 may indicate a state in which the wearable device (101) performs a task and is connected to at least one external electronic device (820, 830, 840). In FIG. 19, state 1902 may indicate a state in which the wearable device (101) suggests a connection to at least one external electronic device (820, 830, 840) via an external display (1910) based on the wearable device (101) detecting the user to be removed. In FIG. 19, the wearable device (101) may be referred to as the "first electronic device (101)."

[0264] Referring to FIG. 19, a wearable device (101) according to one embodiment may transfer at least a portion of the tasks of an application that was running to at least one external electronic device (820, 830, 840) when it detects that a user has removed the wearable device (101). According to one embodiment, the at least one external electronic device (820, 830, 840) may be a device registered to a server with the same user account as the wearable device (101) and may be the nearest device connected to a network. According to one embodiment, the at least one external electronic device (820, 830, 840) may be a device among the devices placed around the wearable device (101) that first detects input from the user (e.g., touch, or mouse click). According to one embodiment, when the wearable device (101) selects a device to continue running the application, it may receive user input from an input device connected to the wearable device (101). For example, the wearable device (101) can run a document editing application, and the document editing application can provide a function for a user to edit a document while a keyboard is connected to the wearable device (101). In this case, the wearable device (101) can select another device connected to a keyboard when selecting the device to continue running the application.

[0265] According to one embodiment, as illustrated in FIG. 19, the wearable device (101) may include an external display (1910). In this case, when the wearable device (101) detects the user removing it, it may display a popup (1911) through the external display (1910) asking the user whether to continue running the application executed on the wearable device (101) on the detected external electronic device (820, 830, 840). According to one embodiment, the wearable device (101) may display a menu through the external display (1910) that allows selection of the device to continue running as well as the application to continue running.

[0266] FIG. 20 is a flowchart illustrating a method of operating a wearable device (101) according to one embodiment.

[0267] The operations illustrated in FIG. 20 can be performed by instructions stored in memory (e.g., memory (130) of FIG. 1). For example, when the instructions are executed by a processor (e.g., processor (120) of FIG. 1), the wearable device (101) (e.g., electronic device (101) of FIG. 1) can perform the operations illustrated in FIG. 20.

[0268] At least some of the operations shown in FIG. 20 may be omitted. At least some operations mentioned in the present disclosure with reference to other drawings may be additionally inserted before or after at least some of the operations shown in FIG. 20.

[0269] According to one embodiment, at least some of the operations illustrated in FIG. 20 may be performed sequentially. According to one embodiment, at least some of the operations illustrated in FIG. 20 may be performed in parallel (simultaneously). According to one embodiment, at least some of the operations illustrated in FIG. 20 may be performed with their order changed.

[0270] Hereinafter, a method of operating a wearable device (101) according to one embodiment will be described with reference to FIG. 20. In FIG. 20, the wearable device (101) may be named as a first electronic device (101) and may be a head-mount device (HMD), a headgear electronic device, a glasses-type electronic device, a video see-through or visible see-through (VST) device, an extended reality (XR) device, a virtual reality (VR) device, and / or an augmented reality (AR) device. In FIG. 20, the external electronic device may be a mobile phone, a tablet PC, a desktop PC, a laptop device, or a TV.

[0271] In operation 2010, a wearable device (101) according to one embodiment may determine whether there is a change in the state of the wearable device (101) using at least one sensor. Operation 2010 may be at least partially similar to operation 1430 described with reference to FIG. 14.

[0272] According to one embodiment, the operation of determining whether the state of the wearable device (101) has changed may include an operation of checking information related to the battery level or information detected from at least one sensor. According to one embodiment, the information related to the battery level may include low battery information or information related to power saving mode settings. According to one embodiment, the low battery information may be determined based at least partially on the expected usage time according to the running application and the current remaining battery capacity. According to one embodiment, the information detected from at least one sensor may include wear detection information of the wearable device (101). For example, the wear detection information may be determined based on information obtained through an eye-tracking camera, a strap fixing sensor, or a proximity sensor. According to one embodiment, the information detected from at least one sensor may include information related to the user's eye fatigue or dizziness. For example, the information related to eye fatigue or dizziness may be information determined using artificial intelligence based on information obtained through an eye-tracking camera.

[0273] In operation 2020, a wearable device (101) according to one embodiment may determine at least one task to be executed on at least one external electronic device (820, 830). Operation 2020 may be at least partially similar to operation 1130 described with reference to FIG. 11.

[0274] According to one embodiment, the task may be a task of an application that has been partially executed on a wearable device (101), or a task of an application that is currently running.

[0275] According to one embodiment, an external electronic device may configure and display a selection menu using at least some of the information related to at least one application running on a wearable device (101) and at least some of the information related to at least one external electronic device (820, 830). For example, the external electronic device may determine a task to be performed next based on user input to the selection menu.

[0276] According to one embodiment, the wearable device (101) may include an external display (1910), and when the user's removal is detected, a popup may be displayed through the external display (1910) asking the user whether to continue running an application executed on the wearable device (101) on a detected external electronic device. According to one embodiment, the wearable device (101) may display information displayed through the external display (1910) for a specified time, for example, a first time. According to one embodiment, the wearable device (101) may operate in a first low-power mode during the first time, and after the first time has elapsed, it may operate in a second low-power mode. According to one embodiment, the second low-power mode may be a mode with lower power consumption than the first low-power mode. According to one embodiment, user input for a selection menu may include at least one of air gesture input, touch input, or voice input.

[0277] According to one embodiment, the external electronic device may be another electronic device registered with the same account (e.g., a first user account) as the wearable device (101). According to one embodiment, the external electronic device may be a mobile phone, a tablet PC, a desktop PC, a laptop device, or a TV.

[0278] According to one embodiment, an external electronic device can be determined through an operation of grouping, via user input, one external electronic device existing in the actual environment displayed on the display and at least one application-related image generated and displayed on the display.

[0279] According to one embodiment, the user input may be an input including at least one air gesture input, gaze input, or voice input. According to one embodiment, the air gesture may include a drag and drop motion. According to one embodiment, the air gesture may be an input received from an auxiliary input device.

[0280] According to one embodiment, an external electronic device may be determined through an operation in which a wearable device (101) collects attribute information of at least one external electronic device (820, 830) and groups at least one application and at least one external electronic device (820, 830) using the attribute information of the external electronic device and at least one application information. According to one embodiment, the attribute information of the external electronic device may be obtained from an external account management server. According to one embodiment, the attribute information of the external electronic device may be obtained from each external electronic device via a local network. For example, the local network may include BLE, BT, or Wi-Fi. According to one embodiment, the attribute information of the external electronic device may include at least some of the application information installed on the external electronic device, information related to the available input means of the external electronic device, information related to the power of the external electronic device, and information on the current operating status of the external electronic device. According to one embodiment, the grouping operation may be performed using an AI function.

[0281] According to one embodiment, the external electronic device may be an electronic device that can be connected to the wearable device (101).

[0282] According to one embodiment, the external electronic device may be an electronic device registered with the same account (e.g., a first user account) or a public account as the wearable device (101).

[0283] According to one embodiment, the external electronic device may be an electronic device connected to the same network (AP) as the wearable device (101).

[0284] According to one embodiment, the external electronic device may be an electronic device capable of receiving P2P service signals with the wearable device (101). For example, the P2P-based service may be a BLE-based service or a UWB-based service.

[0285] In operation 2030, a wearable device (101) according to one embodiment may enable at least one external electronic device (820, 830) to execute at least a portion of a task by transmitting at least one task-related information and commands to at least one external electronic device (820, 830). Operation 2030 may be at least partially similar to operation 770 described with reference to FIG. 7.

[0286] According to one embodiment, the operation of transmitting task-related information to an external electronic device may be an operation performed directly through a direct communication connection with the external electronic device (e.g., D2D method).

[0287] According to one embodiment, the operation of transmitting task-related information to an external electronic device may be an operation performed indirectly through a connection with an external server.

[0288] According to one embodiment, task-related information may include information related to the current context of the application. According to one embodiment, context information may include at least some of file information used by the task and screen display information of the application. According to one embodiment, context information may include information related to the operation of a task currently being performed or information related to an operation scheduled to be performed by the task. According to one embodiment, information related to the operation of the task may include operation information for the task operation. According to one embodiment, the operation for the task operation may include operations related to artificial intelligence.

[0289] According to one embodiment, the task may include a content creation application or a content consumption application. For example, the content creation application may include at least one of a document creation application, a video and image editing application, and CAD. For example, the content consumption application may include at least one of a video player, a music player, an image viewer, or a text viewer.

[0290] According to various embodiments, the wearable device (101) of the present invention may display a UI / UX that allows the user to select playback by connecting the video player to a portion of the currently turned-on TV screen so that when the user removes the wearable device (101), the user can continue watching the video player that was being viewed in full screen on the current screen on the TV.

[0291] According to various embodiments, the wearable device (101) of the present invention may, when a low battery or overheating situation is detected while performing a video editing task while the device stands alone, allow some of its computational functions to be performed on an external electronic device such as a connectable portable terminal and / or a laptop.

[0292] According to various embodiments, the wearable device (101) of the present invention may display a list of transferable applications and an external electronic device to transfer and execute an application running on the wearable device (101) on the internal display screen of the wearable device (101) before turning off the power of the wearable device (101) in an extreme low battery situation where the battery level is very low, so that the user can select.

[0293] According to various embodiments, the wearable device (101) of the present invention may display an external electronic device that can transfer and execute an application running on the wearable device (101) and a list of transferable and executable applications on the external display (1910) screen of the wearable device (101) so that the user can select one.

[0294] According to various embodiments, the wearable device (101) of the present invention may display an external electronic device that can transfer and execute an application running on the wearable device (101) when the wearable device (101) is removed, and a list of transferable and executable applications on the display screen of the external electronic device so that the user can select.

[0295] According to various embodiments, the wearable device (101) of the present invention may collect information about an external electronic device when the battery level drops below a specified level and may perform a preparatory operation to perform a connected task when requested by the wearable device (101). For example, when a document application is connected to and executed on a laptop, the document application may be executed in the background of the laptop in advance, or the data stored may be uploaded in advance to a server accessible from the laptop by automatically saving the document application currently being worked on by the wearable device (101).

[0296] According to various embodiments, the wearable device (101) of the present invention can automatically assign an external electronic device capable of running a currently running application without user selection when the wearable device (101) is removed, activate the assigned external electronic device, and allow the external electronic device to continue running the application currently running. At this time, the wearable device (101) can check the compatibility of the external electronic device.

[0297] According to one embodiment, the wearable device (101) can check the locations of external electronic devices in advance in see-through mode and set an arrangement of screens to be continuously displayed on the external electronic devices based on an arrangement of execution screens of an application currently being displayed on the wearable device (101).

[0298] According to various embodiments, when the wearable device (101) of the present invention transmits and executes an application running on the wearable device (101) to an external electronic device, the external electronic device to which the input device connected to the wearable device (101) is connected can be determined as the transmission target.

[0299] According to various embodiments, the wearable device (101) of the present invention can transmit information related to the internet browsing to the TV and enable continuous use when the wearable device (101) is removed while browsing the internet with a wireless mouse and a wireless keyboard connected to the wearable device (101) and the only external electronic device present in the vicinity is a TV. At this time, the wearable device (101) can switch the connection of the keyboard and mouse, which were connected to the external electronic device, to the TV.

[0300] According to various embodiments, the wearable device (101) of the present invention can continuously execute an application currently running on the wearable device (101) on an external electronic device present on the see-through screen by selecting or / and moving the application using user input (e.g., gesture-drag, voice) in the see-through environment of the wearable device (101).

[0301] According to various embodiments, the wearable device (101) of the present invention may be configured to continuously execute only the application that was used immediately before detecting a change in the state of the wearable device (101) on the current wearable device (101) on another device.

[0302] According to various embodiments, the wearable device (101) of the present invention may suggest stopping the mirroring when heat is detected while mirroring and displaying a task executed by an external electronic device. According to one embodiment, the wearable device (101) may display a UI / UX that allows the user to select whether to mirror the function to another external electronic device or to execute it on the wearable device (101).

[0303] According to various embodiments, the wearable device (101) of the present invention may induce the user to choose whether to continue the application work through the display of a nearby external device (e.g., a watch, or a mobile phone) or to do so on any connected device when the wearable device (101) is removed without the application being terminated.

[0304] According to various embodiments, the wearable device (101) of the present invention may detect a change in the user's biosignal (e.g., information related to the user's fatigue level) and display a UI / UX that recommends running an application currently running on the wearable device (101) on another device. According to one embodiment, the user's biosignal may be obtained based on information obtained from a wearable device such as a watch or a ring, or from at least one sensor embedded in the wearable device.

[0305] FIG. 21 is a flowchart illustrating a method of driving an electronic device according to one embodiment. For example, FIG. 21 may be a flowchart illustrating the operation of a second electronic device (820) or a third electronic device (830) described with reference to FIG. 8.

[0306] The operations illustrated in FIG. 21 can be performed by instructions stored in memory (e.g., memory (130) of FIG. 1). For example, when the instructions are executed by a processor (e.g., processor (120) of FIG. 1), an electronic device (e.g., electronic device (101) of FIG. 1) can perform the operations illustrated in FIG. 21.

[0307] At least some of the operations shown in FIG. 21 may be omitted. At least some operations mentioned in the present disclosure with reference to other drawings may be additionally inserted before or after at least some of the operations shown in FIG. 21.

[0308] According to one embodiment, at least some of the operations illustrated in FIG. 21 may be performed sequentially. According to one embodiment, at least some of the operations illustrated in FIG. 21 may be performed in parallel (simultaneously). According to one embodiment, at least some of the operations illustrated in FIG. 21 may be performed with their order changed.

[0309] Hereinafter, a method of operating an electronic device (820, 830) according to one embodiment is described with reference to FIG. 21. In FIG. 21, the electronic device (820, 830) may be a mobile phone, a tablet PC, a desktop PC, a laptop device, or a TV. In FIG. 21, the wearable device (101) may be named as the first electronic device (101) and may be a head-mount device (HMD), a headgear electronic device (820, 830), a glasses-type electronic device (820, 830), a video see-through (VST) device, an extended reality (XR) device, a virtual reality (VR) device, and / or an augmented reality (AR) device.

[0310] In operation 2110, an electronic device (820, 830) according to one embodiment may receive information and commands related to at least one task being executed by the wearable device (101) from the wearable device (101).

[0311] In operation 2120, an electronic device (820, 830) according to one embodiment may display at least some of the information regarding at least one task.

[0312] In operation 2130, an electronic device (820, 830) according to one embodiment can select one task from at least one task in response to user input.

[0313] In operation 2140, an electronic device (820, 830) according to one embodiment can execute a selected task and display a screen related to the executed task.

[0314] A wearable device according to one embodiment of the present disclosure includes a processor and a memory for storing instructions, wherein the instructions, when executed by the processor, allow the wearable device to determine whether there is a change in the state of the wearable device using at least one sensor, determine at least one task to be executed by at least one external electronic device, and transmit information and instructions related to the at least one task to the at least one external device, thereby enabling the at least one external electronic device to subsequently execute at least a part of the determined at least one task.

[0315] When the above instructions are executed by the processor, the wearable device may obtain information regarding the battery level or information regarding the power saving mode setting as an operation to determine whether the state of the wearable device changes.

[0316] When the above instructions are executed by the processor, the wearable device may detect whether a user is wearing the wearable device by using at least one sensor as an operation to determine whether the state of the wearable device has changed.

[0317] When the above instructions are executed by the processor, the wearable device may acquire information related to the user's eye fatigue or dizziness using at least one sensor as an operation to determine whether the state of the wearable device changes.

[0318] The above processor can obtain information related to the user's eye fatigue or dizziness using artificial intelligence.

[0319] When the above instructions are executed by the processor, the wearable device may communicate directly with the at least one external electronic device as an operation of transmitting information related to the at least one task.

[0320] When the above commands are executed by the processor, the wearable device can transmit information related to the at least one task to the at least one external electronic device through a server as an operation of transmitting information related to the at least one task.

[0321] The information related to at least one task above includes context-related information of an application running on the wearable device, and the context-related information may include information related to the operation of a task currently being performed or information related to the operation of a task scheduled to be performed.

[0322] Information related to the operation of the above task may include computational information for the operation of the task and computational information related to artificial intelligence.

[0323] The above at least one task may include a first application group for creating content, and a second application group for displaying or playing content created by the first application group.

[0324] An electronic device according to one embodiment of the present disclosure includes a processor and a memory for storing instructions, wherein, when executed by the processor, the electronic device receives information and instructions related to at least one task being executed by the wearable device from the wearable device, displays at least some of the information related to the at least one task, selects one task among the at least one task in response to user input, executes the selected task, and displays a screen related to the executed task.

[0325] When the above instructions are executed by the processor, the electronic device may display a selection menu based on information related to the received at least one task.

[0326] The above-mentioned wearable device and the above-mentioned electronic device may be devices registered on a server through a first user account.

[0327] The above-mentioned wearable device and the above-mentioned electronic device may be devices connected to a first short-range network.

[0328] The above-mentioned wearable device and the above-mentioned electronic device can communicate directly with each other through short-range communication.

[0329] A driving method for a wearable device according to one embodiment of the present disclosure may include an operation of determining whether the state of the wearable device changes using at least one sensor, an operation of determining at least one task to be executed by at least one external electronic device, and an operation of transmitting information and commands related to the at least one task to the at least one external device so that the at least one external electronic device subsequently executes at least a part of the determined at least one task.

[0330] The operation of determining whether the state of the above-mentioned wearable device has changed may include an operation of obtaining information regarding the battery level or information regarding the power saving mode setting.

[0331] The operation of determining whether there is a change in the state of the wearable device may include the operation of detecting whether a user is wearing the wearable device using at least one sensor.

[0332] The operation of determining whether there is a change in the state of the above-mentioned wearable device may include the operation of obtaining information related to the user's eye fatigue or dizziness using at least one sensor.

[0333] It may further include an operation to obtain information related to the user's eye fatigue or dizziness using artificial intelligence.

[0334] For one or more embodiments, at least one of the components described in one or more of the prior art drawings may be configured to perform one or more operations, techniques, processes and / or methods as described in the present disclosure. For example, a processor (e.g., a baseband processor) described in the present disclosure in relation to one or more of the prior art drawings may be configured to operate according to one or more examples described in the present disclosure. As another example, circuits associated with user equipment (UE), a base station, a network element, etc., as described above in relation to one or more of the prior art drawings may be configured to operate according to one or more examples described herein.

[0335] Any of the embodiments described above may be combined with any other embodiment (or combination of embodiments) unless otherwise explicitly stated. The foregoing description of one or more embodiments is for illustrative and explanatory purposes only, and is not intended to limit or exhaust the scope of the embodiments in the exact form disclosed. Modifications and variations are possible in light of the foregoing teachings or may be obtained from the practice of various embodiments.

Claims

1. In a wearable device, One or more processors; and It includes memory for storing instructions, When the above instructions are executed alone or together by the one or more processors, the wearable device: Determining whether the state of the wearable device changes using at least one sensor, and Determining at least one task to be executed on at least one external electronic device, By transmitting information and commands related to the at least one task to the at least one external electronic device, the at least one external electronic device subsequently executes at least a portion of the determined at least one task. Wearable device.

2. In Paragraph 1, When the above instructions are executed alone or together by the one or more processors, the wearable device determines whether the state of the wearable device changes, as an operation. To obtain information regarding battery level or power saving mode settings, Wearable device.

3. In Paragraph 1, When the above instructions are executed alone or together by the one or more processors, the wearable device determines whether the state of the wearable device changes, as an operation. Using the above at least one sensor to detect whether a user is wearing the wearable device, Wearable device.

4. In Paragraph 1, When the above instructions are executed alone or together by the one or more processors, the wearable device determines whether the state of the wearable device changes, as an operation. Using the above-mentioned at least one sensor to obtain information related to the user's eye fatigue or dizziness, Wearable device.

5. In Paragraph 4, Acquiring information related to the user's eye strain or dizziness is performed using artificial intelligence, Wearable device.

6. In Paragraph 1, When the above instructions are executed alone or together by the one or more processors, the wearable device transmits information related to the at least one task as an operation, To communicate directly with at least one external electronic device, Wearable device.

7. In Paragraph 1, When the above instructions are executed alone or together by the one or more processors, the wearable device transmits information related to the at least one task as an operation, Information related to the at least one task is transmitted to the at least one external electronic device through the server, Wearable device.

8. In Paragraph 1, Information related to at least one of the above tasks includes context-related information of an application running on the wearable device, and The above context-related information includes information related to the operation of a task currently being performed or information related to the operation of a task scheduled to be performed. Wearable device.

9. In Paragraph 8, Information related to the operation of the above task includes computational information for the operation of the task and computational information related to artificial intelligence. Wearable device.

10. In Paragraph 1, The above at least one task includes a first application group for generating content and a second application group for displaying or playing the content generated by the first application group. Wearable device.

11. A method performed by a wearable device, An operation to determine whether there is a change in the state of a wearable device using at least one sensor; An action that determines at least one task to be executed on at least one external electronic device; and The operation of transmitting information and commands related to the at least one task to the at least one external electronic device, thereby causing the at least one external electronic device to subsequently execute at least a portion of the determined at least one task. method.

12. In Paragraph 11, The operation of determining whether the state of the above-mentioned wearable device changes includes the operation of obtaining information regarding the battery level or information regarding the power saving mode setting. method.

13. In Paragraph 11, The operation of determining whether there is a change in the state of the wearable device includes the operation of detecting whether a user is wearing the wearable device using at least one sensor. method.

14. In Paragraph 11, The above method includes an operation of detecting the detachment of the wearable device from the user, and The operation of transmitting information and commands related to the above at least one task to the above at least one external electronic device is performed based on the detection of the above detachment. method.

15. In Paragraph 11, The above method is, An operation to determine that there is no second external electronic device preconnected to the above-mentioned wearable device; An operation to perform a connection with a connectable second electronic device based on the determination that the second external electronic device is not previously connected; and The operation of transmitting information and commands related to at least one task to the connected second electronic device, method.

Images