Wearable device and driving method therefor

Abstract

Embodiments of the present disclosure relate to a wearable device and a driving method therefor, the wearable device comprising at least one sensor, at least one infrared camera, at least one imaging camera, a display, a processor, and a memory for storing instructions, wherein, when executed by the processor, the instructions can instruct the wearable device to: identify that the illuminance acquired using the at least one sensor is less than a first reference value; acquire a first image by using the at least one infrared camera; display, on the display, a second image including at least a part of the first image; select at least one region of the second image on the basis of a first user input; and add a visual effect to the region selected in the second image.

Description

Wearable device and method of operating the same

[0001] The embodiments of the present disclosure relate to a wearable device and a method of operating the same.

[0002] With recent technological advancements, electronic devices are moving away from uniform rectangular shapes and gradually transforming into various shapes. For example, electronic devices may include wearable devices that can be worn on a part of the body. Wearable devices may include head-mounted displays (HMDs) that can be worn on a user's head. Wearable devices may be referred to as head-mounted devices (HMDs), headgear electronic devices, glasses-type electronic devices, video see-through (VST) or visible see-through (VST) devices, extended reality (XR) devices, virtual reality (VR) devices, and / or augmented reality (AR) devices.

[0003] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art in relation to the present disclosure.

[0004] A wearable device can provide augmented reality, virtual reality, or mixed reality (a combination of augmented reality and virtual reality) to a user wearing the wearable device. For example, the wearable device can perform a pass-through function that displays images acquired through a camera by overlaying them on at least a portion of the screen of a display.

[0005] When a wearable device displays see-through video using a pass-through function, it may not be able to accurately capture external objects through the camera in low-light environments with insufficient light or in bad weather environments such as fog or heavy rain. If the wearable device fails to accurately capture external objects through the camera, it may not be able to accurately convey and display the color or shape of the external objects to the user.

[0006] The embodiments of the present disclosure can provide a wearable device and a method of operating the same that can increase user convenience and provide an enhanced user experience by using an infrared camera to determine the characteristics of an external object corresponding to the user's gaze and applying visual effects to a portion of the space corresponding to the user's gaze based on the determined characteristics of the external object.

[0007] The technical problems to be solved in this disclosure are not limited to those mentioned above, and other unmentioned technical problems will be clearly understood by those skilled in the art from the description below.

[0008] A wearable device according to one embodiment of the present disclosure comprises at least one sensor, at least one infrared camera, at least one shooting camera, a display, a processor, and a memory for storing instructions, wherein when the instructions are executed by the processor, the wearable device may: determine that an illuminance obtained using the at least one sensor is less than a first reference value, obtain a first image using the at least one infrared camera, display a second image including at least a portion of the first image through the display, select at least a portion of the second image based on a first user input, and add visual effects to the selected portion of the second image.

[0009] A method for driving a wearable device according to one embodiment of the present disclosure may include: confirming that an illuminance obtained using at least one sensor is smaller than a first reference value; acquiring a first image using at least one infrared camera; displaying a second image that includes at least a portion of the first image through a display; selecting at least a portion of the second image based on a first user input; and adding a visual effect to the selected portion of the second image.

[0010] According to the embodiments of the present disclosure, by applying visual effects to a portion of a space corresponding to the user's line of sight in a low-light environment with insufficient light or in bad weather environments such as fog or heavy rain, user convenience can be enhanced and an improved user experience can be provided.

[0011] In addition, various effects that can be identified directly or indirectly through this document may be provided.

[0012] Other aspects, features, and advantages according to specific embodiments of the present disclosure will become more apparent from the accompanying drawings and description.

[0013] FIG. 1 is a block diagram of an electronic device in a network environment according to one embodiment.

[0014] FIG. 2a illustrates an example of a perspective view of a wearable device.

[0015] FIG. 2b illustrates an example of one or more hardware components placed within a wearable device.

[0016] FIGS. 3a and 3b illustrate an example of the appearance of a wearable device.

[0017] FIG. 4 illustrates an example of a block diagram of a wearable device.

[0018] Figure 5 shows an example of a block diagram of a wearable device for displaying an image in a virtual space.

[0019] Figure 6 illustrates an example of a structure of multiple layers.

[0020] FIG. 7 is a flowchart illustrating the operation of a wearable device according to one embodiment.

[0021] FIG. 8 is a conceptual diagram illustrating a visual effect provided by a wearable device according to one embodiment.

[0022] FIG. 9 is a flowchart illustrating the operation of a wearable device providing a light source effect according to one embodiment.

[0023] FIG. 10 is a conceptual diagram illustrating a light source effect provided by a wearable device according to one embodiment.

[0024] FIG. 11 is a flowchart illustrating the operation of a wearable device according to one embodiment applying a visual effect according to a light source object.

[0025] FIG. 12 is a conceptual diagram illustrating user interaction provided by a wearable device according to one embodiment.

[0026] FIG. 13 is a flowchart illustrating the operation of a wearable device according to one embodiment in a low-light environment.

[0027] FIG. 14 is a flowchart illustrating the operation of a wearable device according to one embodiment based on the time the user's gaze rests.

[0028] FIG. 15 is a flowchart illustrating the operation of a wearable device according to one embodiment based on the movement of a user's gaze.

[0029] Figure 16 is a flowchart illustrating the visual effects provided by a wearable device according to the movement of the user's gaze.

[0030] FIG. 17 is a flowchart illustrating the operation of a wearable device according to one embodiment.

[0031] FIG. 18 is a block diagram illustrating a generative artificial intelligence system according to one embodiment.

[0032] Each of the embodiments described with reference to the drawings of the present disclosure may be configured independently as a single embodiment. For example, the embodiment of FIG. 1 and the embodiment of FIG. 2a may each be configured independently of each other. Each of the embodiments described with reference to the drawings of the present disclosure may operate independently as a single embodiment. For example, the embodiment of FIG. 1 and the embodiment of FIG. 2a may each operate independently of each other.

[0033] At least two of the embodiments described with reference to the drawings of the present disclosure may be combined. For example, at least a part of the embodiment of FIG. 1 and at least a part of the embodiment of FIG. 2a may be combined with each other. At least two of the embodiments described with reference to the drawings of the present disclosure may be combined and operated. For example, at least a part of the embodiment of FIG. 1 and at least a part of the embodiment of FIG. 2a may be combined and operated with each other.

[0034] When at least two of the embodiments described with reference to the drawings of the present disclosure are combined, at least some of the configurations and / or at least some of the operations included in each embodiment may be omitted. For example, when the embodiment of FIG. 1 and the embodiment of FIG. 2a are combined, at least some of the configurations and / or at least some of the operations included in the embodiment of FIG. 1 may be omitted, and at least some of the configurations and / or at least some of the operations included in the embodiment of FIG. 2a may be omitted.

[0035] FIG. 1 is a block diagram of an electronic device (101) in a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or may communicate with at least one of an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through a server (108). According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (160), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)).

[0036] The processor (120) can control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., a program (140)), and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use lower power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.

[0037] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0038] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, software (e.g., program (140)) and input data or output data for related commands. The memory (130) may include volatile memory (132) or non-volatile memory (134).

[0039] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0040] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0041] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0042] The display module (160) can visually provide information to an external (e.g., user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0043] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (101).

[0044] The sensor module (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0045] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0046] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0047] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that can be perceived by the user through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0048] The camera module (180) can capture still images and video. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0049] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0050] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0051] The communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).

[0052] The wireless communication module (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), external electronic device (e.g., electronic device (104)), or network system (e.g., second network (199)). According to one embodiment, the wireless communication module (192) may support a Peak data rate (e.g., 20 Gbps or more) for eMBB realization, loss coverage (e.g., 164 dB or less) for mMTC realization, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for URLLC realization.

[0053] An antenna module (197) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197).

[0054] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0055] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0056] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (104) or the server (108) may be included within a second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0057] The electronic device according to the various embodiments disclosed in this disclosure may be a device of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. The electronic device according to the embodiments of this disclosure is not limited to the devices described above.

[0058] The various embodiments of the present disclosure and the terms used therein are not intended to limit the technical features described in the present disclosure to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In the present disclosure, phrases such as “A or B,” “at least one of A and B,” “at least one of A or B,” “A, B or C,” “at least one of A, B and C,” and “at least one of A, B, or C” each may include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as “first,” “second,” or “first” or “second” may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as "coupled" or "connected" to another (e.g., 2nd) component, with or without the terms "functionally" or "communicationly," it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0059] The term “module” as used in various embodiments of the present disclosure may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0060] Various embodiments of the present disclosure may be implemented as software (e.g., program (140)) comprising one or more instructions stored in a storage medium (e.g., internal memory (136) or external memory (138)) readable by a machine (e.g., electronic device (101)). For example, a processor (e.g., processor (120)) of the machine (e.g., electronic device (101)) may call at least one of the one or more instructions stored in the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0061] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or an application store (e.g., Play Store). TM It can be distributed online (e.g., downloaded or uploaded) through ) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0062] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0063] In embodiments of the present disclosure, an electronic device for displaying an image in a virtual space (e.g., the electronic device (101) of FIG. 1) may be a wearable device. The wearable device (101) may include a head-mounted display (HMD) that is wearable on a user's head. The wearable device (101) may be referred to as a head-mount device (HMD), a headgear electronic device, a glasses-type electronic device, a video see-through (VST) or visible see-through (VST) device, an extended reality (XR) device, a virtual reality (VR) device, and / or an augmented reality (AR) device. Although the external appearance of the wearable device (101) having the form of glasses is illustrated, the embodiments are not limited thereto. An example of a hardware configuration included within the wearable device (101) is described exemplarily with reference to FIG. 4. An example of the structure of a wearable device (101) that can be worn on the head of a user (110) is described with reference to FIG. 2a, FIG. 2b, FIG. 3a and / or FIG. 3b. The wearable device (101) may be referred to as an electronic device. For example, the electronic device may be combined with an accessory (e.g., a strap) to be attached to the user's head to form an HMD.

[0064] According to one embodiment, a wearable device (101) can perform functions related to augmented reality (AR) and / or mixed reality (MR). For example, while a user (110) is wearing the wearable device (101), the wearable device (101) may include at least one lens positioned adjacent to the user's (110) eyes. The wearable device (101) may combine light emitted from a display of the wearable device (101) with ambient light passing through the lens. The display area of ​​the display may be formed within the lens through which the ambient light passes. Because the wearable device (101) combines the ambient light and the light emitted from the display, the user (110) can see a mixed image of a real object perceived by the ambient light and a virtual object formed by the light emitted from the display. The aforementioned augmented reality, mixed reality, and / or virtual reality may be referred to as extended reality (XR).

[0065] According to one embodiment, a wearable device (101) can perform functions related to VST (video see-through or visible see-through) and / or virtual reality (VR). For example, while a user (110) is wearing the wearable device (101), the wearable device (101) may include a housing that covers the user's (110) eyes. The wearable device (101) may include a display disposed on a first surface of the housing facing the eyes while in the state. The wearable device (101) may include a camera disposed on a second surface opposite to the first surface. Using the camera, the wearable device (101) may acquire an image and / or video representing ambient light. A wearable device (101) may output the image and / or video within a display placed on the first surface, thereby allowing a user (110) to perceive the ambient light through the display. A displaying area (or displaying region) (or active area or active region) of the display placed on the first surface may be formed by one or more pixels included in the display. The wearable device (101) may composite a virtual object with the image and / or video output through the display, thereby allowing the user (110) to perceive the virtual object along with the actual object perceived by the ambient light.

[0066] According to one embodiment, a wearable device (101) can identify or recognize the position or location and / or direction or orientation of the wearable device (101) based on an image (and / or video) obtained or acquired using a camera. The wearable device (101) can obtain information about the external space using one or more cameras and / or one or more sensors. The information may include a geographic location of the external space (e.g., GPS (global positioning system) coordinates) identified by one or more sensors. The information may include an image and / or video of the external space identified by one or more cameras. The wearable device (101) can perform object recognition on the image and / or video to identify external objects contained in the external space from the image and / or video.

[0067] Hereinafter, with reference to FIGS. 2a, FIGS. 2b, FIGS. 3a, FIGS. 3b, and FIGS. 4, an example of a hardware configuration of a wearable device (101) is described.

[0068] FIG. 2a illustrates an example of a perspective view of a wearable device. FIG. 2b illustrates an example of one or more hardware components disposed within the wearable device. According to one embodiment, the wearable device (101) may have the form of glasses that are wearable on a part of a user's body (e.g., head). The wearable device (101) of FIG. 2a and FIG. 2b may be an example of the wearable device (101) of FIG. 1. The wearable device (101) may include a head-mounted display (HMD). For example, the housing of the wearable device (101) may include a flexible material such as rubber and / or silicone that has a shape that adheres to a part of the user's head (e.g., a part of the face covering both eyes). For example, the housing of the wearable device (101) may include one or more straps that can be twined around the user's head, and / or one or more temples that can be attached to the ears of the head.

[0069] Referring to FIG. 2a, a wearable device (101) according to one embodiment may include at least one display (250) and a frame (200) supporting at least one display (250).

[0070] According to one embodiment, a wearable device (101) may be worn on a part of a user's body. The wearable device (101) may provide augmented reality (AR), virtual reality (VR), or mixed reality (MR) that combines augmented reality and virtual reality to a user wearing the wearable device (101). For example, the wearable device (101) may display a virtual reality image provided by at least one optical device (282, 284) of FIG. 2b on at least one display (250) in response to a specified gesture of the user obtained through the motion recognition camera (260-2, 260-3) of FIG. 2b.

[0071] According to one embodiment, at least one display (250) can provide visual information to a user. For example, at least one display (250) may include a transparent or translucent lens. At least one display (250) may include a first display (250-1) and / or a second display (250-2) spaced apart from the first display (250-1). For example, the first display (250-1) and the second display (250-2) may be positioned at locations corresponding to the user's left eye and right eye, respectively.

[0072] Referring to FIG. 2b, at least one display (250) may provide visual information transmitted from external light to a user through a lens included in at least one display (250) and other visual information distinct from said visual information. The lens may be formed based on at least one of a Fresnel lens, a pancake lens, or a multi-channel lens. For example, at least one display (250) may include a first surface (231) and a second surface (232) opposite to the first surface (231). A display area may be formed on the second surface (232) of at least one display (250). When a user wears the wearable device (101), external light may be transmitted to the user by being incident on the first surface (231) and transmitted through the second surface (232). As another example, at least one display (250) can display an augmented reality image combined with a virtual reality image provided by at least one optical device (282, 284) on a real image transmitted through external light in a display area formed on the second surface (232).

[0073] In one embodiment, at least one display (250) may include at least one waveguide (233, 234) that diffracts light emitted from at least one optical device (282, 284) and transmits it to a user. At least one waveguide (233, 234) may be formed based on at least one of glass, plastic, or polymer. A nano pattern may be formed on the exterior or at least a portion of the interior of at least one waveguide (233, 234). The nano pattern may be formed based on a polygonal and / or curved grating structure. Light incident on one end of at least one waveguide (233, 234) may be propagated to the other end of at least one waveguide (233, 234) by the nano pattern. At least one waveguide (233, 234) may include at least one diffractive element (e.g., DOE (diffractive optical element), HOE (holographic optical element)) and at least one reflective element (e.g., a reflective mirror). For example, at least one waveguide (233, 234) may be placed within a wearable device (101) to guide a screen displayed by at least one display (250) to the user's eye. For example, the screen may be transmitted to the user's eye based on total internal reflection (TIR) ​​occurring within at least one waveguide (233, 234).

[0074] A wearable device (101) can analyze an object included in a real-world image collected through a shooting camera (260-4), combine a virtual object corresponding to an object among the analyzed objects that is the target of augmented reality provision, and display it on at least one display (250). The virtual object may include at least one of text and an image regarding various information related to the object included in the real-world image. The wearable device (101) can analyze the object based on a multi-camera such as a stereo camera. For the object analysis, the wearable device (101) can perform spatial recognition (e.g., SLAM (simultaneous localization and mapping)) using a multi-camera and / or time-of-flight (ToF). A user wearing the wearable device (101) can view the image displayed on at least one display (250).

[0075] According to one embodiment, the frame (200) may be formed as a physical structure that allows the wearable device (101) to be worn on the user's body. According to one embodiment, the frame (200) may be configured so that when the user wears the wearable device (101), the first display (250-1) and the second display (250-2) can be positioned corresponding to the user's left and right eyes. The frame (200) may support at least one display (250). For example, the frame (200) may support the first display (250-1) and the second display (250-2) so that they are positioned corresponding to the user's left and right eyes.

[0076] Referring to FIG. 2a, the frame (200) may include an area (220) in which at least a portion of the frame contacts a part of the user's body when the user wears the wearable device (101). For example, the area (220) of the frame (200) in contact with a part of the user's body may include an area in contact with a part of the user's nose, a part of the user's ear, and a part of the side of the user's face that the wearable device (101) contacts. According to one embodiment, the frame (200) may include a nose pad (210) that contacts a part of the user's body. When the wearable device (101) is worn by the user, the nose pad (210) may contact a part of the user's nose. The frame (200) may include a first temple (204) and a second temple (205) that contact a different part of the user's body distinct from the part of the user's body.

[0077] For example, the frame (200) may include a first rim (201) covering at least a portion of a first display (250-1), a second rim (202) covering at least a portion of a second display (250-2), a bridge (203) positioned between the first rim (201) and the second rim (202), a first pad (211) positioned along a portion of the edge of the first rim (201) from one end of the bridge (203), a second pad (212) positioned along a portion of the edge of the second rim (202) from the other end of the bridge (203), a first temple (204) extending from the first rim (201) and fixed to a portion of the wearer's ear, and a second temple (205) extending from the second rim (202) and fixed to a portion of the ear opposite to the first. The first pad (211) and the second pad (212) may come into contact with a part of the user's nose, and the first temple (204) and the second temple (205) may come into contact with a part of the user's face and a part of the ear. The temples (204, 205) may be rotatably connected to the rim through the hinge units (206, 207) of FIG. 2B. The first temple (204) may be rotatably connected to the first rim (201) through a first hinge unit (206) positioned between the first rim (201) and the first temple (204). The second temple (205) may be rotatably connected to the second rim (202) through a second hinge unit (207) positioned between the second rim (202) and the second temple (205). According to one embodiment, a wearable device (101) can identify an external object touching the frame (200) (e.g., a user's fingertip) and / or a gesture performed by said external object by using a touch sensor, a grip sensor, and / or a proximity sensor formed on at least a portion of the surface of the frame (200).

[0078] According to one embodiment, the wearable device (101) may include hardware that performs various functions (e.g., hardware to be described later based on the block diagram of FIG. 4). For example, the hardware may include a battery module (270), an antenna module (275), at least one optical device (282, 284), speakers (e.g., speakers (255-1, 255-2)), a microphone (e.g., microphones (265-1, 265-2, 265-3)), a light-emitting module (not shown), and / or a PCB (printed circuit board) (290) (e.g., a printed circuit board). The various hardware may be placed within a frame (200).

[0079] According to one embodiment, a microphone (e.g., microphones (265-1, 265-2, 265-3)) of a wearable device (101) is positioned on at least a portion of a frame (200) to acquire a sound signal. A first microphone (265-1) positioned on a bridge (203), a second microphone (265-2) positioned on a second rim (202), and a third microphone (265-3) positioned on a first rim (201) are shown in FIG. 2b, but the number and position of the microphones (265) are not limited to the embodiment of FIG. 2b. If there are two or more microphones (265) included in the wearable device (101), the wearable device (101) can identify the direction of the sound signal by using a plurality of microphones positioned on different portions of the frame (200).

[0080] According to one embodiment, at least one optical device (282, 284) may project a virtual object onto at least one display (250) to provide various image information to a user. For example, at least one optical device (282, 284) may be a projector. At least one optical device (282, 284) may be disposed adjacent to at least one display (250) or included within at least one display (250) as part of at least one display (250). According to one embodiment, a wearable device (101) may include a first optical device (282) corresponding to a first display (250-1) and a second optical device (284) corresponding to a second display (250-2). For example, at least one optical device (282, 284) may include a first optical device (282) positioned at the edge of a first display (250-1) and a second optical device (284) positioned at the edge of a second display (250-2). The first optical device (282) may transmit light to a first waveguide (233) positioned on the first display (250-1), and the second optical device (284) may transmit light to a second waveguide (234) positioned on the second display (250-2).

[0081] In one embodiment, the camera (260) may include a shooting camera (260-4), an eye tracking camera (ET CAM) (260-1), and / or a motion recognition camera (260-2, 206-3). The shooting camera (260-4), the eye tracking camera (260-1), and the motion recognition camera (260-2, 260-3) may be positioned at different locations on the frame (200) and may perform different functions. The eye tracking camera (260-1) may output data indicating the position of the eyes or the gaze of a user wearing the wearable device (101). For example, the wearable device (101) may detect the gaze from an image containing the user's pupils obtained through the eye tracking camera (260-1). A wearable device (101) can identify an object focused by a user (e.g., a real object, and / or a virtual object) by using the user's gaze obtained through an eye-tracking camera (260-1). The wearable device (101), having identified the focused object, can perform a function (e.g., gaze interaction) for interaction between the user and the focused object. The wearable device (101) can represent a portion corresponding to the eyes of an avatar representing the user in a virtual space by using the user's gaze obtained through an eye-tracking camera (260-1). The wearable device (101) can render an image (or screen) displayed on at least one display (250) based on the position of the user's eyes. For example, the visual quality of a first area related to the gaze within the image and the visual quality of a second area distinguished from the first area (e.g., resolution, brightness, saturation, grayscale, PPI) may differ from each other. A wearable device (101) can obtain an image having a visual quality of a first area that matches the user's gaze and a visual quality of a second area by using foveated rendering.For example, if the wearable device (101) supports an iris recognition function, user authentication can be performed based on iris information obtained using an eye-tracking camera (260-1). An example in which the eye-tracking camera (260-1) is positioned toward the user's right eye is shown in FIG. 2b, but the embodiment is not limited thereto, and the eye-tracking camera (260-1) may be positioned solely toward the user's left eye or toward both eyes.

[0082] In one embodiment, the camera (260-4) can capture a real image or background to be matched with a virtual image in order to implement augmented reality or mixed reality content. The camera (260-4) can be used to acquire high-resolution images based on HR (high resolution) or PV (photo video). The camera (260-4) can capture an image of a specific object located at the position viewed by the user and provide the image to at least one display (250). The at least one display (250) can display a single image in which information regarding a real image or background including the image of the specific object acquired using the camera (260-4) and a virtual image provided through at least one optical device (282, 284) are superimposed. The wearable device (101) can compensate for depth information (e.g., the distance between the wearable device (101) and an external object acquired through a depth sensor) using the image acquired through the camera (260-4). The wearable device (101) can perform object recognition through an image acquired using a shooting camera (260-4). The wearable device (101) can perform a function of focusing on an object (or subject) in an image (e.g., auto focus) and / or an optical image stabilization (OIS) function (e.g., anti-shake function) using the shooting camera (260-4). The wearable device (101) can perform a pass-through function to superimpose an image acquired through the shooting camera (260-4) onto at least a portion of a screen representing a virtual space while displaying the screen representing a virtual space on at least one display (250). In one embodiment, the shooting camera (260-4) may be placed on a bridge (203) positioned between a first rim (201) and a second rim (202).

[0083] The eye tracking camera (260-1) can achieve more realistic augmented reality by tracking the gaze of a user wearing the wearable device (101), thereby matching the user's gaze with visual information provided to at least one display (250). For example, when the user looks straight ahead, the wearable device (101) can naturally display environmental information related to the user's front on at least one display (250) at the location where the user is situated. The eye tracking camera (260-1) may be configured to capture an image of the user's pupil to determine the user's gaze. For example, the eye tracking camera (260-1) may receive a gaze detection light reflected from the user's pupil and track the user's gaze based on the position and movement of the received gaze detection light. In one embodiment, the eye tracking camera (260-1) may be positioned at locations corresponding to the user's left and right eyes. For example, the eye-tracking camera (260-1) may be positioned within the first rim (201) and / or the second rim (202) to face the direction in which the user wearing the wearable device (101) is located.

[0084] The motion recognition camera (260-2, 260-3) can provide a specific event to a screen provided on at least one display (250) by recognizing the movement of the user's entire body or part thereof, such as the user's torso, hands, or face. The motion recognition camera (260-2, 260-3) can recognize the user's gesture, acquire a signal corresponding to the gesture, and provide a display corresponding to the signal to at least one display (250). The processor can identify the signal corresponding to the gesture and, based on the identification, perform a designated function. The motion recognition camera (260-2, 260-3) can be used to perform spatial recognition functions using SLAM and / or depth maps for a 6-degrees-of-freedom pose (6 dof pose). The processor can use the motion recognition camera (260-2, 260-3) to perform gesture recognition functions and / or object tracking functions. In one embodiment, a motion recognition camera (260-2, 260-3) may be placed on the first rim (201) and / or the second rim (202).

[0085] The camera (260) included in the wearable device (101) is not limited to the eye-tracking camera (260-1) and motion recognition camera (260-2, 260-3) described above. For example, the wearable device (101) can identify external objects included within the FoV by using a camera positioned toward the user's FoV. The identification of external objects by the wearable device (101) can be performed based on a sensor for identifying the distance between the wearable device (101) and the external object, such as a depth sensor and / or a time of flight (ToF) sensor. The camera (260) positioned toward the FoV may support an autofocus function and / or an optical image stabilization (OIS) function. For example, the wearable device (101) may include a camera (260) (e.g., a face tracking camera) positioned toward the face to acquire an image including the face of a user wearing the wearable device (101).

[0086] Although not illustrated, according to one embodiment, the wearable device (101) may further include a light source (e.g., LED) that emits light toward a subject (e.g., user's eye, face, and / or an object outside the FoV) being photographed using a camera (260). The light source may include an LED of infrared wavelength. The light source may be placed in at least one of the frame (200) and hinge units (206, 207).

[0087] According to one embodiment, the battery module (270) can supply power to the electronic components of the wearable device (101). In one embodiment, the battery module (270) may be placed within the first temple (204) and / or the second temple (205). For example, the battery module (270) may be a plurality of battery modules (270). The plurality of battery modules (270) may each be placed in the first temple (204) and the second temple (205). In one embodiment, the battery module (270) may be placed at the end of the first temple (204) and / or the second temple (205).

[0088] The antenna module (275) can transmit a signal or power to the outside of the wearable device (101) or receive a signal or power from the outside. In one embodiment, the antenna module (275) may be placed within the first temple (204) and / or the second temple (205). For example, the antenna module (275) may be placed close to one side of the first temple (204) and / or the second temple (205).

[0089] The speaker (255) can output an acoustic signal to the outside of the wearable device (101). The acoustic output module may be referred to as the speaker. In one embodiment, the speaker (255) may be placed within a first temple (204) and / or a second temple (205) to be positioned adjacent to the ear of a user wearing the wearable device (101). For example, the speaker (255) may include a second speaker (255-2) positioned adjacent to the user's left ear by being placed within the first temple (204), and a first speaker (255-1) positioned adjacent to the user's right ear by being placed within the second temple (205).

[0090] A light-emitting module (not shown) may include at least one light-emitting element. The light-emitting module may emit light of a color corresponding to a specific state or emit light with an action corresponding to a specific state in order to visually provide information regarding a specific state of the wearable device (101) to the user. For example, if the wearable device (101) requires charging, it may emit red light at a constant frequency. In one embodiment, the light-emitting module may be placed on the first rim (201) and / or the second rim (202).

[0091] Referring to FIG. 2b, a wearable device (101) according to one embodiment may include a printed circuit board (PCB) (290). The PCB (290) may be included in at least one of a first temple (204) or a second temple (205). The PCB (290) may include an interposer disposed between at least two sub-PCBs. One or more hardware components included in the wearable device (101) (e.g., hardware components illustrated by different blocks in FIG. 4) may be disposed on the PCB (290). The wearable device (101) may include a flexible PCB (FPCB) for interconnecting the hardware components.

[0092] According to one embodiment, a wearable device (101) may include at least one of a gyroscope sensor, a gravity sensor, and / or an acceleration sensor for detecting the posture of the wearable device (101) and / or the posture of a body part (e.g., head) of a user wearing the wearable device (101). Each of the gravity sensor and the acceleration sensor may measure gravitational acceleration and / or acceleration based on designated three-dimensional axes (e.g., x-axis, y-axis, and z-axis) that are perpendicular to each other. The gyroscope sensor may measure the angular velocity of each of the designated three-dimensional axes (e.g., x-axis, y-axis, and z-axis). At least one of the gravity sensor, the acceleration sensor, and the gyroscope sensor may be referred to as an inertial measurement unit (IMU). According to one embodiment, the wearable device (101) can identify a user's motion and / or gesture performed to execute or stop a specific function of the wearable device (101) based on an IMU.

[0093] FIGS. 3a and 3b illustrate an example of the appearance of a wearable device (e.g., a wearable device (101)). The wearable device (101) of FIGS. 3a and 3b may be an example of the wearable device (101) of FIG. 1. According to one embodiment, an example of the appearance of a first surface (310) of the housing of the wearable device (101) may be illustrated in FIG. 3a, and an example of the appearance of a second surface (320) opposite to the first surface (310) may be illustrated in FIG. 3b.

[0094] Referring to FIG. 3a, according to one embodiment, a first surface (310) of a wearable device (101) may have a shape that is attachable to a part of a user's body (e.g., the face of the user). Although not illustrated, the wearable device (101) may further include a strap for fixing to a part of a user's body and / or one or more temples (e.g., a first temple (204) and / or a second temple (205) of FIG. 2a and FIG. 2b). A first display (250-1) for outputting an image to the left eye among the user's two eyes and a second display (250-2) for outputting an image to the right eye among the two eyes may be disposed on the first surface (310). The wearable device (101) may further include rubber or silicone packing formed on the first surface (310) to prevent interference by light different from light emitted from the first display (250-1) and the second display (250-2) (e.g., ambient light).

[0095] According to one embodiment, a wearable device (101) may include cameras (260-1) for photographing and / or tracking both eyes of a user adjacent to each of the first display (250-1) and the second display (250-2). The cameras (260-1) may be referenced to the eye-tracking camera (260-1) of FIG. 2B. According to one embodiment, a wearable device (101) may include cameras (260-5, 260-6) for photographing and / or recognizing a user's face. The cameras (260-5, 260-6) may be referenced to FT cameras. The wearable device (101) may control an avatar representing the user in a virtual space based on the motion of the user's face identified using the cameras (260-5, 260-6). For example, the wearable device (101) can change the texture and / or shape of a part of an avatar (e.g., a part of an avatar representing a human face) by using information obtained by cameras (260-5, 260-6) (e.g., FT cameras) and representing the facial expression of a user wearing the wearable device (101).

[0096] Referring to FIG. 3b, on a second surface (320) opposite to the first surface (310) of FIG. 3a, a camera (e.g., cameras (260-7, 260-8, 260-9, 260-10, 260-11, 260-12)), and / or a sensor (e.g., a depth sensor (330)) may be placed to acquire information related to the external environment of the wearable device (101). For example, cameras (260-7, 260-8, 260-9, 260-10) may be placed on the second surface (320) to recognize external objects. The cameras (260-7, 260-8, 260-9, 260-10) may be referenced to the motion recognition cameras (260-2, 260-3) of FIG. 2b.

[0097] By using cameras (260-11, 260-12), the wearable device (101) can acquire images and / or videos to be transmitted to each of the user's two eyes. Camera (260-11) may be placed on the second surface (320) of the wearable device (101) to acquire an image to be displayed through a second display (250-2) corresponding to the right eye among the two eyes. Camera (260-12) may be placed on the second surface (320) of the wearable device (101) to acquire an image to be displayed through a first display (250-1) corresponding to the left eye among the two eyes. Cameras (260-11, 260-12) may be referenced to the shooting camera (260-4) of FIG. 2B.

[0098] According to one embodiment, a wearable device (101) may include a depth sensor (330) disposed on a second surface (320) to identify the distance between the wearable device (101) and an external object. Using the depth sensor (330), the wearable device (101) may obtain spatial information (e.g., a depth map) for at least a portion of the FoV of a user wearing the wearable device (101). Although not illustrated, a microphone may be disposed on the second surface (320) of the wearable device (101) to obtain sound output from an external object. The number of microphones may be one or more, depending on the embodiment.

[0099] Hereinafter, with reference to FIG. 4, the hardware or software configuration of the wearable device (101) will be described.

[0100] FIG. 4 illustrates an example of a block diagram of a wearable device (e.g., a wearable device (101)). The wearable device (101) of FIG. 4 may be an example of the electronic device (101) of FIG. 1 and the wearable device (101) of FIG. 2a to FIG. 3b.

[0101] Referring to FIG. 4, a wearable device (101) according to one embodiment may include a processor (410) (e.g., processor (120)), memory (415), a display (250) (e.g., a first display (250-1) and / or a second display (250-2) of FIG. 2a, FIG. 2b, FIG. 3a, and FIG. 3b), a sensor (420) (e.g., an image sensor (421) and / or a motion sensor (422)), and / or a communication circuit (430) (e.g., including at least a part of the communication module (190) of FIG. 1). The processor (410), memory (415), display (250), sensor (420), and / or communication circuit (430) may be electrically and / or operationally connected to each other by an electronic component such as a communication bus (402). In the present disclosure, the operational connection of electronic components may include a direct connection established between the electronic components and / or an indirect connection established between the electronic components such that a first electronic component among the electronic components is controlled by a second electronic component among the electronic components. The type and / or number of electronic components included in the wearable device (101) are not limited to those shown in FIG. 4. For example, the wearable device (101) may include only some of the electronic components shown in FIG. 4.

[0102] A processor (410) of a wearable device (101) according to one embodiment may include a circuit (e.g., a processing circuit) for processing data based on one or more instructions. The circuit for processing data may include, for example, an arithmetic and logic unit (ALU), a field programmable gate array (FPGA), a central processing unit (CPU), and / or an application processor (AP). In one embodiment, the wearable device (101) may include one or more processors. The processor (410) may have a structure of a multi-core processor such as a dual core, a quad core, a hexa core, and / or an octa core. The multi-core processor structure of the processor (410) may include a structure based on a plurality of core circuits (e.g., a big-little structure), distinguished by power consumption, clock, and / or computational power per unit time. In one embodiment comprising a processor (410) having a multi-core processor structure, the operations and / or functions of the present disclosure may be performed individually or collectively by one or more cores included in the processor (410).

[0103] A memory (415) of a wearable device (101) according to one embodiment may include electronic components for storing data and / or instructions that are input to or output from a processor (410). The memory (415) may include, for example, volatile memory such as random-access memory (RAM) and / or non-volatile memory such as read-only memory (ROM). Volatile memory may include, for example, at least one of dynamic RAM (DRAM), static RAM (SRAM), cache RAM, and pseudo SRAM (PSRAM). Non-volatile memory may include, for example, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, hard disk, compact disk, and embedded multi-media card (eMMC). In one embodiment, the memory (415) may be referred to as storage.

[0104] In one embodiment, a display (250) of a wearable device (101) can output visualized information to a user of the wearable device (101). A display (250) arranged in front of the eyes of a user wearing the wearable device (101) may be placed in at least a part of the housing of the wearable device (101) (e.g., a first display (250-1) and / or a second display (250-2) of FIG. 2a, FIG. 2b, FIG. 3a, and FIG. 3b). For example, the display (250) may be included within a display assembly. For example, the display (250) may be controlled by a processor (410) including circuits such as a CPU (411), a GPU (graphic processing unit) (412), and / or a DPU (display processing unit) (413) to output visualized information to a user. The display (250) may include a flexible display, a flat panel display (FPD), and / or electronic paper. The display (250) may include a liquid crystal display (LCD), a plasma display panel (PDP), and / or one or more light emitting diodes (LEDs). The LEDs may include organic LEDs (OLEDs). Embodiments are not limited thereto, for example, if the wearable device (101) includes a lens for transmitting external light (or ambient light), the display (250) may include a projector (or projection assembly) for projecting light onto the lens. In one embodiment, the display (250) may be referred to as a display panel and / or a display module.The pixels included in the display (250) may be positioned toward either of the user's two eyes when worn by the user of the wearable device (101). For example, the display (250) may include display areas (or active areas) corresponding to each of the user's two eyes.

[0105] In one embodiment, a sensor (420) of a wearable device (101) may generate electrical information that can be processed by a processor (410) and / or a memory (415) from non-electronic information associated with the wearable device (101). For example, the sensor (420) may include a global positioning system (GPS) sensor for detecting the geographic location of the wearable device (101). In addition to the GPS method, the sensor (420) may generate information indicating the geographic location of the wearable device (101) based on a global navigation satellite system (GNSS), such as Galileo or Beidou (compass). This information may be stored in the memory (415), processed by the processor (410), and / or transmitted to another electronic device distinct from the wearable device (101) via a communication circuit.

[0106] Referring to FIG. 4, an image sensor (421) and / or a motion sensor (422) are illustrated as examples of sensors (420) included in a wearable device (101). The sensor (420) may include one or more light sensors (e.g., a CCD (charged coupled device) sensor, a CMOS (complementary metal oxide semiconductor) sensor) that generate electrical signals representing the color and / or brightness of light. The image sensor (421) may be referred to as a camera. A plurality of light sensors included in the image sensor (421) may be arranged in the form of a two-dimensional grid (4-dimensional array). The image sensor (421) may acquire the electrical signals of each of the plurality of light sensors substantially simultaneously to generate two-dimensional frame data corresponding to the light reaching the light sensors of the two-dimensional grid. For example, photo data captured using the image sensor (421) may refer to one (a) two-dimensional frame data acquired from the image sensor (421). For example, video data captured using an image sensor (421) may refer to a sequence of multiple two-dimensional frame data obtained from the image sensor (421) along a frame rate. The image sensor (421) may further include a flash light for outputting light in the direction in which the image sensor (421) receives light.

[0107] According to one embodiment, the wearable device (101) may include a plurality of image sensors arranged toward different directions as an example of an image sensor (421). As described above with reference to FIGS. 2A, 2B, 3A, and 3B, the plurality of image sensors may include an eye-tracking camera (e.g., the eye-tracking camera (260-1) of FIGS. 2B and 3A) configured to be arranged toward the eyes of a user wearing the wearable device (101). The plurality of image sensors may include an outward camera. A processor (410) may identify the direction of the user's gaze using an image and / or video obtained from the eye-tracking camera. The eye-tracking camera may include an infrared (IR) sensor. The eye-tracking camera may be referred to as an eye sensor and / or an eye tracker.

[0108] An external camera may be positioned facing forward of a user wearing the wearable device (101) (e.g., a direction in which both eyes may face). The wearable device (101) may include a plurality of external cameras. The embodiments are not limited thereto, and the external camera may be positioned facing an external space. Using images and / or videos obtained from the external camera, the processor (410) may identify external objects. For example, the processor (410) may identify the position, shape, and / or gesture (e.g., hand gesture) of the user wearing the wearable device (101) based on images and / or videos obtained from the external camera. Using images and / or videos of the external environment obtained from the external camera, the processor (410) may recognize or track one or more objects within the external environment.

[0109] According to one embodiment, the motion sensor (422) may output an electrical signal representing gravitational accelerations, accelerations, and / or angular velocities of a plurality of axes (e.g., x-axis, y-axis, and z-axis) that are perpendicular to each other and are based on a designated origin within the wearable device (101) and / or the motion sensor (422). For example, the processor (410) may repeatedly receive or acquire sensor data from the motion sensor (422), including accelerations, angular velocities, and / or magnitudes of the magnetic field of a number of the plurality of axes, based on a designated period (e.g., 1 millisecond). In one embodiment, the motion sensor (422) may be referred to as an inertial measurement unit (IMU). The sensor (420) included in the wearable device (101) is not limited to the above and may include a grip sensor, a proximity sensor, a heart rate sensor, a fingerprint sensor, an ambient light sensor, and / or a ToF sensor. By using a motion sensor (422), the processor (410) can detect motion of the wearable device (101) (e.g., motion of the wearable device (101) caused by a user wearing the wearable device (101)).

[0110] In one embodiment, the communication circuit (430) of the wearable device (101) may include a hardware component for supporting the transmission and / or reception of a signal between the wearable device (101) and an external electronic device (e.g., electronic device (102), electronic device (104)). The communication circuit (430) may include, for example, at least one of a modem, an antenna, and an O / E (optic / electronic) converter. The communication circuit (430) may support the transmission and / or reception of an electrical signal based on various types of protocols such as Ethernet, LAN (local area network), WAN (wide area network), WiFi (wireless fidelity), Bluetooth, BLE (Bluetooth low energy), Zigbee, LTE (long term evolution), and 5G NR (new radio).

[0111] According to one embodiment, within the memory (415) of the wearable device (101), one or more instructions (or commands) representing data to be processed by the processor (410) of the wearable device (101), calculations to be performed, and / or operations may be stored. A set of one or more instructions may be referred to as a program, firmware, operating system, process, routine, sub-routine, and / or software application (hereinafter, application). For example, the wearable device (101), and / or processor (410) may perform at least one of the operations described below when a set of a plurality of instructions distributed in the form of an operating system, firmware, driver, program, and / or software application is executed. In the following, the statement that a software application is installed in the wearable device (101) means that one or more instructions provided in the form of a software application (or package) are stored in memory (415), and that the one or more applications are stored in an executable format (e.g., a file having an extension specified by the operating system of the wearable device (101)) by the processor (410). For example, the application may include a program and / or library related to a service provided to the user.

[0112] Referring to FIG. 4, programs installed on a wearable device (101) may be included in any one of different layers, including an application layer (440), a framework layer (450), and / or a hardware abstraction layer (HAL) (480), based on the target. For example, within the hardware abstraction layer (480), programs (e.g., modules, or drivers) designed to target the hardware of the wearable device (101) (e.g., a display (250), and / or a sensor (420)) may be included. The framework layer (450) may be referred to as an XR framework layer in that it includes one or more programs for providing XR (extended reality) services. For example, the layers illustrated in FIG. 4 are logically (or for convenience of explanation) separated, and may not imply that the address space of memory (415) is separated by said layers.

[0113] Within the framework layer (450), programs designed to target at least one of the hardware abstraction layer (480) and / or the application layer (440) (e.g., a location tracker (471), a spatial recognizer (472), a gesture tracker (473), and / or an eye tracker (474)) may be included. The programs included in the framework layer (450) may provide an application programming interface (API) that is executable (or invokeable) based on other programs.

[0114] Within the application layer (440), programs designed to target users of the wearable device (101) may be included. Examples of programs included in the application layer (440) include an XR (extended reality) system UI (user interface) (441) and / or an XR application (442), but embodiments are not limited thereto. For example, programs included in the application layer (440) (e.g., software applications) may call APIs to cause the execution of functions supported by programs included in the framework layer (450).

[0115] A wearable device (101) may display one or more visual objects on a display (250) to perform interaction with a user based on the execution of an XR system UI (441). A visual object may mean an object that can be placed on a screen for the transmission of information and / or interaction, such as text, images, icons, videos, buttons, checkboxes, radio buttons, text boxes, sliders, and / or tables. A visual object may be referred to as a visual guide, a virtual object, a visual element, a UI element, a view object, and / or a view element. A wearable device (101) may provide the user with functions available in a virtual space based on the execution of an XR system UI (441).

[0116] Referring to FIG. 4, a lightweight renderer (443) and / or an XR plugin (444) are depicted within the XR system UI (441), but are not limited thereto. For example, based on the XR system UI (441), the processor (410) may execute a lightweight renderer (443) and / or an XR plugin (444) within the framework layer (450).

[0117] A wearable device (101) may acquire resources (e.g., APIs, system processes and / or libraries) used to define, create, and / or execute a rendering pipeline, which is permitted to be partially modified, based on the execution of a lightweight renderer (443). The lightweight renderer (443) may be referred to as a lightweight render pipeline in terms of defining a rendering pipeline, which is permitted to be partially modified. The lightweight renderer (443) may include a renderer built prior to the execution of a software application (e.g., a pre-built renderer). For example, the wearable device (101) may acquire resources (e.g., APIs, system processes and / or libraries) used to define, create, and / or execute the entire rendering pipeline based on the execution of an XR plugin (444). The XR plugin (444) may be referred to as an open XR native client in terms of defining (or setting) the entire rendering pipeline.

[0118] The wearable device (101) may display a screen representing at least a portion of a virtual space on a display (250) based on the execution of an XR application (442). The XR plugin (444-1) included in the XR application (442) may include instructions that support functions similar to the XR plugin (444) of the XR system UI (441). Descriptions of the XR plugin (444-1) that overlap with descriptions of the XR plugin (444) may be omitted. The wearable device (101) may trigger the execution of a virtual space manager (451) based on the execution of the XR application (442).

[0119] A wearable device (101) can display an image on a display (250) in a virtual space based on the execution of an application (445). The application (445) may be configured to output image information for displaying a two-dimensional image. A wearable device (101) may trigger the execution of a virtual space manager (451) based on the execution of the application (445). A wearable device (101) may generate dual image information to display the two-dimensional image in a three-dimensional virtual space based on the execution of the application (445). Here, the dual image information may include a first image information for the left eye and a second image information for the right eye, taking into account binocular parallax. To display the two-dimensional image in a three-dimensional virtual space, a wearable device (101) may generate the dual image information based on the image information for displaying the two-dimensional image.

[0120] According to one embodiment, a wearable device (101) can provide a virtual space service based on the execution of a virtual space manager (451). For example, the virtual space manager (451) may include a platform for supporting the virtual space service. Based on the execution of the virtual space manager (451), the wearable device (101) can identify a virtual space formed based on the user's location indicated by data acquired through a sensor (420), and can display at least a portion of the virtual space on a display (250). The virtual space manager (451) may be referred to as a composition presentation manager (CPM).

[0121] The virtual space manager (451) may include a runtime service (452). For example, the runtime service (452) may be referred to as an OpenXR runtime module (or OpenXR runtime program). The wearable device (101) may execute at least one of a user pose prediction function, a frame timing function, and / or a spatial input function based on the execution of the runtime service (452). For example, the wearable device (101) may perform rendering for a virtual space service for the user based on the execution of the runtime service (452). For example, a virtual space-related function, executable by the application layer (440), may be supported based on the execution of the runtime service (452).

[0122] The virtual space manager (451) may include a pass-through manager (453). The wearable device (101) may display an image and / or video representing a real space acquired through an external camera superimposed on at least a portion of the screen while displaying a screen representing a virtual space on the display (250) based on the execution of the pass-through manager (453).

[0123] The virtual space manager (451) may include an input manager (454). The wearable device (101) may identify acquired data (e.g., sensor data) by executing one or more programs included within the recognition service layer (470) based on the execution of the input manager (454). The wearable device (101) may identify user inputs associated with the wearable device (101) using the acquired data. The user inputs may be associated with user motions (e.g., hand gestures), gaze, and / or speech identified by a sensor (420) (e.g., an image sensor (421) such as an external camera). The user inputs may be identified based on an external electronic device connected (or paired) via a communication circuit.

[0124] The perception abstract layer (460) can be used for data exchange between the virtual space manager (451) and the perception service layer (470). In terms of being used for data exchange between the virtual space manager (451) and the perception service layer (470), the perception abstract layer (460) can be referred to as an interface. For example, the perception abstract layer (460) can be referred to as OpenPX. The perception abstract layer (460) can be used for a perception client and a perception service.

[0125] According to one embodiment, the recognition service layer (470) may include one or more programs for processing data obtained from the sensor (420). The one or more programs may include at least one of a location tracker (471), a spatial recognizer (472), a gesture tracker (473), and / or an eye tracker (474). The type and / or number of the one or more programs included in the recognition service layer (470) are not limited to those shown in FIG. 4.

[0126] The wearable device (101) can identify the posture of the wearable device (101) using a sensor (420) based on the operation of a position tracker (471). The wearable device (101) can identify the 6 degrees of freedom pose (6 dof pose) of the wearable device (101) using data acquired using an external camera (e.g., image sensor (421)) and / or an IMU (e.g., motion sensor (422) including a gyroscope, accelerometer, and / or geomagnetic sensor) based on the operation of the position tracker (471). The position tracker (471) may be referred to as a head tracking (HeT) module (or head tracker, head tracking program).

[0127] A wearable device (101) can acquire information to provide a three-dimensional virtual space corresponding to the surrounding environment (e.g., external space) of the wearable device (101) (or the user of the wearable device (101)) based on the execution of a spatial recognition device (472). Based on the execution of the spatial recognition device (472), the wearable device (101) can reproduce the surrounding environment of the wearable device (101) in three dimensions using data acquired using an external camera (e.g., image sensor (421)). Based on the surrounding environment of the wearable device (101) reproduced in three dimensions based on the execution of the spatial recognition device (472), the wearable device (101) can identify at least one of a plane, an incline, and a staircase. The spatial recognition device (472) may be referred to as a scene understanding (SU) module (or scene understanding program).

[0128] The wearable device (101) can identify (or recognize) the pose and / or gesture of the user's hand of the wearable device (101) based on the execution of the gesture tracker (473). For example, the wearable device (101) can identify the pose and / or gesture of the user's hand using data acquired from an external camera (e.g., image sensor (421)) based on the execution of the gesture tracker (473). For example, the wearable device (101) can identify the pose and / or gesture of the user's hand based on data (or images) acquired using an external camera based on the execution of the gesture tracker (473). The gesture tracker (473) may be referred to as a hand tracking (HaT) module (or hand tracking program) and / or a gesture tracking module.

[0129] The wearable device (101) can identify (or track) the movement of the user's eyes of the wearable device (101) based on the execution of the eye tracker (474). For example, the wearable device (101) can identify the movement of the user's eyes using data obtained from an eye-tracking camera (e.g., image sensor (421)) based on the execution of the eye tracker (474). The eye tracker (474) may be referred to as an eye tracking (ET) module (or eye tracking program) and / or a gaze tracking module.

[0130] The recognition service layer (470) of the wearable device (101) may further include a face tracker (475) for tracking the user's face. For example, the wearable device (101) may identify (or track) the movement of the user's face and / or the user's facial expression based on the execution of the face tracker (475). The wearable device (101) may estimate the user's facial expression based on the movement of the user's face based on the execution of the face tracker (475). For example, the wearable device (101) may identify the movement of the user's face and / or the user's facial expression based on data (e.g., images and / or videos) obtained using a camera (425) (e.g., a camera facing at least a part of the user's face) based on the execution of the face tracker (475).

[0131] Referring to FIG. 4, an example of a processor (410) is shown as a CPU (411), a GPU (graphic processing unit) (412), and / or a DPU (display processing unit) (413). A renderer (490) may include instructions for rendering images in a three-dimensional virtual space. A processor (410) (e.g., DPU (413)) that executes the renderer (490) may acquire at least one image to be displayed at least partially in a display area of ​​a display (250) in a software application (e.g., a software application executed by the CPU (411) and / or GPU (412)). For example, a processor (410) that executes the renderer (490) may determine the location of an area where an application (e.g., an XR application (442), an application (445)) will be rendered. The processor (410) that executed the renderer (490) can generate an image of the application to be displayed on the display (250). The renderer (490) can synthesize images to generate a composite image to be displayed on the display (250).

[0132] A processor (410) that executes a renderer (490) can divide the display area of ​​a display (250) into a foveated portion (or may be referred to as a foveated area) and a peripheral portion (or may be referred to as a residual area) using a gaze position calculated using a position tracker (471) and / or a gaze tracker (474). For example, a processor (410) that detects coordinate values ​​of the gaze position can determine the portion of the display area containing said coordinate values ​​as the foveated area. A DPU (413) that executes a renderer (490) can acquire at least one image corresponding to each of said foveated area and said residual area, having a size smaller than the size of the entire display area of ​​the display (250) or having a resolution less than the resolution of the display area.

[0133] A processor (410) that executes a renderer (490) can obtain or generate a composite image to be displayed on a display (250) by synthesizing an image corresponding to a foveated area and an image corresponding to a surrounding area. For example, the processor (410) can perform upscaling to enlarge the image corresponding to the surrounding area to the size of the entire display area of ​​the display (250). On the enlarged image, the processor (410) can combine the image corresponding to the foveated area to generate a composite image to be displayed on the display (250). Along the boundary line of the image corresponding to the foveated area, the processor (410) can mix the enlarged image and the image corresponding to the foveated area by applying a visual effect such as blur.

[0134] FIG. 5 illustrates an example of a block diagram of a wearable device (e.g., the electronic device (101) of FIG. 1) for displaying an image in a virtual space. FIG. 5 describes an example in which multiple programs / instructions for displaying an image in a virtual space are executed. The multiple programs / instructions may all be executed on a single processor (e.g., AP) or may be executed by multiple processors (e.g., AP, GPU (graphic processing unit), NPU (neural processing unit)). The meaning of being able to be executed by multiple processors is that some programs / instructions may be executed by a first processor and other programs / instructions may be executed by a second processor different from the first processor.

[0135] Referring to FIG. 5, a wearable device (101) may execute a virtual space manager (550) (e.g., virtual space manager (451) of FIG. 4, CPM) to render an image in a virtual space. For the virtual space manager (550), at least some of the descriptions of the virtual space manager (451) of FIG. 4 may be referenced. The virtual space manager (550) may include a platform for supporting virtual space services. The virtual space manager (550) may include a runtime service (551) (e.g., OpenXR Runtime), a panel rendering (552) (e.g., 2D Panel Render), and an XR composite unit (553) (XR Compositor). Based on the execution of the runtime service (551), the wearable device (101) may execute at least one of a user pose prediction function, a frame timing function, and / or a spatial input function. For the runtime service (551), at least some of the descriptions of the runtime service (452) of FIG. 4 may be referenced. The wearable device (101) may display at least one image (video) on a panel (e.g., a 2D panel) to enable the implementation of a virtual space through a display, based on the execution of panel rendering (552). For example, the wearable device (101) may display a rendering image corresponding to RGB information (566) for a panel from the spatialization manager (540) described later through a display (e.g., a display (250)). The wearable device (101) may composite an image of a real area (hereinafter, a pass-through image) captured through a camera in a virtual space with an image of a virtual area, based on the execution of an XR composite unit (553) (XR Compositor). For example, the wearable device (101) can generate a composite image by merging the pass-through image and the virtual region image based on the execution of the XR synthesis unit (553).The wearable device (101) can transmit the generated composite image to a display buffer so that the composite image is displayed. The wearable device (101) can identify a virtual space through a virtual space manager (550) and display at least a portion of the virtual space on a display (250). The virtual space manager (550) may be referred to as CPM. The wearable device (101) can execute the virtual space manager (550) to render an image corresponding to at least a portion of the virtual space.

[0136] According to one embodiment, a wearable device (101) can execute a spatialization manager (540). The spatialization manager (540) can perform processing for displaying an image in a three-dimensional virtual space. The wearable device (101) can perform preprocessing based on the execution of the spatialization manager (540) so that an image can be rendered in a three-dimensional virtual space through a virtual space manager (550). For example, the wearable device (101) can perform at least some of the functions of the renderer (490) of FIG. 4 based on the execution of the spatialization manager (540). The wearable device (101) can process image information provided by an application (e.g., an XR application (510), a non-XR general 2D screen application (520), or a system UI (530) based on the execution of the spatialization manager (540). A spatialization manager (540) (e.g., Space Flinger) may include a system screen manager (541) (e.g., System scene), an input manager (542) (e.g., Input Routing), and a lightweight rendering engine (543) (e.g., Impress Engine). The system screen manager (541) may be executed to display the system UI (530). System UI-related information (564) may be transmitted to the system screen manager (541) from a program (e.g., API) that provides the system UI (530). System UI-related information (564) may be obtained through a spatializer API and / or a Same-process private API. The spatialization manager (540) may determine the layout (e.g., position, display order) of the system UI (530) screen in three-dimensional space through pre-allocated resources. The system screen manager (541) can transmit image information (567) for rendering the screen of the system UI (530) according to the above layout to the virtual space manager (550).The input manager (542) may be configured to process user input (e.g., user input on a system screen or an app screen). The lightweight rendering engine (543) may be a renderer for image generation (e.g., lightweight renderer (443)). For example, the lightweight rendering engine (543) may be used to display the system UI (530). According to one embodiment, the spatialization manager (540) may include a lightweight rendering engine (543) for rendering the system UI. According to one embodiment, if the lightweight rendering engine (543) does not have sufficient resources to render an avatar used in an HMD, at least one external rendering engine may be used. In this case, to resolve compatibility issues with external rendering (e.g., a third-party engine), an external rendering engine support module may be added within the spatialization manager (540).

[0137] According to one embodiment, a wearable device (101) may execute an application. For example, in response to the execution of an XR application (510) (e.g., an XR application (442), a 3D game, an XR map, or other immersive application), a virtual space manager (550) may be executed. The wearable device (101) may provide dual image information (561) provided from the XR application (510) to the virtual space manager (550). To display images in three-dimensional space, the dual image information (561) may include two image information that takes into account binocular parallax. For example, the dual image information (561) may include a first image information for the user's left eye and a second image information for the user's right eye to render in three-dimensional virtual space. Hereinafter, the term dual image information is used in the present disclosure as a term referring to image information for displaying images for both eyes in three-dimensional space. In addition to the dual image information, the above dual image information may utilize binocular image information, dual image information, dual image data, dual image, binocular image data, stereoscopic image information, 3D image information, spatial image information, spatial image data, 2D-3D conversion data, dimension conversion image data, binocular parallax image data, and / or equivalent technical terms. The wearable device (101) can generate a composite image by merging image layers through a virtual space manager (550). The wearable device (101) can transmit the generated composite image to a display buffer. The composite image can be displayed on the display (250) of the wearable device (101).

[0138] According to one embodiment, the wearable device (101) can execute at least one application among an XR application (510) and other applications (520) (e.g., a first application (520-1), a second application (520-2), a second application (520-N), and a Nth application (520-N)). According to one embodiment, the application (520) may be configured to output image information for displaying a two-dimensional image. In other words, the application (520) may provide a two-dimensional image. For example, the application (520) may be a video application, a schedule application, or an internet browser application. Let us assume that, in response to the execution of the application (520), image information (562) provided by the application (520) is provided to the virtual space manager (550). Since the image information (562) has only x and y coordinates within a two-dimensional plane, it may be difficult to consider the sequential relationship between other applications centered on the user (i.e., distance from the user). The wearable device (101) may execute a spatialization manager (540) to provide dual image information to a virtual space manager (550) even when displaying an application (520) that provides a general 2D screen. For example, based on the execution of the spatialization manager (540), the wearable device (101) may receive application-related information (563) from the first application (520-1). For example, the application-related information (563) may include image information representing a two-dimensional image of the first application (520-1) (e.g., information including RGB per pixel) and / or content information in the first application (520-1) (e.g., characteristics of the content executed in the first application, type of content). Application-related information (563) can be obtained through the spatializer API.Based on the execution of the spatialization manager (540), the wearable device (101) can identify information regarding the location of the area to be rendered and the size of the area to be rendered (hereinafter, location information). Based on the execution of the spatialization manager (540), the wearable device (101) can generate dual image information (565, e.g., RGBx2) that takes into account the user's binocular parallax through the image information and the location information. Based on the execution of the spatialization manager (540), the wearable device (101) can provide the dual image information (565) to the virtual space manager (550). By converting a simple two-dimensional image into dual image information (565), the problem caused by the image information (562) being directly transmitted to the virtual space manager (550) can be resolved. Additionally, as at least some of the functions for displaying images in virtual space are performed by the spatialization manager (540) instead of the virtual space manager (550), the burden on the virtual space manager (550) may be reduced. However, as image information from the application (520) is not transmitted directly to the virtual space manager (550) but is transmitted through the spatialization manager (540), the quality of the image ultimately output to the user may be lowered. For example, in the first application (520-1), an image is rendered at a resolution of approximately 2756 x 1846, but the image may be downsampled during the process of being transmitted to the virtual space manager (550) through the spatialization manager (540) (e.g., downsampled from a resolution of approximately 2756 x 1846 to a resolution of approximately 1160 x 680). Afterwards, the virtual space manager (550) can upsample the downsampled image (e.g., upsample from a resolution of about 1160 x 680 to a resolution of about 1625 x 1070) and pass the upsampled image to the display buffer.As such, in the process of an image being transmitted from an application (520) to a spatialization manager (540) and from a spatialization manager (540) to a virtual space manager (550), a resolution mismatch may occur, or an aliasing problem or a degradation of image quality may occur during the upsampling process. To resolve the above-mentioned problems, the present disclosure describes techniques for controlling the resolution of an area to be displayed in an application and performing foveation rendering based on the system structure illustrated in FIG. 5.

[0139] Figure 6 illustrates an example of a structure of multiple layers.

[0140] Referring to FIG. 6, programs installed on a wearable device (101) can be classified into one of a platform layer (610), a recognition service layer (620) (e.g., the recognition service layer (470) of FIG. 4), and a sensor service layer (630). For example, the wearable device (101) can operate based on the platform layer (610), the recognition service layer (620), and the sensor service layer (630).

[0141] According to one embodiment, the platform layer (610) may be configured for XR services. For example, the platform layer (610) may include a platform (e.g., an Android platform) for supporting XR services. For example, the platform layer (610) may include the virtual space manager (550) of FIG. 5. The platform layer (610) may include a runtime service (611). For the runtime service (611), the descriptions of the runtime service (551) of FIG. 5 and the descriptions of the runtime service (452) of FIG. 4 may be referenced. For example, the runtime service (611) may be referenced as an OpenXR runtime module. The runtime service (611) may be used to provide at least one of a user pose prediction function, a frame timing function, and / or a spatial input function through the wearable device (101). For example, the runtime service (611) may be used to perform rendering for XR services for the user. For example, based on the runtime service (611), an application (e.g., a Unity or OpenXR native application) can be implemented.

[0142] The perception abstraction layer (612) can be used for data exchange between the platform layer (610) and the perception service layer (620). For the perception abstraction layer (612), the descriptions of the perception abstraction layer (460) of FIG. 4 may be referenced. For example, the perception abstraction layer (612) may be referenced as OpenPX. The perception abstraction layer (612) can be used for a perception client and a perception service.

[0143] According to one embodiment, the recognition service layer (620) may include a service module (621), a recognition plugin layer (622), a sensor management module (623), a playback module (624), and / or an external data management module (625). For example, the recognition service layer (620) may include at least one of the service module (621), the recognition plugin layer (622), the sensor management module (623), the playback module (624), and / or an external data management module (625). For example, at least some of the service module (621), the recognition plugin layer (622), the sensor management module (623), the playback module (624), and the external data management module (625) may be omitted.

[0144] The service module (621) can manage input data of the wearable device (101). The service module (621) can be used to manage data (e.g., gesture information) obtained from a plurality of recognition modules included in the recognition plugin layer (622). As an example, the service module (621) may be referred to as SxrDataService.

[0145] The service module (621) can perform interfacing with an upper layer (e.g., platform layer (610) or runtime service (611)). The service module (621) can exchange data with the upper layer (e.g., platform layer (610) or runtime service (611)) through the recognition abstraction layer (612). For example, the recognition abstraction layer (612) may be referred to as OpenPX. According to an embodiment, the service module (621) may support OpenXR Extension as well as OpenPX. The service module (621) may be used to exchange data (e.g., gesture information) between a plurality of recognition modules. The service module (621) may be configured to manage data processed by the recognition service layer (620). The service module (621) may select data among the data to be recognized as input to the wearable device (101). The above data may include data obtained from a plurality of recognition modules and data obtained through an external data management module (625). The service module (621) may manage data to be used in the recognition abstraction layer (612). The service module (621) may select data to be recognized as an input to the wearable device (101) from the above data and provide it to the recognition abstraction layer (612).

[0146] The recognition plugin layer (622) may include a plurality of recognition modules. The plurality of recognition modules may be referred to as a plurality of perception solutions.

[0147] For example, a plurality of recognition modules may include at least one of a head tracking (HeT) module (622-1), a scene understanding (SU) module (622-2), a hand tracking (HaT) module (622-3), an eye tracking (ET) module (622-4), and a face tracking (FT) module (622-5). Each of the plurality of recognition modules included in the recognition plugin layer (622) may include a common interface for connection (or interaction) with a sensor management module (623). Each of the plurality of recognition modules may include a common interface for connection (or interaction) with a sensor management module (623).

[0148] The head tracking module (622-1) can identify the posture of the wearable device (101) using at least one sensor of the wearable device (101). For example, the head tracking module (622-1) can identify the 6 degrees of freedom pose (6 dof pose) of the wearable device (101) based on data acquired using a camera (e.g., image sensor (421) of FIG. 4) and an IMU.

[0149] The environment recognition module (622-2) can be used to construct the surrounding environment of the wearable device (101) (or the user of the wearable device (101)) into a three-dimensional virtual space. The environment recognition module (622-2) can be used to reconstruct the surrounding environment of the wearable device (101) in three dimensions based on data acquired using a camera (e.g., the image sensor (421) of FIG. 4). Based on the surrounding environment of the wearable device (101) reconstructed in three dimensions, the environment recognition module (622-2) can identify at least one of a plane, an incline, and a staircase.

[0150] A hand tracking module (622-3) may be used to identify (or recognize) the pose and / or gesture of the user's hand of the wearable device (101). For example, the hand tracking module (622-3) may identify the pose and / or gesture of the user's hand based on data acquired from at least one sensor. For example, the hand tracking module (622-3) may identify the pose and / or gesture of the user's hand based on data (e.g., an image) acquired using a camera.

[0151] An eye tracking module (622-4) may be used to identify (or track) the movement of the user's eyes of the wearable device (101). For example, the eye tracking module (622-4) may identify the user's eye movement based on data obtained from at least one sensor. For example, the eye tracking module (622-4) may identify the user's eye movement based on data obtained using a camera (e.g., the eye tracking camera (260-1) of FIG. 2b and FIG. 3a)) and / or an IR LED (infrared light emitting diode).

[0152] The face tracking module (622-5) can be used to identify (or track) the movement of the user's face and / or the user's facial expression. The face tracking module (622-5) can estimate the user's facial expression based on the movement of the user's face. For example, the face tracking module (622-5) can identify the movement of the user's face and / or the user's facial expression based on data (e.g., images) acquired using a camera (e.g., camera (260) in FIG. 2a and FIG. 2b).

[0153] For example, a plurality of recognition modules included in the recognition plugin layer (622) may be configured in a plugin structure. For example, some of the plurality of recognition modules may be replaced with other modules regardless of the sensor service layer (630) and platform layer (610), which are lower layers of the recognition service layer (620).

[0154] According to one embodiment, the sensor management module (623) may be used to provide (or transmit) data to each of the plurality of recognition modules through a common interface. For example, the sensor management module (623) may be used to separate (or eliminate) the dependency between the lower layer, the sensor service layer (630), and the upper layer, the recognition plugin layer (622). For example, the sensor management module (623) may be referred to as SxrSensorSeviceManger.

[0155] The sensor management module (623) can support various modules (or sensor services) of the sensor service layer (630). Multiple recognition modules may not directly interface with the sensor service layer (630). Multiple recognition modules may receive data (e.g., sensor data) through the sensor management module (623). Therefore, even if a module of the sensor service layer (630) is changed, it may not affect the multiple recognition modules.

[0156] The sensor management module (623) may further include a load balancing module. The load balancing module can identify data provided from the sensor service layer (630). Based on the data provided from the sensor service layer (630), the load balancing module can identify at least some of the recognition modules among the plurality of recognition modules. The load balancing module can provide data to the identified at least some of the recognition modules. For example, the load balancing module can distribute data to the plurality of recognition modules based on the status of the plurality of recognition modules and / or the status of the wearable device (101). For example, the load balancing module can filter the data provided to the plurality of recognition modules based on the status of the plurality of recognition modules and / or the status of the wearable device (101). According to an embodiment, the load balancing module may be configured independently of the sensor management module (623). The load balancing module may be referred to as SxrPerceptionLoadBalancer.

[0157] The playback module (624) may be used to provide a stored dataset to at least one of a plurality of recognition modules in real time via playback. For example, the dataset may be stored through the playback module (624) based on specified specifications. The dataset may include first data obtained from the sensor service layer (630) as well as second data obtained based on the first data obtained from the sensor service layer (630) (e.g., virtual object data or synthetic data). For example, the first data may be referenced as sensor data. The second data may be referenced as virtual data.

[0158] According to an embodiment, the wearable device (101) may receive data from an external electronic device. For example, the data received from the external electronic device may include first data obtained from a service layer included in the external electronic device and / or second data obtained based on the first data. The wearable device (101) may perform playback (or a playback function) using the data received from the external electronic device. The wearable device (101) may transmit the result of the playback (or playback function) to the external electronic device. For example, the wearable device (101) may be used to process the data obtained from the external electronic device on its behalf. The wearable device (101) may receive data obtained from at least one sensor of the external electronic device. Based on the received data, the wearable device (101) may obtain information (e.g., information about a 6-degree-of-freedom posture) obtained through a playback module (624) (or a plurality of recognition modules). The wearable device (101) can transmit the acquired information to an external electronic device. The external electronic device can provide XR services based on the acquired information.

[0159] The playback module (624) can perform playback (or a playback function) based on at least one of the first data and the second data. According to an embodiment, the playback module (624) can perform playback by combining (or mixing) real-time data (e.g., runtime data) and pre-stored data.

[0160] For example, playback may refer to a function that utilizes stored data (or gesture information) according to the operation of the wearable device (101). For example, playback may refer to a function that identifies a value regarding the performance of the XR service through a comparison between gesture information obtained based on a specified operation regarding the XR service and reference gesture information according to said specified operation.

[0161] For example, playback may refer to a function for obtaining information on the performance of an XR service provided to a user of a wearable device (101). The playback module (624) may identify information (e.g., gesture information) regarding a user who has performed a designated action (e.g., mission) regarding the XR service. The playback module (624) may identify reference information regarding the designated action. Reference information may refer to information for determining the completion of the performance of the designated action. The playback module (624) may identify the similarity between the information regarding the user who has performed the designated action and the reference information. Based on the similarity, the playback module (624) may identify whether the performance of the action designated by the user has been completed.

[0162] According to an embodiment, the playback module (624) may be included in the sensor management module (623). For example, the playback module (624) can perform playback through the sensor management module (623) without changing the plurality of recognition modules.

[0163] The external data management module (625) can be used to manage data obtained through an external electronic device (e.g., a smart watch, a smartphone, or a tablet PC) (or at least one sensor of the external electronic device) connected to the wearable device (101). For example, the external data management module (625) can improve the accuracy of a plurality of recognition modules using data obtained from the external electronic device. For example, the external data management module (625) can correct data (or gesture information) obtained from a plurality of recognition modules using data obtained from the external electronic device. According to an embodiment, the external data management module (625) may not be included in the recognition service layer (620).

[0164] The sensor service layer (630) may be used to control at least one sensor (e.g., camera, IMU, TOF (time of flight) sensor). For example, the sensor service layer (630) may be used to provide a service for accessing at least one sensor. For example, the sensor service layer (630) may include at least one of a module for VR services (e.g., QVRservice), a module for XR services (e.g., SxrSensorService), a sensor API (e.g., android sensor API), and a sensor hardware abstraction layer (sensor HAL).

[0165] According to one embodiment, the sensor management module (623) can provide sensor data to the recognition plugin layer (622) through a common interface. For example, the sensor management module (623) can provide sensor data to each of a plurality of recognition modules through the same interface. For example, the sensor management module (623) can provide sensor data according to the operation of the recognition module to the recognition module without changing the configuration information of the recognition plugin layer (622) based on changing (or modifying) the configuration information (e.g., configuration file) regarding the sensor management module (623).

[0166] The sensor management module (623) can identify sensor data for at least one recognition module based on the operation of at least one recognition module among a plurality of recognition modules. The sensor management module (623) can provide the identified sensor data to the at least one recognition module.

[0167] According to one embodiment, when the head tracking module (622-1) is driven, the sensor management module (623) can acquire camera data and IMU data through at least one of a module for VR services, a module for XR services, a sensor API, and a sensor hardware abstraction layer in the sensor service layer (630). The sensor management module (623) can provide the camera data and IMU data to the head tracking module (622-1). According to an embodiment, the camera data and IMU data may be acquired through different modules.

[0168] According to one embodiment, when the environment recognition module (622-2) is operated in playback mode, the sensor management module (623) can identify stored camera data and stored pose data. The sensor management module (623) can provide the camera data and pose data to the environment recognition module (622-2).

[0169] According to one embodiment, the service module (621) may be configured to eliminate dependency on the upper layer of the recognition plugin layer (622). For example, the upper layer of the recognition plugin layer (622) may include a platform layer (610) (e.g., Android XR) and / or an application layer (e.g., the application layer (440) of FIG. 4).

[0170] The service module (621) can manage input data of the wearable device (101). The service module (621) can be configured to integrate and manage information (e.g., gesture information or tracking data) obtained from multiple recognition modules. The service module (621) can convert the information (e.g., gesture information or tracking data) according to the requirements of the upper layer without changing the multiple recognition modules, and then provide the converted information to the upper layer.

[0171] For example, a service module (621) can obtain information about a 6-degree-of-freedom attitude from a head tracking module (622-1). The information about the 6-degree-of-freedom attitude obtained from the head tracking module (622-1) can be configured in a quaternion format. On the other hand, an upper layer (e.g., platform layer (610)) can request information about the 6-degree-of-freedom attitude configured in an axis-angle representation format. The service module (621) can change (or convert) the information about the 6-degree-of-freedom attitude configured in a quaternion format into information about the 6-degree-of-freedom attitude configured in an axis-angle representation format. The service module (621) can provide information about the 6-degree-of-freedom attitude configured in an axis-angle representation format to the upper layer (e.g., platform layer (610)). However, it is not limited thereto. For example, the service module (621) can change (or convert) information about a 6-degree-of-freedom attitude configured in an axis-angle representation format into information about a 6-degree-of-freedom attitude configured in a quaternion format and provide it to the upper layer.

[0172] For example, a service module (621) can obtain information about the movement of the hand from a hand tracking module (622-3). Information about the movement of the hand can be obtained based on the movement of a first number of joints. On the other hand, an upper layer (e.g., platform layer (610)) can request information about the movement of the hand obtained based on the movement of a second number of joints. The service module (621) can perform either a joint interpolation procedure or a simplification procedure. Based on performing either a joint interpolation procedure or a simplification procedure, the service module (621) can support the structure of the joints required by the upper layer.

[0173] FIG. 7 is a flowchart illustrating the operation of a wearable device (101) according to one embodiment.

[0174] The operations illustrated in FIG. 7 can be performed by instructions stored in memory (e.g., memory (130) of FIG. 1). For example, when the instructions are executed by a processor (e.g., processor (120) of FIG. 1), the wearable device (101) (e.g., electronic device (101) of FIG. 1) can perform the operations illustrated in FIG. 7.

[0175] At least some of the operations shown in FIG. 7 may be omitted. At least some operations mentioned in the present disclosure with reference to other drawings may be additionally inserted before or after at least some of the operations shown in FIG. 7.

[0176] According to one embodiment, at least some of the operations illustrated in FIG. 7 may be performed sequentially. According to one embodiment, at least some of the operations illustrated in FIG. 7 may be performed in parallel (simultaneously). According to one embodiment, at least some of the operations illustrated in FIG. 7 may be performed with their order changed.

[0177] In operation 710, a wearable device (101) according to one embodiment can activate a shooting camera (e.g., the shooting camera (260-4) of FIG. 2b) (e.g., the first camera) and display an immersive space. According to one embodiment, the wearable device (101) can acquire a real-world image using the shooting camera (260-4). The wearable device (101) can analyze at least one object (e.g., 811 of FIG. 8) included in the real-world image using the shooting camera (260-4) and place the analyzed at least one object (e.g., 811 of FIG. 8) in an immersive space (e.g., 801 of FIG. 8) for augmented reality, virtual reality, or mixed reality combining augmented reality and virtual reality. For example, the wearable device (101) can perform a pass-through function of superimposing and displaying an image acquired through a shooting camera (260-4) on at least a part of the screen of the display.

[0178] In operation 720, a wearable device (101) according to one embodiment can sense illuminance using at least one sensor (e.g., sensor module (176) of FIG. 1). For example, the wearable device (101) can detect external illuminance while performing a pass-through function.

[0179] In operation 730, a wearable device (101) according to one embodiment may determine whether the detected illuminance is less than a first reference value. According to one embodiment, the wearable device (101) may perform operation 740 if the detected illuminance is less than the first reference value (e.g., the result of operation 730 is yes). According to one embodiment, the wearable device (101) may perform operation 720 if the detected illuminance is greater than or equal to the first reference value (e.g., the result of operation 720 is no). According to one embodiment, the wearable device (101) may set the first reference value based on user input.

[0180] In operation 740, a wearable device (101) according to one embodiment may activate an infrared camera (e.g., a second camera). According to one embodiment, the wearable device (101) may acquire an infrared image using the infrared camera. For example, the wearable device (101) may use the infrared camera to shine infrared light onto a real object (e.g., 811 in FIG. 8) and acquire a signal reflected from the real object (e.g., 811 in FIG. 8). According to one embodiment, the wearable device (101) may generate an infrared image based on the acquired signal. According to one embodiment, the infrared camera of the wearable device (101) may not be limited to using a method of shining infrared light onto a real object (e.g., 811 in FIG. 8) in acquiring an infrared image. For example, a wearable device (101) can use an infrared camera to detect the thermal energy of a real object (e.g., 811 in FIG. 8) and generate an infrared image based on the detected thermal energy.

[0181] In operation 750, a wearable device (101) according to one embodiment can detect a user's gaze (e.g., 803 in FIG. 8) using an eye-tracking camera (260-1) (e.g., the eye-tracking camera (260-1) in FIG. 2b).

[0182] In operation 760, a wearable device (101) according to one embodiment can generate a color for a gaze area (e.g., 810 in FIG. 8) corresponding to a user's gaze (e.g., 803 in FIG. 8). According to one embodiment, the wearable device (101) can determine the gaze area (810) based on the detected user's gaze (e.g., 803 in FIG. 8). For example, the wearable device (101) can determine a portion of the immersive space (e.g., 801 in FIG. 8) corresponding to the user's gaze (e.g., 803 in FIG. 8) in the immersive space being displayed (e.g., 801 in FIG. 8) as the gaze area (810).

[0183] According to one embodiment, when a gaze area (810) is determined, the wearable device (101) can generate a color for the gaze area (810) using an infrared image corresponding to the gaze area (810).

[0184] The wearable device (101) can collect surrounding information as much as possible using a camera (260-4) and determine the characteristics of a perceived object (e.g., 811 in FIG. 8) by limiting the user's gaze to a focused gaze area (810). Depending on the characteristics of the perceived object (e.g., 811 in FIG. 8), the wearable device (101) can generate a color of the gaze area (810) as if in a lighted situation by using an artificial intelligence model that has pre-trained data for various illuminances for each perceived object.

[0185] According to one embodiment, a wearable device (101) may use an artificial intelligence model to correct the color of an image corresponding to a gaze area (810). The wearable device (101) may input an infrared image corresponding to a gaze area (810) into an artificial intelligence model and cause the artificial intelligence model to generate the color of at least one object included in the gaze area (810). The artificial intelligence model of the wearable device (101) can correct the shape and color of an object by determining the characteristics of each object based on the shape information of each recognized object and applying a color suitable for each object to the object using previously learned data.

[0186] In operation 770, a wearable device (101) according to one embodiment may display a screen (e.g., 802 in FIG. 8) with a color applied to a gaze area (810) corresponding to the user's gaze. Thus, the wearable device (101) may provide an effect similar to a user turning on a light in a dark environment to see only the highlighted area.

[0187] According to one embodiment, the wearable device (101) can provide a corrected image for a gaze area (810) corresponding to the user's gaze when the user is in a situation where it is difficult to secure a view due to low light, darkness, fog, heavy rain, etc. For example, if the surrounding environment is nighttime, the wearable device (101) can correct the image as if looking at the gaze area (810) during the day. For example, if the surrounding environment is a dark indoor environment, the wearable device (101) can correct the image as if a bright light is illuminating the gaze area (810).

[0188] According to one embodiment, the wearable device (101) displays an immersive space (e.g., 802 in FIG. 8), but does not perform color correction on the entire area of ​​the immersive space (e.g., 802 in FIG. 8), and performs color correction only on a part of the area corresponding to the line of sight (810), so that the processing speed is fast and battery efficiency can be increased.

[0189] FIG. 8 is a conceptual diagram illustrating the visual effect provided by a wearable device (101) according to one embodiment.

[0190] Referring to FIG. 8, a user may wear a wearable device (101) in various situations and use the pass-through function provided by the wearable device (101). For example, the user may be in an environment where it is difficult to obtain a normal field of vision because the surroundings are dark, and in this case, the user may not be able to easily perceive real objects (811) through the pass-through function provided by the wearable device (101).

[0191] According to one embodiment, a wearable device (101) can sense illuminance and generate a color for a gaze area (810) in a low-light environment where the sensed illuminance is less than a first reference value. For example, the wearable device (101) can provide visual information in which at least one of the illuminance, color, and shape of an object is corrected only for a gaze area (810) corresponding to the direction of the user's gaze, rather than for the entire foreground (e.g., immersive space) that the user is looking at.

[0192] According to one embodiment, a wearable device (101) may acquire a first image using at least one infrared camera and display a second image (801) that includes at least a portion of the first image through a display. The wearable device (101) may acquire user gaze information (e.g., user gaze (803)) as a first user input and select at least a portion of the second image (801) as a gaze area (810) based on the user gaze (803). The wearable device (101) may add a visual effect to the selected gaze area (810) in the second image (801). The visual effect may include an image in which at least one of the illuminance, color, and shape of a real object (811) is corrected only for the gaze area (810) currently being viewed by the user.

[0193] According to one embodiment, when the wearable device (101) detects that a user is wearing the wearable device (101), it can automatically measure the illuminance using an illuminance sensor, and if the measured illuminance is less than a first reference value, it can determine the gaze area (810) using the user's gaze information. The wearable device (101) can crop the gaze area (810) in the immersive space (801) and add visual effects to the cropped area using an artificial intelligence model. The wearable device (101) can increase processing speed and battery efficiency by providing a screen with visual effects applied only to the gaze area (810) rather than the entire immersive space (801, 802).

[0194] FIG. 9 is a flowchart illustrating the operation of a wearable device (101) according to one embodiment providing a light source effect.

[0195] The operations illustrated in FIG. 9 can be performed by instructions stored in memory (e.g., memory (130) of FIG. 1). For example, when the instructions are executed by a processor (e.g., processor (120) of FIG. 1), the wearable device (101) (e.g., electronic device (101) of FIG. 1) can perform the operations illustrated in FIG. 9.

[0196] At least some of the operations shown in FIG. 9 may be omitted. At least some operations mentioned in the present disclosure with reference to other drawings may be additionally inserted before or after at least some of the operations shown in FIG. 9.

[0197] According to one embodiment, at least some of the operations illustrated in FIG. 9 may be performed sequentially. According to one embodiment, at least some of the operations illustrated in FIG. 9 may be performed in parallel (simultaneously). According to one embodiment, at least some of the operations illustrated in FIG. 9 may be performed with their order changed.

[0198] In operation 910, a wearable device (101) according to one embodiment can detect a user's gaze (e.g., user's gaze (803) of FIG. 8). Operation 910 may be at least partially similar or substantially identical to operation 750 described with reference to FIG. 7. For example, the wearable device (101) can detect the user's gaze (803) using an eye-tracking camera (e.g., eye-tracking camera (260-1) of FIG. 2b).

[0199] In operation 920, a wearable device (101) according to one embodiment may crop a gaze area (e.g., gaze area (810) of FIG. 8) corresponding to the user's gaze (803) in a real-world image acquired. For example, the wearable device (101) may determine the gaze area (810) based on the detected user's gaze (803). The wearable device (101) may crop a portion of the immersive space (801, 802) corresponding to the user's gaze (803) in an immersive space (e.g., immersive space (801, 802) of FIG. 8) (e.g., real-world image) being displayed as a gaze area (810).

[0200] In operation 930, a wearable device (101) according to one embodiment can generate and apply a color for a gaze area (810). Operation 930 may be at least partially similar or substantially identical to operation 760 described with reference to FIG. 7. For example, when the gaze area (810) is determined, the wearable device (101) can generate a color for the gaze area (810) using an infrared image corresponding to the gaze area (810). The wearable device (101) can collect surrounding information using a camera (e.g., camera (260-4) in FIG. 2b) and determine the characteristics of a perceived object limited to the gaze area (810) where the user's gaze (803) is focused. Depending on the characteristics of the perceived object, the wearable device (101) can generate a color for the gaze area (810) as if in a lighted situation by using an artificial intelligence model that has data learned for various illuminances for each perceived object.

[0201] In operation 940, a wearable device (101) according to one embodiment can determine the location of a virtual light source and generate a light source effect (1042) that illuminates a line of sight area (810) from the virtual light source. For example, if the surrounding environment is a dark indoor space, the wearable device (101) can add a light source effect (1042) as if a bright light were illuminating the line of sight area (810). The wearable device (101) can determine the location of a virtual light source (e.g., virtual light source (1041) of FIG. 10) based on an image (e.g., second image) acquired through a shooting camera (260-4) or user input. The wearable device (101) can generate a light source effect (e.g., light source effect (1042) of FIG. 10) that illuminates a line of sight area (810) from the determined virtual light source (1041). The light source effect (1042) may include, for example, an effect in which light is emitted from a virtual light source (1041) and the emitted light illuminates a line of sight area (810), but the present invention is not limited thereto.

[0202] In operation 950, a wearable device (101) according to one embodiment may display a screen in which a color-generated gaze area (810) and a light source effect (1042) are combined. The wearable device (101) may display a screen in which a color generated in the gaze area (810) corresponding to the user's gaze (803) is applied, and may provide a light source effect (1042) that illuminates the gaze area (810). Thus, the wearable device (101) may provide an effect similar to a user turning on a light in a dark environment to see only the highlighted area.

[0203] According to one embodiment, when the wearable device (101) detects that the user's gaze (803) is moving, it can move the area to which the light source effect (1042) is applied according to the movement of the user's gaze (803). For example, the wearable device (101) can change the color-corrected area according to the movement of the user's gaze (803), and can move the area to which the light source effect (1042) is applied to correspond to the changed area.

[0204] FIG. 10 is a conceptual diagram for explaining a light source effect (1042) provided by a wearable device (101) according to one embodiment. For example, FIG. 10 is a conceptual diagram for explaining a light source effect (1042) described with reference to FIG. 9.

[0205] Operation 1010 of FIG. 10 may be related to operations 910 and 920 described with reference to FIG. 9. With reference to operation 1010 of FIG. 10, a wearable device (101) according to one embodiment may detect a user's gaze (803) and crop a gaze area (810) corresponding to the user's gaze (803) in a real image (e.g., an immersive space (801)).

[0206] Operation 1020 of FIG. 10 may be related to operation 930 described with reference to FIG. 9. Referring to operation 1020 of FIG. 10, a wearable device (101) according to one embodiment may crop a gaze area (810) corresponding to the user's gaze and input the cropped image (811) into an artificial intelligence model (1021). The artificial intelligence model (1021) may be a generative artificial intelligence model (1021). The artificial intelligence model (1021) may recognize at least one object included in the input image (811) and determine the characteristics of each object based on the shape information of each recognized object. The artificial intelligence model (1021) may generate a corrected image (812) that corrects the shape and color of an object by applying a color suitable for each object to the object using previously learned data. The artificial intelligence model (1021) can output a generated corrected image (812), and the corrected image (812) can be synthesized into a part of a screen displayed through the display of the wearable device (101).

[0207] Operation 1030 of FIG. 10 may be related to operation 930 described with reference to FIG. 9. With reference to operation 1030 of FIG. 10, a wearable device (101) according to one embodiment may generate and apply a color to a gaze area (810). For example, the wearable device (101) may display an image (e.g., an immersive space (802)) in which a corrected image (812) generated by an artificial intelligence model (1021) is combined with at least a portion of a real image.

[0208] Operation 1040 of FIG. 10 may be related to operations 940 and 950 described with reference to FIG. 9. Referring to operation 1040 of FIG. 10, a wearable device (101) according to one embodiment may determine the position of a virtual light source (1041) and generate a light source effect (1042) that illuminates a line of sight area (810) from the virtual light source (1041). For example, if the surrounding environment is a dark indoor space, the wearable device (101) may add a light source effect (1042) as if a bright light were illuminating the line of sight area (810). The light source effect (1042) may include, for example, an effect in which light is output from the virtual light source (1041) and the output light illuminates the line of sight area (810), but the present invention is not limited thereto.

[0209] FIG. 11 is a flowchart illustrating the operation of a wearable device (101) according to one embodiment applying a visual effect according to a light source object.

[0210] The operations illustrated in FIG. 11 can be performed by instructions stored in memory (e.g., memory (130) of FIG. 1). For example, when the instructions are executed by a processor (e.g., processor (120) of FIG. 1), the wearable device (101) (e.g., electronic device (101) of FIG. 1) can perform the operations illustrated in FIG. 11.

[0211] At least some of the operations shown in FIG. 11 may be omitted. At least some operations mentioned in the present disclosure with reference to other drawings may be additionally inserted before or after at least some of the operations shown in FIG. 11.

[0212] According to one embodiment, at least some of the operations illustrated in FIG. 11 may be performed sequentially. According to one embodiment, at least some of the operations illustrated in FIG. 11 may be performed in parallel (simultaneously). According to one embodiment, at least some of the operations illustrated in FIG. 11 may be performed with their order changed.

[0213] In operation 750, a wearable device (101) according to one embodiment may detect a user's gaze (e.g., user's gaze (803) of FIG. 8) using an eye-tracking camera (e.g., the eye-tracking camera (260-1) of FIG. 2b). Operation 750 of FIG. 11 may be at least partially similar or substantially identical to operation 750 described with reference to FIG. 7. For example, the wearable device (101) may perform operation 1110 after performing operation 750 described with reference to FIG. 7.

[0214] In operation 1110, a wearable device (101) according to one embodiment can determine whether a light source object exists in a line of sight area (810). For example, the light source object may include a lighting device (e.g., the lighting device (1210) of FIG. 12), and the wearable device (101) can determine whether the lighting device (1210) is located in the line of sight area (e.g., the line of sight area (810) of FIG. 12). According to one embodiment, the wearable device (101) can perform operation 1120 if a light source object exists in the line of sight area (810) (e.g., the result of operation 1110 is yes). According to one embodiment, the wearable device (101) can perform operation 760 if a light source object does not exist in the line of sight area (810) (e.g., the result of operation 1110 is no).

[0215] In operation 760, a wearable device (101) according to one embodiment can generate a color for a gaze area (810) corresponding to the user's gaze (803). Operation 760 of FIG. 11 may be at least partially similar or substantially identical to operation 760 described with reference to FIG. 7.

[0216] In operation 1120, a wearable device (101) according to one embodiment can determine whether there is one light source object detected in the line of sight area (810). For example, a wearable device (101) according to one embodiment can detect one light source object in the line of sight area (810). For example, the wearable device (101) can determine that there is one lighting device (1210) in the line of sight area (810). For example, a wearable device (101) according to one embodiment can detect multiple light source objects in the line of sight area (810). For example, the wearable device (101) can determine that there are multiple lighting devices (1210) in the line of sight area (810).

[0217] A wearable device (101) according to one embodiment may perform operation 1131 when there is only one light source object detected in the line of sight area (810) (e.g., the result of operation 1120 is yes). A wearable device (101) according to one embodiment may perform operation 1132 when there are multiple light source objects detected in the line of sight area (810) (e.g., the result of operation 1120 is no).

[0218] In operation 1131, a wearable device (101) according to one embodiment can determine the type of light source object and the direction of the light source. For example, the type of light source object may refer to the type of lighting device (1210). For example, the direction of the light source may refer to the direction in which light is output from the light source of the lighting device (1210).

[0219] In operation 1132, a wearable device (101) according to one embodiment may select fewer than a specified number of light source objects among a plurality of light source objects. For example, when a plurality of light source objects exist in a line of sight area (810), the wearable device (101) may not recognize all of the plurality of light source objects, but may recognize only some of the light source objects among the plurality of light source objects. According to one embodiment, the wearable device (101) may detect the size of each of the plurality of light source objects and select fewer than a specified number of light source objects according to the order of size. For example, the wearable device (101) may select the top N light source objects that are relatively larger in size among the plurality of light source objects.

[0220] In operation 1140, a wearable device (101) according to one embodiment can generate a lighting effect (e.g., 812 in FIG. 12) in which a light source object is lit. For example, although the light source object is not lit in the real world, the wearable device (101) can generate color for the light source object and the surrounding light source object as if the light source object were lit. The wearable device (101) can generate a lighting effect (e.g., 812 in FIG. 12) depending on the type of light source object and the direction of the light source. If there are multiple light source objects, the wearable device (101) can select the top N light source objects that are relatively larger in size among the multiple light source objects and generate a lighting effect (e.g., 812 in FIG. 12) in which the selected light source objects are lit.

[0221] In operation 1150, a wearable device (101) according to one embodiment may display a screen (e.g., 803 of FIG. 12) in which a color-generated viewing area (810) and a lighting effect (e.g., 812 of FIG. 12) are combined.

[0222] FIG. 12 is a conceptual diagram for explaining user interaction provided by a wearable device (101) according to one embodiment.

[0223] Referring to FIG. 12, a user may wear a wearable device (101) in various situations and use the pass-through function provided by the wearable device (101). For example, the user may be in an environment where it is difficult to obtain a normal field of vision because the surroundings are dark, and in this case, the user may not be able to easily perceive a real object (811) through the pass-through function provided by the wearable device (101).

[0224] According to one embodiment, a wearable device (101) can sense illuminance and generate a color for a gaze area (810) in a low-light environment where the sensed illuminance is less than a first reference value. For example, the wearable device (101) can provide visual information in which at least one of the illuminance, color, and shape of an object is corrected only for a gaze area (810) based on the user's gaze direction, rather than for the entire foreground (e.g., immersive space (801)) that the user is looking at.

[0225] According to one embodiment, a wearable device (101) can detect the presence of a light source object (e.g., a lighting device (1210)) in a line of sight area (810). If the light source object is present, the wearable device (101) can generate a lighting effect (812). For example, even though the light source object is not lit in the real world, the wearable device (101) can generate color for the light source object and the area around the light source object as if the light source object were lit. The wearable device (101) can generate a lighting effect (812) depending on the type of light source object and the direction of the light source.

[0226] According to one embodiment, when a wearable device (101) detects a light source object (e.g., a lighting device (1210)) that is turned off or has no light activated in a line of sight area (810), it can determine the characteristics, direction, and light intensity of the light source object using a pre-trained artificial intelligence model. The wearable device (101) can determine a virtual light intensity, color, or light spreading degree according to lighting characteristics appropriate to the type of light source object analyzed using the artificial intelligence model, and provide an image correction effect (i.e., a lighting effect (1210)) in which the light source of the light source object is activated.

[0227] According to one embodiment, a wearable device (101) can detect user input (1220) for a gaze area (810). The user input (1220) may include, for example, a specified user gesture for the gaze area (810). The user gesture may be a user gesture that controls a light source object (e.g., a lighting device (1210)). The user gesture may be information obtained using a motion recognition camera (e.g., the motion recognition camera (260-2, 206-3) of FIG. 2b).

[0228] According to one embodiment, the wearable device (101) can adjust a lighting effect (812) based on user input (1220) controlling a light source object (e.g., a lighting device (1210)). For example, the wearable device (101) can provide a dimming effect (813), such as adjusting the brightness of the light source object based on a user gesture. According to one embodiment, the wearable device (101) can enable or disable the lighting effect (812) of the light source object based on a user gesture.

[0229] According to one embodiment, the wearable device (101) can provide a virtual on-off function for a light source object through user interaction such as operation or touch interaction at an interface to the detected light source object (e.g., lighting device (1210)), and can provide a function to change the amount of light of the light source object virtually.

[0230] FIG. 13 is a flowchart illustrating the operation of a wearable device (101) according to one embodiment in a low-light environment.

[0231] The operations illustrated in FIG. 13 can be performed by instructions stored in memory (e.g., memory (130) of FIG. 1). For example, when the instructions are executed by a processor (e.g., processor (120) of FIG. 1), the wearable device (101) (e.g., electronic device (101) of FIG. 1) can perform the operations illustrated in FIG. 13.

[0232] At least some of the operations shown in FIG. 13 may be omitted. At least some operations mentioned in the present disclosure with reference to other drawings may be additionally inserted before or after at least some of the operations shown in FIG. 13.

[0233] According to one embodiment, at least some of the operations illustrated in FIG. 13 may be performed sequentially. According to one embodiment, at least some of the operations illustrated in FIG. 13 may be performed in parallel (simultaneously). According to one embodiment, at least some of the operations illustrated in FIG. 13 may be performed with their order changed.

[0234] In operation 720, a wearable device (101) according to one embodiment may sense illuminance using at least one sensor (e.g., sensor module (176) of FIG. 1). Operation 720 of FIG. 13 may be at least partially similar or substantially identical to operation 720 described with reference to FIG. 7.

[0235] In operation 730, a wearable device (101) according to one embodiment may determine whether the detected illuminance is less than a first reference value. According to one embodiment, the wearable device (101) may perform operation 740 if the detected illuminance is less than the first reference value (e.g., the result of operation 730 is yes). According to one embodiment, the wearable device (101) may perform operation 1310 if the detected illuminance is greater than or equal to the first reference value (e.g., the result of operation 730 is no). Operation 730 of FIG. 13 may be at least partially similar or substantially identical to operation 730 described with reference to FIG. 7.

[0236] In operation 740, a wearable device (101) according to one embodiment may activate an infrared camera (e.g., a second camera). Operation 740 of FIG. 13 may be at least partially similar or substantially identical to operation 740 described with reference to FIG. 7.

[0237] In operation 1310, a wearable device (101) according to one embodiment can determine whether the recognition rate for a real object (e.g., a real object (811) of FIG. 8) is less than a second reference value. The recognition rate for a real object (811) may be an indicator indicating whether a user can easily distinguish the real object (811). Situations in which the recognition rate for a real object (811) is low may include the following situations. For example, situations in which the recognition rate for a real object (811) is low may include a situation where there is thick fog but it is not dark, a situation where there is heavy snow or rain but it is not dark, or a situation where visibility is secured or a specific area is dark and difficult to see. According to one embodiment, the wearable device (101) may acquire a real image through a shooting camera (e.g., a shooting camera (260-4) of FIG. 2b) and input the real image into an artificial intelligence model to determine the recognition rate for a real object (811). For example, the artificial intelligence model can determine the recognition rate of real objects (811) by analyzing the input real-world image.

[0238] According to one embodiment, the wearable device (101) according to the embodiment may perform operation 1320 if the recognition rate for a real object (811) is less than a second reference value (e.g., the result of operation 1310 is yes). According to one embodiment, the wearable device (101) according to the embodiment may perform operation 720 if the recognition rate for a real object (811) is greater than or equal to the second reference value (e.g., the result of operation 1310 is no).

[0239] In operation 1320, a wearable device (101) according to one embodiment may simultaneously activate an infrared camera and a shooting camera (e.g., the shooting camera (260-4) of FIG. 2b). For example, if the recognition rate of a real object (811) is lower than a second reference value, the wearable device (101) may simultaneously activate the infrared camera and the shooting camera (260-4) to acquire an infrared image and a real image.

[0240] According to one embodiment, a wearable device (101) inputs an infrared image and a real-world image into an artificial intelligence model, and the artificial intelligence model analyzes the infrared image and the real-world image to correct the image for the gaze area (810).

[0241] According to one embodiment, the wearable device (101) can correct the image by inputting an infrared image and a real image into an artificial intelligence model so that the line of sight area (810) can be seen clearly when the real image is in a foggy situation.

[0242] According to one embodiment, the wearable device (101) can separate a first layer of an object corresponding to heavy rain or heavy snow that obstructs the user's gaze (803) and a second layer excluding the first layer when the real-world image is in a situation of heavy rain or heavy snow. According to one embodiment, the wearable device (101) can use an artificial intelligence model to correct the image so that the first layer of an object corresponding to heavy rain or heavy snow appears translucent or disappears.

[0243] In operation 750, a wearable device (101) according to one embodiment may detect a user's gaze (803) using an eye-tracking camera (e.g., the eye-tracking camera (260-1) of FIG. 2b). Operation 750 of FIG. 13 may be at least partially similar or substantially identical to operation 750 described with reference to FIG. 7. For example, the wearable device (101) may perform operation 760 of FIG. 7 after performing operation 750 of FIG. 13.

[0244] FIG. 14 is a flowchart illustrating the operation of a wearable device (101) according to one embodiment based on the time the user's gaze (803) stays.

[0245] The operations illustrated in FIG. 14 can be performed by instructions stored in memory (e.g., memory (130) of FIG. 1). For example, when the instructions are executed by a processor (e.g., processor (120) of FIG. 1), the wearable device (101) (e.g., electronic device (101) of FIG. 1) can perform the operations illustrated in FIG. 14.

[0246] At least some of the operations shown in FIG. 14 may be omitted. At least some operations mentioned in the present disclosure with reference to other drawings may be additionally inserted before or after at least some of the operations shown in FIG. 14.

[0247] According to one embodiment, at least some of the operations illustrated in FIG. 14 may be performed sequentially. According to one embodiment, at least some of the operations illustrated in FIG. 14 may be performed in parallel (simultaneously). According to one embodiment, at least some of the operations illustrated in FIG. 14 may be performed with their order changed.

[0248] In operation 770, a wearable device (101) according to one embodiment may display a screen with a color generated by an artificial intelligence model applied to a gaze area (e.g., gaze area (810) of FIG. 8) corresponding to the user's gaze (e.g., user's gaze (803) of FIG. 8). Operation 770 of FIG. 14 may be at least partially similar or substantially identical to operation 770 described with reference to FIG. 7. For example, the wearable device (101) may perform operation 1410 after performing operation 770 described with reference to FIG. 7.

[0249] In operation 1410, a wearable device (101) according to one embodiment can count the time the user’s gaze (803) stays. For example, the wearable device (101) can count the time the user’s gaze (803) stays while displaying a color-generated image in a gaze area (810) corresponding to the user’s gaze (803).

[0250] In operation 1420, a wearable device (101) according to one embodiment can confirm that the counted time exceeds a third reference value. For example, the wearable device (101) can confirm that the user's gaze (803) is fixed on the gaze area (810) for a certain period of time or longer.

[0251] In operation 1430, a wearable device (101) according to one embodiment may expand the gaze area (810) and display a screen with color applied to the expanded gaze area (810). According to one embodiment, the wearable device (101) may consider that if the user's gaze (803) is fixed on the gaze area (810) for a certain period of time or longer, the user has a need to see the information and related visual information around it in more detail. If the user's gaze (803) is fixed on the gaze area (810) for a certain period of time or longer, the wearable device (101) may expand and display the gaze area (810) with color applied. According to one embodiment, when the wearable device (101) receives user interaction such as a user gesture, it may provide a user experience of widening or narrowing the size of the gaze area (810).

[0252] FIG. 15 is a flowchart illustrating the operation of a wearable device (101) according to one embodiment according to the movement of the user's gaze (803).

[0253] The operations illustrated in FIG. 15 can be performed by instructions stored in memory (e.g., memory (130) of FIG. 1). For example, when the instructions are executed by a processor (e.g., processor (120) of FIG. 1), the wearable device (101) (e.g., electronic device (101) of FIG. 1) can perform the operations illustrated in FIG. 15.

[0254] At least some of the operations shown in FIG. 15 may be omitted. At least some operations mentioned in the present disclosure with reference to other drawings may be additionally inserted before or after at least some of the operations shown in FIG. 15.

[0255] According to one embodiment, at least some of the operations illustrated in FIG. 15 may be performed sequentially. According to one embodiment, at least some of the operations illustrated in FIG. 15 may be performed in parallel (simultaneously). According to one embodiment, at least some of the operations illustrated in FIG. 15 may be performed with their order changed.

[0256] In operation 770, a wearable device (101) according to one embodiment may display a screen with a color generated by an artificial intelligence model applied to a gaze area (e.g., gaze area (810) of FIG. 8) corresponding to the user's gaze (e.g., user's gaze (803) of FIG. 8). Operation 770 of FIG. 15 may be at least partially similar or substantially identical to operation 770 described with reference to FIG. 7. For example, the wearable device (101) may perform operation 1510 after performing operation 770 described with reference to FIG. 7.

[0257] In operation 1510, a wearable device (101) according to one embodiment can confirm that the user's gaze (803) is sustained in an initial first gaze area (e.g., 810A in FIG. 16). For example, the wearable device (101) can confirm that the user's gaze (803) is fixed in the initial first gaze area (810A) for a certain period of time or longer.

[0258] In operation 1520, a wearable device (101) according to one embodiment can detect that a user's gaze (803) moves from an initial first gaze area (810A) to a final third gaze area (e.g., the third gaze area (810E) of FIG. 16) via at least one second gaze area (e.g., the second gaze areas (810B, 810C, 810D) of FIG. 16). For example, at least one second gaze area may be named as at least one intermediate area.

[0259] In operation 1530, a wearable device (101) according to one embodiment can check whether the movement speed of the user's gaze (803) exceeds a fourth reference value. For example, the wearable device (101) can calculate the movement speed of the user's gaze (803) when the user's gaze (803) moves. The wearable device (101) can consider that the user's gaze (803) is moving fast if the movement speed of the user's gaze (803) exceeds the fourth reference value. The wearable device (101) according to one embodiment can perform operation 1541 if the movement speed of the user's gaze (803) exceeds the fourth reference value (e.g., the result of operation 1530 is an example).

[0260] A wearable device (101) according to one embodiment can confirm that the movement speed of a user's gaze (e.g., 803A, 803B, 803C, 803D, 803E in FIG. 16) is less than or equal to a fourth reference value. For example, the wearable device (101) can calculate the movement speed of the user's gaze when the user's gaze moves. The wearable device (101) can consider that the user's gaze is moving slowly if the movement speed of the user's gaze is less than or equal to the fourth reference value. A wearable device (101) according to one embodiment can perform operation 1542 if the movement speed of the user's gaze (803) is less than or equal to the fourth reference value (e.g., the result of operation 1530 is not).

[0261] In operation 1541, a wearable device (101) according to one embodiment may switch directly from a screen with color applied to a first line of sight area (810A) (e.g., 812A of FIG. 16) to a screen with color applied to a third line of sight area (810) (e.g., 812E of FIG. 16). For example, the wearable device (101) may not display a screen with color applied (e.g., 812B, 812C, 812D of FIG. 16) for at least one second line of sight area (e.g., 810B, 810C, 810D of FIG. 16) located between the first line of sight area (810A) and the third line of sight area (810E).

[0262] According to one embodiment, the wearable device (101) can perform an efficient processing method tailored to the speed of movement of the user's gaze (e.g., 803A, 803B, 803C, 803D, 803E in FIG. 16). For example, when the user's gaze moves quickly, the wearable device (101) can omit the generation of color images (e.g., 812B, 812C, 812D in FIG. 16) for the intermediate area of ​​the moving range (e.g., 810B, 810C, 810D in FIG. 16) so that it does not appear as an afterimage.

[0263] In operation 1542, a wearable device (101) according to one embodiment can sequentially switch screens from a screen with color applied to a first line of sight area (810) (e.g., 812A of FIG. 16) to a screen with color applied to at least one second line of sight area (810) (e.g., 812B, 812C, 812D of FIG. 16) and a screen with color applied to a third line of sight area (810) (e.g., 812E of FIG. 16).

[0264] According to one embodiment, the wearable device (101) can perform an efficient processing method tailored to the speed of movement of the user's gaze (e.g., 803A, 803B, 803C, 803D, 803E in FIG. 16). For example, when the user's gaze moves slowly, the wearable device (101) can provide a afterimage effect that gradually weakens the intensity of the color applied to the previous gaze area according to the movement of the gaze.

[0265] FIG. 16 is a flowchart illustrating the visual effects provided by the wearable device (101) according to the movement of the user's gaze (803).

[0266] Referring to FIG. 16, a user may wear a wearable device (101) in various situations and use the pass-through function provided by the wearable device (101). For example, the user may be in an environment where it is difficult to obtain a normal field of vision because the surroundings are dark, and in this case, the user may not be able to easily perceive a real object (811) through the pass-through function provided by the wearable device (101).

[0267] According to one embodiment, a wearable device (101) can sense illuminance and generate a color for a line of sight area (810) in a low-light environment where the sensed illuminance is less than a first reference value.

[0268] According to one embodiment, the wearable device (101) can detect the first gaze (803A) of a user looking at area A as a first gaze area (810A).

[0269] According to one embodiment, a wearable device (101) can detect a user's second gaze (803B) looking at area B as at least one intermediate area. The user's second gaze (803B) may be moved from the user's first gaze (803A).

[0270] According to one embodiment, the wearable device (101) can detect a user's third gaze (803C) looking at area C as at least one intermediate area. The user's third gaze (803C) may be moved from the user's second gaze (803B).

[0271] According to one embodiment, the wearable device (101) can detect a user's fourth gaze (803D) looking at area D as at least one intermediate area. The user's fourth gaze (803D) may be moved from the user's third gaze (803C).

[0272] According to one embodiment, the wearable device (101) can detect a user's fifth gaze (803E) looking at area E as a third gaze area (810E). The user's fifth gaze (803E) may be moved from the user's fourth gaze (803D).

[0273] According to one embodiment, the wearable device (101) can perform an efficient processing method tailored to the movement speed of the user's gaze (803A, 803B, 803C, 803D, 803E). For example, when the user's gaze (803A, 803B, 803C, 803D, 803E) moves quickly, the wearable device (101) can omit the generation of a color image for an intermediate area of ​​the moving range so that it does not appear as an afterimage. If the user's gaze (803A, 803B, 803C, 803D, 803E) moves from the first gaze area (810A) to the last third gaze area (810E) via at least one second gaze area (810B, 810C, 810D), the wearable device (101) can immediately switch from a color screen (812A) in the first gaze area (810A) to a color screen (812E) in the third gaze area (810E). The wearable device (101) may not display a color screen for at least one second gaze area (810B, 810C, 810D) located between the first gaze area (810A) and the third gaze area (810E).

[0274] According to one embodiment, the wearable device (101) can perform an efficient processing method tailored to the movement speed of the user's gaze (803A, 803B, 803C, 803D, 803E). For example, when the user's gaze (803A, 803B, 803C, 803D, 803E) moves slowly, the wearable device (101) can provide a residual effect that gradually weakens the intensity of the color applied to the previous gaze area according to the movement of the gaze. If the user's gaze (803A, 803B, 803C, 803D, 803E) moves from the first gaze area (810A) to the last third gaze area (810E) via at least one second gaze area (810B, 810C, 810D), the wearable device (101) can sequentially switch screens from a screen (812A) with color applied to the first gaze area (810A) to a screen (812B, 812C, 812D) with color applied to at least one second gaze area (810B, 810C, 810D) and a screen (812E) with color applied to the third gaze area (810E).

[0275] FIG. 17 is a flowchart illustrating the operation of a wearable device (101) according to one embodiment.

[0276] The operations illustrated in FIG. 17 can be performed by instructions stored in memory (e.g., memory (130) of FIG. 1). For example, when the instructions are executed by a processor (e.g., processor (120) of FIG. 1), the wearable device (101) (e.g., electronic device (101) of FIG. 1) can perform the operations illustrated in FIG. 17.

[0277] At least some of the operations shown in FIG. 17 may be omitted. At least some operations mentioned in the present disclosure with reference to other drawings may be additionally inserted before or after at least some of the operations shown in FIG. 17.

[0278] According to one embodiment, at least some of the operations illustrated in FIG. 17 may be performed sequentially. According to one embodiment, at least some of the operations illustrated in FIG. 17 may be performed in parallel (simultaneously). According to one embodiment, at least some of the operations illustrated in FIG. 17 may be performed with their order changed.

[0279] In operation 1710, a wearable device (101) according to one embodiment can acquire a first image using at least one infrared camera. The wearable device (101) can acquire a first image using at least one infrared camera when it confirms that the illuminance acquired using at least one sensor is smaller than a first reference value.

[0280] In operation 1720, a wearable device (101) according to one embodiment may display a second image (e.g., an immersive space (801, 802) of FIG. 8) that includes at least a portion of a first image through a display.

[0281] In operation 1730, a wearable device (101) according to one embodiment may select at least a portion of a second image based on a first user input. According to one embodiment, the second image may be an image generated using a first image and a third image (e.g., a real image) obtained using at least one shooting camera (e.g., the shooting camera (260-4) of FIG. 2b).

[0282] According to one embodiment, the first user input may include user gaze information (e.g., user gaze (803) of FIG. 8) obtained using an eye-tracking camera (e.g., eye-tracking camera (260-1) of FIG. 2b). The user gaze information may include at least some of first information indicating the location where the user's gaze (803) is directed, or second information indicating the time the user's gaze (803) stays.

[0283] According to one embodiment, the first user input may include a user gesture. The user gesture may be information obtained using a motion recognition camera (e.g., the motion recognition camera (260-2, 206-3) of FIG. 2b).

[0284] In operation 1740, a wearable device (101) according to one embodiment can add visual effects to a selected portion of the second image.

[0285] According to one embodiment, the visual effect may include an effect that changes at least some of the color type, saturation, brightness, or luminance for a selected portion of the second image.

[0286] According to one embodiment, the visual effect may include an effect that gradually increases the intensity of color information for a selected portion of the second image depending on the time the user's gaze (803) stays.

[0287] According to one embodiment, the visual effect may include an effect that changes the size of a selected portion of the second image depending on the time the user's gaze (803) stays.

[0288] According to one embodiment, the visual effect may include an effect that increases the size of a selected portion of the second image in proportion to the time the user's gaze (803) stays.

[0289] According to one embodiment, the wearable device (101) can divide a selected portion of the second image into a central area and an outer area surrounding the central area, and the intensity of the visual effect can be reduced as it goes from the central area to the outer area.

[0290] According to one embodiment, the wearable device (101) may divide a selected portion of a second image into a first portion area and a second portion area, apply a first visual effect to the first portion area, and apply a second visual effect to the second portion area.

[0291] According to one embodiment, the wearable device (101) can detect a second user input while providing a visual effect. The wearable device (101) can change a selected portion of the second image based on the second user input. For example, the previously entered first user input may be a multimodal input including the user's gaze (803) and user gestures. For example, the second user input detected by the wearable device (101) while providing a visual effect may include the user's gaze (803).

[0292] According to one embodiment, the wearable device (101) can change color information for a previously selected area when changing a selected area in a second image based on a second user input.

[0293] According to one embodiment, the wearable device (101) can determine the position of a virtual light source (1041) in the second image based on the second image or a third user input. The wearable device (101) can additionally display a light source effect (1042) that illuminates a selected area in the second image from the determined virtual light source (1041). The wearable device (101) can move the area to which the light source effect (1042) is applied when the selected area in the second image moves.

[0294] FIG. 18 is a block diagram illustrating a generative artificial intelligence system according to one embodiment.

[0295] Referring to FIG. 18, a generative artificial intelligence system according to one embodiment (e.g., server (108) of FIG. 1) may include a user interface (1860), a database (1865), an application and service component (1870), an artificial intelligence framework (1880), and a generative AI model (1890) (e.g., artificial intelligence model (1021) of FIG. 10). According to one embodiment, the generative artificial intelligence system may be included in an electronic device (e.g., electronic device (101) of FIG. 1) (e.g., AI engine) or / or in an intelligent server (e.g., an external server (e.g., server (108) of FIG. 1).

[0296] The user interface (1860) can receive user queries.

[0297] The above input may include user input and / or data obtained or generated by an electronic device (e.g., the electronic device (101) of FIG. 1).

[0298] The above data may include images, videos, and / or sensor data generated by at least one processor of the electronic device (101) (e.g., processor (120) of FIG. 1).

[0299] The sensor data may include, for example, illuminance data around the electronic device (101) obtained from the sensor module (176) or sensor hub of FIG. 1, attitude data (or orientation data) of the electronic device (101), temperature inside the electronic device (101) (e.g., temperature of the display module (160) of FIG. 1 and / or temperature of the processor (120)), size information of the display area of ​​the display module (160), and / or an image obtained through the image sensor of the electronic device (101) (e.g., camera module (180) of FIG. 1).

[0300] The above user query may be in the form of natural language, touch data obtained through a touch circuit included in the display module (160) (e.g., used to identify input from a finger and / or stylus), an image, audio, and / or video. Additionally, context information may be transmitted along with the user query. The context information may include various side information related to the time when the user query is input into the generative artificial intelligence system. For example, the side information may include information such as application information currently being used by the user and / or location information of the user. As another example, the user query may also be a non-natural language input that does not generate natural language, such as a design request or modification. Additionally, a mixed form of the natural language, image, sound, and context information described above is also possible.

[0301] The user interface (1860) can output results of the generative artificial intelligence system to the user. The output may include results (or result information) generated or obtained by the generative artificial intelligence system based on at least part of the input. The output may be in the form of natural language or specific content, and may also be provided in the form of an action requested by the user. For example, the output may have a format according to the user settings of the electronic device (101).

[0302] The AI ​​framework (1880) can receive user queries and coordinate and control each component necessary to perform the user's intent. This AI framework (1880) may include a prompt design component (1881), an APIs / Plugins Management component (1883), and an output modification component (1885).

[0303] User queries or actions entered in the user interface (1860) can be transmitted to a prompt design component (1881). The prompt design component (1881) can be used to generate prompts suitable for input into a large language model (LLM), a large vision model (LVM), or a large multimodal model (LMM). The prompt design component (1881) may be an AI component that uses machine learning algorithms or neural networks to develop better prompts over time. The prompt design component (1881) can generate prompts by accessing a database (1865) (e.g., a knowledge component) containing user preference data, a prompt library, and prompt examples, and can transmit them to the large language model (LLM), the large vision model (LVM), and / or the large multimodal model (LMM).

[0304] The application and plugin management component (1883) can perform the role of communicating with external information when there is a request for additional information when user input is passed as input to the generative AI model (1890). The application and plugin management component (1883) establishes a channel to communicate with the outside of the generative artificial intelligence system through an application programming interface (API), thereby enabling access to various data sources. For example, the application and plugin management component (1883) can be used to request other components (e.g., application and service components (1870)) that perform feedback (or response) according to the prompt. The acquired information can be used to generate a prompt by the prompt design component (1881) together with the user input, or can be used as input to the generative AI model (1890). Additionally, the application and plugin management component (1883) can request the action through the API if the application or service needs to perform an action that ultimately executes the user query rather than an intermediate result. Information obtained from an external source can be transmitted as input to a generative AI model (1890) along with user input.

[0305] The output modification component (1885) can fine-tune (or adjust or change) the output produced by the generative AI model (1890). For example, the output modification component (1885) can determine the relevance (e.g., score) between the output (e.g., content) of the generative AI model (1890) and the user input. For example, the output modification component (1885) can verify whether the content generated through a large language model (LLM), a large vision model (LVM), or a large multimodal model (LMM) is relevant, contains biased information (e.g., selective information), or contains harmful information (e.g., violent content or profanity). Additionally, the output modification component (1885) can determine the extent to which the output matches the desired result and, if additional processing is required, proceed with that process. Additionally, the output modification component (1885) can configure and provide to the user a hint to avoid unwanted output.

[0306] A generative AI model (1890) generally refers to an artificial intelligence neural network that generates new forms of data based on user input information. A generative AI model (1890) may include an image generation model and / or a language generation model. An image generation model may include a generative adversarial network (GAN) and / or a variational autoencoder (VAE). An example of an image generation model is a diffusion-based generative model that uses the structures of a VAE and a Transformer. Additionally, a language generation model is a model trained to output the statistically most appropriate output value based on input values, and representative examples include models such as CHAT-GPT 3 and CHAT-GPT 4. Additionally, it may include a large-scale multimodal model (LMM) capable of recognizing various forms of data input, such as text, images, and / or speech, and generating new data corresponding to them.

[0307] In one embodiment, the AI ​​framework (1880) and / or generative AI model (1890) may be included within an AI module (e.g., including a processing circuit) within the electronic device (101). For example, the AI ​​module may be operatively coupled with at least one processor (120) of the electronic device (101). For example, the AI ​​module may be operatively coupled with a sensor hub of the electronic device (101) for one or more sensors within the electronic device (101).

[0308] A wearable device according to one embodiment of the present disclosure comprises at least one sensor, at least one infrared camera, at least one shooting camera, a display, a processor, and a memory for storing instructions, wherein when the instructions are executed by the processor, the wearable device may: determine that an illuminance obtained using the at least one sensor is less than a first reference value, obtain a first image using the at least one infrared camera, display a second image including at least a portion of the first image through the display, select at least a portion of the second image based on a first user input, and add visual effects to the selected portion of the second image.

[0309] The second image above may be an image generated using the first image and the third image obtained using at least one camera.

[0310] The first user input above may include user gaze information obtained using an eye-tracking camera.

[0311] The user's gaze information may include at least some of the first information indicating the location where the user's gaze is directed, or the second information indicating the time the user's gaze stays.

[0312] The above visual effect may include an effect that gradually increases the intensity of color information for a selected portion of the second image according to the time the user’s gaze rests.

[0313] The above visual effect may include an effect that changes the size of a selected portion of the second image according to the time the user’s gaze rests.

[0314] The above visual effect may include an effect that increases the size of a selected portion of the second image in proportion to the time the user’s gaze stays.

[0315] The above first user input may further include user gestures.

[0316] The wearable device further includes a motion recognition camera, and the instructions, when executed by the processor, may enable the wearable device to acquire the user gesture using the motion recognition camera.

[0317] When the above commands are executed by the processor, the wearable device can: generate the visual effect using an artificial intelligence model.

[0318] The above visual effect may include an effect that changes at least some of the color type, saturation, brightness, or luminance for a selected portion of the second image.

[0319] When the above commands are executed by the processor, the wearable device may: divide a selected portion of the second image into a central area and an outer area surrounding the central area, and cause the intensity of the visual effect to decrease as it moves from the central area to the outer area.

[0320] When the above instructions are executed by the processor, the wearable device may: divide a selected portion of the second image into a first portion and a second portion, apply a first visual effect to the first portion, and apply a second visual effect to the second portion.

[0321] When the above commands are executed by the processor, the wearable device may: change a selected portion of the second image based on a second user input.

[0322] The first user input above may include the user's gaze and user gestures as a multimodal input.

[0323] The above second user input may include the user's gaze.

[0324] When the above commands are executed by the processor, the wearable device may: change the color information for the previously selected area when the area selected in the second image is changed.

[0325] When the above commands are executed by the processor, the wearable device may: determine the position of a virtual light source in the second image based on the second image or a third user input, additionally display a light source effect that illuminates a selected area in the second image from the determined virtual light source, and, when the selected area in the second image moves, move the area to which the light source effect is applied.

[0326] A method for driving a wearable device according to one embodiment of the present disclosure may include: confirming that an illuminance obtained using at least one sensor is smaller than a first reference value; acquiring a first image using at least one infrared camera; displaying a second image that includes at least a portion of the first image through a display; selecting at least a portion of the second image based on a first user input; and adding a visual effect to the selected portion of the second image.

[0327] The first user input above may include user gaze information obtained using an eye-tracking camera.

Claims

1. In a wearable device, At least one sensor; At least one infrared camera; At least one shooting camera; display; processor; and It includes memory for storing instructions, When the above instructions are executed by the processor, the wearable device: It is confirmed that the illuminance obtained using the above at least one sensor is smaller than the first reference value, and A first image is obtained using the above-mentioned at least one infrared camera, and A second image including at least a portion of the first image is displayed through the display, and Based on the first user input, at least a portion of the second image is selected, and Adding visual effects to a selected portion of the second image above, Wearable device.

2. In Paragraph 1, The second image above is an image generated using the first image and the third image obtained using the at least one camera, Wearable device.

3. In Paragraph 1, The above first user input includes user gaze information obtained using an eye-tracking camera, Wearable device.

4. In Paragraph 3, The above user's gaze information is, at least a portion of the first information indicating the location where the user’s gaze is directed, or the second information indicating the time the user’s gaze stays there. Wearable device.

5. In Paragraph 4, The above visual effect is, The effect of gradually increasing the intensity of color information for a selected portion of the second image according to the time the user’s gaze rests, Wearable device.

6. In Paragraph 4, The above visual effect is, The effect of changing the size of a selected portion of the second image according to the time the user’s gaze rests thereon, Wearable device.

7. In Paragraph 6, The above visual effect is, The effect of increasing the size of a selected portion of the second image in proportion to the time the user’s gaze rests thereon, Wearable device.

8. In Paragraph 3, The above first user input further includes a user gesture, Wearable device.

9. In Paragraph 8, The above wearable device Including additional motion recognition cameras, When the above instructions are executed by the processor, the wearable device: Acquiring the user gesture using the motion recognition camera, Wearable device.

10. In Paragraph 1, When the above instructions are executed by the processor, the wearable device: Generating the above visual effect using an artificial intelligence model, Wearable device.

11. In Paragraph 1, The above visual effect The effect of changing at least some of the color type, saturation, brightness, or luminance for a selected portion of the second image above, Wearable device.

12. In Paragraph 1, When the above instructions are executed by the processor, the wearable device: A selected portion of the second image above is divided into a central region and an outer region surrounding the central region, and The intensity of the visual effect decreases as it moves from the central area to the outer area. Wearable device.

13. In Paragraph 1, When the above instructions are executed by the processor, the wearable device: A selected portion of the second image above is divided into a first portion and a second portion, and A first visual effect is applied to the above first partial area, and Applying a second visual effect to the above second partial area, Wearable device.

14. In Paragraph 1, When the above instructions are executed by the processor, the wearable device: Based on the second user input, changing a selected portion of the second image, Wearable device.

15. In a method for operating a wearable device, An operation to confirm that the illuminance obtained using at least one sensor is smaller than a first reference value, The operation of acquiring a first image using at least one infrared camera, The operation of displaying a second image containing at least a portion of a first image through a display, An operation of selecting at least a portion of the second image based on a first user input, and A method including adding visual effects to a selected portion of the second image, method.

Images