Hmd device for displaying information received from external electronic device

Abstract

Provided is an electronic device wearable on the face of a user. The electronic device comprises: a display disposed to illuminate the eyes of a user which are located inside the electronic device when worn on the face of the user; a camera disposed on the front surface of the electronic device and configured to capture the outside; a communication circuit for wireless communication with an external electronic device; a memory for storing instructions; and a processor. The instructions, when executed by the processor, instruct the electronic device to: acquire a lock screen image from an image acquired using the camera; compare a reference image received from a first external electronic device through the communication circuit with the lock screen image; transmit, to the first external electronic device through the communication circuit, a first message requesting activation of a function of relaying notification content to a second external electronic device, which is a source of the reference image, on the basis of the lock screen image and the reference image matching as a result of the comparison; receive a second message from the first external electronic device through the communication circuit in response to the transmission of the first message; and display at least a part of the second message on the display.

Description

HMD device for displaying information received from an external electronic device

[0001] The present disclosure relates to a head-mounted display (HMD) device for displaying information received from an external electronic device.

[0002] HMD devices can implement augmented reality (AR) and / or virtual reality (VR). One type of HMD device is the so-called AR headset (or OST (optical see-through) device). An AR headset can be configured to allow external light to reach the user's eyes through the glasses when worn. Another type of HMD device is the so-called VR headset (or VST (video see-through) device). A VR headset can be configured to allow light emitted from the display to reach the user's eyes when worn, while blocking external light so that external light does not reach the user's eyes. AR headsets and VR headsets can be collectively referred to as mixed reality (MR) devices or extended reality (XR) devices.

[0003] The HMD device is equipped with a communication circuit and can display information received from an external electronic device on a display through this communication circuit.

[0004] The information described above is provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art related to the present disclosure.

[0005] In an embodiment of the present disclosure, an electronic device wearable on the face (e.g., an HMD device) may receive notifications from an external electronic device to a user wearing the electronic device and display them on a display provided on the electronic device. The technical problems to be solved by the present disclosure are not limited to those mentioned above, and other unmentioned technical problems will be clearly understood by those skilled in the art from the description below.

[0006] According to one embodiment, an electronic device wearable on a user's face is provided. The electronic device comprises: a display positioned to illuminate the user's eyes located inside the electronic device when worn on the user's face; a camera positioned on the front of the electronic device and configured to photograph the outside; a communication circuit for wirelessly communicating with an external electronic device; a memory for storing instructions; and at least one processor. When the instructions are executed individually or collectively by the at least one processor, the electronic device may acquire a lock screen image from an image acquired using the camera, compare a reference image received from a first external electronic device via the communication circuit with the lock screen image, and, based on the result of the comparison that the lock screen image and the reference image match, transmit a first message to the first external electronic device via the communication circuit requesting the activation of a function to relay notification content to a second external electronic device that is the source of the reference image, and in response to the transmission of the first message, receive a second message from the first external electronic device via the communication circuit and display at least a portion of the second message on the display.

[0007] According to one embodiment, a method for operating an electronic device wearable on a user's face is provided. The electronic device may include: a display positioned to illuminate the user's eyes located inside the electronic device when worn on the user's face; a camera positioned on the front of the electronic device and configured to photograph the outside; and a communication circuit for wirelessly communicating with an external electronic device. The method may include: an operation of acquiring a lock screen image from an image acquired using the camera; an operation of comparing a reference image received from a first external electronic device through the communication circuit with the lock screen; an operation of transmitting a first message to the first external electronic device through the communication circuit, based on the result of the comparison that the lock screen image and the reference image match, requesting the activation of a function to relay notification content to a second external electronic device, which is the source of the reference image; an operation of receiving a second message from the first external electronic device through the communication circuit in response to the transmission of the first message; and an operation of displaying at least a portion of the second message on the display.

[0008] According to an embodiment of the present disclosure, an electronic device may provide external notification information to a wearer through a display. In addition, various effects that can be identified directly or indirectly through this document may be provided.

[0009] FIG. 1 is a block diagram of an electronic device in a network environment according to various embodiments.

[0010] FIG. 2a is a perspective view of an HMD device (201) according to one embodiment. FIG. 2b shows hardware placed inside a frame (200) in the HMD device (201) of FIG. 2a.

[0011] FIG. 3a illustrates an HMD device (301) according to one embodiment. A rear (or, second) side (320) of the housing of the HMD device (301) (e.g., the electronic device (101) of FIG. 1) is illustrated in FIG. 3a. A front (or, first) side (310) opposite to the rear (310) is illustrated in FIG. 3b.

[0012] FIG. 4 is a block diagram of a system for notifying a user of the reception of information and providing information to a user using an HMD device (401) according to one embodiment.

[0013] FIG. 5 is a flowchart illustrating the operation of a first electronic device (401) for confirming that the user of a second electronic device (402) is the user of a first electronic device (401), according to one embodiment.

[0014] FIGS. 6a and FIGS. 6b are drawings illustrating examples regarding the angle of view of a camera and the direction in which the camera is facing in a first electronic device (401) according to one embodiment.

[0015] FIGS. 7a and 7b are drawings illustrating a lock screen displayed on the second electronic device (402).

[0016] FIG. 8 is a flowchart illustrating the operation of a first electronic device (401) to enable a user of the first electronic device (401) to recognize a notification generated by a second electronic device (402) according to one embodiment.

[0017] FIG. 9 is a flowchart illustrating the operation of a first electronic device (401) to enable a user of the first electronic device (401) to recognize a notification generated by a second electronic device (402) according to one embodiment.

[0018] FIG. 10 is a flowchart illustrating the operation of a first electronic device (401) to enable a user of the first electronic device (401) to recognize a notification generated by a second electronic device (402) according to one embodiment.

[0019] FIG. 11 is a flowchart for illustrating operations performed in an electronic device (e.g., a first electronic device (401) or a second electronic device (402)) according to one embodiment.

[0020] Figure 12 is a diagram illustrating the operation of generating a prompt to be fed into an artificial intelligence model.

[0021] FIG. 13 is a flowchart illustrating the operation of a first electronic device (401) for providing a notification received from a second electronic device (402) to a user, according to one embodiment.

[0022] FIGS. 14a, 14b, 14c, 14d, 14e, 14f, 14g, and 14h are drawings for illustrating various examples of notification content displayed on the display (1401) of an HMD device (e.g., the first electronic device (401)).

[0023] Hereinafter, embodiments of the present disclosure are described in detail with reference to the drawings so that those skilled in the art can easily practice them. However, the present disclosure may be embodied in various different forms and is not limited to the embodiments described herein. In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and brevity.

[0024] FIG. 1 is a block diagram of an electronic device (101) in a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or may communicate with at least one of an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through a server (108). According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (160), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)).

[0025] The processor (120) can control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., a program (140)), and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use lower power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.

[0026] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0027] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, input data or output data for software (e.g., program (140)) and related commands. The memory (130) may include volatile memory (132) or non-volatile memory (134).

[0028] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0029] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0030] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0031] The display module (160) can visually provide information to an external (e.g., user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0032] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (101).

[0033] The sensor module (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0034] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0035] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0036] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that can be perceived by the user through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0037] The camera module (180) can capture still images and video. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0038] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0039] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0040] The communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).

[0041] The wireless communication module (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), external electronic device (e.g., electronic device (104)), or network system (e.g., second network (199)). According to one embodiment, the wireless communication module (192) may support a Peak data rate (e.g., 20 Gbps or more) for eMBB realization, loss coverage (e.g., 164 dB or less) for mMTC realization, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for URLLC realization.

[0042] An antenna module (197) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197).

[0043] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0044] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0045] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (104) or the server (108) may be included within a second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0046] According to various embodiments of the present disclosure, the above-described components of the electronic device (101) may be applied to a wearable electronic device for displaying images in a virtual space. The wearable electronic device may include an HMD device that implements AR and / or VR as described above. The HMD device may be referred to as an AR headset or a VR headset depending on whether it blocks external light. The VR headset may implement a VR mode that blocks external light and displays a virtual image through a display, and an ST (see through) mode that displays an actual external image captured using a camera through a display. Here, the camera may include a camera mounted on an external electronic device (e.g., a smartphone, a digital camera) and / or a camera mounted on the HMD device. In the ST mode, only the actual image may be provided, or the actual image and the virtual image may be provided together.

[0047] According to one embodiment, an HMD device can perform functions related to AR and / or MR. For example, when a user wears the HMD device, the HMD device may include at least one lens positioned adjacent to the user's eyes. The HMD device may combine light emitted from the display of the HMD device with ambient light passing through the lens. The display area of ​​the display may be formed within the lens through which the ambient light passes. Because the HMD device combines the ambient light and the light emitted from the display, the user can see an image in which a real object perceived by the ambient light and a virtual object formed by the light emitted from the display are mixed. The aforementioned augmented reality, mixed reality, and / or virtual reality may also be referred to as extended reality (XR).

[0048] According to one embodiment, an HMD device may perform functions related to VST and / or virtual reality (VR). For example, when a user wears the HMD device, the HMD device may include a housing that covers the user's eyes. When worn, the HMD device may include a display disposed on a first surface of the housing facing the eyes. The HMD device may include a camera disposed on a second surface opposite to the first surface. Using the camera, the HMD device may acquire an image and / or video representing ambient light. The HMD device may output an image and / or video within the display disposed on the first surface so that the user perceives the ambient light through the display. A displaying area (or displaying region) (or active area or active region) of the display disposed on the first surface may be formed by one or more pixels included in the display. The HMD device can composite virtual objects onto images and / or videos output through a display, allowing the user to perceive virtual objects along with real objects perceived by ambient light.

[0049] According to one embodiment, an HMD device can identify or recognize the position or location and / or direction or orientation of the HMD device based on an image (and / or video) obtained or acquired using a camera. The HMD device can acquire information about the external space using one or more cameras and / or one or more sensors. The acquired information may include the geographic location of the external space (e.g., Global Positioning System (GPS) coordinates) identified by one or more sensors. The acquired information may include images and / or videos of the external space identified by one or more cameras. The HMD device can perform object recognition on the images and / or videos to identify external objects contained in the external space from the images and / or videos.

[0050] In this document, 'inside' may refer to the inner space of the HMD device where the user's eyes are located when the HMD device is worn on the user's face (201a, referring to FIG. 2a to be described later). 'Outside' may refer to the outer space of the HMD device when the HMD device is worn on the user's face (201b, referring to FIG. 2a to be described later). In this document, 'front' may refer to the front of the display formed on the outside when the HMD device is worn on the user's face (e.g., the surface facing the -y-axis direction, referring to FIG. 2b, 3a, and 3b to be described later). 'Side' may refer to the side of the display formed on the outside when the HMD device is worn on the user's face. 'Rear' may refer to the surface facing the user's eyes when the HMD device is worn on the user's face (e.g., the surface facing the y-axis direction, referring to FIG. 2b, 3a, and 3b to be described later).

[0051] FIG. 2a is a perspective view of an HMD device (201) according to one embodiment. FIG. 2b illustrates hardware placed inside a frame (200) in the HMD device (201) of FIG. 2a. The HMD device (201) (e.g., the electronic device (101) of FIG. 1) may have the form of glasses that are wearable on a part of a user's body (e.g., head). For example, the housing of the HMD device (201) may include a flexible material such as rubber and / or silicone that has a form that adheres to a part of the user's head (e.g., a part of the face covering both eyes). For example, the housing of the HMD device (201) may include a strap that can be twined around the user's head and / or temples that are attachable to the ears of the head.

[0052] Referring to FIGS. 2a and 2b, an HMD device (e.g., an AR headset) (201) may include a display (250) and a frame (200). The frame (or, housing) (200) may be configured to support the display (250) and accommodate a number of hardware.

[0053] The HMD device (201) can provide augmented reality (AR), virtual reality (VR), or mixed reality (MR) to a user wearing the HMD device (201). For example, the HMD device (201) can display a virtual reality image provided by an optical device (282, 284; see FIG. 2b) on a display (250) in response to a specified gesture of the user obtained through the motion recognition camera (260-2, 260-3) of FIG. 2b.

[0054] The display (250) may provide visual information to the user. For example, the display (250) may include a transparent or translucent lens. The display (250) may include a first display (250-1) and / or a second display (250-2). For example, the first display (250-1) may be placed on the glass for the left eye and the second display (250-2) may be placed on the glass for the right eye. The display (250) may provide the user with visual information transmitted from external light through the lens and other visual information distinct from said visual information. The lens may be formed based on at least one of a Fresnel lens, a pancake lens, or a multi-channel lens. Referring to FIG. 2b, the display (250) may include a front surface (or, first surface) (231) and a rear surface (or, second surface) (232) opposite to the front surface (231). A display area may be formed on the rear (232) of the display (250). When a user wears the HMD device (201), external light may be transmitted to the user by being incident on the front (231) and transmitted through the rear (232). As another example, the display (250) may display an augmented reality image combined with a virtual reality image provided by an optical device (282, 284) on a real-world image transmitted through external light, in the display area formed on the rear (232).

[0055] The display (250) may include a waveguide (233, 234) that diffracts light emitted from an optical device (282, 284) and transmits it to the user. The waveguide (233, 234) may be formed based on at least one of glass, plastic, or polymer. A nano pattern may be formed on the exterior or at least a portion of the interior of the waveguide (233, 234). The nano pattern may be formed based on a polygonal and / or curved grating structure. Light incident on one end of the waveguide (233, 234) may be propagated to the other end of the waveguide (233, 234) by the nano pattern. The waveguide (233, 234) may include a diffractive element (e.g., DOE (diffractive optical element), HOE (holographic optical element)) and / or a reflective element (e.g., a reflective mirror). For example, waveguides (233, 234) may be placed within the HMD device (201) to guide a screen displayed by the display (250) to the user's eyes. For example, the screen may be transmitted to the user's eyes based on total internal reflection (TIR) ​​occurring within the waveguides (233, 234).

[0056] The HMD device (201) may be configured to identify and / or analyze subjects (or objects) in real-world images collected through a shooting camera (260-4). The HMD device (201) may combine a virtual subject with a real subject among the subjects that is the target of augmented reality provision and display it on a display (250). The virtual subject may include at least one of text and an image regarding information related to the real subject. The HMD device (201) may analyze and / or identify the real subject based on a multi-camera, such as a stereo camera. For analysis and / or identification, the HMD device (201) may perform spatial recognition (e.g., simultaneous localization and mapping (SLAM)) using a multi-camera and / or time-of-flight (ToF).

[0057] The frame (200) may be formed as a physical structure that allows the HMD device (201) to be worn on the user's body. The frame (200) may be configured such that when the user wears the HMD device (201), the first display (250-1) is positioned in front of the user's left eye and the second display (250-2) is positioned in front of the user's right eye. Referring to FIG. 2a, the frame (200) may include an area (220) in which at least a portion contacts a part of the user's body when the user wears the HMD device (201). For example, the area (220) of the frame (200) in contact with a part of the user's body may include an area in contact with a part of the user's nose, a part of the user's ear, and a part of the side of the user's face that the HMD device (201) contacts. According to one embodiment, the frame (200) may include a nose pad (210) that contacts a part of the user's body. For example, when the HMD device (201) is worn by a user, the nose pad (210) may come into contact with the nose. The frame (200) may include a first temple (204) and a second temple (205) that come into contact with other parts of the body.

[0058] The frame (200) may include a first rim (201) covering at least a portion of a first display (250-1), a second rim (202) covering at least a portion of a second display (250-2), a bridge (203) positioned between the first rim (201) and the second rim (202), a first pad (211) positioned along a portion of the edge of the first rim (201) from one end of the bridge (203), a second pad (212) positioned along a portion of the edge of the second rim (202) from the other end of the bridge (203), a first temple (204) extending from the first rim (201) and fixed to a portion of the wearer's ear, and a second temple (205) extending from the second rim (202) and fixed to a portion of the ear opposite to the ear. The first pad (211) and the second pad (212) may come into contact with a part of the nose. The first temple (204) and the second temple (205) may come into contact with a part of the face and a part of the ear. The temples (204, 205) may be rotatably connected to the rim through the hinge units (206, 207) of FIG. 2B. The first temple (204) may be rotatably connected to the first rim (201) through the first hinge unit (206) positioned between the first rim (201) and the first temple (204). The second temple (205) may be rotatably connected to the second rim (202) through the second hinge unit (207) positioned between the second rim (202) and the second temple (205). The HMD device (201) can identify an external object (e.g., a user's fingertip) touching the frame (200) and / or a gesture performed by said external object by using a touch sensor, a grip sensor, and / or a proximity sensor formed on at least a portion of the surface of the frame (200).

[0059] The HMD device (201) may include a number of hardware components to perform various functions. For example, the HMD device (201) may include a battery module (270), an antenna module (275), an optical device (282, 284), a speaker (255-1, 255-2), a microphone (265-1, 265-2, 265-3), a light-emitting module (not shown), and / or a PCB (printed circuit board) (290).

[0060] The HMD device (201) may include microphones (265-1, 265-2, 265-3) configured to convert sound into an electrical signal (audio signal) and output it. Referring to FIG. 2b, the first microphone (265-1) may be placed on the bridge (203). The second microphone (265-2) may be placed on the second rim (202). The third microphone (265-3) may be placed on the first rim (201). The number and / or placement of the microphones are not limited to the embodiment of FIG. 2b. When two or more microphones are included in the HMD device (201) and placed at different locations, the HMD device (201) may be configured to identify the direction from which sound is coming (or the location of the sound source) using the plurality of microphones.

[0061] Optical devices (282, 284) can project a virtual object onto a display (250). For example, optical devices (282, 284) may include a projector. Optical devices (282, 284) may be positioned adjacent to the display (250) or may be components of the display (250). According to one embodiment, the HMD device (201) may include a first optical device (282) corresponding to a first display (250-1) and a second optical device (284) corresponding to a second display (250-2). For example, the first optical device (282) may be positioned at the edge of the first display (250-1). The second optical device (284) may be positioned at the edge of the second display (250-2). The first optical device (282) can transmit light to the first waveguide (233) placed in the first display (250-1). The second optical device (284) can transmit light to the second waveguide (234) placed in the second display (250-2).

[0062] The camera (260) may include a shooting camera (260-4), an eye tracking camera (ET CAM) (260-1), and / or a motion recognition camera (260-2, 206-3). The shooting camera (260-4), the eye tracking camera (260-1), and the motion recognition camera (260-2, 260-3) may be positioned at different locations in the frame (200) and may perform different functions. The eye tracking camera (260-1) may output data indicating the position of the eyes or the gaze of a user wearing the HMD device (201). For example, the HMD device (201) may detect the pupils and track the gaze (e.g., the subject the pupils are directed toward) from an image obtained through the eye tracking camera (260-1). The HMD device (201) can identify the subject of interest (e.g., a real subject and / or a virtual subject) (or a region of interest) that the user is looking at (focused on by the user) based on the gaze being tracked through the gaze tracking camera (260-1). The HMD device (201) can perform functions for interaction between the subject of interest (or region of interest) (e.g., gaze interaction). The HMD device (201) can display an animation of the eyes of an avatar representing the user in virtual space moving based on the gaze being tracked through the gaze tracking camera (260-1). The HMD device (201) can render an image (or screen) to be displayed on the display (250) based on the position of the user's eyes. For example, the visual quality of a first region related to the gaze within an image and the visual quality of a second region distinct from the first region (e.g., resolution, brightness, saturation, grayscale, PPI (pixels per inch)) may differ from each other.The HMD device (201) can acquire an image having a first visual quality and a second visual quality by using foveated rendering. For example, if the HMD device (201) supports an iris recognition function, user authentication can be performed based on iris information acquired using an eye-tracking camera (260-1). An example in which the eye-tracking camera (260-1) is positioned toward the user's right eye is shown in FIG. 2b, but the embodiment is not limited thereto, and the eye-tracking camera (260-1) may be positioned alone toward the user's left eye or toward both eyes.

[0063] The eye tracking camera (260-1) can achieve more realistic augmented reality by tracking the gaze of a user wearing the HMD device (201), thereby matching the user's gaze with the visual information provided to the display (250). For example, when the user looks straight ahead, the HMD device (201) can naturally display environmental information related to the user's front on the display (250) at the location where the user is situated. The eye tracking camera (260-1) may be configured to capture an image of the user's pupil to determine the user's gaze. For example, the eye tracking camera (260-1) may receive a gaze detection light reflected from the user's pupil and track the user's gaze based on the position and movement of the received gaze detection light. In one embodiment, the eye tracking camera (260-1) may be positioned at locations corresponding to the user's left and right eyes. For example, the eye-tracking camera (260-1) may be positioned within the first rim (201) and / or the second rim (202) to face the direction in which the user wearing the HMD device (201) is located.

[0064] The motion recognition camera (260-2, 260-3) may be used to recognize movements or gestures of the user's entire body or parts of the user's body, such as the user's torso, hands, or face. For example, the HMD device (201) may recognize gestures of specific parts of the body (e.g., fingertips) in images acquired through the motion recognition camera (260-2, 260-3) and perform designated functions based on the recognition of the gestures. For example, the HMD device (201) may display indicators corresponding to the gestures on the display (250). The motion recognition camera (260-2, 260-3) may be used to perform spatial recognition functions using SLAM and / or depth maps for 6 degrees of freedom poses (6 dof pose). The HMD device (201) may perform gesture recognition functions and / or object tracking functions using the motion recognition camera (260-2, 260-3). In one embodiment, a motion recognition camera (260-2, 260-3) may be placed on the first rim (201) and / or the second rim (202).

[0065] The camera (260-4) can capture a real image or background to be matched with a virtual subject (e.g., image and / or text) to implement augmented reality or mixed reality content. The camera (260-4) can be used to acquire high-resolution images based on HR (high resolution) or PV (photo video). The camera (260-4) can capture a specific object located at the position viewed by the user and provide the captured image of the specific object to the display (250). The display (250) can display a single image in which information regarding a real image or background including the image of the specific object acquired using the camera (260-4) and a virtual image provided through the optical device (282, 284) are superimposed. The HMD device (201) can compensate (or correct) for depth information (e.g., the distance between the HMD device (201) and an external object acquired through a depth sensor) using the image acquired through the camera (260-4). The HMD device (201) can perform the operation of identifying an object (or subject) in an image acquired using a shooting camera (260-4). The HMD device (201) can perform a function of focusing on a specific subject (e.g., auto focus) and / or an optical image stabilization (OIS) function (e.g., anti-shake function). The HMD device (201) can perform a pass-through function to display an image acquired through the shooting camera (260-4) superimposed on at least a portion of the screen while displaying a screen representing a virtual space on the display (250). In one embodiment, the shooting camera (260-4) may be placed on a bridge (203) located between the first rim (201) and the second rim (202).

[0066] The camera (260) included in the HMD device (201) is not limited to the eye-tracking camera (260-1), motion recognition camera (260-2, 260-3), and shooting camera (260-4) as described above. For example, the HMD device (201) can identify external objects included in the field of view (FoV) by using a camera positioned toward the user's field of view (FoV). The identification of external objects by the HMD device (201) can be performed based on a sensor for identifying the distance between the HMD device (201) and the external object, such as a depth sensor and / or a time of flight (ToF) sensor. The camera positioned toward the FoV can support an autofocus function and / or an optical image stabilization (OIS) function. For example, the HMD device (201) may include a camera (e.g., a face tracking camera) positioned toward the face to acquire an image including the face of a user wearing the HMD device (201).

[0067] The HMD device (201) may further include a light source (e.g., LED) that emits light toward a subject (e.g., user's eyes, face, and / or external objects within the FoV) being captured using a camera (260). This light source may include an LED of infrared wavelength. The light source may be placed in at least one of the frame (200) and the hinge unit (206, 207).

[0068] The battery module (270) can supply power to the electronic components of the HMD device (201). The battery module (270) may be placed within the first temple (204) and / or the second temple (205). For example, the battery module (270) may include a first battery (271) placed within the first temple (204) and a second battery (272) placed in the second temple (205).

[0069] The antenna module (275) can transmit a signal or power to the outside of the HMD device (201) or receive a signal or power from the outside. In one embodiment, the antenna module (275) may be placed within the first temple (204) and / or the second temple (205). For example, the antenna module (275) may be placed close to one side of the first temple (204) and / or the second temple (205).

[0070] The speaker (255) can output an acoustic signal to the outside of the HMD device (201). In one embodiment, the speaker (255) may be placed within a first temple (204) and / or a second temple (205) to be positioned adjacent to the ears of a user wearing the HMD device (201). For example, the speaker (255) may include a first speaker (255-1) positioned adjacent to the user's left ear by being placed within the first temple (204) and a second speaker (255-2) positioned adjacent to the user's right ear by being placed within the second temple (205).

[0071] A light-emitting module (not shown) may include at least one light-emitting element. The light-emitting module may emit light of a color corresponding to a specific state or emit light with an action corresponding to a specific state in order to visually provide information regarding a specific state of the HMD device (201) to the user. For example, if the HMD device (201) requires charging, it may emit red light at a constant frequency. In one embodiment, the light-emitting module may be placed on the first rim (201) and / or the second rim (202).

[0072] The HMD device (201) may include a printed circuit board (PCB) (290). The PCB (290) may be placed on a first temple (204) and / or a second temple (205). The HMD device (201) may include an interposer placed between sub-PCBs on the PCB (290). Multiple hardware may be placed on the PCB (290). The HMD device (201) may include a flexible PCB (FPCB) for interconnecting multiple hardware.

[0073] The HMD device (201) may include an inertia measurement unit (IMU) configured to include a sensor (e.g., an acceleration sensor, a gyroscope (in other words, an angular velocity sensor), and a magnetometer) for detecting the posture of the HMD device (201) and / or the posture of a body part (e.g., head) of a user wearing the HMD device (201). The acceleration sensor may be configured to generate a first acceleration value corresponding to motion in the x-axis direction, a second acceleration value corresponding to motion in the y-axis direction, and a third acceleration value corresponding to motion in the z-axis direction. The gyroscope may be configured to generate a first angular velocity value corresponding to rotational motion with respect to the x-axis, a second angular velocity value corresponding to rotational motion with respect to the y-axis, and a third angular velocity value corresponding to rotational motion with respect to the z-axis. The HMD device (201) can obtain (e.g., calculate) tilt values ​​(e.g., Euler angles (roll, pitch, yaw)) using data obtained through a sensor (e.g., data indicating acceleration values ​​of each axis and data indicating the direction in which the Earth's magnetic force is directed). The HMD device (201) can identify a user's motion or gesture performed to execute or interrupt a specific function of the HMD device (201) based on data obtained through an IMU.

[0074] FIG. 3a illustrates an HMD device (301) according to one embodiment. A rear (or, second) side (320) of the housing of the HMD device (301) (e.g., the electronic device (101) of FIG. 1) is illustrated in FIG. 3a. A front (or, first) side (310) opposite to the rear (310) is illustrated in FIG. 3b.

[0075] Referring to FIG. 3a, the rear (320) of the HMD device (e.g., VR headset) (301) may have a shape that is attachable to a part of the user's body (e.g., the user's face). Although not illustrated, the HMD device (301) may include a strap and / or one or more temples (e.g., the first temple (204) and / or the second temple (205) of FIG. 2a and FIG. 2b) for securing to a part of the user's body. A first display (350-1) (e.g., the first display (250-1)) for outputting an image to the left eye and a second display (350-2) (e.g., the second display (250-2)) for outputting an image to the right eye may be placed on the rear (320). The HMD device (301) may include a structure (e.g., rubber or silicone packing) to prevent interference by light different from light emitted from the first display (350-1) and the second display (350-2) (e.g., ambient light).

[0076] The HMD device (301) may include a camera (360-1) (e.g., an eye-tracking camera (260-1)) for capturing and / or tracking both eyes of a user adjacent to each of the first display (350-1) and the second display (350-2). The HMD device (301) may include a camera (360-5, 360-6) for capturing and / or recognizing the user's face. The camera (360-5, 360-6) may include a face tracking (FT) camera. The HMD device (301) may control an avatar representing the user in a virtual space based on the motion of the user's face identified using the camera (360-5, 360-6). For example, the HMD device (301) can change the texture and / or shape of a part of an avatar (e.g., a part of an avatar representing a human face) by using information obtained by a camera (360-5, 360-6) and representing the facial expression of a user wearing the HMD device (301).

[0077] Referring to FIG. 3b, the HMD device (301) may include a camera (e.g., cameras (260-7, 260-8, 260-9, 260-10, 260-11, 260-12)) and / or a sensor (e.g., a depth sensor (330)) for acquiring information related to the external environment. For example, the camera (360-7, 360-8, 360-9, 360-10) may be positioned on the front (310) to recognize external objects. The camera (360-7, 360-8, 360-9, 360-10) may be a camera corresponding to the motion recognition camera (260-2, 260-3) of FIG. 2b.

[0078] The HMD device (301) may include cameras (360-11, 360-12) for acquiring images and / or videos to be transmitted to each of the user's two eyes. The camera (360-11) may be positioned on the front (310) of the HMD device (301) to acquire an image to be displayed through a second display (350-2) corresponding to the right eye. The camera (360-12) may be positioned on the front (310) of the HMD device (301) to acquire an image to be displayed through a first display (350-1) corresponding to the left eye. The cameras (360-11, 360-12) may be cameras corresponding to the shooting camera (260-4) of FIG. 2B.

[0079] According to one embodiment, the HMD device (301) may include a depth sensor (330) positioned on the front (310) to identify the distance between the HMD device (301) and an external object. Using the depth sensor (330), the HMD device (301) may obtain spatial information (e.g., a depth map) for at least a portion of the FoV of a user wearing the HMD device (301). Although not illustrated, a microphone may be positioned on the front (310) of the HMD device (301) to obtain sound output from an external object. The number of microphones may be one or more, depending on the embodiment.

[0080] FIG. 4 is a block diagram of a system for notifying a user of the reception of information and providing information to a user using an HMD device (401) according to one embodiment. The system of FIG. 4 may include a first electronic device (401) wearable on the face of a user, a second electronic device (402) of the user, and a third electronic device (403) for relaying between the two electronic devices (401, 402). The first electronic device (401) may correspond to the electronic device (101) of FIG. 1, the HMD device (201) of FIG. 2, or the HMD device (301) of FIG. 3 as described above. The second electronic device (402) may correspond to the electronic device (102) or electronic device (104) of FIG. 1. The third electronic device (403) may correspond to the server (108) of FIG. 1. The first electronic device (401) and the second electronic device (402) may be interconnected via a network that supports short-range wireless communication (e.g., Bluetooth, Wi-Fi) (e.g., the first network (198) of FIG. 1). The first electronic device (401) and the third electronic device (403) may be interconnected via a network that supports short-range wireless communication or a network that supports long-range wireless communication (e.g., LTE, 5G) (e.g., the second network (199) of FIG. 1). The second electronic device (402) and the third electronic device (403) may be interconnected via a network that supports short-range wireless communication or a network that supports long-range wireless communication.

[0081] The first electronic device (401) may include a first camera (411), a second camera (412), a first communication circuit (421), a first sensor (431), a first display (441), a microphone (451), a first memory (481), and a first processor (491). The first camera (411) may include a lens located at the front of the first electronic device (401) and configured to collect light through the front. The second camera (412) may include a lens located at the rear of the first electronic device (401) and configured to collect light through the rear.

[0082] In the first electronic device (401), the camera (411, 412), the first communication circuit (421), the first sensor (431), the first display (441), the first memory (481), and the first processor (491) are each implemented substantially identically to the camera module (180), communication module (190), sensor module (176), display module (160), memory (130), and processor (120) in the electronic device (101) of FIG. 1, so as to perform the same functions. The microphone (451) may correspond to the input module (150) in FIG. 1. The microphone (451) may be configured to receive external sound and convert the sound into an analog electrical signal (audio signal). The audio signal can be converted into a digital electrical signal (audio data) through an analog-to-digital converter (ADC) of the first electronic device (401) and output to other components of the first electronic device (401) (e.g., a first processor (491) and / or a first communication circuit (421)).

[0083] According to one embodiment, the first camera (411) may include at least one of a plurality of cameras (260-2, 260-3, 260-4) in FIG. 2b. The second camera (412) may include an eye-tracking camera (260-1) in FIG. 2b. According to one embodiment, the first camera (411) may include at least one of a plurality of cameras (360-7, 360-8, 360-9, 360-10, 360-11, 360-12) in FIG. 3b. The second camera (412) may include at least one of a plurality of cameras (360-1, 360-5, 360-6) in FIG. 3a.

[0084] Instructions may be stored in the first memory (481). When executed by the first processor (491), the instructions may cause the first electronic device (401) to perform a given operation. According to one embodiment, the first account management module (461) and / or the first artificial intelligence module (471) may be composed of instructions and stored in the first memory (481). According to one embodiment, a portion of the instructions may be stored in the first memory (481) and another portion may be stored in the internal memory of the first processor (491).

[0085] The first electronic device (401) can operate in a first mode (or user mode) that allows a person registered as the primary user of the device to use the first electronic device (401), and in a second mode (or guest mode) that allows another person to use the first electronic device (401). While operating in the first mode, the first electronic device (401) can verify user information of the second electronic device (402) (e.g., email ID as information regarding a user account) through the first communication circuit (421). The first electronic device (401) can verify that the verified user information matches the information of the primary user registered in the first electronic device (401) (e.g., stored in the first memory (481)). Based on the fact that the two user information matches, the first electronic device (401) can share information with the second electronic device (402) through the first communication circuit (421). For example, the second electronic device (402) may receive a message from an external device through the second communication circuit (422). The received message may include, for example, a short message, a request message for a call, a message notifying that there is an update to a program (e.g., an application, an operating system, or firmware), or a message regarding an online service using the program (e.g., advertisement, music, movies, weather). The second electronic device (402) may transmit a relevant message (e.g., the received message itself (original) or a notification message indicating that the message has been received) to the first electronic device (401) through the second communication circuit (422). As another example, the second electronic device (402) may transmit a message regarding the status of the second electronic device (402) (e.g., a notification message indicating that the battery has finished charging) to the first electronic device (401) through the second communication circuit (422). The first electronic device (402) can display a message received from the second electronic device (401) through the first communication circuit (421) on the first display (441).

[0086] According to one embodiment, the first electronic device (401) can confirm that the user wearing the first electronic device (401) is the user of the second electronic device (402) based on information regarding a subject collected through the first camera (411) while operating in the second mode. The first electronic device (401) can confirm that the user wearing the first electronic device (401) is an auxiliary user (e.g., a family member of the main user) registered with the first electronic device (401) by communicating with the second electronic device (402) through relay of the third electronic device (403) (or communicating directly with the second electronic device (402) without relay). As described above, if a user wearing the first electronic device (401) is identified as a user of the second electronic device (402) and as an auxiliary user of the first electronic device (401), the first electronic device (401) can share information with the second electronic device (402) through the first communication circuit (421).

[0087] The first account management module (461) can manage user account information registered in the first electronic device (401) (e.g., stored in the first memory (481)). The first account management module (461) can update account information by communicating with the third electronic device (403) through the first communication circuit (421). For example, the first account management module (461) can synchronize account information managed by the first account management module (461) with account information managed by the third account management module (463). Account information may include user information. For example, user information may include the user's name, a web address (e.g., a URL (uniform resource locator)), and an ID for logging into the web.

[0088] Account information may include information regarding a device owned by the user. For example, device information associated with user information may include a phone number, product name, model name, or serial number. According to one embodiment, account information may include a lock screen image associated with the device. For example, a first electronic device (401) may receive an image displayed on the second electronic device (402) (e.g., home screen image, lock screen image) from the second electronic device (402) via a third electronic device (403), and may store the received image in the first memory (481) by including it in the device information of the second electronic device (402).

[0089] Account information managed by the first account management module (461) may include account information of a primary user for the first electronic device (401) and account information of a secondary user for the first electronic device (402). A secondary user is a user grouped with the primary user and may be, for example, a family member of the primary user. Account information may also include group information (e.g., group name) indicating which group the secondary user belongs to. According to one embodiment, account information may include the user information, device information, and group information described above for each user, be managed by the first account management module (461), and be stored in the first memory (481).

[0090] The first artificial intelligence module (471) may include an artificial intelligence model. The first artificial intelligence module (471) may generate an input value (e.g., a prompt) and feed the input value into the artificial intelligence model. The artificial intelligence model may process the input value and output a result value.

[0091] According to one embodiment, the first electronic device (401) can analyze a subject in an image obtained through the first camera (411) using the first artificial intelligence module (471) (e.g., a vision model). For example, the first electronic device (401) can obtain an image from the first camera (411) and include the obtained image in a prompt and input it into the model. The first electronic device (401) can recognize the subject and its state or action based on the result value output from the model. For example, the first electronic device (401) can recognize that the first subject is a smartphone and the second subject is a finger, and that the action of the finger (second subject) is to unlock the screen of the smartphone (first subject). The first electronic device (401) can recognize that the wearer of the first electronic device (401) is the user of the first subject (smartphone) through the action of the user attempting to unlock the screen.

[0092] The first electronic device (401) can receive a lock screen image of the second electronic device (402) from the third electronic device (403) via the first communication circuit (421) and store it in the first memory (481) as a reference image for comparison with other images. The first electronic device (401) may also receive a reference image directly from the second electronic device (402) via the first communication circuit (421). The first electronic device (401) can use the first artificial intelligence module (471) (e.g., a vision model) to compare the reference image with a subject image (lock screen image) obtained from the first camera (411), and based on the comparison result, recognize that the second electronic device (402) is the previously recognized first subject (smartphone). For example, the first artificial intelligence module (471) can include the subject image (lock screen) and the reference image in a prompt and input them into the model. This model may be a model trained to measure the similarity between two images and output a result value indicating whether the two images match based on the measurement result. The first electronic device (401) can check whether the subject image and the reference image match based on the result value output from the model. Based on whether the two images (subject image, reference image) match, the first electronic device (401) can recognize that the second electronic device (402) is the first subject being captured through the first camera (411).

[0093] The first electronic device (401) can receive account information along with a lock screen image of the second electronic device (402) from the third electronic device (403) through the first communication circuit (421). The first electronic device (401) may also receive account information directly from the second electronic device (402) through the first communication circuit (421). When the two images (subject image, reference image) match, the first electronic device (401) can determine whether the user of the second electronic device (402) is a user (e.g., auxiliary user) registered with the first electronic device (401) based on the account information (e.g., user information) received along with the reference image. For example, the first electronic device (401) can determine that the user of the second electronic device (402) is a user registered in the first electronic device (401) based on the fact that the account information of an auxiliary user that matches the account information of the second electronic device (402) is stored in the first electronic device (401).

[0094] According to the above, the wearer of the first electronic device (401) can be identified as the user of the first subject (smartphone), the first subject can be identified as the second electronic device (402), and the user of the second electronic device (402) can be identified as a user registered with the first electronic device (401) (e.g., an auxiliary user). Based on this identification, the first electronic device (401) can share information with the second electronic device (402) through communication with the second electronic device (402) via relay of the third electronic device (403) (or direct communication with the second electronic device (402) without relay). For example, the second electronic device (402) may receive a message (e.g., a short message or a request message for a call) from an external device through the second communication circuit (422) and transmit a related message (e.g., the message itself or a notification indicating that the message has been received) to the third electronic device (403) through the second communication circuit (422). As another example, the second electronic device (402) may also transmit a message regarding the status of the second electronic device (402) (e.g., a notification message indicating that the battery has finished charging) to the third electronic device (403) through the second communication circuit (422). The first electronic device (401) may receive a message from the third electronic device (403) through the first communication circuit (421) and display the received message on the first display (441). As another example, the first electronic device (401) may receive a message directly from the second electronic device (402) through the first communication circuit (421) without relaying the third electronic device (403) and display the received message on the first display (441).

[0095] The first electronic device (401) can recognize that a notification has occurred in the second electronic device (402) from an image obtained from the first camera (411) using the first artificial intelligence module (471) (e.g., a vision model). For example, the first electronic device (401) can obtain an image from the first camera (411). The obtained image may include a situation corresponding to the occurrence of a notification. For example, the image may include a situation where the screen of the second electronic device (402) is turned on, or a situation where a notification window appears on the screen. The first electronic device (401) can include the obtained image in a prompt and input it into the model. The first electronic device (401) can recognize that a notification has occurred in the second electronic device (402) based on the result value output from the model. For example, the result value may include information indicating that “a notification window to inform that a message has been received by the second electronic device (402) is being displayed on the lock screen of the second electronic device (402).” As another example, the result value may include information indicating that “a notification window to indicate that the state of the second electronic device (402) has changed is being displayed on the lock screen of the second electronic device (402).”

[0096] The first electronic device (401) can recognize that a notification has occurred in the second electronic device (402) from audio obtained from the microphone (451) using the first artificial intelligence module (471) (e.g., audio model). For example, the first electronic device (401) can obtain audio from the microphone (451). The obtained audio may include sound corresponding to the notification (e.g., vibration sound, guidance voice). The first electronic device (401) can include the obtained audio in a prompt and input it into the model. The first electronic device (401) can recognize that a notification has occurred in the second electronic device (402) based on the result value output from the model. For example, the result value may include information indicating that “sound is being output to notify that a message has been received from the second electronic device (402).” As another example, the result value may include information indicating that “a sound is being output to indicate that the state of the second electronic device (402) has changed.”

[0097] The first electronic device (401) may recognize that a notification has been generated in the second electronic device (402) from an image obtained from the first camera (411) and / or audio obtained from the microphone (451) using the first artificial intelligence module (471) (e.g., multimodal model).

[0098] Based on the recognition that a notification has occurred in the second electronic device (402), the first electronic device (401) may transmit a message requesting information regarding the notification generated in the second electronic device (402) to the third electronic device (403) or the second electronic device (402) via the first communication circuit (421). As an operation resulting from this request, the third electronic device (403) may receive the message from the second electronic device (402) via the third communication circuit (423) and provide it to the first electronic device (401). If the second electronic device (402) receives the request message directly from the first electronic device (401), the second electronic device (402) may transmit the message to the first electronic device (401) via the second communication circuit (422). The first electronic device (401) can receive a message from the third electronic device (403) or the second electronic device (402) through the first communication circuit (421) and display it on the first display (441).

[0099] The first electronic device (401) may display a menu on the first display (441) to allow the user to select whether to receive notification content from an external electronic device (e.g., the second electronic device (402)). When the user is set to receive notification content through the menu, the first electronic device (401) may display a message (e.g., text) on the first display (441) suggesting that the user unlock the screen on the second electronic device (402), and / or output a voice message suggesting unlocking through a speaker provided in the first electronic device (401).

[0100] The second electronic device (402) may include a second communication circuit (422), a second sensor (432), a second display (442), a speaker (452), a second memory (482), and a second processor (492). The second communication circuit (422), the second sensor (432), the second display (442), the speaker (452), the second memory (482), and the second processor (492) can each perform the same functions by being implemented substantially identically to the communication module (190), sensor module (176), display module (160), sound output module (155), memory (130), and processor (120) in the electronic device (101) of FIG. 1.

[0101] Instructions may be stored in the second memory (482). When executed by the second processor (492), the instructions may cause the second electronic device (402) to perform a given operation. According to one embodiment, the second account management module (462), the notification management module (464), and the second artificial intelligence module (472) may be composed of instructions and stored in the second memory (482). According to one embodiment, some of the instructions may be stored in the second memory (482) and other parts may be stored in the internal memory of the second processor (492). The second account management module (462) may perform the same function as the first account management module (461) described above. In response to a request from the third electronic device (403) or the second electronic device (403), the second electronic device (402) can transmit account information managed by the second account management module (462), along with a lock screen image, to the third electronic device (403) or the first electronic device (401) through the second communication circuit (422).

[0102] The notification management module (464) can manage notifications generated by the second electronic device (402). The second electronic device (402) receiving a message from an external device and / or a change in the state of the second electronic device (402) may be events that trigger a notification. For example, when a schedule registered in a calendar application arrives, the second electronic device (402) may generate a notification to inform the user that the schedule has arrived. The notification management module (464) may generate a notification to inform the user when an event related to an update of an application installed on the second electronic device (402) or an event related to security occurs. The second electronic device (402) may provide services to the user based on support from an external server (e.g., a cloud server). The notification management module (464) may also generate notifications regarding server-based services. The notification management module (464) may output sound corresponding to the notification through the speaker (452). The notification management module (464) can display an image corresponding to the notification on the screen (e.g., the lock screen or the indication bar at the top of the screen).

[0103] The notification management module (464) can receive a request message from the first electronic device (401) requesting that relevant information be provided when a notification occurs, from the third electronic device (403) via the second communication circuit (422). If the second electronic device (402) and the first electronic device (401) are interconnected to enable data communication via a short-range wireless communication channel (e.g., WIFI channel, Bluetooth channel), the notification management module (464) may also receive the request message directly from the first electronic device (401) via the second communication circuit (422). In response to the reception of the request message, the notification management module (464) can activate a relay function, also known as a relay, to provide information related to the notification to an external electronic device.

[0104] When the relay function is activated, when a notification occurs, the notification management module (464) can transmit information related to the notification to the third electronic device (403) via the second communication circuit (422). The third electronic device (403) can transmit information related to the notification to the first electronic device (401). When the second electronic device (402) and the first electronic device (401) are interconnected to enable data communication via a short-range wireless communication channel (e.g., a Bluetooth channel), the notification management module (464) may also directly transmit information related to the notification to the first electronic device (401) via the second communication circuit (422) when a notification occurs.

[0105] The second artificial intelligence module (472) (e.g., a large language model (LLM) or a small LLM) may include an artificial intelligence model. The second artificial intelligence module (472) may generate an input value (e.g., a prompt) and feed the input value into the artificial intelligence model. The artificial intelligence model may process the input value and output a result value. According to one embodiment, the second electronic device (402) may use the second artificial intelligence module (472) to summarize content to be communicated to the first electronic device (401) or extract keywords from said content. For example, the second electronic device (402) may feed content corresponding to the notification (e.g., a message received from an external electronic device) into the model by including it in the prompt. The second electronic device (402) may obtain a summary and / or keywords from the result value output from the model. The second artificial intelligence module (472) may summarize the content to be communicated (e.g., summarized into a single sentence) and output it. The second artificial intelligence module (472) may collect and summarize content of the same type (e.g., content regarding a single application or service). The second artificial intelligence module (472) may also extract keywords from the content to be announced. The second artificial intelligence module (472) may summarize and output all content to be announced, regardless of where the source of the content (e.g., application) is.

[0106] According to one embodiment, when a notification occurs while the relay function is activated, the notification management module (464) can transmit information (summary and / or keywords) generated using the second artificial intelligence module (472) to the third electronic device (403) via the second communication circuit (422). The third electronic device (403) can transmit the information to the first electronic device (401). When the second electronic device (402) and the first electronic device (401) are interconnected to enable data communication via a short-range wireless communication channel (e.g., a Bluetooth channel), the notification management module (464) may also directly transmit the information generated using the second artificial intelligence module (472) to the first electronic device (401) via the second communication circuit (422) when a notification occurs.

[0107] According to one embodiment, a third artificial intelligence module identical to the second artificial intelligence module (472) described above may be configured as instructions, stored in the first memory (481) and / or the internal memory of the first processor (491) in the first electronic device (401), and executed by the first processor (491). For example, the first electronic device (401) may receive details of a notification from the third electronic device (403) or the second electronic device (402) via the first communication circuit (421). The first electronic device (401) may summarize the details or extract keywords from the details using the third artificial intelligence module. The first electronic device (401) may display the information (summary and / or keywords) generated using the third artificial intelligence module on the first display (441).

[0108] The third electronic device (403) may include a third communication circuit (423), a third memory (483), and a third processor (493). The third communication circuit (423), the third memory (483), and the third processor (493) can each perform the same function by being implemented substantially identically to the communication module (190), memory (130), and processor (120) in the electronic device (101) of FIG. 1.

[0109] Instructions may be stored in the third memory (483). When executed by the third processor (493), the instructions may cause the third electronic device (403) to perform a given operation. According to one embodiment, the third account management module (463) may be composed of instructions and stored in the third memory (483). According to one embodiment, a portion of the instructions may be stored in the third memory (483) and another portion may be stored in the internal memory of the third processor (493). The third account management module (463) may store and manage information about devices registered to an account. The third account management module (463) may manage multiple accounts as a group (e.g., family). The third account management module (463) may set one account in the group as the primary user's account and manage other accounts as secondary user accounts (e.g., registration and / or deletion). In addition, the third account management module (463) can perform the same function as the first account management module (461) described above.

[0110] According to one embodiment, users of the first electronic device (401) may be classified into primary users and secondary users. Account information related to the first electronic device (401) may include user information and device information for each user. User information may include, for example, a name, a web address, and an ID. Device information may include a phone number, a product name, a model name, and a serial number. Device information may include a lock screen image associated with the device. Additionally, secondary users may be classified by group (e.g., family, friends) based on the primary user. The third account management module (463) may manage the account information of the first electronic device (401) as shown in Table 1 below. The third account management module (463) may synchronize the account information managed in the first electronic device (401) by transmitting the account information (e.g., Table 1 below) to the first electronic device (401) through the third communication circuit (423). Wearing the HMD device on the face may be considered unlocking, and accordingly, the lock screen image of device #1 in Table 1 below may be excluded from management.

[0111] User Information (Group Name) Device Information Main User Device #1 (First Electronic Device (401)) Lock screen image of Device #1 Device #2 Lock screen image of Device #2 Device #3 Lock screen image of Device #3 Auxiliary User 1 (Family) Device #4 (Second Electronic Device (402)) Lock screen image of Device #4 Device #5 Lock screen image of Device #5 Auxiliary User 2 (Friend) Device #6 Lock screen image of Device #6.........

[0112] FIG. 5 is a flowchart illustrating the operation of a first electronic device (401) to verify that a user of a second electronic device (402) is a user of a first electronic device (401) according to one embodiment. According to one embodiment, a first processor (491) in the first electronic device (401) may be configured to perform operations 520, 530, and 540 in FIG. 5. According to one embodiment, instructions (e.g., modules (461, 471)) stored in memory (e.g., first memory (481) and / or internal memory of the first processor (491)) may cause the electronic device to perform operations 520, 530, and 540 when executed by a processor (e.g., first processor (491)). According to one embodiment, operations 520, 530, and 540 may be performed based on the fact that a front-facing camera (e.g., the first camera (411)) in the first electronic device (401) is in an active state. According to one embodiment, operations 520, 530, and 540 may be performed based on the fact that the first electronic device (401) operates in a second mode (guest mode). FIGS. 6A and 6B are drawings illustrating examples regarding the angle of view of the camera and the direction in which the camera is facing in the first electronic device (401) according to one embodiment. FIG. 6A illustrates the case where the first electronic device (401) is viewed from above to below (in the -z-axis direction, with reference to the drawing), and FIG. 6B illustrates the case where the first electronic device (401) is viewed from the side (in the -x-axis direction, with reference to the drawing). FIGS. 6c and 6d are drawings illustrating examples of a method for unlocking a screen. FIGS. 7a and 7b are drawings illustrating a lock screen displayed on a second electronic device (402).

[0113] In operation 501, the second electronic device (402) can transmit an image of the lock screen to the third electronic device (403). For example, device information of the second electronic device (402) may be associated with a user account managed by the third account management module (463) of the third electronic device (403), and the device information may be stored in the third memory (483) by the third account management module (463). Accordingly, the third electronic device (403) can transmit a message requesting an image of the lock screen to the second electronic device (402). The third electronic device (403) can transmit the request message at specified intervals. In response to the reception of the request message, the second electronic device (402) can transmit an image of the lock screen to the third electronic device (403) which is set to be displayed when the second display (442) is turned on. The third electronic device (403) may delete the image of the associated lock screen and stop the transmission of the request message to the second electronic device (402) based on the fact that the device information has been deleted from the user account. According to one embodiment, the second electronic device (402) may transmit status information of the second electronic device (402) to the third electronic device (403) along with the lock screen image. For example, the status information may include information indicating the completion of unlocking. The status information may include information regarding the image displayed on the second display (422) after unlocking and / or information regarding the application executed after unlocking.

[0114] In operation 502, the third electronic device (403) can store the image of the lock screen (additionally, status information) received from the second electronic device (403) in the third memory (483) to be linked to the corresponding user account.

[0115] In addition to the device information of the second electronic device (402), device information of the first electronic device (401) is associated with a user account (e.g., an account of an auxiliary user of the first electronic device (402)) managed by the third account management module (463) of the third electronic device (403), and the device information can be stored in the third memory (483) by the third account management module (463). Accordingly, in operation 503, the third electronic device (403) can transmit the account information of the user associated therewith, along with an image of the lock screen (additionally, status information), to the first electronic device (401).

[0116] Operation 510, including the above-described operations 501, 502, and 503, may be performed before operations 520 and 530, which will be described later, are performed. Alternatively, operation 510 may be performed while operations 520 or 530 are being performed.

[0117] In operation 520, the first electronic device (401) can acquire an image from a camera. Referring to FIGS. 6a and 6b, the first electronic device (401) may include a first front camera (601) corresponding to the right eye and / or a second front camera (602) corresponding to the left eye. The angle indicated by reference numeral 611 signifies the horizontal angle of view of the first front camera (601). The angle indicated by reference numeral 612 signifies the horizontal angle of view of the second front camera (602). The angle indicated by reference numeral 621 signifies the vertical angle of view of the first front camera (601). The vertical angle of view of the second front camera (601) may be the same as the vertical angle of view of the first front camera (601). An object may exist within the viewable area that enters the field of view of the first front camera (601) and / or the second front camera (602). For example, the second electronic device (402) may be present within a visible area. In this state, the second electronic device (402) may display a lock image (402a). The user may perform an action to manipulate the lock image (402a) to unlock the second electronic device (402) (e.g., a touch gesture (402b) of sliding the screen from left to right). The aforementioned visible area may be a concept that includes a horizontal visible area determined by a horizontal angle of view and a vertical visible area determined by a vertical angle of view. Referring to FIG. 6c, the second electronic device (402) may display a lock image (402a) that includes a pattern (630) for unlocking the screen. The user may touch (640) the pattern (630) to unlock the second electronic device (402). Referring to FIG. 6d, the second electronic device (402) can display a lock image (402a) including a keypad (650) and a password input field (660).

[0118] In operation 530, the first electronic device (401) can recognize the unlocking of the screen lock based on the acquired image. According to one embodiment, the first electronic device (401) can recognize the subject and its state or action from the acquired image using the first artificial intelligence module (471).

[0119] In operation 540, the first electronic device (401) can recognize that the user of the second electronic device (402) is the same person as the user of the first electronic device (401) based at least on the result of comparing the image of the lock screen received from the third electronic device (403) with the image of the subject recognized as the lock screen by the camera.

[0120] According to one embodiment, the first electronic device (401) may obtain status information (e.g., information indicating the completion of unlocking or an image displayed after unlocking and / or information on an executed application) from the subject image. The first electronic device (401) may also recognize that the user of the second electronic device (402) is the same person as the user of the first electronic device (401) based further on the fact that the status information obtained from the subject image matches the status information received from the third electronic device (403).

[0121] According to one embodiment, the first electronic device (401) can identify a user's operation (action) to unlock the screen of the second electronic device (402) (e.g., a touch gesture (402b) in FIG. 6a, a touch (640) on a pattern (630) in FIG. 6c, or a touch on a keypad (650) in FIG. 6d) in an image obtained from a camera. The first electronic device (401) can receive information from the third electronic device (403) indicating the user's unlocking action (e.g., information indicating the trajectory of the touch (640) or information regarding a number selected on the keypad (650). The first electronic device (401) compares the received information with the user's unlocking action obtained through the camera, and based further on the result of the comparison, may recognize that the user of the second electronic device (402) is the same person as the user of the first electronic device (401).

[0122] According to one embodiment, the first electronic device (401) may obtain a pattern used by a user for unlocking (e.g., a pattern for unlocking a screen lock as in FIG. 6c) or a passcode (e.g., a password entered via a keypad (650) as in FIG. 6d) and send a request message for user verification containing the obtained information to a server (e.g., a third electronic device (403)). The server may verify the user information in the request message and compare it with reference information present on the server (e.g., pattern and / or passcode information in user account information). The server may send a response message to the first electronic device (401) containing information indicating whether the two pieces of information match as a result of the comparison. The first electronic device (401) may confirm from the response message that the two pieces of information match and, accordingly, finally determine that the user of the second electronic device (402) is the wearer of the first electronic device (401).

[0123] According to one embodiment, a server (e.g., a third electronic device (403)) can verify user information in a request message for user verification and compare it with reference information. As a result of the comparison, the server can send a response message to the first electronic device (401) indicating that it is difficult to determine whether the two pieces of information match. In this way, if it is determined through verification via the server that it is difficult to verify the contents of the pattern or passcode, the first electronic device (401) can perform an operation to determine whether the user of the second electronic device (402) is the wearer of the first electronic device (401) based solely on whether it is unlocked and whether the lock state has changed.

[0124] According to one embodiment, the first electronic device (401) can measure the similarity of two images using the first artificial intelligence module (471) and determine whether the two images match based on the result. Referring to FIGS. 7a and 7b, the first electronic device (401) can receive the first lock screen image (701) of FIG. 7a from the third electronic device (403). The second electronic device (401) can extract the second lock screen image (702) of FIG. 7b from an image obtained from a camera. The first lock screen image (701) may include an indication bar (711), a navigation bar (712), and a background image (713). The second lock screen image (702) may include an indication bar (721), a navigation bar (722), and a background image (723), along with a pop-up window (e.g., a window for displaying information indicating the status of an application running on the first electronic device (401)) (724). The first electronic device (401) may generate a prompt including two images (701, 702), the source of the first lock screen image (701) (e.g., user information, device information), and a list of user accounts managed by the first account management module (461), and may input the prompt (input value) into the first artificial intelligence module (471). The first electronic device (401) may recognize from the result value output from the first artificial intelligence module (471) that the user of the second electronic device (402) is the same person as the user of the first electronic device (e.g., an auxiliary user).

[0125] FIG. 8 is a flowchart illustrating the operation of a first electronic device (401) to enable a user of the first electronic device (401) to recognize a notification generated by a second electronic device (402) according to one embodiment. According to one embodiment, a first processor (491) in the first electronic device (401) may be configured to perform the operation of FIG. 8. According to one embodiment, instructions (e.g., modules (461, 471)) stored in memory (e.g., first memory (481) and / or internal memory of the first processor (491)) may cause the electronic device to perform the operation of FIG. 8 when executed by a processor (e.g., first processor (491)). According to one embodiment, the operation of FIG. 8 may be performed based on the fact that a front-facing camera (e.g., first camera (411)) in the first electronic device (401) is in an active state. According to one embodiment, the operation of FIG. 8 can be performed based on the first electronic device (401) operating in a second mode (guest mode).

[0126] In operation 810 (e.g., operation 540), the first electronic device (401) can recognize that the user of the second electronic device (402) is the same person as the user of the first electronic device (401) based on the result of comparing the image of the lock screen received from the third electronic device (403) with the image of the subject recognized as the lock screen by the camera. According to one embodiment, the first electronic device (401) can set the second electronic device (402) as an object of interest and track it using the camera based on the fact that the user of the second electronic device (402) is identified as the user of the first electronic device (401).

[0127] In operation 820, the first electronic device (401) can transmit a message requesting the activation of a relay function to the second electronic device (402) through the third electronic device (403). In response to the activation request, in operation 821, the second electronic device (402) can activate the relay function.

[0128] In operation 830, the first electronic device (401) may receive notification content from the second electronic device (402) via the third electronic device (403) in response to requesting a relay from the second electronic device (402). The notification content received by the first electronic device (401) may be the source (detailed content) that is the subject of the notification generated by the second electronic device (402) and / or simple content obtained from the source (e.g., summary and / or keywords). Here, the source may include a message received by the second electronic device (402) from the outside and / or a message regarding the status of the second electronic device (402). According to one embodiment, the second electronic device (402) may transmit the notification content to the first electronic device (401) via the third electronic device (403) whenever a notification occurs. According to one embodiment, the second electronic device (402) acquires the notification content whenever a notification occurs, and when the number of notification occurrences reaches a specified value, the acquired multiple notification contents can be transmitted to the first electronic device (401) through the third electronic device (403). According to one embodiment, the second electronic device (402) acquires the notification content whenever a notification occurs, and at specified intervals, the acquired multiple notification contents can be transmitted to the first electronic device (401) through the third electronic device (403).

[0129] In operation 840, the first electronic device (401) can display notification content on the first display (441). According to one embodiment, if the notification content received from the second electronic device (402) is the original (detailed content), the first electronic device (401) can perform an operation to simplify the original to obtain simple content (e.g., an operation to summarize the original and / or an operation to extract keywords from the original) and, as a result of the operation, display the obtained simple content (summary and / or keywords) on the first display (441). According to one embodiment, the first electronic device (401) can identify an empty area (or empty space) among the display areas of the first display (441) where visual information (e.g., a virtual image) is not being displayed and compare the amount of the empty area with the amount of notification content (detailed content). As a result of the comparison, the first electronic device (401) can determine that the size of the empty area is insufficient to display the entire notification content. In accordance with this decision, the first electronic device (401) may perform an operation to obtain simple content (summary and / or keywords) by simplifying the notification content (detailed content) and display the obtained simple content in an empty area as a result of the operation. According to one embodiment, the first electronic device (401) may display the corresponding original (detailed content) on the first display (441) in response to the simple content being selected by the user (e.g., it is confirmed that the user's gaze has stopped for a specified time on the summary or keywords).

[0130] In operation 850, the first electronic device (401) may determine to release the relay based on the wearing status of the first electronic device (401) or the result of tracking the second electronic device (402) using a camera. For example, the first electronic device (401) may confirm that the first electronic device (401) has been unworn based on data received from the first sensor (431). This unworn status may serve as a condition for determining to release the relay. As another example, it may be unworn and then worn again within a certain period of time. Accordingly, if the first electronic device (401) is not worn again by the user until a specified time has elapsed from the time it was unworn, the first electronic device (400) may determine to release the relay. As another example, the first electronic device (401) may send a message notifying the unworn status to the third electronic device (403). The third electronic device (403) can determine whether the second electronic device (402) is in use through data communication with the second electronic device (402) based on the receipt of a wear release notification message. The first electronic device (401) can receive information from the third electronic device (403) indicating that the second electronic device (402) is in use. If it is confirmed that the second electronic device (402) is in use, the first electronic device (400) can decide to release the relay. As another example, the first electronic device (401) can determine from an image received from the camera that the second electronic device (402) has moved out of the camera's field of view for more than a specified time, and accordingly decide to release the relay.

[0131] In operation 860, the first electronic device (401) may transmit a message requesting relay release to the second electronic device (402) via the third electronic device (403) based on the condition for relay release being satisfied. In response to the request message, the second electronic device (402) may disable the relay function.

[0132] FIG. 9 is a flowchart illustrating the operation of a first electronic device (401) to enable a user of the first electronic device (401) to recognize a notification generated by a second electronic device (402) according to one embodiment. According to one embodiment, a first processor (491) in the first electronic device (401) may be configured to perform the operation of FIG. 9. According to one embodiment, instructions (e.g., modules (461, 471)) stored in memory (e.g., first memory (481) and / or internal memory of the first processor (491)) may cause the electronic device to perform the operation of FIG. 9 when executed by a processor (e.g., first processor (491)). According to one embodiment, the operation of FIG. 9 may be performed based on the fact that a front-facing camera (e.g., first camera (411)) in the first electronic device (401) is in an active state. According to one embodiment, the operation of FIG. 9 can be performed based on the first electronic device (401) operating in a second mode (guest mode).

[0133] In operation 910 (e.g., operation 540), the first electronic device (401) can recognize that the user of the second electronic device (402) is the same person as the user of the first electronic device (401) based on the result of comparing the image of the lock screen received from the third electronic device (403) with the image of the subject recognized as the lock screen by the camera. According to one embodiment, the first electronic device (401) can set the second electronic device (402) as an object of interest and track it using the camera based on the fact that the user of the second electronic device (402) is identified as the user of the first electronic device (401).

[0134] In operation 920, the first electronic device (401) can recognize that a notification has occurred in the second electronic device (402) through the analysis of an image obtained from a camera and / or audio obtained from a microphone. The first artificial intelligence module (471) as described above may be used for the analysis of the image and / or audio.

[0135] In operation 930, the first electronic device (401) may transmit a message requesting notification content to the second electronic device (402) through the third electronic device (403) based on recognizing the occurrence of a notification. According to one embodiment, the first electronic device (401) may transmit a message requesting notification content to the second electronic device (402) through the third electronic device (403) in response to user input received from an input device (e.g., a touch pad) provided in the first electronic device (401).

[0136] In operation 940, the first electronic device (401) may receive notification content from the second electronic device (402) via the third electronic device (403) in response to the request from the second electronic device (402). The notification content received by the first electronic device (401) may be the source (detailed content) that is the subject of the notification generated by the second electronic device (402) and / or simple content obtained from the source (e.g., summary and / or keywords). Here, the source may include a message received by the second electronic device (402) from an external source and / or a message regarding the status of the second electronic device (402).

[0137] In operation 950, the first electronic device (401) can display the notification content on the first display (441). According to one embodiment, if the notification content received from the second electronic device (402) is the original, the first electronic device (401) can perform an operation to simplify the original to obtain simple content (e.g., summary and / or keywords) and, as a result of the operation, display the obtained summary and / or keywords on the first display (441). According to one embodiment, the first electronic device (401) can identify an empty area among the display areas of the first display (441) where visual information (e.g., virtual image) is not being displayed and compare the empty area with the amount of the notification content. As a result of the comparison, the first electronic device (401) can determine that the size of the empty area is insufficient to display the entire notification content. Based on this determination, the first electronic device (401) can perform an operation to simplify the notification content (original) to obtain simple content (e.g., summary and / or keywords) and, as a result of the operation, display the obtained simple content in the empty area. According to one embodiment, the first electronic device (401) can display the original content on the first display (441) in response to a simple content being selected by the user (e.g., the user's gaze being stopped for a specified time on a summary or keyword).

[0138] FIG. 10 is a flowchart illustrating the operation of a first electronic device (401) to enable a user of the first electronic device (401) to recognize a notification generated by a second electronic device (402) according to one embodiment. According to one embodiment, a first processor (491) in the first electronic device (401) may be configured to perform the operation of FIG. 10. According to one embodiment, instructions (e.g., modules (461, 471)) stored in memory (e.g., first memory (481) and / or internal memory of the first processor (491)) may cause the electronic device to perform the operation of FIG. 10 when executed by a processor (e.g., first processor (491)). According to one embodiment, the operation of FIG. 10 may be performed based on the fact that a front-facing camera (e.g., first camera (411)) in the first electronic device (401) is in an active state. According to one embodiment, the operation of FIG. 10 can be performed based on the first electronic device (401) operating in a second mode (guest mode).

[0139] In operation 1010 (e.g., operation 540), the first electronic device (401) can recognize that the user of the second electronic device (402) is the same person as the user of the first electronic device (401) based on the result of comparing the image of the lock screen received from the third electronic device (403) with the image of the subject recognized as the lock screen by the camera. According to one embodiment, the first electronic device (401) can set the second electronic device (402) as an object of interest and track it using the camera based on the fact that the user of the second electronic device (402) is identified as the user of the first electronic device (401).

[0140] In operation 1020, the first electronic device (401) can transmit a message to the third electronic device (403) requesting a setting for wireless communication with the second electronic device (402) based on the recognition described above.

[0141] In response to a request from the first electronic device (401), in operation 1021, the third electronic device (403) may transmit a message to the second electronic device (402) requesting the activation of a wireless AP (access point) function (e.g., mobile hotspot). According to one embodiment, the third electronic device (403) may generate an ID (e.g., SSID (service set identifier)) and a password (e.g., one-time) and include them in the request message and transmit them to the second electronic device (402). According to one embodiment, the third electronic device (403) may encrypt the ID and password to be transmitted to the second electronic device (402) with a designated first encryption key and transmit them to the second electronic device (402). According to one embodiment, the ID and password may be predetermined. In this case, the third electronic device (403) may transmit a request message to the second electronic device (402) for the activation of the wireless AP function without an ID and password.

[0142] In response to a request from the first electronic device (401), in operation 1023, the third electronic device (403) may transmit a response message to the first electronic device (401) containing information (ID and password) necessary for establishing a channel for wireless communication with the second electronic device (402). Here, the ID and password may be one-time generated by the third electronic device (403). Alternatively, the ID and password may be verified from the account information of the user concerned. For example, the third electronic device (403) may verify the ID and password from the account information of the user concerned and transmit a response message containing them to the first electronic device (401). According to one embodiment, the third electronic device (403) may encrypt the ID and password to be transmitted to the first electronic device (401) with a designated second encryption key (e.g., the first encryption key described above or another encryption key) and transmit them to the first electronic device (401).

[0143] In response to a request from the third electronic device (403), in operation 1025, the second electronic device (402) can activate an AP function for direct wireless communication with the first electronic device (401).

[0144] In operation 1030, the first electronic device (401) can establish a channel for short-range wireless communication with the second electronic device (402) using information (ID and password) provided by the third electronic device (403). After the wireless communication channel is established, the first electronic device (401) can transmit a message requesting the activation of a relay function to the second electronic device (402) through the channel. In response to the request from the first electronic device (401), the second electronic device (402) can activate the relay function.

[0145] In operation 1040, the first electronic device (401) may receive notification content from the second electronic device (402) in response to requesting a relay from the second electronic device (402). The notification content received by the first electronic device (401) may be the source (detailed content) that is the subject of the notification generated by the second electronic device (402) and / or simple content obtained from the source (e.g., summary and / or keywords). Here, the source may include a message received from the outside by the second electronic device (402) and / or a message regarding the status of the second electronic device (402). According to one embodiment, the second electronic device (402) may transmit the notification content to the first electronic device (401) whenever a notification occurs. According to one embodiment, the second electronic device (402) may obtain the notification content whenever a notification occurs and, when the number of notification occurrences reaches a specified value, transmit the obtained multiple notification contents to the first electronic device (401). According to one embodiment, the second electronic device (402) can acquire the notification content whenever a notification occurs and, at designated intervals, transmit the acquired multiple notification contents to the first electronic device (401).

[0146] In operation 1050 (e.g., operation 840), the first electronic device (401) can display notification content on the first display (441).

[0147] In operation 1060 (e.g., operation 850), the first electronic device (401) may determine to release the relay based on the wearing status of the first electronic device (401) or the result of tracking the second electronic device (402) using a camera.

[0148] In operation 1070 (e.g., operation 860), the first electronic device (401) may transmit a message requesting relay release to the second electronic device (402) based on the fact that the conditions for relay release are satisfied. In response to the request message, the second electronic device (402) may disable the relay function. Additionally, the second electronic device (402) may disable the wireless AP function.

[0149] FIG. 11 is a flowchart illustrating an operation performed in an electronic device (e.g., a first electronic device (401) or a second electronic device (402)) according to one embodiment. According to one embodiment, instructions may be stored in the memory of the electronic device. When the instructions are executed by a processor, the electronic device may be able to perform the operation of FIG. 11. According to one embodiment, when performing the operation of FIG. 11, the electronic device may use an artificial intelligence model (e.g., a second artificial intelligence module (472) or a third artificial intelligence module identical thereto). FIG. 12 is a diagram illustrating an operation to generate a prompt to be fed into an artificial intelligence model.

[0150] In operation 1110, the electronic device can obtain notification content (detailed content). Referring to FIG. 12, a plurality of notification storage units (1210) may store notification content managed by a notification management module (1201) (e.g., notification management module (464)). The plurality of notification storage units (1210) may be separated by the source of the notification content (e.g., application). The notification content stored in each of the plurality of notification storage units (1210) may include source information, the time of notification occurrence, and detailed content (original). If the source of the notification content is a message application, the notification content may include sender information of the message (e.g., mobile phone number, name), message content, and attachments. If the source of the notification content is a calendar application, the notification content may include schedule content, date, location, and contact information associated with the schedule. The electronic device can obtain notification content (e.g., notification content 1, 2, 3… newly added to the multiple notification stores (1210)) from multiple notification stores (1210).

[0151] In operation 1120, the electronic device can obtain simple content (e.g., a summary and / or keywords) by simplifying the acquired notification content. Referring to FIG. 12, the electronic device can complete a prompt (1240) to be fed into an artificial intelligence model (1230) by adding the notification content obtained as a result of performing operation 1110 to a pre-prepared prompt template (1220). The prompt template (1220) may include instructions (1221) indicating what the artificial intelligence model (1230) should do. For example, the artificial intelligence model (1230) may perform an action of summarizing detailed content (original) and / or an action of extracting keywords from the notification content based on the instructions (1221). For example, the instructions (1221) may be: “Summarize the notification content in bullet points by sender (or by application), extract three keywords from the notification content, and add # before the keywords.” Instructions (1221) can be configured in various ways. For example, a phrase such as “arrange multiple summaries in chronological order” may be added to instructions (1221). The electronic device can input the completed prompt (1240) into the artificial intelligence model (1230). As a result, the artificial intelligence model (1230) can summarize the notification content and extract keywords from the notification content based on the instructions (1220). The output produced by the artificial intelligence model (1230) can be grouped by designated topics based on the content of the prompt (e.g., instructions). For example, the instructions may include content such as “sort the notification content by its source (e.g., application)” or “sort the notification content by sender if the source of the notification content is a phone application or a messenger application.”

[0152] In operation 1130, the electronic device can output simple content. If the electronic device is a second electronic device (402), the second electronic device (402) can transmit a message containing simple content (e.g., a summary and / or keywords) to the first electronic device (401) via (or directly) the third electronic device (403). If the electronic device is a first electronic device (401), the first electronic device (401) can display simple content on the first display (441).

[0153] FIG. 13 is a flowchart illustrating the operation of a first electronic device (401) for providing a user with notification content received from a second electronic device (402) according to one embodiment. According to one embodiment, a first processor (491) in the first electronic device (401) may be configured to perform the operation of FIG. 13. According to one embodiment, instructions stored in memory (e.g., first memory (481) and / or internal memory of the first processor (491)) (e.g., a third artificial intelligence module identical to the second artificial intelligence module (472)) may cause the electronic device to perform the operation of FIG. 13 when executed by a processor (e.g., the first processor (491)).

[0154] In operation 1310 (e.g., operation 830, 940, or 1040), the first electronic device (401) can receive notification content from the second electronic device (402).

[0155] In operation 1320, the first electronic device (401) can determine whether the notification content needs to be simplified based on the result of comparing the amount of the provided notification content and the size of the display area.

[0156] If it is determined that simplification is not necessary, the first electronic device (401) in operation 1330 may display the notification content on the first display (441).

[0157] If it is determined that simplification is necessary, in operation 1340, the first electronic device (401) can simplify the notification content (detailed content) and display the simplified content provided as a result of the simplification on the first display (441). For example, the first electronic device (401) can identify an empty area (or empty space) in the display area of ​​the first display (441) where visual information (e.g., a virtual image) is not being displayed, and compare the amount of the empty area with the amount of the notification content. As a result of the comparison, the first electronic device (401) can determine that the size of the empty area is insufficient to display the entire notification content. Based on this determination, the first electronic device (401) can obtain a summary and / or keywords through the operation of FIG. 11 as described above. The first electronic device (401) can display the obtained simplified content (e.g., summary and / or keywords) on the first display (441).

[0158] FIGS. 14a, 14b, 14c, 14d, 14e, 14f, 14g, and 14h are drawings for illustrating various examples of notification content displayed on the display (1401) of an HMD device (e.g., a first electronic device (401)). FIGS. 14i and 14j are drawings for illustrating examples of messages displayed on the HMD device and the external electronic device when the HMD device requests relay activation from the external electronic device (e.g., a second electronic device (402)) and the external electronic device receives such request from the HMD device.

[0159] The HMD device can receive notifications from an external electronic device (e.g., a second electronic device (402) or a third electronic device (403)).

[0160] Referring to FIGS. 14a and 14b, the HMD device can display a message (1410) on a display (1401) to allow the user to recognize the notification content. The HMD device can display user interface (UI) elements (e.g., a 'View' button and a 'Close' button as illustrated) within the message (1410) to allow the user to select whether to display the notification content. If the user selects to view it (e.g., when the 'View' button is selected via an input device provided in the HMD device), the HMD device can display the corresponding notification content (e.g., a message received from an external source via an internet banking application) (1411) on the display (1401).

[0161] The HMD device may receive multiple notifications from an external electronic device (e.g., a second electronic device (402) or a third electronic device (403)). The multiple notifications received may have been acquired from the external electronic device (e.g., the second electronic device (402)) over a certain period of time.

[0162] Referring to FIGS. 14c and 14d, the HMD device may display multiple notification contents on the display (1401) selectively or sequentially one by one. For example, the HMD device may display notification content 1 (e.g., a message received from an external source via an internet banking application) (1421) on the display (1401). The HMD device may display a first UI element (1431) on the display (1401) to display other notification contents along with notification content 1 (1421). For example, multiple notification contents may be arranged in the order of acquisition time, and notification content 1 (1421) may be acquired first and displayed on the display (1401) with the highest priority. The HMD device may display a first UI element (1431) to move to the next notification content along with notification content 1 (1421). When the first UI element (1431) is selected by the user, the HMD device can display notification content 2 (1422) on the display (1401). The HMD device can display a second UI element (1432) for moving to the previous notification content on the display (1401) along with notification content 2 (e.g., a message received from outside through an application providing a notification service regarding class progress) (1422). When another notification content exists, the HMD device can display the first UI element (1431) on the display (1401) along with notification content 2 (1422) and the second UI element (1432).

[0163] A number of notification contents provided to the HMD device from an external electronic device (e.g., a second electronic device (402)) may each include information regarding the source (application).

[0164] Referring to FIG. 14e, the HMD device can classify a number of provided notification contents (1440) by their source and display them on the display (1401). For example, the HMD device can group and display notification contents (1441) sourced from application 1 (e.g., internet banking application) together with source information (e.g., name of the application) (1442), and group and display notification contents (1443) sourced from application 2 (e.g., school application) together with source information (1444).

[0165] A number of notification contents provided to the HMD device from an external electronic device (e.g., a second electronic device (402)) may each include simple content (e.g., a summary and / or keywords). Alternatively, the HMD device may obtain simple content by simplifying the detailed content provided from the external electronic device.

[0166] Referring to FIG. 14f and 14g, the HMD device can classify a number of simple contents by their source and display them on the display (1401). For example, the HMD device can group and display simple contents (e.g., summary and / or keywords) (1451) sourced from Application 1 along with source information (e.g., name of the application) and simple contents (e.g., summary and / or keywords) (1452) sourced from Application 2 along with source information. Referring to FIG. 14g, the HMD device can display the corresponding detailed contents (1461) on the display (1401) when the simple contents (1451) are selected by the user.

[0167] The outputs produced by the artificial intelligence model (e.g., artificial intelligence model (1230)) on the HMD device can be grouped by designated topics based on the content of the prompt (e.g., instructions). For example, multiple notification contents may be classified based on a designated topic (e.g., caller) even though their sources (applications) are different. Referring to FIG. 14h, the HMD device can display on the display (1401) a caller's notification contents (1470) including missed calls and messages received from the caller, along with the caller's identification information (e.g., caller's name “Kim Samsung”). For example, the missed call and message may be displayed together on the display (1401) as the artificial intelligence model determines that they are related, even though their sources are different.

[0168] Referring to FIG. 14i, an HMD device (e.g., a first electronic device (401)) may display a menu (1481) on a display (1401) that allows a user to select whether to send a message requesting the activation of a relay function. When the transmission of the request message is determined by the user selecting the 'OK' button in the menu (1481), the HMD device may send the request message. The corresponding notification provider (e.g., a second electronic device (402)) that receives the request message may notify the user that the request message has arrived and display a menu (1482) that allows the user to select whether to activate the relay function. If the user selects the 'Accept' button in the menu (1482), the notification provider may activate the relay function in response to the request of the HMD device. If the user selects the 'Reject' button in the menu (1482), the notification provider may not respond to the request of the HMD device. The notification provider may send a message to the HMD device indicating that the request has been rejected.

[0169] The HMD device and / or the device providing notification content to the HMD device may notify the user that the relay function has been terminated. Alternatively, the HMD device and / or the notification providing device may provide a menu that allows the user to select whether to terminate (disable) the relay function. Referring to FIG. 14j, when the HMD device is unworn, a message indicating unworn is transmitted from the HMD device to the notification providing device, and accordingly, the notification providing device may display a menu (1491) to allow the user to select whether to terminate the relay. Since the device can be unworn and then put back on, the notification providing device may also allow the user to select to maintain the connection through the menu (1491). The HMD device may set the notification providing device as an object of interest to be tracked. Even if the HMD device is not unworn, if it is confirmed that the object of interest has disappeared from the HMD device's field of view for a certain period of time or longer, the HMD device may display a menu (1492) on the display (1401) to allow the user (wearer) to select whether to terminate the relay. The user may terminate or maintain the relay through the menu (1492). When the termination of the relay is selected through the menu (1492) (e.g., the 'OK' button is selected), the HMD device can send a message to the notification provider requesting the deactivation of the relay function.

[0170] According to one embodiment, an electronic device wearable on a user's face is provided. The electronic device comprises: a display positioned to illuminate the user's eyes located inside the electronic device when worn on the user's face; a camera positioned on the front of the electronic device and configured to photograph the outside; a communication circuit for wirelessly communicating with an external electronic device; a memory for storing instructions; and at least one processor. When the above command is executed individually or collectively by the at least one processor, the electronic device may acquire a lock screen image from an image acquired using the camera, compare a reference image received from a first external electronic device (e.g., a third electronic device (403)) through the communication circuit with the lock screen image, and based on the result of the comparison that the lock screen image and the reference image match, transmit a first message through the communication circuit to the first external electronic device requesting the activation of a function to relay notification content to a second external electronic device (e.g., a second electronic device (402)) which is the source of the reference image, and in response to the transmission of the first message, receive a second message from the first external electronic device through the communication circuit and display at least a portion of the second message on the display.

[0171] When the above command is executed individually or collectively by the at least one processor, the electronic device may detect a user's action of unlocking the screen in an image obtained using the camera, and based on the detection of said action, obtain the lock screen image in the image obtained using the camera.

[0172] When the above command is executed individually or collectively by the at least one processor, the electronic device may input an image acquired using the camera into an artificial intelligence model as an input value, and verify the action in the output value output from the artificial intelligence model.

[0173] When the above command is executed individually or collectively by the at least one processor, the electronic device may input the lock screen image and the reference image as input values ​​to an artificial intelligence model and check for a match between the lock screen image and the reference image in the output value output from the artificial intelligence model.

[0174] When the above instruction is executed individually or collectively by the at least one processor, the electronic device may receive first account information from the first external electronic device through the communication circuit and transmit the first message to the first external electronic device through the communication circuit based on the existence of second account information matching the first account information in the electronic device.

[0175] The above second message may include the above notification content. The above notification content may include detailed content and simple content obtained from the above detailed content. When the above command is executed individually or collectively by the at least one processor, the electronic device may display the simple content on the display and, based on the simple content selected by the user, display the detailed content.

[0176] When the above command is executed individually or collectively by the at least one processor, the electronic device may input the notification content included in the second message as an input value to an artificial intelligence model along with specified instructions, obtain simple content from the output value output from the artificial intelligence model, display the simple content on the display, and display the notification content based on the simple content selected by the user.

[0177] When the above command is executed individually or collectively by the at least one processor, the electronic device inputs an image acquired using the camera into an artificial intelligence model, confirms that a notification has occurred in the second external electronic device from the output value output from the artificial intelligence model, and based on confirming that a notification has occurred in the second external electronic device, transmits a third message to the first external electronic device through the communication circuit requesting the transmission of the notification content to the second external electronic device (e.g., operation 930), and in response to the transmission of the third message, receives a fourth message from the first external electronic device through the communication circuit and displays at least a portion of the fourth message on the display.

[0178] When the above instruction is executed individually or collectively by the at least one processor, the electronic device may receive information necessary for establishing a channel for short-range wireless communication with the second external electronic device through the communication circuit, receive a fifth message from the second external electronic device through the short-range wireless communication channel established based on the information (e.g., operation 1040), and display at least a portion of the fifth message on the display.

[0179] When the above command is executed individually or collectively by the at least one processor, the electronic device may perform the operation of classifying a plurality of notification contents included in the second message by the source of the notification contents and displaying them on the display, or classifying the plurality of notification contents by a designated topic and displaying them on the display.

[0180] According to one embodiment, a method for operating an electronic device wearable on a user's face is provided. The electronic device may include: a display positioned to illuminate the user's eyes located inside the electronic device when worn on the user's face; a camera positioned on the front of the electronic device and configured to photograph the outside; and a communication circuit for wirelessly communicating with an external electronic device. The method may include: an operation of acquiring a lock screen image from an image acquired using the camera; an operation of comparing a reference image received from a first external electronic device through the communication circuit with the lock screen; an operation of transmitting a first message to the first external electronic device through the communication circuit, based on the result of the comparison that the lock screen image and the reference image match, requesting the activation of a function to relay notification content to a second external electronic device, which is the source of the reference image; an operation of receiving a second message from the first external electronic device through the communication circuit in response to the transmission of the first message; and an operation of displaying at least a portion of the second message on the display.

[0181] The operation of acquiring the lock screen image may include: an operation of confirming a user’s action of unlocking the screen in an image acquired using the camera; and an operation of acquiring the lock screen image in an image acquired using the camera based on the confirmation of the action.

[0182] The operation of confirming the above action may include: an operation of inputting an image obtained using the camera into an artificial intelligence model as an input value; and an operation of confirming the above action from an output value output from the artificial intelligence model.

[0183] The operation of transmitting the first message may include: an operation of inputting the lock screen image and the reference image as input values ​​into an artificial intelligence model; an operation of verifying the match between the lock screen image and the reference image in the output value output from the artificial intelligence model; and an operation of transmitting the first message to the first external electronic device through the communication circuit based on the verification of the match between the lock screen image and the reference image.

[0184] The operation of transmitting the first message may include: receiving first account information from the first external electronic device through the communication circuit; and transmitting the first message to the first external electronic device through the communication circuit based on the existence of second account information matching the first account information in the electronic device.

[0185] In the above explanation, prefixes such as “first,” “second,” and “third” are intended merely to distinguish components of the same name and are not assigned any special meaning in themselves, such as importance or order.

[0186] In the present disclosure, the expression 'connection' means not only a direct connection between components, but also an electrical connection where other components (e.g., resistors, inductors, etc.) are present between components.

[0187] The electronic device according to the various embodiments disclosed in this document may be of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. The electronic device according to the embodiments of this document is not limited to the devices described above.

[0188] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as "coupled" or "connected" to another (e.g., 2nd) component, with or without the terms "functionally" or "communicationly," it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0189] The term “module” as used in the various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. According to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0190] Various embodiments of the present document may be implemented as software (e.g., program (140)) comprising one or more instructions stored in a storage medium (e.g., internal memory (136) or external memory (138)) readable by a machine (e.g., electronic device (101)). For example, a processor (e.g., processor (120)) of the machine (e.g., electronic device (101)) may call at least one of the one or more instructions stored in the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0191] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or an application store (e.g., Play Store). TM It can be distributed online (e.g., downloaded or uploaded) through ) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0192] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In an electronic device wearable on a user's face, A display positioned to shine light into the user's eyes located inside the electronic device when the electronic device is worn on the user's face; A camera positioned on the front of the electronic device and configured to photograph the outside; Communication circuit for wireless communication with an external electronic device; Memory for storing instructions; and It includes at least one processor, When the above instruction is executed individually or collectively by the at least one processor, the electronic device, A lock screen image is obtained from an image obtained using the above camera, and A reference image received from a first external electronic device through the above communication circuit is compared with the lock screen image, and Based on the fact that the lock screen image and the reference image match as a result of the above comparison, a first message requesting the activation of a function to relay notification content to a second external electronic device, which is the source of the reference image, is transmitted to the first external electronic device through the communication circuit, and In response to the transmission of the first message, a second message is received from the first external electronic device through the communication circuit, and An electronic device that displays at least a portion of the above second message on the display.

2. In claim 1, when the instruction is executed individually or collectively by the at least one processor, the electronic device, Confirming the user's action of unlocking the screen in an image obtained using the above camera, and An electronic device that obtains the lock screen image from an image obtained using the camera, based on the confirmation of the above action.

3. In claim 2, when the instruction is executed individually or collectively by the at least one processor, the electronic device, The image obtained using the above camera is fed as an input value into an artificial intelligence model, and An electronic device that verifies the above action from the output value output from the above artificial intelligence model.

4. In claim 1, when the instruction is executed individually or collectively by the at least one processor, the electronic device, The above lock screen image and the above reference image are fed as input values ​​into an artificial intelligence model, and An electronic device that verifies the match between the lock screen image and the reference image in the output value output from the artificial intelligence model.

5. In claim 1, when the instruction is executed individually or collectively by the at least one processor, the electronic device, Receive first account information from the first external electronic device through the communication circuit above, and An electronic device that transmits the first message to the first external electronic device through the communication circuit based on the existence of second account information matching the first account information in the electronic device.

6. In Paragraph 1, The above second message includes the above notification content, and The above notification includes detailed information and brief information obtained from the above detailed information, and When the above instruction is executed individually or collectively by the at least one processor, the electronic device, Display the above simple content on the above display, and An electronic device that displays the above detailed content based on the above simple content selected by the user.

7. In claim 1, when the instruction is executed individually or collectively by the at least one processor, the electronic device, The notification content included in the second message above is input into the artificial intelligence model along with the specified instructions, and Obtain simple content from the output value output from the above artificial intelligence model, and Display the above simple content on the above display, and An electronic device that displays the notification content based on the simple content selected by the user.

8. In claim 1, when the instruction is executed individually or collectively by the at least one processor, the electronic device, The image obtained using the above camera is fed as an input value into an artificial intelligence model, and Confirming that a notification has occurred from the second external electronic device in the output value output from the above artificial intelligence model, Based on confirming that a notification has occurred in the second external electronic device, a third message requesting the delivery of the notification content to the second external electronic device is transmitted to the first external electronic device through the communication circuit, and In response to the transmission of the third message, a fourth message is received from the first external electronic device through the communication circuit, and An electronic device that displays at least a portion of the above-mentioned fourth message on the display.

9. In claim 1, when the instruction is executed individually or collectively by the at least one processor, the electronic device, Information necessary for establishing a channel for short-range wireless communication with the second external electronic device is received from the first external electronic device through the communication circuit, and Receiving a fifth message from the second external electronic device through a short-range wireless communication channel established based on the above information, and An electronic device that displays at least a portion of the above fifth message on the display.

10. In claim 1, when the instruction is executed individually or collectively by the at least one processor, the electronic device, An operation of classifying multiple notification contents included in the second message above by source of notification contents and displaying them on the display, or An electronic device that performs the operation of classifying the above multiple notification contents by designated topics and displaying them on the above display.

11. A method for operating an electronic device wearable on a user's face, The above electronic device is, A display positioned to shine light into the user's eyes located inside the electronic device when the electronic device is worn on the user's face; A camera positioned on the front of the electronic device and configured to photograph the outside; and It includes a communication circuit for wirelessly communicating with an external electronic device, and The operation of obtaining a lock screen image from an image obtained using the above camera; The operation of comparing a reference image received from a first external electronic device through the above communication circuit with the lock screen; The operation of transmitting a first message to the first external electronic device through the communication circuit, requesting the activation of a function to relay notification content to the second external electronic device, which is the source of the reference image, based on the fact that the lock screen image and the reference image match as a result of the above comparison; The operation of receiving a second message from the first external electronic device through the communication circuit in response to the transmission of the first message; and A method comprising the operation of displaying at least a portion of the above second message on the display.

12. In claim 11, the operation of acquiring the lock screen image is, An operation to confirm a user's action of unlocking the screen in an image obtained using the above camera; and A method comprising the operation of obtaining the lock screen image from an image obtained using the camera, based on the confirmation of the above action.

13. In Clause 12, the action of confirming the above act is, The operation of inputting an image acquired using the above camera into an artificial intelligence model as an input value; and A method including an action to verify the action in the output value output from the artificial intelligence model.

14. In claim 11, the operation of transmitting the first message is, The operation of inputting the above lock screen image and the above reference image as input values ​​into an artificial intelligence model; An operation to verify the match between the lock screen image and the reference image in the output value output from the artificial intelligence model; and A method comprising the operation of transmitting a first message to a first external electronic device through the communication circuit based on the confirmation that the lock screen image and the reference image match.

15. In claim 11, the operation of transmitting the first message is, The operation of receiving first account information from the first external electronic device through the communication circuit; and A method comprising the operation of transmitting the first message to the first external electronic device through the communication circuit, based on the existence of second account information matching the first account information in the electronic device.

Images