Electronic device, method, and non-transitory storage medium for providing notification information indicating state change of observation target

Abstract

The present document relates to an electronic device, a method, and a non-transitory storage medium for providing notification information indicating a state change of an observation target. According to one embodiment, an electronic device wearable on a user's body may comprise at least one camera, a display, at least one processor, and a memory for storing instructions. The instructions, when executed individually or collectively by the at least one processor, may instruct the electronic device to: display a screen for an external environment of the electronic device through the display on the basis of images captured through the at least one camera; control the display to display at least one virtual object in a partial area of the screen; designate at least one actual object located in the external environment as an observation target; monitor a state change of the observation target by using the at least one camera; acquire information related to the state change of the observation target on the basis of identifying the state change of the observation target; acquire notification information indicating the state change of the observation target by using an artificial intelligence model (530) on the basis of the information related to the state change of the observation target; and control the display to display the notification information on the screen. Various other embodiments are also possible.

Description

Electronic device, method, and non-transient storage medium for providing notification information indicating a change in the state of an observed object

[0001] The present disclosure relates to an electronic device, a method, and a non-transient storage medium for providing notification information indicating a change in the state of an observed object using artificial intelligence.

[0002] With the advancement of digital technology, electronic devices are being provided in various forms, such as smartphones, tablet PCs, and PDAs. Electronic devices are also being developed in wearable forms to enhance portability and user accessibility. Electronic devices can be configured in various forms to be worn on parts of the user's body, and as technology advances, technologies are being developed to provide real-world spaces that correspond to the actual external environment (e.g., virtual reality, augmented reality, or mixed reality).

[0003] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art related to the present disclosure.

[0004] According to one embodiment of the present disclosure, an electronic device wearable on a user's body includes at least one camera, a display, at least one processor including a processing circuit, and a memory for storing instructions.

[0005] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device is made to display a screen of the external environment of the electronic device through the display based on images captured through the at least one camera.

[0006] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device causes at least one virtual object to be displayed through the display in a portion of the screen area.

[0007] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device is configured to designate at least one actual object located in the external environment as an observation target.

[0008] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device is configured to monitor a change in the state of the object of observation using the at least one camera.

[0009] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device obtains information related to a change in the state of the observed object based on identifying a change in the state of the observed object.

[0010] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device is enabled to obtain notification information indicating the state change of the observed object using an artificial intelligence model based on information related to the state change of the observed object.

[0011] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device causes the notification information to be displayed on the screen through the display.

[0012] According to one embodiment, a method of operation in an electronic device wearable on a user's body includes an operation of displaying a screen of the external environment of the electronic device through a display of the electronic device based on images captured through at least one camera of the electronic device.

[0013] According to one embodiment, the method includes the operation of displaying at least one virtual object through the display in a portion of the screen area.

[0014] According to one embodiment, the method includes the operation of designating at least one actual object located in the external environment as an observation target.

[0015] According to one embodiment, the method includes the operation of monitoring a change in the state of the observed object using at least one camera.

[0016] According to one embodiment, the method includes an operation of obtaining information related to a change in the state of the observed object based on identifying a change in the state of the observed object.

[0017] According to one embodiment, the method includes the operation of obtaining notification information indicating the state change of the observed object using an artificial intelligence model based on information related to the state change of the observed object.

[0018] According to one embodiment, the method includes the operation of displaying the notification information on the screen through the display.

[0019] According to one embodiment, in a non-transient storage medium storing one or more programs, the one or more programs include instructions that, when executed by at least one processor of an electronic device, cause the electronic device to perform an operation of displaying a screen of the external environment of the electronic device through a display of the electronic device based on images captured through at least one camera of the electronic device.

[0020] According to one embodiment, the one or more programs include an instruction that causes the electronic device to perform an operation of displaying at least one virtual object through the display in a part area of ​​the screen when executed by at least one processor of the electronic device.

[0021] According to one embodiment, the one or more programs include an instruction that causes the electronic device to execute an operation of designating at least one actual object located in the external environment as an observation target when executed by at least one processor of the electronic device.

[0022] According to one embodiment, the one or more programs include an instruction that causes the electronic device to perform an operation of monitoring a change in the state of the observation target using the at least one camera when executed by at least one processor of the electronic device.

[0023] According to one embodiment, the one or more programs include instructions that, when executed by at least one processor of an electronic device, cause the electronic device to execute an operation of obtaining information related to a change in the state of an object to be observed based on identifying a change in the state of the object to be observed, and an operation of obtaining notification information indicating the change in the state of the object to be observed using an artificial intelligence model based on the information related to the change in the state of the object to be observed.

[0024] According to one embodiment, the one or more programs include an instruction that causes the electronic device to perform an operation of displaying the notification information on the screen through the display when executed by at least one processor of the electronic device.

[0025] FIG. 1 is a block diagram of an electronic device in a network environment according to various embodiments.

[0026] FIG. 2 is a perspective view showing the structure of an electronic device according to one embodiment.

[0027] FIGS. 3A, FIGS. 3B, and FIGS. 3C are perspective views showing the structure of an electronic device according to one embodiment.

[0028] FIG. 4 is a block diagram showing an example of the configuration of an electronic device according to one embodiment.

[0029] FIG. 5 is a diagram illustrating an example of providing notification information indicating a change in the state of an observed object in an electronic device according to one embodiment.

[0030] FIGS. 6a, FIGS. 6b, and FIGS. 6c are drawings illustrating examples of providing notification information indicating a change in the state of an observed object in an electronic device according to one embodiment.

[0031] FIGS. 7a, FIGS. 7b, and FIGS. 7c are drawings illustrating examples of providing notification information indicating a change in the state of an observed object in an electronic device according to one embodiment.

[0032] FIGS. 8A, FIGS. 8B, and FIGS. 8C are drawings illustrating examples of providing notification information indicating a change in the state of an observed object in an electronic device according to one embodiment.

[0033] FIGS. 9a, FIGS. 9b, and FIGS. 9c are drawings illustrating examples of providing notification information indicating a change in the state of an observed object in an electronic device according to one embodiment.

[0034] FIGS. 10a and FIGS. 10b are drawings illustrating an example of providing notification information indicating a change in the state of an observed object in an electronic device according to one embodiment.

[0035] FIG. 11 is a drawing showing an example of a method of operation in an electronic device according to one embodiment.

[0036] FIG. 12 is a diagram illustrating an example of transmitting notification information from an electronic device to an external electronic device according to one embodiment.

[0037] FIG. 13 is a diagram illustrating an example of a method of operation in an electronic device according to one embodiment.

[0038] FIG. 14 is a diagram illustrating an example of a method of operation in an electronic device according to one embodiment.

[0039] FIG. 15 is a diagram illustrating an example of a method of operation in an electronic device according to one embodiment.

[0040] FIG. 16 is a diagram showing an example of a method of operation in an electronic device according to one embodiment.

[0041] In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components.

[0042] Hereinafter, embodiments of the present disclosure are described in detail with reference to the drawings so that those skilled in the art can easily implement them. However, the present disclosure may be embodied in various different forms and is not limited to the embodiments described herein. In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and brevity. The term "user" as used in the embodiments of the present disclosure may refer to a person using an electronic device or a device using an electronic device (e.g., an artificial intelligence electronic device).

[0043] FIG. 1 is a block diagram of an electronic device (101) in a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or may communicate with at least one of an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through a server (108). According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (160), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)).

[0044] The processor (120) can control at least one other component (e.g., hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., program (140)), for example, and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., sensor module (176) or communication module (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., central processing unit or application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., graphics processing unit, neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use lower power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.

[0045] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0046] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, input data or output data for software (e.g., program (140)) and related commands. The memory (130) may include volatile memory (132) or non-volatile memory (134).

[0047] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0048] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0049] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0050] The display module (160) can visually provide information to an external (e.g., user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0051] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (101).

[0052] The sensor module (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0053] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0054] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0055] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0056] The camera module (180) can capture still images and video. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0057] The power management module (188) can manage the power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0058] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0059] The communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).

[0060] The wireless communication module (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), external electronic device (e.g., electronic device (104)), or network system (e.g., second network (199)). According to one embodiment, the wireless communication module (192) can support a Peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mMTC, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for realizing URLLC.

[0061] An antenna module (197) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197).

[0062] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0063] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0064] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (104) or the server (108) may be included within a second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0065] FIG. 2 is a perspective view showing the structure of an electronic device according to one embodiment.

[0066] Referring to FIGS. 1 and FIGS. 2, an electronic device (200) according to one embodiment may be an electronic device (101) of FIG. 1, an electronic device (102 or 104) communicating with the electronic device (101) of FIG. 1, or a device capable of providing services related to virtual reality technology that provides a virtual environment similar to the electronic device (101) of FIG. 1. Virtual reality (VR) technology, which is a technology that provides a virtual environment, may be developed into augmented reality (AR), mixed reality (MR), and / or extended reality (XR) that encompasses these.

[0067] The electronic device (200) may be a device configured to be wearable on a user's body, as illustrated in FIG. 2 (e.g., a head-mounted display (HMD) or a glasses-type AR glasses device). For example, the electronic device (200) may be configured to combine with an external electronic device, such as a mobile device, and may utilize components of the external electronic device (e.g., a display module, a camera module, an audio output module, or other components). Not limited thereto, the electronic device (200) may be implemented in various forms that can be worn on a user's body.

[0068] According to one embodiment, the electronic device (200) may configure a real space (e.g., virtual reality space, augmented reality space, or mixed reality space) that displays a real space corresponding to an actual external environment captured in the surrounding environment where the user is located (e.g., augmented reality image) or a virtual image provided (e.g., 2D or 3D image), and may control a display module (160) to display at least one virtual object corresponding to the user and / or at least one virtual object corresponding to an object for user interaction in the real space.

[0069] According to one embodiment, the electronic device (200) may include a processor (120), memory (130), display module (160), sensor module (176), camera module (180), charging module (e.g., battery (189) of FIG. 1) and communication module (190) as shown in FIG. 1. The electronic device (200) may further include an audio output device (155), an input module (150) as shown in FIG. 1, or other components as shown in FIG. 1. In addition, the electronic device (200) may be configured to include other components necessary to provide virtual reality functions, augmented reality functions, or mixed reality functions (e.g., services or methods).

[0070] According to one embodiment, the processor (120) is electrically connected to other components and can control other components. The processor (120) can perform various data processing or operations in accordance with the execution of various functions (e.g., operations, services, or programs) provided by the electronic device (200). The processor (120) can perform various data processing or operations to display at least one virtual object related to real objects included in an image captured in real space and / or a virtual object corresponding to a user (e.g., an avatar) in a virtual reality space. The processor (120) can perform various data processing or operations to express user interaction or movement of the virtual object displayed in the virtual reality space.

[0071] Again, with reference to FIG. 2, an electronic device (200) according to one embodiment will be described. As described above, the electronic device (200) is not limited to a glasses-type (e.g., AR glasses) augmented reality device, and can be implemented as various devices capable of providing immersive content (e.g., content based on XR technology) to the user's eyes (e.g., AR head-mounted display type, 2D / 3D head-mounted display device or VR head-mounted display device).

[0072] According to one embodiment, a camera module of an electronic device (200) (e.g., camera module (180) of FIG. 1 or camera circuit) can capture still images and / or video. According to one embodiment, the camera module may be placed within a lens frame and around a first display (251) and a second display (252). According to one embodiment, the camera module may include one or more first cameras (211-1, 211-2), one or more second cameras (212-1, 212-2), and one or more third cameras (213). According to one embodiment, images acquired through one or more first cameras (211-1, 211-2) may be used for detecting hand gestures by a user, tracking the user's head, and / or spatial recognition. One or more first cameras (211-1, 211-2) may be a GS (global shutter) camera or an RS (rolling shutter) camera. One or more first cameras (211-1, 211-2) can perform simultaneous localization and mapping (SLAM) operations through depth imaging. One or more first cameras (211-1, 211-2) can perform spatial recognition and / or motion recognition for 3DoF (depth of field) and / or 6DoF. According to one embodiment, the first cameras (211-1, 211-2) can periodically or non-periodically transmit information (e.g., trajectory information) related to the user's eyes (e.g., left eye and right eye) or the trajectory of the gaze (e.g., eye tracking) to a processor (e.g., processor (120) of FIG. 1). The first camera (211-1, 211-2) can be used to position the center of a virtual image projected onto an electronic device (200) (e.g., AR glasses) according to the direction in which the user's pupil gazes, and a GS camera can be primarily used to detect the pupil and track rapid pupil movements.The first camera (211-1, 211-2) can be configured for the left eye and the right eye, respectively, and the first camera (211-1, 211-2) configured for the left eye and the right eye, respectively, may have the same performance and specifications.

[0073] According to one embodiment, the electronic device (200) may use another camera (e.g., a third camera (213)) for hand detection and tracking and user gesture recognition. According to one embodiment, at least one of the first camera (211-1, 211-2) to the third camera (213) may be replaced with a sensor module (e.g., a LiDAR sensor). For example, the sensor module may include at least one of a vertical cavity surface emitting laser (VCSEL), an infrared sensor, and / or a photodiode.

[0074] According to one embodiment, images acquired through one or more second cameras (212-1, 212-2) may be used to detect and track the user's pupils. One or more second cameras (212-1, 212-2) may be GS cameras. One or more second cameras (212-1, 212-2) may correspond to the left eye and the right eye, respectively, and the performance of one or more second cameras (212-1, 212-2) may be substantially identical. One or more third cameras (213) may be relatively high-resolution cameras. One or more third cameras (213) may perform auto-focusing (AF) and optical image stabilization (OIS) functions. One or more third cameras (213) may be GS (global shutter) cameras or RS (rolling shutter) cameras. One or more third cameras (213) may be color cameras.

[0075] According to one embodiment, the electronic device (200) may include one or more light-emitting elements (214-1, 214-2). The light-emitting elements (214-1, 214-2) are different from the light source described below, which irradiates light onto a screen output area of ​​a display. According to one embodiment, the light-emitting elements (214-1, 214-2) may irradiate light to facilitate pupil detection in detecting and tracking a user's pupil through one or more second cameras (212-1, 212-2). According to one embodiment, the light-emitting elements (214-1, 214-2) may each include an LED. According to one embodiment, the light-emitting elements (214-1, 214-2) may irradiate light in the infrared region. According to various embodiments, the light-emitting elements (214-1, 214-2) may be attached around the frame of the augmented reality device (200). According to one embodiment, a light-emitting element (214-1, 214-2) is positioned around one or more first cameras (211-1, 211-2) and can assist gesture detection, head tracking, and spatial recognition by one or more first cameras (211-1, 211-2) when the augmented reality device (200) is used in a dark environment. According to one embodiment, a light-emitting element (214-1, 214-2) is positioned around one or more third cameras (213) and can assist image acquisition by one or more third cameras (213) when the augmented reality device (200) is used in a dark environment.

[0076] According to one embodiment, the electronic device (200) may include a battery (235-1, 235-2) (e.g., battery (189) of FIG. 1). The battery (235-1, 235-2) may store power to operate the remaining components of the augmented reality device (200).

[0077] According to one embodiment, a display module of an electronic device (200) (e.g., a display module (160) of FIG. 1) may include a first display (251), a second display (252), one or more input optical members (253-1, 253-2), one or more transparent members (290-1, 290-2), and one or more screen display portions (254-1, 254-2). According to one embodiment, the first display (251) and the second display (252) may be light output modules and may include, for example, a liquid crystal display (LCD), a digital mirror device (DMD), a liquid crystal on silicon (LCoS), an organic light emitting diode (OLED), or a micro light emitting diode (micro LED). According to one embodiment, if the first display (251) and the second display (252) are composed of a liquid crystal display device, a digital mirror display device, or a silicon liquid crystal display device, the augmented reality device (200) may include a light source that irradiates light onto a screen output area of ​​the display. According to one embodiment, if the first display (251) and the second display (252) can generate light themselves, for example, if they are composed of an organic light-emitting diode or a micro LED, the augmented reality device (200) may provide a user with a good quality virtual image (e.g., an image of a virtual reality space) without including a separate light source.

[0078] According to one embodiment, one or more transparent members (290-1, 290-2) included in the electronic device (200) may be positioned to face the user's eyes (e.g., left and right eyes) when the user wears the augmented reality device (200). The one or more transparent members (290-1, 290-2) may include at least one of a glass plate, a plastic plate, or a polymer. When the user wears the augmented reality device (e.g., the electronic device (200)), the user can view the external environment through the one or more transparent members (290-1, 290-2).

[0079] According to one embodiment, one or more input optical members (253-1, 253-2) included in the electronic device (200) can guide light generated from a first display (251) and a second display (252) to the user's eye. An image based on the light generated from the first display (251) and the second display (252) is formed on one or more screen display portions (254-1, 254-2) on one or more transparent members (290-1, 290-2), and the user can see the image formed on the one or more screen display portions (254-1, 254-2).

[0080] According to one embodiment, the electronic device (200) may include one or more optical waveguides (not shown). The optical waveguides may transmit light generated from the first display (251) and the second display (252) to the user's eye. The electronic device (200) may include one optical waveguide corresponding to the left eye and one optical waveguide corresponding to the right eye. According to one embodiment, the optical waveguides may include at least one of glass, plastic, or polymer. The optical waveguides may include a nano-pattern formed on an inner or outer surface, for example, a polygonal or curved grating structure. The optical waveguides may include a free-form prism, in which case the optical waveguides may provide incident light to the user through a reflective mirror. According to one embodiment, the optical waveguide includes at least one of a diffractive element (e.g., a diffractive optical element (DOE), a holographic optical element (HOE)) or a reflective element (e.g., a reflective mirror), and can guide display light emitted from a light source to the user's eye using at least one diffractive element or reflective element included in the optical waveguide. According to one embodiment, the diffractive element may include an input / output optical member. According to one embodiment, the reflective element may include a member that causes total internal reflection (TIR) ​​(e.g., a total internal reflection optical element or a total internal reflection waveguide). For example, total internal reflection is a method of guiding light, which may mean creating an angle of incidence such that light (e.g., a virtual image) input through an input grating area is 100% reflected from one surface (e.g., a specific surface) of the waveguide and is transmitted 100% to an output grating area.

[0081] In one embodiment, light emitted from a display (e.g., a first display (251) and a second display (252)) may have its light path guided to a waveguide through an input optical member (e.g., an optical waveguide). Light traveling inside the waveguide may be guided toward the user's eye through an output optical member. A screen display may be determined based on the light emitted toward the eye.

[0082] According to one embodiment, the electronic device (200) may include one or more voice input devices (262-1, 262-2, 262-3) and one or more voice output devices (263-1, 263-2).

[0083] According to one embodiment, the electronic device (200) may include a first PCB (270-1) and a second PCB (270-2). The first PCB (270-1) and the second PCB (270-2) may transmit electrical signals to components included in the electronic device (200), such as a first camera (211-1, 211-2), a second camera (212-1, 212-2), a third camera (213), a display (251, 252), an audio module (e.g., the audio module (170) of FIG. 1), and a sensor module (e.g., the sensor module (176) of FIG. 1). According to one embodiment, the first PCB (270-1) and the second PCB (270-2) may be flexible printed circuit boards (FPCB). According to one embodiment, the first PCB (270-1) and the second PCB (270-2) may each include a first substrate, a second substrate, and an interposer disposed between the first substrate and the second substrate.

[0084] FIG. 3a is a perspective view showing the structure of an electronic device according to one embodiment.

[0085] Referring to FIG. 3a, an electronic device (300) according to one embodiment (e.g., the electronic device (101) of FIG. 1 or the electronic device (200) of FIG. 2) may be a wearable device such as a head-mounted device (HMD) that can be worn on a user's head to provide an image (e.g., a virtual reality space image) in front of the eyes. The configuration of the electronic device (300) of FIG. 3 may be all or partly the same as the configuration of the electronic device (200) of FIG. 2.

[0086] According to one embodiment, the electronic device (300) may include a housing (310, 320, 330) that can form an exterior and provide a space in which components of the electronic device (300) can be placed.

[0087] According to one embodiment, the electronic device (300) may include a first housing (310) that can surround at least a portion of the user's head. According to one embodiment, the first housing (310) may include a first surface (300a) facing the outside of the electronic device (300) (e.g., in the +X direction).

[0088] According to one embodiment, the first housing (310) may surround at least a portion of the internal space (I). For example, the first housing (310) may include a second surface (300b) facing the internal space (I) of the electronic device (300) and a third surface (300c) opposite to the second surface (300b). According to one embodiment, the first housing (310) may be combined with a third housing (330) to form a closed curve shape surrounding the internal space (I).

[0089] According to one embodiment, the first housing (310) may accommodate at least some of the components of the electronic device (300). For example, a light output module, a circuit board, and a speaker module may be placed within the first housing (310).

[0090] According to one embodiment, a display member (340) corresponding to the left and right eyes of the electronic device (300) may be included. The display member (340) may be placed in a first housing (310). The configuration of the display member (340) of FIG. 3 may be all or partly the same as the configuration of the screen display portion (254-1, 254-2) of FIG. 2.

[0091] According to one embodiment, the electronic device (300) may include a second housing (320) that can be placed on the user's face. According to one embodiment, the second housing (320) may include a fourth surface (300d) that can face at least partially the user's face. According to one embodiment, the fourth surface (300d) may be a surface facing the internal space (I) of the electronic device (300) (e.g., -X direction). According to one embodiment, the second housing (320) may be combined with the first housing (310).

[0092] According to one embodiment, the electronic device (300) may include a third housing (330) that can be seated on the back of the user's head. According to one embodiment, the third housing (330) may be combined with the first housing (310). According to one embodiment, the third housing (330) may accommodate at least some of the components of the electronic device (300). For example, a battery (e.g., the battery (235-1, 235-2) of FIG. 2) may be placed within the third housing (330).

[0093] In order to enhance the user's overall user experience, usage environment, and usability of the head-mounted wearable electronic device (300), it may be necessary for the sensations felt and experienced by the user in VR (virtual reality), AR (augmented reality), and MR (mixed reality) spaces to be as similar as possible to the sensations of the real world.

[0094] FIGS. 3B and FIGS. 3C are perspective views showing the structure of an electronic device according to one embodiment.

[0095] Referring to FIG. 3b and FIG. 3c, in one embodiment, camera modules (311, 312, 313, 314, 315, 316) and / or a depth sensor (317) for acquiring information related to the surrounding environment of an electronic device (300) (e.g., a wearable device) may be disposed on a first surface (310) of the housing.

[0096] In one embodiment, camera modules (311, 312) can acquire images related to the surrounding environment of a wearable electronic device.

[0097] In one embodiment, camera modules (313, 314, 315, 316) can acquire images while the electronic device (300) is worn by a user. Camera modules (313, 314, 315, 316) can be used for hand detection, tracking, and user gesture (e.g., hand movements) recognition. Camera modules (313, 314, 315, 316) can be used for 3DoF, 6DoF head tracking, position (space, environment) recognition, and / or movement recognition. In one embodiment, camera modules (311, 312) may be used for hand detection and tracking and user gestures.

[0098] In one embodiment, the depth sensor (317) may be configured to transmit a signal and receive a signal reflected from the subject, and may be used for determining the distance to the object, such as time of flight (TOF). In place of or additionally to the depth sensor (317), camera modules (313, 314, 315, 316) may determine the distance to the object.

[0099] According to one embodiment, a face recognition camera module (325, 326) (e.g., FT (Face Tracking) camera) and / or a display (321) (and / or a lens) may be disposed on the second surface (320) of the housing.

[0100] In one embodiment, a face recognition camera module (325, 326) adjacent to the display may be used to recognize the user's face or to recognize and / or track both of the user's eyes. In one embodiment, the lens may serve to adjust the focus so that the screen output to the display (321) can be seen by the user's eyes, and may be composed of, for example, a Fresnel lens, a Pancake lens, or a multi-channel lens.

[0101] In one embodiment, the display (321) (and / or lens) may be disposed on a second surface (320) of the wearable electronic device (300). In one embodiment, the wearable electronic device (300) may not include camera modules (315, 316) among a plurality of camera modules (313, 314, 315, 316). Although not illustrated in FIG. 3b and FIG. 3c, the electronic device (300) may further include at least one of the configurations illustrated in FIG. 2.

[0102] As described above, according to one embodiment, the electronic device (300) may have a form factor for being worn on a user's head. The electronic device (300) may further include a strap and / or a wearing member for being secured on a part of the user's body. The electronic device (300) may provide a user experience based on augmented reality, virtual reality, and / or mixed reality while being worn on the user's head.

[0103] The electronic device described herein may be an electronic device that can be worn on the head by a user, such as augmented reality (AR) glasses, a head-mounted display (HMD) device, and / or a VST device, for example, as described with reference to FIGS. 2, 3a, 3b, and 3c. Herein, the electronic device may also be referred to as a wearable electronic device.

[0104] FIG. 4 is a block diagram showing an example of the configuration of an electronic device according to one embodiment, and FIG. 5 is a diagram showing an example of providing notification information indicating a change in the state of an observed object in an electronic device according to one embodiment.

[0105] Referring to FIGS. 4 and FIGS. 5, an electronic device (401) according to one embodiment may be a device capable of providing services related to virtual reality technology that provides a virtual environment similar to the electronic device (101) of FIG. 1, or the electronic device (200) of FIG. 2 and the electronic device (300) of FIGS. 3a, 3b, and 3c. Virtual reality (VR) technology, which is a technology that provides a virtual environment, may be developed into augmented reality (AR), mixed reality (MR), and / or extended reality (XR) that encompasses these. According to one embodiment, virtual reality (VR) technology may be described in the sense that it includes augmented reality (AR), mixed reality (MR), and / or extended reality (XR). The electronic device (401) may be a device configured to be wearable on a user's body (e.g., a head-mounted display (HMD) or an AR glasses device) as illustrated in FIGS. 2 and FIGS. 3a, FIGS. 3b and FIGS. 3c. For example, the electronic device (401) may be configured to combine with an external electronic device, such as a mobile device, and may utilize components of the external electronic device (e.g., the electronic device (102 or 104) of FIG. 1) (e.g., a display module, a camera module, an audio output module, or other components). Not limited thereto, the electronic device (401) may be implemented in various forms that can be worn on a user's body (e.g., the head).

[0106] According to one embodiment, the electronic device (401) may display a screen (e.g., a display module (160) of FIG. 1 or a display (321) of FIG. 3) based on images captured using at least one camera included in a camera circuit (440) (e.g., a camera module (180) of FIG. 1, a first camera (211-1, 211-2) of FIG. 2, a third camera (213) of FIG. 3, or a camera (313, 314, 315, 316) of FIG. 3) in the external environment surrounding the electronic device (401). The electronic device (401) may display at least one virtual object on the screen based on information (e.g., content) related to applications currently running. According to one embodiment, when the electronic device (401) makes the real space of the real environment visible to the user through a transparent member, it can display at least one virtual object (e.g., image) anchored on a screen corresponding to the real space.

[0107] An electronic device (401) according to one embodiment may include at least one processor (410), memory (420), display (430), camera circuit (440), communication circuit (450), microphone (460), and speaker (470). Not limited thereto, the electronic device (401) may be implemented identically or similarly to the electronic device (101) of FIG. 1, the electronic device (200) of FIG. 2, or the electronic device (300) of FIG. 3, and may further include other components of the electronic device (101) of FIG. 1, the electronic device (200) of FIG. 2, or the electronic device (300) of FIG. 3a, 3b, and 3c. In addition, the electronic device (200) may be configured to include other components necessary for the method of operation of the present document.

[0108] According to one embodiment, a processor (410) (e.g., processor (120) of FIG. 1) may acquire images of the external environment of an electronic device (401) through at least one camera included in a camera circuit (440) and, based on the acquired images, display a screen of the external environment on a display (430). The screen may be shown to the user through the display (430) using the camera circuit (440) of the electronic device (401) worn by the user, or may be shown to the user's eyes through a transparent member (e.g., one or more transparent members (290-1, 290-2)).

[0109] According to one embodiment, the processor (410) can control the display (430) using a camera circuit (440) so that a real space including a screen is visible through the display (430) when the electronic device (401) is, for example, a VR device. According to one embodiment, the processor (410) can control the display (430) (e.g., a transparent member (e.g., one or more transparent members (290-1, 290-2)) so that a real space including a screen is visible through the user's eyes when the electronic device (401) is, for example, an AR device.

[0110] According to one embodiment, the processor (410) may control the display (430) to display at least one virtual object related to a running application in a portion of the screen. For example, the at least one virtual object may display the content of the running application (e.g., an execution screen) on the screen in the form of a window or a pop-up window.

[0111] According to one embodiment, the processor (410) may designate at least one real object located in an external environment (e.g., real space) as an observation target. The processor (410) may designate a real object included in a portion of a screen obscured by at least one virtual object, a real object included in a portion of a screen designated based on user input, a real object included in a portion of a screen where the user's gaze is directed, or a real object designated in an external environment not included in the screen, or an observation area as an observation target. According to one embodiment, when the processor (410) automatically designates an observation target using at least one camera, it may analyze the recognized screen based on images of the external environment, classify objects such as objects, animals, or people included in the screen, and designate some of the classified objects as observation targets. The processor (410) may perform the operation of analyzing the screen and classifying objects using an artificial intelligence model (e.g., a CNN (convolutional neural network) model). According to one embodiment, the processor (410) may control the display to display visual effects (e.g., outlines or stickers) on the observation target included in the screen.

[0112] According to one embodiment, the processor (410) can monitor changes in the state of an object of observation using at least one camera. The processor (410) can track an object of observation included in the screen using a screen analysis module (510) and continuously analyze and record the state of the object of observation. The processor (410) can crop an area of ​​the object of observation, transmit the cropped information to the screen analysis module (510), and through the screen analysis module (510), check for changes in the object of observation or the occurrence of a designated event based on the cropped information. When a change in the object of observation or a designated event occurs, the processor (410) can transmit information related to the change in the state of the object of observation or information related to the event to an artificial intelligence model (530). According to one embodiment, the screen analysis module (510) may include a scene tracking detector and a multiple object tracking function. The scene tracking detector recognizes a scene for a space designated by a user or a space automatically designated by an electronic device, classifies the recognized scene by object such as a thing, person, or animal, and then analyzes or tracks the state of the object. The screen analysis module (510) can visually process classification by using a CNN model to outline each object in a semantic segmentation manner. The multi-object tracking function can track multiple observed objects and analyze their state to identify changes. Objects visible in the camera's field of view are recognized and recorded using a CNN model, and the recorded scenes are summarized into frames with specific changes using video summarization. The point in time when a change occurs in the summarized frame information is determined using video summarization, and the frame in which a state change occurs can be transmitted to an artificial intelligence model (e.g., a multi-modal model).

[0113] According to one embodiment, the processor (410) uses the screen analysis module (510) to record initial frame information at the start time of monitoring the state change of the object being observed, and compares the monitored frame information with the initial frame information to check whether the difference between the two frames exceeds a specified threshold level. According to one embodiment, the processor (410) uses the screen analysis module (510) to analyze each frame at regular intervals to check for changes between each frame and the previous frame, or to check for the occurrence of a specified anomaly in the analyzed frame. For example, the screen analysis module (510) can check whether the shape of an object corresponding to the object being observed included in the monitored frame changes by more than a threshold level compared to the object corresponding to the object being observed included in the initial frame, or whether the location changes by more than a threshold level. For example, the screen analysis module (510) can check whether a new object that was not present in the initial frame is identified in the monitored frame to track the state change of the object being observed.

[0114] According to one embodiment, the processor (410) stores images captured by at least one camera and, if there is a change between the frames of the images, can transmit at least one frame, either the frame before the change (e.g., initial frame) or the changed frame (e.g., a frame obtained by monitoring), to an artificial intelligence model (530) (e.g., multimodal) as information related to the state change. Here, the artificial intelligence model (530) may be a generative artificial intelligence model.

[0115] According to one embodiment, the processor (410) may obtain notification information to inform the user of a change in the state of an object of observation by using an artificial intelligence model (530) based on information related to a change in the state of an object of observation. According to one embodiment, the artificial intelligence model (530) of the electronic device (401) may generate notification information based on information related to a change in state as input information. Here, the notification information may be information generated as at least one of text, image, or audio (e.g., voice) (e.g., message, window, or PIP (picture-in-picture type virtual object). According to one embodiment, the processor (410) may control a display to display the notification information on the screen. According to one embodiment, the processor (420) may change at least one setting value for a virtual object based on information related to a change in the state of an object of observation. For example, if the transparency, color, or shape of a hidden virtual object changes, a visual effect indicating the change in transparency, color, or shape may be displayed in conjunction with the notification information.

[0116] According to one embodiment, the processor (410) may be a hardware component (function) or a software element (program) comprising at least one component provided in the electronic device (401), such as a hardware module or a software module (e.g., an application program). According to one embodiment, the processor (410) may include, for example, one or more combinations of hardware, software, or firmware. The processor (410) may be configured to omit at least some of the components or to include additional components for performing image processing operations in addition to the components.

[0117] According to one embodiment, the memory (420) (e.g., the memory (130) of FIG. 1) may store applications. For example, the memory (420) may store applications (functions or programs) related to images (or image generation), applications related to image management, or applications related to generative AI. The memory (420) may store a first image (e.g., original image), a second image (e.g., image for editing), a third image (e.g., edited result image) and / or information related to image editing that are captured through an external electronic device or camera circuit (440).

[0118] According to one embodiment, the memory (420) may store various data generated during the execution of the program (140), including a program used for functional operation (e.g., the program (140) of FIG. 1). For example, the memory (420) may include a program (140) area and a data area (not shown). The program (140) area may store related program information for operating the electronic device (401), such as an operating system (OS) that boots the electronic device (401) (e.g., the operating system (142) of FIG. 1). The data area (not shown) may store transmitted and / or received data and generated data according to various embodiments. Additionally, the memory (420) may be configured to include at least one storage medium among flash memory, hard disk, multimedia card micro type memory (e.g., secure digital (SD) or extreme digital (XD) memory), RAM, and ROM.

[0119] According to one embodiment, a display (430) (e.g., the display module (160) of FIG. 1, the display (251, 252) of FIG. 2, or the display (321) of FIG. 3) may display a screen of the external environment of an electronic device based on images captured through at least one camera, and may display at least one virtual object in a part area of ​​the screen. The display (430) may display notification information indicating a change in the state of an object of observation on the screen under the control of the processor (410). The display (430) may display visual effects (e.g., outlines or stickers) on an object of observation included on the screen under the control of the processor (410). According to one embodiment, the display (430) may be implemented in the form of a touch screen. When the display (430) is implemented in the form of a touch screen together with an input module, it may display various information generated according to the user's touch actions. According to one embodiment, the display (430) may be composed of at least one of an LCD (liquid crystal display), a TFT-LCD (thin film transistor LCD), an OLED (organic light emitting diodes), an LED, an AMOLED (active matrix organic LED), a flexible display, and a 3-dimensional display. Additionally, some of these displays may be configured to be transparent or light-transmitting so that the outside can be seen through them. This may be configured in the form of a transparent display including a TOLED (transparent OLED). According to one embodiment, in addition to the display (430), other display modules (e.g., an extended display or a flexible display) may be further included.

[0120] According to one embodiment, a camera circuit (440) (e.g., camera module (180) of FIG. 1, camera (211-1, 211-2) of FIG. 2, or camera (311, 312, 313, 314, 315, 316) of FIG. 3b) may include at least one camera and may capture images (e.g., 2D images or 3D images) of an external environment so that the actual external environment is displayed through a display in a real space (e.g., virtual reality space, augmented reality space, or mixed reality space) or on a screen corresponding to the real space (e.g., to display a screen). The configuration and operation of at least one camera included in the camera circuit (440) may be the same or similar to the camera (211-1, 211-2) of FIG. 2 or the camera (311, 312, 313, 314, 315, 316) of FIG. 3b.

[0121] According to one embodiment, a communication circuit (450) (e.g., a communication module (190) of FIG. 1) can communicate with an external electronic device (e.g., an electronic device (102, 104) of FIG. 1, a server (108) of FIG. 1, or another user's electronic device). For example, the communication circuit (450) can receive at least one object displayed in a portion of a screen from an external electronic device and transmit notification information to the external electronic device. According to one embodiment, the communication circuit (450) may include a cellular module, a Wi-Fi (wireless-fidelity) module, a Bluetooth module, or a near field communication (NFC) module.

[0122] FIGS. 6a, FIGS. 6b, and FIGS. 6c are drawings illustrating examples of providing notification information indicating a change in the state of an observed object in an electronic device according to one embodiment.

[0123] Referring to FIGS. 4, FIGS. 5 and FIGS. 6a, FIGS. 6b and FIGS. 6c, a processor (410) (e.g., at least one processor) of an electronic device (401) according to one embodiment (e.g., the electronic device (101) of FIG. 1, or the electronic device (200) of FIG. 2 and the electronic device (300) of FIGS. 3a, FIGS. 3b and FIGS. 3c) may control a display (430) to display a screen (601) of an external environment. The processor (410) of the electronic device (401) may control the display (430) to display at least one virtual object (e.g., an application execution screen or window) (e.g., a first virtual object (611) and / or a second virtual object (613)) related to at least one application currently running on the screen (601). The screen (601) may be a screen configured based on images of an external environment captured using at least one camera.

[0124] According to one embodiment, the processor (410) may designate an observation target (603) (e.g., a kettle) based on at least one camera, the user's gaze, or user input. Since the first virtual object (611) (e.g., the execution screen of a messenger application) is displayed overlapping at least a part of the observation target (603), the observation target (603) may not be visible to the user. The observation target (631) is described as an example of a real object (e.g., a kettle) for convenience of explanation, and the electronic device (401) may further designate one or more other real objects included in the screen (601) as observation targets.

[0125] According to one embodiment, the processor (410) can monitor an observation target (603) (e.g., a portion of the screen corresponding to the observation target) to track changes in the state of the observation target (603). When the electronic device (401) detects a change in the state of the observation target (603), it can transmit information related to the change in the state of the observation target (e.g., an initial frame and a converted frame) as input to the artificial intelligence model (530). In addition to information related to the change in state (e.g., frames), the electronic device (401) can transmit various multimedia information, such as sound information or notification information, to the artificial intelligence model (530). The electronic device (401) can display notification information (621) (e.g., “The kettle is boiling.”) generated by the artificial intelligence model (530) based on information related to the change in the state of the observation target on the screen (601). The artificial intelligence model (530) can verify the change in the state of the observation target based on information related to the change in the state of the observation target and / or additional information (e.g., sound information or notification information). For example, the artificial intelligence model (530) can identify the sound of boiling water or the sound coming from the kettle based on sound information, even if it confirms that the frame change of the kettle is not significant based on information related to the state change. The artificial intelligence model (530) can increase the accuracy of the state change by confirming the state change of the object of observation based further on sound information along with information related to the state change of the object of observation. For example, the notification information (621) may be in the form of a window (e.g., a pop-up window) that can be displayed in a part of the screen (601) that the user is watching or in an area that does not obscure at least one virtual object (611, 613), and may include text messages, images and / or voice messages.For example, the notification information (621) may be displayed in an area adjacent to the actual object (e.g., TV) (e.g., an area that does not obscure the actual object being watched) so that the user can easily check it while watching the actual object (e.g., TV) within the screen (601).

[0126] According to one embodiment, the processor (410) can detect a user interaction (e.g., gesture input) requesting detailed information from the notification information (621) displayed on the display (430), and based on identifying the priority or emergency situation, display detailed information about the state change of the object being observed on the screen (601). For example, when checking the state change of the object being observed, if the processor (410) obtains notification information regarding the emergency situation through the artificial intelligence model (530), it can set the priority of the notification information regarding the emergency situation high, change the first virtual object even without user interaction, and display the notification information regarding the emergency situation on the display (430). For example, the processor (410) may adjust the transparency of the first virtual object (611) or move the first virtual object (611) so that the user can check the state of the observation target (603), and may display visual graphic effects (631) (e.g., outline, highlight, or sticker) on the observation target (603) where the state change occurred to visually provide the location (e.g., area) where the state change occurred on the screen (601). For example, the processor (410) may display detailed information along with notification information (621) in a part of the screen (601) (e.g., expanded window or expanded pop-up window). For example, the processor (410) may output detailed information as a voice message through the speaker (470). For example, when the observation target (603) is obscured by the first virtual object (611), the processor (410) may display a virtual object corresponding to the obscured observation target (603) (e.g., an image of a boiling kettle) in a part of the screen (601) (e.g., a location corresponding to the obscured observation target (603) or an area that does not overlap with at least one virtual object).

[0127] FIGS. 7a, FIGS. 7b, and FIGS. 7c are drawings illustrating examples of providing notification information indicating a change in the state of an observed object in an electronic device according to one embodiment.

[0128] Referring to FIGS. 4, FIGS. 5 and FIGS. 7a, FIGS. 7b and FIGS. 7c, the processor (410) of an electronic device (401) according to one embodiment (e.g., the electronic device (101) of FIG. 1, or the electronic device (200) of FIG. 2 and the electronic device (300) of FIGS. 3a, FIG. 3b and FIG. 3c) can display a screen (601) of an external environment on a display (430) in FIG. 7a and display a plurality of virtual objects (611, 613, 711) (e.g., an application execution screen or window) (e.g., a first virtual object (611), a second virtual object (613), and a third virtual object (711)) on the screen (601).

[0129] According to one embodiment, with reference to FIG. 7b, the processor (410) may designate an observation target (603) (e.g., a kettle) based on at least one camera, the user's gaze, or user input. Since the first virtual object (611) (e.g., the execution screen of a messenger application) is displayed superimposed on the observation target (603), the observation target (603) may not be visible to the user. For convenience of explanation, the observation target (603) is described as an example of a real object (e.g., a kettle), and the processor (410) may designate one or more other real objects included in the screen (601) as observation targets.

[0130] According to one embodiment, the processor (410) can monitor a change in the state of the observation target (603). When the processor (410) detects a change in the state of the observation target (603), it can adjust the transparency of the first virtual object (611) and / or the third virtual object (711) so that the observation target (603) is visible to the user, or move the first virtual object (611) and / or the third virtual object (711) to another location that does not overlap with the observation target (603). The processor (410) can display a visual graphic effect (721) (e.g., outline, highlight, zoom effect, or sticker) indicating that a change in the state of the observation target (603) has occurred. When the processor (410) detects a change in the state of the observation target (603), it can display notification information (723) generated by the artificial intelligence model (530) on the screen (601). Here, the notification information (723) may be displayed in an area adjacent to the actual object (e.g., TV) (e.g., an area adjacent to the actual object being watched) so that the user can easily check it while watching the actual object (e.g., TV) within the screen (601).

[0131] According to one embodiment, when a user is looking at a third virtual object (711) displayed at the top among a plurality of virtual objects (611, 613, 711) overlapping on a screen (601), the processor (410) can monitor the state change of the first virtual object (611) obscured by the third virtual object (711) through a camera circuit (440), and can detect scenes where the state change of the obscured first virtual object (611) is large or significant through a scene tracking detector of the screen analysis module (510). For example, the virtual object monitoring the state change may be an application that requires notification to the user, such as CCTV video, messenger, or email. According to one embodiment, when the processor (410) identifies (e.g., identifies or detects) a significant or meaningful change in the first virtual object (611) or identifies (e.g., identifies or detects) a change in the state by analyzing frames, it may adjust the transparency of the third virtual object (711) (e.g., adjusts the transparency to a high level) so that the first virtual object (611) is visible to the user, or move the third virtual object (711) to another location that does not overlap with the first virtual object (611). According to one embodiment, the processor (410) may move the first virtual object (611) to another location that does not overlap with the third virtual object (711). According to one embodiment, the processor (410) may display a visual graphic effect (731) (e.g., outline, highlight, zoom effect, or sticker) indicating that a change in the state has occurred in the first virtual object (611).When the processor (410) confirms (e.g., identifies or detects) a change in state, it transmits information related to the first virtual object (611) (e.g., transmitted and received messages or previous and changed images of the first virtual object (611)) to the artificial intelligence model (530), and based on the information related to the first virtual object (611), it uses the artificial intelligence model (530) to obtain (e.g., generate) notification information (733) indicating that a change in state has occurred in the first virtual object (611), and can display the obtained notification information (733) (e.g., “A change has occurred in the message.”) on the screen (601). Here, the notification information (733) can be displayed in an area adjacent to the actual object (e.g., TV) (e.g., an area that does not obscure the actual object being watched) so that the user can easily check it while watching the actual object (e.g., TV) within the screen (601).

[0132] FIGS. 8A, FIGS. 8B, and FIGS. 8C are drawings illustrating examples of providing notification information indicating a change in the state of an observed object in an electronic device according to one embodiment.

[0133] Referring to FIGS. 4, FIGS. 5 and FIGS. 8a, FIGS. 8b and FIGS. 8c, the processor (410) of an electronic device (401) according to one embodiment (e.g., the electronic device (101) of FIG. 1, or the electronic device (200) of FIG. 2 and the electronic device (300) of FIGS. 3a, FIGS. 3b and FIGS. 3c) can control a display (430) to display a screen (601) of an external environment.

[0134] According to one embodiment, with reference to FIG. 8, the processor (410) may designate a real object (e.g., a kettle) that the user’s gaze is looking at as an observation target (603) (e.g., a kettle). For example, if the processor (410) detects a blink of the user’s eye or a gesture of the user after the user’s gaze has looked at a specific area of ​​the screen (601) for a certain period of time, the processor (410) may designate a real object located in the specific area that was looked at as an observation target (603). With reference to FIG. 8b, the processor (410) may identify that the user’s gaze has moved to another real object (823) (e.g., a TV) within the screen (601). For convenience of explanation, the observation target (603) is described as an example of a real object (e.g., a kettle), and the processor (410) may designate one or more other real objects included in the screen (601) as observation targets.

[0135] According to one embodiment, with reference to FIG. 8c, a processor (410) can monitor a change in the state of an observation target (603) using at least one camera. When the processor (410) confirms a change in the state of the observation target (603), it can display a visual graphic effect (821) (e.g., outline, highlight, zoom effect, or sticker) indicating that a change in the state of the observation target (603) has occurred. When the processor (410) confirms a change in the state of the observation target (603), it can obtain notification information (831) generated by an artificial intelligence model (530) based on information related to the change in state, and display the obtained notification information (831) (e.g., a window or pop-up window containing text and / or voice information) on a screen (601). Here, the notification information (831) may be displayed in an area adjacent to the actual object (e.g., TV) (823) (e.g., an area that does not obscure the actual object being watched) so that the user can easily see it while watching the actual object (e.g., TV) (823) within the screen (601).

[0136] FIGS. 9a, FIGS. 9b, and FIGS. 9c are drawings illustrating examples of providing notification information indicating a change in the state of an observed object in an electronic device according to one embodiment.

[0137] Referring to FIGS. 4, FIGS. 5 and FIGS. 9a, FIGS. 9b and FIGS. 9c, a processor (410) of an electronic device (401) according to one embodiment (e.g., the electronic device (101) of FIG. 1, or the electronic device (200) of FIG. 2 and the electronic device (300) of FIGS. 3a, FIGS. 3b and FIGS. 3c) can control a display (430) to display a first screen (601) of an external environment.

[0138] According to one embodiment, the processor (410) may designate a real object not included in the first screen (601) as an observation target. Referring to FIG. 9a, the processor (410) may display a second screen (901) of another space in the external environment on the display (430) based on images of the external environment captured using at least one camera as the user’s gaze moves to another space in the external environment not included in the first screen (601). When the processor (410) identifies that the user’s gaze on the second screen (901) is directed at a real object (e.g., a washing machine), it may designate the real object (e.g., a washing machine) as an observation target (911) and designate the space where the observation target (911) is located as a space for monitoring state changes.

[0139] According to one embodiment, the processor (410) may pre-designate at least one observation target (911) by the user’s gaze or user gesture on a second screen (901) for another space in an external environment (e.g., a laundry room), and store information about the designated at least one observation target (911) in memory (420). The processor (410) may check the pre-designated observation target (911) while the user is looking at the first screen (601) and start monitoring for changes in the state of the observation target (911). According to one embodiment, when an observation target is designated by an external electronic device, the processor (410) receives information about the observation target from the external electronic device and may designate the observation target based on the received information about the observation target (911). For example, when designating an observation target in another space (e.g., a laundry room) corresponding to the second screen (911), the processor (410) may designate the observation target by having the user move to the other space (e.g., a laundry room) and gaze at an actual object or select it via a user gesture. For example, the processor (410) may designate a pre-designated observation target to be monitored based on information about the observation target stored in memory (420) while the user is looking at the space (e.g., a living room) corresponding to the first screen (601) without moving to the other space (e.g., a laundry room) corresponding to the second screen (901).

[0140] Referring to FIG. 9b, according to one embodiment, the processor (410) can identify that after designating the observation target (911), the user’s gaze is directed toward a real object (921) (e.g., a TV) within the screen (601). When the user shifts their gaze, the camera circuit (440) may have a wider field of view than the user’s vision or may maintain orientation toward the designated space in a manner similar to a gimbal using a gyro sensor. When the user gazes at the real object (921), the observation target (911) may not be visible within the user’s field of view because it is not included in the first screen (601). According to one embodiment, after designating the observation target (911), the processor (410) may monitor the observation target (911) using at least one camera if the observation target (911) is included within the field of view of at least one camera. For example, at least one camera may be a camera capable of capturing an area outside the user's field of view (e.g., having a field of view wider than the user's field of view) (e.g., a wide-angle or 360-degree camera). According to one embodiment, the processor (410) may monitor the observation target (911) using a microphone (460) after designating the observation target (911). According to one embodiment, the processor (410) may receive information monitored by an external electronic device (e.g., a smartphone, a wearable device, or an IoT device) connected to communication through a communication circuit (450) from the external electronic device.

[0141] Referring to FIG. 9c, according to one embodiment, the processor (410) can determine whether there is a change in the state of the observation target (911) based on information obtained by monitoring the observation target (911). According to one embodiment, when the processor (410) confirms a change in the state of the observation target (911), it can display notification information (931) (e.g., an image of a washing machine and text indicating the remaining time of the washing machine or the end of washing) generated by an artificial intelligence model (e.g., a CNN MODEL) (530) based on information related to the change in state on the first screen (601). For example, the processor (410) can recognize and detect the number display of the washing machine using a CNN model and provide the actual scene to the user in real time in a PIP format through a camera circuit (440). If voice information is included in the notification information (931), the electronic device (401) can output the voice information through a speaker (470). Here, the information regarding the monitoring of the observation target (911) may include information about events that occurred in the observation target (911) (e.g., a numerical display indicating the remaining time of the washing machine or an alert sound generated by the washing machine). The processor (410) may check whether there is a change in state based on the information regarding the monitoring of the observation target (911). Here, the alert information (931) may be displayed in an area adjacent to the actual object (921) (e.g., TV) (e.g., an area that does not obscure the actual object being watched) so that the user can easily check it while watching the actual object (921) (e.g., TV) within the screen (601).

[0142] FIGS. 10a and FIGS. 10b are drawings illustrating an example of providing notification information indicating a change in the state of an observed object in an electronic device according to one embodiment.

[0143] Referring to FIGS. 4, FIGS. 5, FIGS. 10a and FIGS. 10b, a processor (410) of an electronic device (401) according to one embodiment (e.g., the electronic device (101) of FIG. 1, or the electronic device (200) of FIG. 2 and the electronic device (300) of FIGS. 3a, FIG. 3b and FIG. 3c) can control a display (430) to display a screen (601) of an external environment.

[0144] Referring to FIG. 10a, according to one embodiment, the processor (410) may automatically designate an area of ​​real space corresponding to an external environment outside the screen (601) (e.g., a blind spot or an area outside the user's field of view) as an observation area for monitoring using at least one camera without shifting the user's gaze. The processor (410) may designate the designated observation area as an observation target and monitor changes in the state of the observation target using at least one camera and / or microphone (460) while looking at a real object (1011) (e.g., a TV) included in the screen (601). According to one embodiment, changes in the state of the observation target may be monitored by an external electronic device (e.g., an IoT device or a wearable device) located in the observation area. In this case, the processor (410) may obtain information regarding the monitoring of the observation target from the external electronic device.

[0145] Referring to FIG. 10b, according to one embodiment, when the state of the observation area changes (e.g., a change in the state of the actual object included in the observation area) while the processor (410) is watching a real object (1011) (e.g., a TV) included in the first screen (601), the processor (410) can acquire information related to the change in the state of the observation area and transmit the acquired information related to the change in the state of the observation area to an artificial intelligence model (530). According to one embodiment, when a designated event occurs (e.g., an event such as another person staring at the user or a sound calling the user, or an emergency situation), the processor (410) can acquire information related to the event and transmit the acquired information related to the event to an artificial intelligence model (530).

[0146] The processor (410) can provide notification information (1021) generated using an artificial intelligence model (530) in real time (e.g., display on a screen (601) or output through a speaker (470). Here, the notification information (1021) can be displayed in an area adjacent to the actual object (1011) (e.g., TV) (e.g., an area that does not obscure the actual object being watched) so that the user can easily check it while watching the actual object (1011) (e.g., TV) within the screen (601).

[0147] A software module of an electronic device (101, 200, 300, 401) according to one embodiment may be configured to include a kernel (or HAL), a framework (e.g., middleware (144) of FIG. 1), and an application (e.g., application (146) of FIG. 1). At least some of the software modules may be preloaded onto the electronic device (101, 200, 300, 401) or downloadable from a server (e.g., server (108)).

[0148] According to one embodiment, the kernel may include, for example, a system resource manager or a device driver, but may be configured to include other modules, not limited thereto. The system resource manager may perform control, allocation, or reclamation of system resources. The device driver may include, for example, a display driver, a camera driver, a Bluetooth driver, a shared memory driver, a USB driver, a keypad driver, a WIFI driver, an audio driver, or an IPC (inter-process communication) driver.

[0149] According to one embodiment, the framework may provide functions commonly required by the application, or provide various functions to the application through an application programming interface (API) (not shown) so that the application can efficiently use limited system resources within the electronic device (101, 200, 300, 401). The framework may include modules that form combinations of various functions of the components. The framework may provide modules specialized for each type of operating system to provide differentiated functions. The framework may dynamically delete some existing components or add new components.

[0150] According to one embodiment, the application may be configured to include an application (e.g., a module, a manager, or a program) for displaying an image of the external environment in real space. The application may include an application received from an external electronic device (e.g., a server (108) or an electronic device (102, 104)). According to one embodiment, the application may include a preloaded application or a third-party application downloadable from a server. The components of the software module and the names of the components according to the illustrated embodiments may vary depending on the type of operating system. According to one embodiment, at least a portion of the software module may be implemented as software, firmware, hardware, or a combination of at least two of these. At least a portion of the software module may be implemented (e.g., executed) by a processor (e.g., AP). At least a portion of the software module may include, for example, a module, a program, a routine, a set of instructions, or a process for performing at least one function.

[0151] As such, in one embodiment, the main components of an electronic device (101, 200, 300, 401) have been described through the electronic device of FIGS. 1, 2, 3a, 3b, 3c, and 4. However, in various embodiments, the components illustrated in FIGS. 1, 2, 3a to 3c, and 4 are not all essential components, and the electronic device (101, 200, 300, 401) may be implemented by more components than those illustrated, or by fewer components. Additionally, the positions of the main components of the electronic device (101, 200, 300, 401) described above through FIGS. 1, 2, 3a, 3b, 3c, and 4 may be changed according to various embodiments.

[0152] According to one embodiment, an electronic device wearable on a user's body (e.g., electronic device (101) of FIG. 1, electronic device (200) of FIG. 2, electronic device (300) of FIG. 3a, FIG. 3b, FIG. 3c, or electronic device (401) of FIG. 4) comprises at least one camera (e.g., camera module (180) of FIG. 1, first camera (211-1, 211-2) of FIG. 2, third camera (213), camera (313, 314, 315, 316) of FIG. 3b, or at least one camera included in the camera circuit (440) of FIG. 4), a display (e.g., first display (251) of FIG. 2, second display (252), display member (340) of FIG. 3a, display (321) of FIG. 3c, or display (430) of FIG. 4), and at least one processor (processor (120) of FIG. 1 or processor (410) of FIG. 4). It may include memory for storing instructions (e.g., memory (130) of FIG. 1 or memory (420) of FIG. 4).

[0153] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be able to display a screen of the external environment of the electronic device through the display based on images captured through the at least one camera.

[0154] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may control the display to display at least one virtual object in a part area of ​​the screen.

[0155] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be configured to designate at least one actual object located in the external environment as an observation target.

[0156] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be enabled to monitor a change in the state of the object of observation using the at least one camera.

[0157] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be enabled to obtain information related to a change in the state of the observed object based on identifying a change in the state of the observed object.

[0158] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be enabled to obtain notification information indicating the state change of the observed object using an artificial intelligence model (e.g., the artificial intelligence model (530) of FIG. 5) based on information related to the state change of the observed object.

[0159] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be able to control the display to display the notification information on the screen.

[0160] According to one embodiment, the object of observation may be a real object included in a portion of the screen obscured by at least one virtual object, a real object included in a portion of the screen specified based on user input, or a real object specified in the external environment not included in the screen.

[0161] According to one embodiment, the notification information may include at least one of text, image, or voice information.

[0162] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be enabled to monitor changes in the state of the object being observed using the microphone of the electronic device (e.g., the acoustic output module (155) of FIG. 1 or the speaker (470) of FIG. 4) or an external electronic device located around the object being observed, and to control the display to display visual graphic effects on the object being observed included in the screen.

[0163] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be able to identify a first real object included in the screen obscured by the at least one virtual object, designate the first real object as the object of observation, and, based on identifying that the state of the object of observation has changed, adjust the transparency of the at least one virtual object or move the at least one virtual object so that the first real object is visible.

[0164] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be able to identify a first actual object to be designated as the object of observation in a part of the screen based on the user’s gaze looking at a part of the screen and identifying a designated gesture input.

[0165] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be enabled to analyze the images captured through the at least one camera to identify a second actual object to be designated as the observation target.

[0166] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be able to identify a third actual object located in the external environment not included in the screen using the user's gaze, the designated gesture input, or the at least one camera as the at least one actual object to be designated as the object of observation.

[0167] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may identify an area that is included within the field of view of the at least one camera and not included in the screen as an observation area while the user is looking at the screen, and designate the observation area as the observation target.

[0168] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may monitor a change in the state of a first virtual object obscured by a second virtual object among the plurality of virtual objects based on identifying that a plurality of virtual objects are superimposed and displayed on the screen, and adjust the transparency of the second virtual object or move the second virtual object so that the first virtual object becomes visible based on confirming that the state of the first virtual object has changed.

[0169] FIG. 11 is a diagram illustrating an example of an operation method in an electronic device according to one embodiment. In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0170] Referring to FIG. 11, an electronic device according to one embodiment (e.g., electronic device (101) of FIG. 1, electronic device (200) of FIG. 2, electronic device (300) of FIG. 3a, 3b, and 3c, or electronic device (401) of FIG. 4) displays, in operation 1101, a screen of an external environment (e.g., screen (601) of FIG. 6 to 10) on a display (e.g., display module (160) of FIG. 1, display (321) of FIG. 3, or display (430) of FIG. 4) based on images captured using at least one camera (e.g., camera module (180) of FIG. 1, first camera (211-1, 211-2) of FIG. 2, third camera (213), camera (313, 314, 315, 316) of FIG. 3b, or at least one camera included in the camera circuit (440) of FIG. 4). It is possible.

[0171] In operation 1103, the electronic device may control the display to display at least one virtual object in a portion of the screen. The electronic device may display at least one virtual object on the screen configured based on information (e.g., content or execution screen) related to one or more applications running.

[0172] In operation 1105, the electronic device may designate at least one real object located in an external environment as an observation target. According to one embodiment, the electronic device may designate a real object located in an external environment selected by a user’s gaze or user gesture, or a real object automatically detected by at least one camera in the external environment, as an observation target. According to one embodiment, the electronic device may designate an object in which an emergency situation (e.g., fire or smoke) has occurred as an observation target object by analyzing a video or image acquired through at least one camera. If there are multiple observation target objects, the electronic device may set priorities and designate the observation target objects in order of highest priority. Here, the observation target may be a real object included in a part of the screen that is obscured by at least one virtual object, a real object included in a part of the screen scene designated based on user input (e.g., gesture input), or a real object designated in an external environment not included in the screen. Not limited thereto, the observation target may be designated as an observation area set to be monitored on the screen or an observation area set in a space of the external environment outside the screen. The electronic device may display visual graphic effects (e.g., outline, highlight, or sticker) on the observation target included in the screen. For example, visual graphic effects may be displayed when a change in the state of the observed object is confirmed, or after it has been designated as an observed object so that the user can know that it is an observed object.

[0173] In operation 1107, the electronic device may monitor changes in the state of the object being observed using at least one camera and / or microphone (e.g., the acoustic output module (155) of FIG. 1 or the speaker (470) of FIG. 4). According to one embodiment, if the object being observed is a real object included in the screen (e.g., a first real object), the electronic device may monitor the object being observed using at least one camera within the user's field of view. According to one embodiment, if the object being observed is a real object not included in the screen (e.g., a second real object), the electronic device may monitor the object being observed using at least one camera having a wider field of view than the user's field of view. According to one embodiment, if the object being observed (e.g., a real object not included in the screen) cannot be monitored within the field of view of at least one camera while the user is looking at the screen, the object being observed may be monitored by an external electronic device located nearby, and the electronic device may receive information regarding the monitoring of the object being observed from the external electronic device through a communication circuit (e.g., the communication module (190) of FIG. 1 or the communication circuit (450) of FIG. 4).

[0174] In operation 1109, the electronic device can acquire (e.g., identify or confirm) information related to the change in the state of the observed object (e.g., initial frame and transformed frame) as it confirms that the state of the observed object has changed. According to one embodiment, the observed object is a real object (e.g., a first real object) obscured by at least one virtual object, and when the electronic device confirms that the state of the observed object has changed, it can adjust the transparency of at least one virtual object so that the real object obscured by at least one virtual object becomes visible. According to one embodiment, when the electronic device receives information regarding the monitoring of the observed object from an external electronic device located around the observed object, it can acquire information related to the change in the state of the observed object to be transmitted to an artificial intelligence model (e.g., the artificial intelligence model of FIG. 5) based on the information regarding the monitoring of the observed object.

[0175] In operation 1111, the electronic device transmits information related to the state change of the observed object to an artificial intelligence model, and can obtain notification information generated by the artificial intelligence model based on the information related to the state change of the observed object.

[0176] In operation 1113, the electronic device can control the display to display notification information on the screen. If the notification information includes voice information, the electronic device can output the voice information through a speaker (e.g., the sound output module (155) of FIG. 1 or the speaker (470) of FIG. 4). The notification information is information that informs the user of a change in the state of an object of observation, and may include at least one of text, image, or voice information, and may be displayed on the screen as a virtual object in the form of a window or a pop-up window.

[0177] FIG. 12 is a diagram illustrating an example of transmitting notification information from an electronic device to an external electronic device according to one embodiment.

[0178] Referring to FIG. 12, according to one embodiment, an electronic device (401) can transmit notification information (1210) displayed on a screen (601) to an external electronic device (1201, 1203) (e.g., a wearable device). The external electronic device (1201, 1203) can display text information and / or an image (1210a) included in the received notification information (1210) through a display, or output voice information (1210b) included in the notification information (1210) (e.g., "Laundry is finished.") through a speaker (e.g., the sound output module (155) of FIG. 1 or the speaker (470) of FIG. 4).

[0179] FIG. 13 is a diagram illustrating an example of a method of operation in an electronic device according to one embodiment.

[0180] Referring to FIG. 13, in operation 1301, according to one embodiment, an electronic device (e.g., electronic device (101) of FIG. 1, electronic device (200) of FIG. 2 and electronic device (300) of FIG. 3a, FIG. 3b and FIG. 3c or electronic device (401) of FIG. 4) can identify that at least one virtual object displayed on the screen (e.g., virtual object (611) of FIG. 6a and FIG. 6b) obscures a real object (e.g., object of observation (603) of FIG. 6c). The electronic device can designate the real object as the object of observation.

[0181] In operation 1303, the electronic device can transmit information about the area of ​​a real object that is obscured by a virtual object (e.g., cropped information about the area of ​​the real object) to a screen analysis module (e.g., screen analysis module (510) of FIG. 5). If there are multiple virtual objects, the electronic device can transmit parts of cropped images of multiple areas to the screen analysis module.

[0182] In operation 1305, the electronic device can monitor the object of observation through a screen analysis module (e.g., the screen analysis module (510) of FIG. 5). The screen analysis module can analyze image frames at all times or at regular intervals in the area where the virtual object is located based on cropped information.

[0183] In operation 1307, the electronic device may transmit information related to a change in the state of the observed object to an artificial intelligence model (e.g., the artificial intelligence model (530) of FIG. 5 (multimodal engine)) based on confirming that a change in the observed object has occurred (e.g., a trigger or event indicating a change in the observed object has occurred). According to one embodiment, if the electronic device identifies that a designated event (e.g., an emergency situation) has occurred in addition to a change in the observed object, it may transmit information related to the event to the artificial intelligence model.

[0184] In operation 1309, the electronic device can obtain notification information based on information about changes in the observed object through an artificial intelligence model (e.g., the artificial intelligence model (530) of FIG. 5 (multimodal engine)). For example, if a notable event is identified through the analysis of an image frame of the area where a virtual object is located, notification information can be obtained based on information about changes in the observed object through an artificial intelligence model.

[0185] In operation 1311, the electronic device may display notification information (e.g., notification information (621) of FIG. 6c) generated based on information about changes in the object being observed through an artificial intelligence model, while making the real object obscured by the virtual object visible (e.g., adjusting the transparency of the virtual object or displaying the virtual object being moved to a different location). According to one embodiment, the electronic device may display a visual graphic effect (e.g., visual graphic effect (631) of FIG. 6c) (e.g., outline, highlight, zoom effect, or sticker) indicating that a change in state has occurred in the object being observed.

[0186] FIG. 14 is a diagram illustrating an example of a method of operation in an electronic device according to one embodiment.

[0187] Referring to FIG. 14, in operation 1401, according to one embodiment, an electronic device (e.g., electronic device (101) of FIG. 1, electronic device (200) of FIG. 2 and electronic device (300) of FIG. 3a, FIG. 3b and FIG. 3c or electronic device (401) of FIG. 4) can identify that a first virtual object (e.g., first virtual object (611) of FIG. 7a), which is one of a plurality of virtual objects displayed on the screen, is obscured by a third virtual object (e.g., third virtual object (711) of FIG. 7a), which is another object.

[0188] In operation 1403, the electronic device can transmit information about the area of ​​the first hidden virtual object to a screen analysis module (e.g., the screen analysis module (510) of FIG. 5).

[0189] In operation 1405, the electronic device can monitor the area of ​​the first virtual object through the screen analysis module.

[0190] In operation 1407, the electronic device can transmit information about the first virtual object (e.g., information related to the event) to an artificial intelligence model (e.g., the artificial intelligence model (530) of FIG. 5) based on identifying that a specified event for the first virtual object occurs during monitoring.

[0191] 1409 In operation, the electronic device can obtain notification information through an artificial intelligence model.

[0192] In operation 1411, the electronic device may display notification information (e.g., notification information (733) of FIG. 7c) generated based on information about changes in the observed object through an artificial intelligence model while making the first virtual object obscured by the third virtual object visible (e.g., adjusting the transparency of the third virtual object or displaying the first virtual object moved to a different location). The electronic device may display a visual graphic effect (e.g., visual graphic effect (731) of FIG. 7c) (e.g., outline, highlight, zoom effect, or sticker) indicating that a state change has occurred in the first virtual object while making the first virtual object obscured by the third virtual object visible.

[0193] FIG. 15 is a diagram illustrating an example of a method of operation in an electronic device according to one embodiment.

[0194] Referring to FIG. 15, in operation 1501, an electronic device (e.g., electronic device (101) of FIG. 1, electronic device (200) of FIG. 2 and electronic device (300) of FIG. 3a, FIG. 3b and FIG. 3c or electronic device (401) of FIG. 4) identifies that a real object displayed on the screen is identified by a user and can designate the identified real object (e.g., observation target (603) of FIG. 8a to 8c) as an observation area. For example, if the electronic device detects a blink of the user's eye or a gesture of the user after the user's gaze has been fixed on a specific area of ​​the screen (e.g., screen (601) of FIG. 8a to 8c) for a certain period of time, it can designate a real object located in the specific area that was gazed at as an observation area.

[0195] In operation 1503, the electronic device can transmit information about the area of ​​the object to be observed (e.g., cropping the area of ​​the object to be observed and information about the cropped area) to a screen analysis module (e.g., the screen analysis module (510) of FIG. 5). If there are multiple virtual objects, the electronic device can transmit parts of cropped images of multiple areas to the screen analysis module.

[0196] In operation 1505, the electronic device can monitor an observation target designated by the user through a screen analysis module (e.g., the screen analysis module (510) of FIG. 5). The screen analysis module can analyze image frames at all times or at regular intervals in the area where the virtual object is located based on cropped information.

[0197] In operation 1507, the electronic device may transmit information about a change in the state of the observed object to an artificial intelligence model (e.g., the artificial intelligence model (530) of FIG. 5 (multimodal engine)) based on confirming that a change in the observed object has occurred (e.g., a trigger indicating a change in the observed object). In one embodiment, if the electronic device confirms that a designated event (e.g., an emergency situation) has occurred in addition to a change in the observed object, it may transmit information related to the event to the artificial intelligence model.

[0198] In operation 1509, the electronic device can obtain notification information based on information about changes in the observed object through an artificial intelligence model (e.g., the artificial intelligence model (530) of FIG. 5 (multimodal engine)). For example, if a notable event is identified through the analysis of an image frame of the area where a virtual object is located, notification information can be obtained based on information about changes in the observed object through an artificial intelligence model.

[0199] In operation 1511, the electronic device may display notification information (e.g., notification information (831) of FIG. 8c). The electronic device may display visual graphic effects (e.g., visual graphic effects (821) of FIG. 8b and FIG. 8c) (e.g., outline, highlight, zoom effect, or sticker) indicating that a change in state has occurred in the object being observed.

[0200] FIG. 16 is a diagram showing an example of a method of operation in an electronic device according to one embodiment.

[0201] Referring to FIG. 16, in operation 1601, according to one embodiment, an electronic device (e.g., electronic device (101) of FIG. 1, electronic device (200) of FIG. 2 and electronic device (300) of FIG. 3a, FIG. 3b and FIG. 3c or electronic device (401) of FIG. 4) can identify a first area that is outside the user's field of view (e.g., field of view) that is not visible to the user but is monitored by a camera. The electronic device can designate the identified first area as an observation target.

[0202] In operation 1603, the electronic device can crop a first area and transmit information about the cropped first area to a screen analysis model.

[0203] In operation 1605, the electronic device can monitor changes in the state of the observed object using at least one camera and / or microphone (e.g., the microphone (460) of FIG. 4). For example, if the electronic device uses an external camera device (CCTV, webcam) other than the electronic device's camera, it can receive and process images from the other camera and transmit the processed images to a screen analysis module to monitor changes in the state of the observed object.

[0204] In operation 1507, the electronic device may transmit information about a change in the state of the observed object to an artificial intelligence model (e.g., the artificial intelligence model (530) of FIG. 5) based on identifying that a change in the observed object has occurred (e.g., a trigger indicating a change in the observed object). According to one embodiment, if the electronic device identifies that a specified event (e.g., an emergency situation) has occurred in addition to a change in the observed object, it may transmit information related to the event to the artificial intelligence model.

[0205] In operation 1509, the electronic device can obtain notification information based on information about changes in the observed object or information related to a specified event through an artificial intelligence model.

[0206] In operation 1511, the electronic device may display notification information (e.g., notification information (1021) of FIG. 10b) in an area adjacent to the actual object (e.g., TV) (e.g., an area that does not obscure the actual object being watched) so that the user can easily check the notification information (e.g., notification information (1021) of FIG. 10b) while watching the actual object (e.g., the actual object (1011) of FIG. 10b) within the screen (e.g., the screen (601) of FIG. 10a and FIG. 10b).

[0207] According to one embodiment, a method of operation of an electronic device wearable on a user's body (e.g., electronic device (101) of FIG. 1, electronic device (200) of FIG. 2, electronic device (300) of FIG. 3a, FIG. 3b, and FIG. 3c, or electronic device (401) of FIG. 4) is based on images captured through at least one camera of the electronic device (e.g., camera module (180) of FIG. 1, first camera (211-1, 211-2) of FIG. 2, third camera (213), camera (313, 314, 315, 316) of FIG. 3b, or at least one camera included in the camera circuit (440) of FIG. 4), through a display of the electronic device (e.g., first display (251) of FIG. 2, second display (252), display member (340) of FIG. 3a, display (321) of FIG. 3c, or display (430) of FIG. 4). It may include an action that displays a screen of the external environment.

[0208] According to one embodiment, the method may include the operation of displaying at least one virtual object in a part area of ​​the screen.

[0209] According to one embodiment, the method may include an operation of designating at least one actual object located in the external environment as an observation target.

[0210] According to one embodiment, the method may include an operation of monitoring a change in the state of the observed object using at least one camera.

[0211] According to one embodiment, the method may include an operation of obtaining information related to a change in the state of the observed object based on identifying a change in the state of the observed object.

[0212] According to one embodiment, the method may include an operation of obtaining notification information indicating the state change of the observed object using an artificial intelligence model (e.g., the artificial intelligence model of FIG. 5) based on information related to the state change of the observed object.

[0213] According to one embodiment, the method may include an operation of displaying the notification information on the screen.

[0214] According to one embodiment, the object to be observed may be a real object included in a portion of the screen obscured by at least one virtual object, a real object included in a portion of the screen specified based on user input, or a real object specified in the external environment not included in the screen.

[0215] According to one embodiment, the notification information may include at least one of text, image, or voice information.

[0216] According to one embodiment, the method may further include an operation of monitoring a change in the state of the object being observed using a microphone of the electronic device (e.g., the acoustic output module (155) of FIG. 1 or the speaker (470) of FIG. 4) or an external electronic device located around the object being observed.

[0217] According to one embodiment, the method may further include an operation of displaying a visual graphic effect on the observation target included in the screen.

[0218] According to one embodiment, the operation of designating at least one actual object located in the external environment as an observation target may include the operation of identifying a first actual object included in the screen obscured by the at least one virtual object and the operation of designating the first actual object as an observation target.

[0219] According to one embodiment, the method may further include the operation of adjusting the transparency of the at least one virtual object or moving the at least one virtual object so that the first actual object is visible, based on identifying that the state of the observed object has changed.

[0220] According to one embodiment, the operation of designating at least one actual object located in the external environment as an observation target may further include the operation of identifying a first actual object to be designated as an observation target in a part of the screen based on the user’s gaze looking at a part of the screen and identifying a designated gesture input.

[0221] According to one embodiment, the operation of designating at least one actual object located in the external environment as an observation target may include the operation of identifying a second actual object to be designated as an observation target by analyzing the images captured through the at least one camera, and the operation of designating the second actual object as an observation target.

[0222] According to one embodiment, the operation of designating at least one actual object located in the external environment as an observation target may include the operation of identifying a third actual object located in the external environment that is not included in the screen using the user's gaze, the designated gesture input, or the at least one camera as the at least one actual object to be designated as an observation target, and the operation of designating the third actual object as an observation target.

[0223] According to one embodiment, the method may further include the action of identifying an area that is included within the field of view of at least one camera and not included in the screen as an observation area while the user is looking at the screen, and the action of designating the observation area as the observation target.

[0224] According to one embodiment, the method may further include an operation of identifying that a plurality of virtual objects are superimposed and displayed on the screen, an operation of monitoring a change in the state of a first virtual object that is obscured by a second virtual object among the plurality of virtual objects, and an operation of adjusting the transparency of the second virtual object or moving the second virtual object so that the first virtual object becomes visible based on confirming that the state of the first virtual object has changed.

[0225] According to one embodiment, in a non-transient storage medium storing one or more programs, the one or more programs may include instructions that, when executed by at least one processor of an electronic device, cause the electronic device to perform an operation of displaying a screen of the external environment of the electronic device through a display of the electronic device based on images captured through at least one camera of the electronic device.

[0226] According to one embodiment, the one or more programs may include an instruction that causes the electronic device to perform an operation of displaying at least one virtual object in a part area of ​​the screen when executed by at least one processor of the electronic device.

[0227] According to one embodiment, the one or more programs may include an instruction that causes the electronic device to execute an operation of designating at least one actual object located in the external environment as an observation target when executed by at least one processor of the electronic device.

[0228] According to one embodiment, the one or more programs may include instructions that, when executed by at least one processor of the electronic device, cause the electronic device to perform an operation of monitoring a change in the state of the observation target using the at least one camera.

[0229] According to one embodiment, the one or more programs may include instructions that, when executed by at least one processor of an electronic device, cause the electronic device to execute an operation of obtaining information related to a change in the state of an object to be observed based on identifying a change in the state of the object to be observed, and an operation of obtaining notification information indicating the change in the state of the object to be observed using an artificial intelligence model (e.g., the artificial intelligence model of FIG. 5) based on the information related to the change in the state of the object to be observed.

[0230] According to one embodiment, the one or more programs may include instructions that cause the electronic device to perform an operation of displaying the notification information on the screen when executed by at least one processor of the electronic device.

[0231] An electronic device according to one embodiment of the present document designates at least one physical object in an external environment as an observation target and monitors changes in the state of the designated observation target; when a change in state occurs, it may use artificial intelligence to provide notification information indicating the change in the state of the observation target so that the user can easily confirm it. The present document provides the effect of enabling the user to easily confirm the state of an observation target that is obscured by a virtual object, a space outside the screen, or a blind spot, and to easily confirm the state of the observation target even without the user looking at the observation target. In addition, various effects that can be identified directly or indirectly through the present document may be provided. The effects obtainable from the present document are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art to which the present disclosure belongs from the description below.

[0232] Furthermore, the embodiments disclosed in this document are presented for the purpose of explaining and understanding the disclosed technical content and are not intended to limit the scope of the technology described in this document. Accordingly, the scope of this document should be interpreted to include all modifications or various other embodiments based on the technical concept of this document.

[0233] The electronic device according to the various embodiments disclosed in this document may be of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. The electronic device according to the embodiments of this document is not limited to the devices described above.

[0234] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as “coupled” or “connected” to another (e.g., 2nd) component, with or without the terms “functionally” or “communicationly,” it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0235] The term “module” as used in the various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0236] Various embodiments of the present document may be implemented as software (e.g., program (140)) comprising one or more instructions stored in a storage medium (e.g., internal memory (136) or external memory (138)) readable by a machine (e.g., electronic device (101)). For example, a processor (e.g., processor (120)) of the machine (e.g., electronic device (101)) may call at least one of the one or more instructions stored in the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0237] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or an application store (e.g., Play Store). TM It can be distributed online (e.g., downloaded or uploaded) through ) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0238] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In an electronic device (101, 200, 300, 401) wearable on a user's body, At least one camera (180, 211-1, 211-2, 213, 313, 314, 315, 316, 440); Display(160, 251, 252, 340, 321, 430); At least one processor (120, 410) including a processing circuit; and It includes memory (130, 420) for storing instructions, When the above instructions are executed individually or collectively by the at least one processor, the electronic device: Based on images captured through the above at least one camera, a screen of the external environment of the electronic device is displayed through the display, and At least one virtual object is displayed through the display in a portion of the above screen, and At least one actual object located in the above external environment is designated as an observation target, and Using at least one camera, the state change of the observation target is monitored, and Based on identifying a change in the state of the above-mentioned object of observation, information related to the change in the state of the above-mentioned object of observation is obtained, and Based on information related to the state change of the above-mentioned observation target, an artificial intelligence model (530) is used to obtain notification information indicating the state change of the above-mentioned observation target, and An electronic device that causes the above notification information to be displayed on the screen via the display.

2. In Paragraph 1, The observation object is a real object included in a portion of the screen obscured by at least one virtual object, a real object included in a portion of the screen designated based on user input, or a real object designated in the external environment not included in the screen. The above notification information includes at least one of text information, image or voice information, and when the above instructions are executed individually or collectively by the at least one processor, the electronic device: Monitoring changes in the state of the object to be observed using the microphone (155, 470) of the electronic device or an external electronic device located around the object to be observed, and An electronic device that causes the display to control the display to display visual graphic effects on the observation target included in the screen.

3. In paragraph 1 or 2, when the instructions are executed individually or collectively by the at least one processor, the electronic device: Identifying a first real object included in the screen obscured by at least one virtual object, and Designate the above first actual object as the observation target, and An electronic device that, based on identifying a change in the state of the observed object, causes the transparency of at least one virtual object to be adjusted or the at least one virtual object to be moved so that the first actual object is visible.

4. In any one of claims 1 to 3, when the instructions are executed individually or collectively by the at least one processor, the electronic device: An electronic device that causes the user’s gaze to look at a portion of the screen and, based on identifying a designated gesture input, to identify a first actual object to be designated as an observation target in a portion of the screen.

5. In any one of claims 1 to 4, when the instructions are executed individually or collectively by the at least one processor, the electronic device: An electronic device that analyzes images captured through at least one camera to identify a second actual object to be designated as an object of observation, and causes to identify a third actual object located in an external environment not included in the screen using the user's gaze, the designated gesture input, or the at least one camera as the at least one actual object to be designated as an object of observation.

6. In any one of claims 1 to 4, when the instructions are executed individually or collectively by the at least one processor, the electronic device: While the user is looking at the screen, an area included within the field of view of at least one camera and not included in the screen is identified as an observation area, An electronic device that causes the above observation area to be designated as the above observation target.

7. In any one of claims 1 through 6, when the instructions are executed individually or collectively by the at least one processor, the electronic device: Based on identifying that a plurality of virtual objects are displayed in overlap on the screen above, the state change of a first virtual object obscured by a second virtual object among the plurality of virtual objects is monitored, and An electronic device that, based on confirming that the state of the first virtual object has changed, causes the transparency of the second virtual object to be adjusted so that the first virtual object is visible or causes the second virtual object to be moved.

8. A method of operation in an electronic device (101, 200, 300, 401) wearable on a user's body, An operation of displaying a screen of the external environment of the electronic device through a display (160, 251, 252, 340, 321, 430) of the electronic device based on images captured through at least one camera (180, 211-1, 211-2, 213, 313, 314, 315, 316, 440) of the electronic device; The operation of displaying at least one virtual object through the display in a portion of the above screen; An action of designating at least one actual object located in the above external environment as an observation target; An operation of monitoring a change in the state of the observation target using at least one camera; An operation to obtain information related to a change in the state of the observed object based on identifying a change in the state of the observed object; An operation to obtain notification information indicating the state change of the observed object using an artificial intelligence model (530) based on information related to the state change of the observed object; and A method comprising the operation of displaying the above notification information on the screen through the display.

9. In paragraph 8, the above method is, An operation of monitoring a change in the state of the observation target using a microphone (470) of the electronic device or an external electronic device located around the observation target; and It further includes an operation of displaying visual graphic effects on the observation target included in the above screen, and The observation object is a real object included in a portion of the screen obscured by at least one virtual object, a real object included in a portion of the screen designated based on user input, or a real object designated in the external environment not included in the screen. A method in which the above notification information includes at least one of text, image, or voice information.

10. In claim 8 or 10, the action of designating at least one actual object located in the external environment as an observation target is, An operation to identify a first actual object included in the screen obscured by at least one virtual object; and It includes an action of designating the first actual object as the observation target, The above method is, A method further comprising the operation of adjusting the transparency of at least one virtual object or moving at least one virtual object so that the first actual object is visible, based on identifying that the state of the observed object has changed.

11. In any one of claims 8 to 10, the operation of designating at least one actual object located in the external environment as an observation target is, A method further comprising the action of identifying a first actual object to be designated as an observation target in a part of the screen based on the user’s gaze looking at a part of the screen and identifying a designated gesture input.

12. In any one of claims 8 to 11, the operation of designating at least one actual object located in the external environment as an observation target is, The operation of analyzing the images captured through the at least one camera to identify a second actual object to be designated as the observation target, and designating the second actual object as the observation target; and A method comprising identifying a third actual object located in an external environment not included in the screen using the user's gaze, the designated gesture input, or the at least one camera as the at least one actual object to be designated as the object of observation, and designating the third actual object as the object of observation.

13. In any one of paragraphs 8 to 12, the method is, An operation of identifying an area as an observation area that is included within the field of view of at least one camera and not included in the screen while the user is looking at the screen; and A method further comprising the action of designating the observation area as the observation target.

14. In any one of paragraphs 8 to 13, the above method is, An operation to identify that a plurality of virtual objects are displayed in an overlapping manner on the above screen; An operation to monitor the state change of a first virtual object obscured by a second virtual object among the plurality of virtual objects; and A method further comprising the operation of adjusting the transparency of the second virtual object or moving the second virtual object so that the first virtual object is visible, based on confirming that the state of the first virtual object has changed.

15. In a non-transient storage medium storing one or more programs, the one or more programs, when executed by at least one processor (120, 410) of an electronic device (101, 200, 300, 401), cause the electronic device: An operation of displaying a screen of the external environment of the electronic device through a display (160, 251, 252, 340, 321, 430) of the electronic device based on images captured through at least one camera (180, 211-1, 211-2, 213, 313, 314, 315, 316, 440) of the electronic device; The operation of displaying at least one virtual object through the display in a portion of the above screen; An action of designating at least one actual object located in the above external environment as an observation target; An operation of monitoring a change in the state of the observation target using at least one camera; An operation to obtain information related to a change in the state of the observed object based on identifying a change in the state of the observed object; An operation to obtain notification information indicating the state change of the observed object using an artificial intelligence model (530) based on information related to the state change of the observed object; and A non-transient storage medium comprising an executable command to perform an operation of displaying the above notification information on the screen via the display.

Images