Electronic device and method for identifying movement of object in electronic device

Abstract

An electronic device according to an embodiment comprises a communication circuit, a user interface, a display, at least one processor, and a memory that stores instructions, wherein, when executed individually or collectively by the at least one processor, the instructions may instruct the electronic device to: identify, on the basis of a user instruction obtained via the user interface, a first object, which is a target to be monitored, and a first movement of the first object; if the distance between the electronic device and the first object exceeds a specified distance, control a sensor of at least one external device located in a first region, which is within a first distance from the first object, to be turned on; receive, via the communication circuit, a sensor value obtained from the at least one external device by means of the sensor thereof; and, if the current movement of the first object is identified as the first movement of the first object on the basis of the sensor value, output notification information reporting the occurrence of the first movement of the first object. Other embodiments may be included.

Description

Methods for verifying electronic devices and the behavior of objects in electronic devices

[0001] The present disclosure relates to an electronic device and a method for verifying the operation of an object in an electronic device.

[0002] Electronic devices used for monitoring security or safety, such as home cameras, baby cameras, pet cameras, and / or dash cameras, include additional features beyond their basic functions. In addition to the basic function of monitoring or recording the current state of the home environment when the user is away, these devices track the movement and / or sound of specific objects. When an event occurs, such as noise or significant movement, a notification is sent to the user's electronic device, allowing the user to check the status in real-time or view stored footage when necessary. These devices may be fixed in a designated location, equipped with wheels for mobility, or manufactured in the form of drones.

[0003] An electronic device according to one embodiment may include a communication circuit, a user interface, a display, at least one processor, and a memory for storing instructions. When the instructions according to one embodiment are executed individually or collectively by the at least one processor, the electronic device may identify a first object to be monitored and a first operation of the first object based on user instructions obtained through the user interface. When the instructions according to one embodiment are executed individually or collectively by the at least one processor, the electronic device may control the electronic device to turn on a sensor of at least one external device located in a first area within a first distance from the first object when the distance between the electronic device and the first object exceeds a specified distance. When the instructions according to one embodiment are executed individually or collectively by the at least one processor, the electronic device may receive a sensor value obtained from the at least one external electronic device through the sensor of the at least one external device via the communication circuit. When the above commands according to one embodiment are executed individually or collectively by the at least one processor, the electronic device can output notification information indicating the occurrence of the first operation of the first object when it identifies the current operation of the first object as the first operation of the first object based on the sensor value.

[0004] A method for verifying the operation of an object in an electronic device according to one embodiment may include an operation of verifying a first object to be monitored and a first operation of the first object based on a user command obtained through a user interface of the electronic device. The method according to one embodiment may include an operation of controlling to turn on a sensor of at least one external device located in a first area within a first distance from the first object when the distance between the electronic device and the first object exceeds a specified distance. The method according to one embodiment may include an operation of receiving a sensor value obtained from the at least one external electronic device through the sensor of the at least one external device via a communication circuit of the electronic device. The method according to one embodiment may include an operation of outputting notification information indicating the occurrence of the first operation of the first object when the current operation of the first object is confirmed as the first operation of the first object based on the sensor value.

[0005] In a non-volatile storage medium storing commands according to one embodiment, the commands are configured to cause the electronic device to perform at least one operation when executed by the electronic device, wherein the at least one operation may include an operation of confirming a first object to be monitored and a first operation of the first object based on a user command obtained through the user interface of the electronic device. The at least one operation according to one embodiment may include an operation of controlling to turn on a sensor of at least one external device located in a first area within a first distance from the first object when the distance between the electronic device and the first object exceeds a specified distance. The at least one operation according to one embodiment may include an operation of receiving a sensor value obtained from the at least one external electronic device through the sensor of the at least one external device via the communication circuit of the electronic device. The at least one operation according to one embodiment may include an operation of outputting notification information indicating the occurrence of the first operation of the first object when the current operation of the first object is confirmed as the first operation of the first object based on the sensor value.

[0006] FIG. 1 is a block diagram of an electronic device in a network environment according to one embodiment.

[0007] FIG. 2 is a block diagram of an electronic device according to one embodiment.

[0008] FIG. 3 is a block diagram of an object monitoring unit in an electronic device according to one embodiment.

[0009] FIGS. 4a and FIGS. 4b are drawings for explaining the operation of monitoring an object in an electronic device according to one embodiment.

[0010] FIGS. 5A and FIGS. 5B are drawings for explaining an operation for determining the position of an object in an electronic device according to one embodiment.

[0011] FIG. 6 is a diagram illustrating an operation for monitoring an object in an electronic device according to one embodiment.

[0012] FIGS. 7a, FIGS. 7b, and FIGS. 7c are drawings illustrating an operation for monitoring an object in an electronic device according to one embodiment.

[0013] FIG. 8 is a diagram illustrating an operation for monitoring an object in an electronic device according to one embodiment.

[0014] FIG. 9 is a diagram illustrating an operation for monitoring a plurality of objects in an electronic device according to one embodiment.

[0015] FIGS. 10a, FIGS. 10b, FIGS. 10c, FIGS. 10d, and FIGS. 10e are drawings for illustrating the provision of a user interface for monitoring an object in an electronic device according to a temporary example.

[0016] FIGS. 11a, FIGS. 11b, FIGS. 11c and FIGS. 11d are drawings for illustrating the provision of a user interface for monitoring an object in an electronic device according to a temporary example.

[0017] FIGS. 12a, FIGS. 12b, FIGS. 12c, FIGS. 12d, FIGS. 12e, FIGS. 12f, FIGS. 12g, and FIGS. 12h are drawings for illustrating the provision of a user interface for monitoring an object in an electronic device according to a temporary example.

[0018] FIG. 13 is a flowchart illustrating the operation of identifying an object in an electronic device according to one embodiment.

[0019] FIG. 14 is a flowchart illustrating the operation of identifying an object in an electronic device according to one embodiment.

[0020] FIG. 15 is a perspective view illustrating the internal configuration of a wearable electronic device according to one embodiment.

[0021] FIGS. 16a and FIGS. 16b are drawings showing the front and rear of a wearable electronic device according to one embodiment.

[0022] FIG. 17 is a block diagram of a wearable electronic device according to one embodiment.

[0023] FIG. 18 is a drawing for illustrating a generative artificial intelligence system according to one embodiment.

[0024] FIG. 1 is a block diagram of an electronic device (101) in a network environment (100) according to one embodiment. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or may communicate with at least one of an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through the server (108). According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (160), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)).

[0025] The processor (120) can control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., a program (140)), and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use lower power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.

[0026] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0027] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, input data or output data for software (e.g., program (140)) and related commands. The memory (130) may include volatile memory (132) or non-volatile memory (134).

[0028] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0029] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0030] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0031] The display module (160) can visually provide information to an external (e.g., user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0032] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (101).

[0033] The sensor module (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor. According to one embodiment, the sensor module (176) may include, for example, an inertial measurement unit sensor capable of detecting the movement of a user wearing the electronic device.

[0034] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0035] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0036] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0037] The camera module (180) can capture still images and video. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0038] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0039] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0040] The communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).

[0041] The wireless communication module (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), external electronic device (e.g., electronic device (104)), or network system (e.g., second network (199)). According to one embodiment, the wireless communication module (192) can support a Peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mMTC, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for realizing URLLC.

[0042] An antenna module (197) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197).

[0043] According to one embodiment, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0044] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0045] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (104) or the server (108) may be included within a second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0046] FIG. 2 is a block diagram of an electronic device according to one embodiment.

[0047] According to one embodiment, the electronic device may include various types of electronic devices that can check inputs and outputs through a visualized user interface, such as portable communication devices (e.g., smartphones) and / or wearable electronic devices (e.g., smartwatches and / or smart glasses).

[0048] According to one embodiment, the electronic device (201) may be implemented as a wearable electronic device that can be worn on a user's body (e.g., head). For example, the wearable electronic device (301) may be implemented as augmented reality glasses, a video see-through (VST) device, or a head-mount display (HMD) device. However, this is just one example, and the wearable electronic device (301) may be implemented as various devices.

[0049] Referring to FIG. 2, according to one embodiment, the electronic device (201) may include a processor (220), memory (230), display (260), and communication circuit (290).

[0050] According to one embodiment, the processor (220) can perform overall control operations of the electronic device (201). According to one embodiment, the processor (220) can control at least one other component (e.g., hardware or software component) of the electronic device (201) connected to the processor (220) by executing software (e.g., program (140) of FIG. 1), and can perform data processing or operations based on instructions. According to one embodiment, the instructions may include instructions composed of machine language that can be processed by the electronic device (201) or the processor (220). For example, the instructions may include instructions corresponding to operation instructions used in the program.

[0051] According to one embodiment, the processor (220) can identify a user's touch, user's gaze and / or user's gesture (e.g., hand gesture) obtained through a user interface as a user's command, and identify a first object to be monitored and a first operation of the first object for outputting notification information based on the user's command.

[0052] According to one embodiment, the output of notification information indicating a first operation of a first object may include displaying notification information through a display (260), outputting notification information through a speaker, and / or haptic feedback (vibration output) through an actuator.

[0053] According to one embodiment, the processor (220) can identify the user's voice command or user input for setting a command as a user command instructing the monitoring of an object.

[0054] According to one embodiment, when the processor (220) confirms the reception of a user's command for performing monitoring of an object, it can use an AI model to analyze the user's command and confirm the first object to be monitored and the first operation of the first object for outputting notification information.

[0055] According to one embodiment, the processor (220) can request the user to input additional information for performing monitoring of the object if, as a result of analyzing the user's command using an AI model, only the first object or the first action is confirmed.

[0056] According to one embodiment, the first object may be movable or fixed.

[0057] For example, when the processor (220) receives a user command, “Notify me if my dog ​​causes trouble,” it can use an AI model to classify the first object to be monitored as “my dog,” search for images and / or audio corresponding to “my dog” among the user data stored in memory (230), and classify the searched images and / or audio by matching them with the first object (“my dog).” The processor (220) can confirm “Notify me” as the first action of the first object for outputting notification information.

[0058] For example, when the processor (220) receives a user command, “Notify me if Toto gets into trouble,” it uses an AI model to classify the first object being monitored as “Toto,” and when it checks the image and / or audio stored under the name “Toto” among the user data stored in memory (230), it can classify the confirmed image and / or audio stored under the name “Toto” by matching it with the first object (“Toto”). The processor (220) can identify “Notify me” as the first action of the first object for outputting notification information.

[0059] For example, when the processor (220) receives a user command, “Let me know if Toto gets into an accident,” it uses an AI model to classify the first object being monitored as “Toto,” and if it cannot identify an image and / or audio stored under the name “Toto” among the user data stored in memory (230), it identifies at least one object (e.g., baby, puppy 1, and / or cat 1) among the images stored in memory (230) that include dynamic objects (e.g., puppy, cat and / or person), and can make an additional query to the user to see if there is an object corresponding to “Toto” among the at least one object.

[0060] For example, when the processor (220) receives a user's command "watch my dog," it uses an AI model to identify "my dog" as a monitoring target, but even if it cannot identify a command to analyze the first action of the first object for outputting notification information, it can identify it as a command to monitor the first object, "my dog," as it is a command entered for the purpose of monitoring the object.

[0061] According to one embodiment, the processor (220) can use an AI model to determine the type of sensor (e.g., camera sensor and / or microphone sensor) for verifying the first action of the first object.

[0062] For example, the processor (220) can use an AI model to analyze a user’s command, “Tell me if the puppy has a potty accident,” and if it identifies the first object as “puppy” and the first action of the first object as “puppy’s potty act,” it can use the AI ​​model to identify the types of sensors required to identify the first action of the first object as camera sensors and microphone sensors.

[0063] For example, the processor (220) can use an AI model to analyze a user's command, "Notify me if the dog has a potty accident," and if it is confirmed that the first object is "dog" and the first action of the first object is "dog's potty behavior," then since "potty accident" is a sub-concept of "abnormal behavior," it can use the AI ​​model to additionally ask the user whether to output notification information when confirming abnormal behavior of the dog (e.g., breaking objects) in addition to "potty accident." According to one embodiment, the processor (220) can check the location of the first object based on sensor values ​​received through the sensor (276) when the distance between the electronic device (201) and the first object falls within a specified distance, and can monitor the first object.

[0064] According to one embodiment, the sensor (276) may include a camera sensor and / or a microphone sensor capable of detecting the motion, position, movement and / or audio of the first object.

[0065] According to one embodiment, the processor (220) may provide the motion, position, movement, and / or audio of the first object in real time through a display (260), a speaker, and / or an actuator, based on sensor values ​​(e.g., an image including the first object and / or audio of the first object) received through a sensor (276).

[0066] According to one embodiment, the processor (220) can output notification information indicating the occurrence of the first action of the first object through a display (260) or a speaker when confirming the current action of the first object as the first action of the first object based on a sensor value (e.g., an image including the first object and / or audio of the first object) received through a sensor (276).

[0067] According to one embodiment, when the processor (220) confirms that the electronic device (201) and at least one external electronic device are included in a first area within a first distance from the first object, it can determine the location of the first object through the sensor (276) of the electronic device (201) and control the turn on of at least one external electronic device located in the first area within a first distance from the first object.

[0068] According to one embodiment, when the processor (220) identifies at least one first external electronic device including a sensor of the same type among at least one external electronic device located in a first area, it can control the sensor of the selected first external electronic device to turn on using an AI model based on priority conditions.

[0069] According to one embodiment, priority conditions may include the location of the first object and / or the performance of an external electronic device.

[0070] According to one embodiment, the processor (220) can use an AI model to control the sensor of the first external electronic device located closest to the first object among at least one first external electronic device located in a first area based on the location of the first object.

[0071] According to one embodiment, the processor (220) can control to turn on a first external electronic device selected based on performance and distance from a first object among at least one first external electronic device located in a first area using an AI model.

[0072] For example, the processor (220) can use an AI model to control one of the first external electronic devices located in the first area to turn on one of the first external electronic device that has lower performance but is closest to the location of the first object, the first external electronic device that is closest to the location of the first object, or the first external device that is closest to the location of the first object and has higher performance.

[0073] According to one embodiment, when the processor (220) confirms that the electronic device (201) and at least one external electronic device are included in a first area within a first distance from the first object, it may control the sensor of at least one external electronic device located in a second area within a second distance from the first object to switch to a standby state. According to one embodiment, the second area may represent an area that does not overlap with the first area.

[0074] According to one embodiment, the standby state may represent a state that minimizes power consumption, such as a power saving mode.

[0075] According to one embodiment, the processor (220) can control the sensor of at least one external electronic device located in a third area within a third distance from the first object to turn off when it confirms that the electronic device (201) and at least one external electronic device are included in a first area within a first distance from the first object. According to one embodiment, the third area may represent an area that does not overlap with the first area and the second area.

[0076] According to one embodiment, the processor (220) can receive information related to a plurality of external electronic devices from a first server that manages a plurality of external electronic devices registered under the same account as the electronic device (201) and store it in memory (230).

[0077] According to one embodiment, a plurality of external electronic devices may include at least one external electronic device located in a first region, at least one external electronic device located in a second region, and at least one external electronic device located in a third region.

[0078] According to one embodiment, information related to a plurality of external electronic devices may include at least one of type information of the external electronic device, function information of the external electronic device, location information of the external electronic device, or sensor information of the external electronic device.

[0079] According to one embodiment, the processor (220) can distinguish a first area, a second area, and a third area by using an AI model based on information related to a plurality of external electronic devices and the location of a first object confirmed through a sensor (276) of an electronic device.

[0080] According to one embodiment, the processor (220) can determine the location, movement, and / or operation of the first object based on sensor values ​​received from at least one external electronic device located in the first area when it determines that the distance between the electronic device (201) and the first object is outside a specified distance or that the electronic device (201) is not located in the first area.

[0081] According to one embodiment, the processor (220) can determine the location of the first object based on multiple sensors of multiple external electronic devices registered to the server with the same account as the electronic device (201) when the distance between the electronic device (201) and the first object exceeds a specified distance, is not located in a first area within a first distance from the first object, or is outside a specified space in which the first object can move.

[0082] According to one embodiment, the processor (220) can determine that the distance between the electronic device (201) and the first object has exceeded a specified distance if, due to the movement of the electronic device (201) or the first object, it is not possible to obtain a sensor value that can monitor the first object through the sensor (246) or to monitor (confirm) the current operation of the first object based on the sensor value obtained through the sensor (246).

[0083] According to one embodiment, when a monitoring operation for a first object needs to be performed when the distance between the electronic device (201) and the first object is outside a specified distance, or is not located in a first area within a first distance from the first object, or is outside a specified space in which the first object can move, the processor (220) controls the multiple sensors of multiple external electronic devices registered to the first server with the same account as the electronic device (201) to be temporarily turned on, and can determine the location of the first object based on the multiple sensors of the multiple external electronic devices.

[0084] According to one embodiment, the processor (220) can control to turn on at least one sensor of an external electronic device located in a first area within a first distance from the first object when it determines the location of the first object based on a plurality of sensors of a plurality of external electronic devices.

[0085] According to one embodiment, when the processor (220) determines the location of a first object based on a plurality of sensors of a plurality of external electronic devices, it can control at least one sensor of an external electronic device located in a second area within a second distance from the first object to switch to a standby state.

[0086] According to one embodiment, when the processor (220) determines the location of a first object based on a plurality of sensors of a plurality of external electronic devices, it can control to turn off at least one sensor of an external electronic device located in a third area within a third distance greater than a second distance from the first object.

[0087] According to one embodiment, the processor (220) may, based on information related to a plurality of external electronic devices, request the first server to control the sensor of at least one external electronic device located in the first area to turn on, control the sensor of at least one external electronic device located in the second area to switch to a standby state, and control the sensor of at least one external electronic device located in the third area to turn off.

[0088] According to one embodiment, the processor (220) can distinguish a first area, a second area, and a third area based on information related to a plurality of external electronic devices and the location of a first object identified based on a plurality of sensors of a plurality of external electronic devices.

[0089] According to one embodiment, the processor (220) can receive a sensor value obtained from a sensor of at least one external electronic device located in a first area through a first server.

[0090] According to one embodiment, the processor (220), based on information related to a plurality of external electronic devices, can establish a communication connection with a plurality of external electronic devices, transmit a command to at least one external electronic device located in a first area to instruct the sensor to turn on, transmit a command to at least one external electronic device located in a second area to instruct the sensor to switch to a standby state, and transmit a command to at least one external electronic device located in a third area to instruct the sensor to turn off.

[0091] According to one embodiment, the processor (220) can receive a sensor value obtained from a sensor of at least one external electronic device located in a first area based on a communication connection with a plurality of external electronic devices.

[0092] According to one embodiment, the processor (220) can control the first sensor of at least one first external electronic device to turn on when it identifies the type of sensor capable of verifying the first operation of the first object as the first sensor, and identifies at least one first external electronic device including the first sensor among at least one external electronic device located in the first area based on information related to a plurality of external electronic devices.

[0093] According to one embodiment, when the processor (220) identifies at least one first external electronic device including the same first sensor among at least one first external electronic device including at least one first sensor, it can control the selected first external electronic device to turn on using an AI model based on priority conditions (e.g., location of the first object and / or performance of the external electronic device, etc.).

[0094] For example, if the processor (220) determines the location of the first object based on a sensor value (e.g., an image containing the first object) received through a camera sensor of the second external electronic device located in the first area when the first external electronic device and the second external electronic device located in the first area both include a microphone sensor necessary to verify the first action of the first object, the processor (220) can use an AI model to control the microphone sensor of the first external electronic device located close to the first object among the first external electronic device and the second external electronic device to turn on and control the microphone sensor of the second external electronic device to turn off.

[0095] For example, if the first external electronic device and the second external electronic device located in the first area both include the same microphone sensor required to verify the first operation of the first object, and the microphone sensor of the first audio external electronic device has better performance than the microphone sensor of the second audio external electronic device, the processor (220) can use an AI model to control the microphone sensor of the first external electronic device to turn on and control the microphone sensor of the second external electronic device to turn off.

[0096] According to one embodiment, if the processor (220) identifies an external electronic device among at least one external electronic device located in a first region that is performing an action (task) instructed by the user, it may not control the sensor of the external electronic device so that the external electronic device can perform the action (task) instructed by the user.

[0097] For example, if the processor (220) identifies an external electronic device located in the first area that is performing an action (task) instructed by the user, such as outputting or storing a sensor value (e.g., audio) obtained through a microphone sensor when the front doorbell is pressed, the processor (220) may not control the microphone sensor of the external electronic device so that the external electronic device can perform the action (task) instructed by the user.

[0098] According to one embodiment, the processor (220) can detect the movement of a first object based on a sensor of at least one external electronic device located in a first area, and if the sensor in the first area within a first distance from the first object is switched to a standby state, the processor can control the sensor of at least one external electronic device in the standby state to turn on.

[0099] For example, the processor (220) can control the sensors of the first external electronic device and the second external electronic device located in a first area within a first distance from the first object to turn on, control the sensors of the third external electronic device and the fourth external electronic device located in a second area within a second distance from the first object to switch to a standby state, and control the sensors of the fifth external electronic device and the sixth external electronic device located in a third area within a third distance from the first object to turn off, and when the first object moves and is located near the third external electronic device and the fourth external electronic device, the processor (220) can control the sensors of the third external electronic device and the fourth external electronic device located in a first area within a first distance from the first object to turn on. The processor (220) can control the sensors of the first external electronic device and the second external electronic device or the sensors of the fifth external electronic device and the sixth external electronic device located in a second area within a second distance from the first object to switch to a standby state.

[0100] According to one embodiment, the processor (220) may output a message indicating that the first operation of the first object cannot be temporarily verified if there is no external electronic device including a sensor for verifying the first operation of the first object while the first object is moving.

[0101] According to one embodiment, the processor (220) may switch to a remote monitoring (tracking) mode when the electronic device (201) moves out of a first area within a first distance from the first object or moves out of a designated space (e.g., a house) where the first object can move.

[0102] According to one embodiment, the processor (220) receives a sensor value obtained through a sensor of at least one external electronic device from at least one external electronic device located in a first area within a first distance from the first object based on the movement of the first object in a remote monitoring mode, and can check the current operation of the first object based on the received sensor value.

[0103] For example, when the processor (220) confirms a user command for monitoring an object, such as "Notify me when the puppy causes trouble," it can use an AI model to identify the first object as "puppy" and identify the first action of the first object as "when the puppy causes trouble." The processor (220) can identify the first object being monitored by detecting data (video and / or audio) that can identify the first object, "puppy," among the data of the electronic device and / or the data of an external electronic device connected to the electronic device. When the processor confirms, based on user input, whether the "puppy" is blocking the first object being monitored, it can perform monitoring of the first object. The processor (220) can provide a cross-sectional view of an area of ​​a designated space (e.g., a house) where the first object, the puppy, can move through the display (260), and can display location information of the first object, the puppy, superimposed on the cross-sectional view. The processor (220) can monitor the first object, the dog, by tracking it based on sensor values ​​obtained through at least one external electronic device, such as a home cam, microphone, home assistant, and / or a vacuum cleaner camera sensor, included in "Room 1," which is a first area within a first distance from the first object, the dog. When the processor (220) confirms that the first object, the dog, which was in Room 1, is heading toward the hallway based on sensor values ​​obtained through the vacuum cleaner camera sensor received from the vacuum cleaner after a certain period of time has passed, the processor (220) can monitor the first object by changing the sensor of at least one external electronic device included in the hallway from a standby state to a turned-on state via a server. When the first object, the dog, has completely moved toward the hallway, the processor (220) controls the sensor of the home cam, microphone, home assistant, and / or vacuum cleaner included in Room 1 to return to a standby state or turn off, and can continue monitoring the first object, the dog, based on the sensor of at least one external electronic device included in the hallway.

[0104] For example, when the processor (220) confirms a user’s command for monitoring a first object, such as “track the child in a dangerous situation,” and determines that a person presumed to be a child is located in Room 1 based on data from an electronic device or data from at least one external connection device connected to the electronic device, it may display a pop-up window confirming whether the person located in Room 1 is the “child” included in the user’s command. When the processor (220) confirms, based on the user’s input, that the person located in Room 1 is the “child” included in the user’s command, it uses an AI model. Based on the video and audio of the child, the "child" is identified as the first object, and a "dangerous situation" corresponding to the first action of the first object can be identified as "when a loud sound is made" and / or "when a large movement is detected." The processor (220) identifies a camera sensor and a microphone sensor as the types of sensors required to identify the first action of the first object, and can control, via a server, to turn on the sensor of at least one first external electronic device including a camera sensor and / or a microphone sensor among at least one external electronic device included in room 1 where the "child" is located. While monitoring the "child", the first object located in room 1, the processor (220) can change to a remote tracking mode based on the movement of the electronic device, and if the electronic device moves out of the area of ​​room 1 or out of the designated space (e.g., house), the processor (220) can control to turn on, standby, or turn off the sensor of at least one external electronic device to identify the first action of the first object centered on the "child", the first object.When the processor (220) confirms the first action of the "child in a dangerous situation" first object based on sensor values ​​obtained through the camera sensor and / or microphone sensor of at least one external electronic device received from at least one external electronic device, it may output notification information indicating that the "child is in a dangerous situation" through the display (260) of the electronic device or the speaker of the electronic device. After outputting the notification information, the processor (220) may provide a user interface that can control the "child" which is the first object, and through the user interface, remotely control the external electronic device around the "child" which is the first object to block the danger, or output a voice of use through the speaker of the external electronic device around the "child" which is the first object so that the child can feel safe.

[0105] According to one embodiment, the processor (220) can monitor the first object to check the first operation of the first object, and check the second object to check the first operation of the second object based on the user's command, and monitor the second object to check the first operation of the second object.

[0106] According to one embodiment, the processor (220) can monitor a first object based on a sensor value obtained from at least one external electronic device located in a first region through a sensor of at least one external electronic device.

[0107] According to one embodiment, a sensor of at least one external electronic device may include a camera sensor and / or a microphone sensor capable of detecting the motion, position, movement and / or audio of a first object.

[0108] According to one embodiment, the processor (220) can provide the motion, position, movement, and / or audio of the first object in real time through a display (260), a speaker, and / or haptic feedback (vibration output) using an actuator, based on sensor values ​​(e.g., an image including the first object and / or audio of the first object) received through a sensor of at least one external electronic device located in a first area.

[0109] According to one embodiment, the processor (220) can output notification information indicating the occurrence of the first action of the first object through haptic feedback (vibration output) using a display (260), a speaker and / or an actuator, based on sensor values ​​(e.g., an image including the first object and / or audio of the first object) received through a sensor of at least one external electronic device located in the first area.

[0110] According to one embodiment, the processor (220) may provide a user interface that can control a first object located in a first area after outputting notification information indicating the occurrence of a first action of a first object through haptic feedback (vibration output) using a display (260), a speaker and / or an actuator.

[0111] For example, when the first object is a kettle and the first action of the first object is boiling water inside the kettle, the processor (220) receives a sensor value (audio) obtained through a microphone sensor of at least one external electronic device located in the first area, and when the first action of the first object, boiling water inside the kettle, is confirmed based on the sensor value, it can output notification information indicating the occurrence of the first action of the first object. The processor (220) can display a user interface through the display (260) that can control the first object by turning off or reducing the power of the induction on which the kettle is placed.

[0112] According to one embodiment, the processor (220) can display the location of the first object in real time as text or an icon through the display (260) while monitoring the first object.

[0113] According to one embodiment, the processor (220) may provide a user interface that controls the first object based on a user's selection while displaying the location of the first object in real time as text or an icon through the display (260) while monitoring the first object.

[0114] According to one embodiment, a user interface capable of controlling a first object may include a first menu for checking video and / or audio of the first object in real time, a second menu for outputting a user's voice so that the first object can hear it, a third menu for outputting audio selected by the user so that the first object can hear it, a fourth menu for controlling a sensor of at least one external electronic device located in a first area within a first distance from the first object based on user input, a fifth menu for checking the operation of the first object stored after notification information indicating the occurrence of a first operation of the first object is output, and / or a sixth menu for stopping the monitoring operation of the first object.

[0115] A memory (230) according to one embodiment may be implemented substantially identically or similarly to the memory (130) of FIG. 1.

[0116] According to one embodiment, an AI model may be stored in the memory (230), and the AI ​​model may be a generative AI model, may include a Large Multi Modal Model (LMM), and may be an on-device model or an external AI model.

[0117] At least some of the on-device AI models according to one embodiment can be operated as external AI models.

[0118] According to one embodiment, an on-device AI model performs processing of sensitive information such as personal information, and an external AI model can perform processing of information excluding sensitive information such as personal information.

[0119] An on-device AI model according to one embodiment is an artificial intelligence model implemented in an electronic device (201) and can provide various functions without a network.

[0120] According to one embodiment, a plurality of artificial intelligence models may be stored in the memory (230).

[0121] According to one embodiment, each of the plurality of AI models may be a model trained based on a specified type of learning algorithm, and may be an AI model implemented to receive various types of data (or content) as input, perform calculations, and output (or obtain) result data.

[0122] According to one embodiment, a plurality of AI models may include generative AI models.

[0123] A generative AI model according to one embodiment can generate and output new content (e.g., text, images, and / or computer code, etc.) based on learned content in response to an input prompt. For example, in an electronic device (601), learning is performed to output specific types of result data as output data using data of specified types based on a machine learning algorithm or a deep learning algorithm, thereby generating multiple AI models (e.g., machine learning models and deep learning models) that are stored in the electronic device (201), or AI models learned from an external electronic device (e.g., an external server) can be transmitted to and stored in the electronic device (201). For example, the electronic device (201) can output input data (input data) as output data of a model learned through artificial intelligence of specified types based on a machine learning algorithm or a deep learning algorithm. Machine learning algorithms include supervised learning algorithms such as linear regression and logistic regression, unsupervised learning algorithms such as clustering, visualization and dimensionality reduction, and association rule learning, and reinforcement learning algorithms, and deep learning algorithms may include Artificial Neural Networks (ANN), Deep Neural Networks (DNN), and Convolutional Neural Networks (CNN), and may further include various learning algorithms not limited to those described.The trained AI model includes at least one operation (e.g., a convolution layer or a pooling layer) for processing input data, and can be implemented to output result data by performing operations on the input data based on at least one operation.

[0124] According to one embodiment, an application that can be connected to an external AI model may be stored in the memory (230).

[0125] A display (260) according to one embodiment may be implemented substantially identically or similarly to the display (160) of FIG. 1.

[0126] Notification information indicating the occurrence of a first action of a first object can be output through a display (260) according to one embodiment.

[0127] The sensor (276) according to one embodiment may be implemented substantially identically or similarly to the sensor (276) of FIG. 1.

[0128] A sensor (276) according to one embodiment may include a camera sensor (e.g., camera module (180) of FIG. 1) and / or a microphone sensor, etc.

[0129] A communication circuit (290) according to one embodiment can form a communication connection with an external electronic device (e.g., another electronic device, or a server) using various types of communication methods and transmit and / or receive data. As described above, the communication methods may include a communication method that establishes a direct communication connection such as Bluetooth and Wi-Fi Direct, a communication method using an access point (AP) (e.g., Wi-Fi communication), or a communication method using cellular communication using a base station (e.g., 3G, 4G / LTE, 5G). Since the communication circuit (290) can be implemented as described above in the communication module (190) in FIG. 1, a redundant description is omitted.

[0130] FIG. 3 is a block diagram of an object monitoring unit in an electronic device according to one embodiment.

[0131] Referring to FIG. 3, according to one embodiment, the object monitoring unit (301) may include a user input unit (311), an object verification unit (313), a sensor management unit (315), an area management unit (317), a sensor control unit (319), an object analysis unit (321), a notification information output unit (323), and an object control unit (325).

[0132] According to one embodiment, the processor of the electronic device (e.g., the processor (220) of FIG. 2) can perform the same function as the object monitoring unit (301).

[0133] According to one embodiment, the object monitoring unit (301) or at least a part of the object monitoring unit (301) may be included in the processor of the electronic device (e.g., the processor (220) of FIG. 2).

[0134] According to one embodiment, the object monitoring unit (301) is configured separately from the processor of the electronic device (e.g., the processor (220) of FIG. 2) and can operate under the control of the processor.

[0135] According to one embodiment, the user input unit (311) may receive a user command (e.g., a user's voice command or a command received based on user input) that instructs the monitoring of an object.

[0136] According to one embodiment, the object verification unit (313) can verify the first object to be monitored and the first operation of the first object for outputting notification information by using an AI model to analyze the user's command when it confirms the reception of a user's command for performing monitoring of an object.

[0137] According to one embodiment, the sensor management unit (315) can manage information related to a plurality of external electronic devices received from a first server configured to manage a plurality of external electronic devices registered with the same account as the electronic device.

[0138] According to one embodiment, information related to a plurality of external electronic devices may include at least one of type information of the external electronic device, function information of the external electronic device, location information of the external electronic device, or sensor information of the external electronic device.

[0139] According to one embodiment, the area management unit (317) may set a first area within a first distance from a first object to turn on a sensor of at least one external electronic device, a second area within a second distance from a first object to switch a sensor of at least one external electronic device to a standby state, and a third area within a third distance from a first object to turn off a sensor of at least one external electronic device.

[0140] According to one embodiment, the area management unit (317) can determine whether the first object is static or dynamic using an AI model, and if the first object is dynamic, it can be divided into a first area, a second area, and a third area based on the movement of the first object.

[0141] According to one embodiment, the sensor control unit (319) can control multiple sensors of multiple external electronic devices registered with the same account as the electronic device.

[0142] According to one embodiment, the sensor control unit (319) can control to turn on a sensor of at least one external electronic device located in a first area within a first distance from a first object.

[0143] According to one embodiment, the sensor control unit (319) can control the sensor of at least one external electronic device located in a second area within a second distance from the first object to switch to a standby state.

[0144] According to one embodiment, the sensor control unit (319) can control to turn off the sensor of at least one external electronic device located in a third area within a third distance from the first object that is further than the second distance.

[0145] According to one embodiment, the object analysis unit (321) can analyze the current operation of the first object based on the sensor value when it receives a sensor value obtained through at least one sensor from at least one external electronic device located in a first area within a first distance from the first object.

[0146] According to one embodiment, if the object analysis unit (321) confirms the current operation of the first object analyzed based on the sensor value as the first operation of the first object, it can transmit information indicating that the first operation of the first object has occurred to the notification information output unit (323).

[0147] According to one embodiment, when the notification information output unit (323) receives information from the object analysis unit (319) indicating that a first operation of the first object has occurred, it can output notification information indicating the occurrence of a first operation of the first object through the display of the electronic device or the speaker of the electronic device.

[0148] According to one embodiment, the object control unit (325) may provide a user interface that can control the first object located in the first area after outputting notification information indicating the occurrence of a first operation of the first object through the display of the electronic device or the speaker of the electronic device.

[0149] FIGS. 4a and FIGS. 4b are drawings for explaining the operation of monitoring an object in an electronic device according to one embodiment.

[0150] Referring to FIGS. 4a and 4b, according to one embodiment, an electronic device (201) (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) can monitor a first object (401) by confirming a first operation of the first object for outputting notification information and a first object to be monitored based on a user command for monitoring the object.

[0151] According to one embodiment, the electronic device (201) can confirm the inclusion of the electronic device (201) and at least one external electronic device (411, 413, 415, and 417) in a first area (A1) within a first distance from a first object (401).

[0152] According to one embodiment, the electronic device (201) can identify a first external electronic device (411) and a second external electronic device (413) including camera sensors (411a, 413a) of the same type among at least one external electronic device (411, 413, 415, and 417) located in a first area (A1).

[0153] According to one embodiment, each of the external electronic devices (411, 413, 415, and 417) may include a plurality of sensors.

[0154] According to one embodiment, the electronic device (201) can identify a third external electronic device (415) and a second external electronic device (417) including a microphone sensor (415a, 417a) of the same type among at least one external electronic device (411, 413, 415, and 417) located in the first area (A1).

[0155] According to one embodiment, the electronic device (201) can determine the location of the first object (401) through the sensor of the electronic device (201) (e.g., the sensor (267) of FIG. 2).

[0156] Referring to FIG. 4a, according to one embodiment, an electronic device (201) can control to turn on the camera sensor (413a) of the second external electronic device (413) closest to the first object (401) and the microphone sensor (417a) of the fourth external electronic device (417) based on the location of the first object (401) in the first area (A1).

[0157] According to one embodiment, the electronic device (201) can control the camera sensor (411a) of the first external electronic device (411) and the microphone sensor (415a) of the third external electronic device (415) among at least one external electronic device (411, 413, 415, and 417) to switch to a standby state.

[0158] According to one embodiment, the electronic device (201) can check the operation of the first object (401) based on sensor values ​​received from the second external electronic device (413) and the fourth external electronic device (417) located in the first area (A1).

[0159] Referring to FIG. 4b, according to one embodiment, an electronic device (201) can control to turn on the camera sensor (411a) of the first external electronic device (411) and the microphone sensor (415a) of the third external electronic device (415) that are closest to the first object (401) among at least one external electronic device (411, 413, 415, and 417) based on the location of the first object (401) in a first area (A1).

[0160] According to one embodiment, the electronic device (201) can control the camera sensor (413a) of the second external electronic device (413) and the microphone sensor (417a) of the fourth external electronic device (417) among at least one external electronic device (411, 413, 415, and 417) to switch to a standby state.

[0161] According to one embodiment, the electronic device (201) can check the operation of the first object (401) based on sensor values ​​received from the first external electronic device (411) and the third external electronic device (415) located in the first area (A1).

[0162] FIGS. 5A and FIGS. 5B are drawings for explaining an operation for determining the position of an object in an electronic device according to one embodiment.

[0163] According to one embodiment, an electronic device (201) (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) can monitor a first object (501) by confirming a first operation of the first object for outputting notification information and a first object to be monitored based on a user's command for monitoring the object.

[0164] Referring to FIG. 5a, according to one embodiment, an electronic device (201) can control to temporarily turn on a plurality of sensors of a plurality of external electronic devices (511, 513, 515, 517, 519, 521, 523, and 525) registered to a first server with the same account as the electronic device (201) when it is determined that the distance between the electronic device (201) and the first object (501) exceeds a specified distance or that the electronic device (201) is not located in a space (550) where the first object (501) can move.

[0165] According to one embodiment, the electronic device (201) can determine the location of the first object (501) based on sensor values ​​received from a plurality of external electronic devices (511, 513, 515, 517, 519, 521, 523, and 525).

[0166] Referring to FIG. 5b, according to one embodiment, an electronic device (201) can turn on the sensor of the first external electronic device (511), the sensor of the second external electronic device (513), the sensor of the fifth external electronic device (519), and the sensor of the sixth external electronic device (521) located in a first area within a first distance from the first object (501) among the plurality of external electronic devices (511, 513, 515, 517, 519, 521, 523, and 525) when determining the location of the first object (501) based on sensor values ​​received from a plurality of external electronic devices (511, 513, 515, 517, 519, 521, 523, and 525).

[0167] According to one embodiment, the electronic device (201) can control the sensor of the third external electronic device (515) and the sensor of the seventh external electronic device (523), which are located in a second area within a second distance from the first object (501) among a plurality of external electronic devices (511, 513, 515, 517, 519, 521, 523, and 525), to switch to a standby state.

[0168] According to one embodiment, the electronic device (201) can control the sensor of the third external electronic device (515) and the sensor of the seventh external electronic device (523), which are located in a second area within a second distance from the first object (501) among a plurality of external electronic devices (511, 513, 515, 517, 519, 521, 523, and 525), to turn off without a standby state.

[0169] According to one embodiment, the electronic device (201) can control the sensor of the fourth external electronic device (517) and the sensor of the eighth external electronic device (525), which are located in a third area within a third distance from the first object (501) among a plurality of external electronic devices (511, 513, 515, 517, 519, 521, 523, and 525), to turn off.

[0170] According to one embodiment, the electronic device (201) can check the operation of the first object (501) based on sensor values ​​received from the first external electronic device (511), the second external electronic device (513), the fifth external electronic device (519) and / or the sixth external electronic device (521) located in the first area (A1).

[0171] FIG. 6 is a diagram illustrating an operation for monitoring an object in an electronic device according to one embodiment.

[0172] According to one embodiment, an electronic device (201) (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) can monitor a first object (601) by confirming a first operation of the first object for outputting notification information and a first object to be monitored based on a user's command for monitoring the object.

[0173] Referring to FIG. 6, according to one embodiment, when the electronic device (201) confirms that the distance between the electronic device (201) and the first object (501) is outside the specified distance (d1), the electronic device (201) controls the multiple sensors of multiple external electronic devices registered on the first server with the same account as the electronic device (201) to temporarily turn on, and can confirm the location of the first object based on the multiple sensors of the multiple external electronic devices.

[0174] According to one embodiment, the electronic device (201) can identify a first external electronic device (611) and a camera sensor (611a), a microphone sensor (613a) of a second external electronic device (613), a microphone sensor (615a) of a third external electronic device (615), and a camera sensor (617a) of a fourth external electronic device (617) located in a first area (A1) within a first distance from a first object (601), based on information related to a plurality of external electronic devices.

[0175] According to one embodiment, the electronic device (201) can identify the external electronic device to turn on the sensor using an AI model based on priority conditions (e.g., the location of the first object and / or the sensor performance of the external electronic device), and control the center of the identified external electronic device to turn on and control the sensors of the remaining external electronic devices to switch to a standby state when it is confirmed that the first external electronic device (611) and the fourth external electronic device (617a) located in the first area (A1) include the same camera sensor and the second external electronic device (613) include the same microphone sensor.

[0176] FIGS. 7a, FIGS. 7b, and FIGS. 7c are drawings illustrating an operation for monitoring an object in an electronic device according to one embodiment.

[0177] According to one embodiment, an electronic device (201) (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) can monitor a first object (701) by confirming a first operation of the first object for outputting notification information and a first object to be monitored based on a user command for monitoring the object.

[0178] Referring to FIG. 7a, according to one embodiment, an electronic device (201) can control to turn on the sensor of the first external electronic device (711), the sensor of the second external electronic device (713), the sensor of the third external electronic device (715), and the sensor of the fourth external electronic device (717) located in a first area (A1) within a first distance from the first object (701).

[0179] According to one embodiment, the electronic device (201) can control the sensor of the fifth external electronic device (719) and the sensor of the sixth external electronic device (721), located in a second area (A2) within a second distance from the first object (701), to switch to a standby state.

[0180] According to one embodiment, the electronic device (201) can control the sensor of the fifth external electronic device (719) and the sensor of the sixth external electronic device (721) located in the second area (A2) within a second distance from the first object (701) to turn off without a standby state.

[0181] According to one embodiment, the electronic device (201) can determine the operation of the first object (701) based on sensor values ​​received from the first external electronic device (711), the second external electronic device (713), the third external electronic device (715) and / or the fourth external electronic device (717) located in the first area (A1).

[0182] Referring to FIG. 7b, according to one embodiment, an electronic device (201) can control the sensor of the third external electronic device (715), the sensor of the fourth external electronic device (717), the sensor of the fifth external electronic device (719), and the sensor of the sixth external electronic device (721) located in the first area (A1) within a first distance from the first object (701) to turn on the sensor of the sixth external electronic device (721) based on the position change resulting from the movement of the first object (701) and the sensor of the second external electronic device (717) located in the first area (A1).

[0183] According to one embodiment, the electronic device (201) can control the sensor of the first external electronic device (711) and the sensor of the second external electronic device (713) located in a second area (A2) within a second distance from the first object (701) to switch to a standby state.

[0184] According to one embodiment, the electronic device (201) can control the sensor of the first external electronic device (711) and the sensor of the second external electronic device (713) located in a second area (A2) within a second distance from the first object (701) to turn off without a standby state.

[0185] According to one embodiment, the electronic device (201) can check the operation of the first object (701) based on sensor values ​​received from the third external electronic device (715), the fourth external electronic device (717), the fifth external electronic device (719), and the sixth external electronic device (721) located in the first area (A1).

[0186] Referring to FIG. 7c, according to one embodiment, the electronic device (201) can control the sensor of the fifth external electronic device (719) and the sensor of the sixth external electronic device (721) located in the first area (A1) within a first distance from the first object (701) to turn on based on the position change resulting from the movement of the first object (701), when the movement of the first object (701) is confirmed based on a sensor value received from at least one external electronic device among the first external electronic device (711), the second external electronic device (713), the third external electronic device (715), and the fourth external electronic device (717) located in the first area (A1).

[0187] According to one embodiment, the electronic device (201) can control the sensor of the third external electronic device (715) and the sensor of the fourth external electronic device (717), located in a second area (A2) within a second distance from the first object (701), to switch to a standby state.

[0188] According to one embodiment, the electronic device (201) can control the sensor of the first external electronic device (711) and the sensor of the second external electronic device (713) located in a third area (A3) within a second distance from the first object (701) to turn off.

[0189] According to one embodiment, the electronic device (201) can check the operation of the first object (701) based on sensor values ​​received from the fifth external electronic device (719) and the sixth external electronic device (721) located in the first area (A1).

[0190] As shown in FIGS. 7a, 7b, and 7c, the first area, second area, and third area are set up in real time based on the movement of the first object, thereby allowing the first object to be continuously monitored (tracked) while efficiently operating the power.

[0191] FIG. 8 is a diagram illustrating an operation for monitoring an object in an electronic device according to one embodiment.

[0192] According to one embodiment, an electronic device (201) (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) can monitor a first object (801) by confirming a first operation of the first object for outputting notification information and a first object to be monitored based on a user command for monitoring the object.

[0193] Referring to FIG. 8, according to one embodiment, the electronic device (201) may switch to a remote monitoring (tracking) mode when it is determined that the distance between the electronic device (201) and the first object (801) has exceeded a specified distance, or when the electronic device (201) has moved out of a first area within a first distance from the first object (801), or when the electronic device (201) has moved out of a specified space (e.g., a house) where the first object (801) can move.

[0194] According to one embodiment, the electronic device (201) can control to turn on the sensor of the first external electronic device (811), the sensor of the second external electronic device (813), the sensor of the third external electronic device (815), and / or the sensor of the fourth external electronic device (817) located in a first area (A1) within a first distance from the first object (801) in a remote monitoring mode.

[0195] According to one embodiment, the electronic device (201) can determine the operation of the first object (801) based on sensor values ​​received from the first external electronic device (811), the second external electronic device (813), the third external electronic device (815) and / or the fourth external electronic device (817) located in the first area (A1).

[0196] FIG. 9 is a diagram illustrating an operation for monitoring a plurality of objects in an electronic device according to one embodiment.

[0197] According to one embodiment, an electronic device (201) (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) can monitor a first object (901) by confirming a first operation of the first object for outputting notification information and a first object to be monitored based on a user's command for monitoring the object.

[0198] According to one embodiment, an electronic device (201) (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) can monitor the second object (990) while monitoring the first object (901) by confirming the second object (990) being monitored and the first operation of the second object for outputting notification information based on a user command for monitoring the object.

[0199] Referring to FIG. 9, according to one embodiment, the electronic device (201) can switch to a remote monitoring (tracking) mode when it detects that the distance between the electronic device (201) and the first object (901) has exceeded a specified distance, or when the electronic device (201) has exceeded a first area within a first distance from the first object (901).

[0200] According to one embodiment, the electronic device (201) can control to turn on the sensor of the first external electronic device (911), the sensor of the second external electronic device (913), the sensor of the third external electronic device (915), and / or the sensor of the fourth external electronic device (917) located in a first area (A1) within a first distance from the first object (901) in a remote monitoring mode.

[0201] According to one embodiment, the electronic device (201) can determine the operation of the first object (901) based on sensor values ​​received from the first external electronic device (911), the second external electronic device (913), the third external electronic device (915) and / or the fourth external electronic device (917) located in the first area (A1).

[0202] According to one embodiment, the electronic device (201) can control the sensor of the fifth external electronic device (919), the sensor of the sixth external electronic device (921), the sensor of the seventh external electronic device (923), and / or the sensor of the eighth external electronic device (925) located in the first area (A1) within the first distance from the second object (990) when it is determined that the distance between the electronic device (201) and the second object (990) is outside the specified distance (d1).

[0203] According to one embodiment, the electronic device (201) can check the operation of the second object (990) based on sensor values ​​received from the fifth external electronic device (919), the sixth external electronic device (921), the seventh external electronic device and / or the eighth external electronic device located in the first area (A1).

[0204] According to one embodiment, the electronic device (201) can monitor a first object (901) based on sensor values ​​received from a first external electronic device (911), a second external electronic device (913), a third external electronic device (915) and / or a fourth external electronic device (917), while monitoring a second object (990) based on sensor values ​​received from a fifth external electronic device (919), a sixth external electronic device (921), a seventh external electronic device and / or an eighth external electronic device.

[0205] FIGS. 10a, FIGS. 10b, FIGS. 10c, FIGS. 10d, and FIGS. 10e are drawings for illustrating the provision of a user interface for monitoring an object in an electronic device according to a temporary example.

[0206] FIGS. 11a, FIGS. 11b, FIGS. 11c and FIGS. 11d are drawings for illustrating the provision of a user interface for monitoring an object in an electronic device according to a temporary example.

[0207] FIGS. 10a, FIGS. 10b, FIGS. 10c, FIGS. 10d, and FIGS. 10c, FIGS. 11a, FIGS. 11b, FIGS. 11c, and FIGS. 11d illustrate a user interface that can be provided when the electronic device (201) is a portable communication device (e.g., a smartphone) according to one embodiment. Referring to FIG. 10a, according to one embodiment, the electronic device (201) (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) can confirm the execution of a first application for monitoring an object, and if there is no specified object through the display (260) (e.g., the display (260) of FIG. 2), display a notification message (1001a) indicating that there is no specified object, and display a button (1001b) that can specify an object to monitor.

[0208] According to one embodiment, the electronic device (201) can confirm the selection of the button (701b) based on user input.

[0209] Referring to FIG. 10b, according to one embodiment, an electronic device (201) (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) can display the user's command (1003) through a display (260) when it receives a user's command (1003) for monitoring an object.

[0210] According to one embodiment, the electronic device (201) can analyze a user's command using an AI model to identify a first object to be monitored (tracked) as "a dog named Beau" and identify a first action of the first object for outputting notification information as "something happening to Beau".

[0211] Referring to FIG. 10c, according to one embodiment, an electronic device (201) (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) can acquire video or audio containing a first object, "a dog named Beau," based on data of the electronic device (201) or data of an external electronic device connected to the electronic device.

[0212] According to one embodiment, the electronic device (201) may request the user to confirm whether the first object to be monitored is correct while displaying an image (1005) containing the first object, a dog named "Beau," through a display (260).

[0213] According to one embodiment, the electronic device (201) can display a user command (1003), an image (1005) containing a first object, a dog named "Beau", a button (1007) for starting monitoring of the object, and a button (1009) for adding an object to be monitored through a display (260).

[0214] According to one embodiment, the electronic device (201) can control to turn on at least one sensor of an external electronic device located in a first area within a first distance from the location of the first object, a dog named "Beau," when it confirms the selection of a button (1007) for starting monitoring of an object based on user input.

[0215] According to one embodiment, the electronic device (201) can determine the location of the first object and / or the current operation of the first object based on sensor values ​​received from at least one external electronic device located in a first area within a first distance from the location of the first object, "a dog named Beau".

[0216] Referring to FIG. 10d, according to one embodiment, an electronic device (201) (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) can display a cross-sectional view of a designated space (e.g., a house) where a first object, a dog named "Beau," can move through a display (260) in real time, and display an icon (1013) at the point where the first object, a dog named "Beau," is located in the cross-sectional view, along with a guidance message (1011) indicating that the current location of the first object, a dog named "Beau," is the "living room."

[0217] Referring to FIG. 10e, according to one embodiment, an electronic device (201) (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) may display an icon (1017) representing the first object at the point where the first object "the dog named Beau" is located in the cross-sectional view, along with a guidance message (1015) indicating that the current location of the first object "the dog named Beau" is the "front door" through the display (260) when the first object "the dog named Beau" moves from the refusal to the front door.

[0218] According to one embodiment, the electronic device (201) can provide a user interface (1105) capable of controlling the first object as shown in FIG. 11b when confirming the selection (1017a) of an icon (1017) representing the first object based on user input.

[0219] Referring to FIG. 11a, according to one embodiment, an electronic device (201) (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) can detect the action of a dog named Beau, which is a first object, urinating on the floor, and, upon confirming this action as a first action of the first object for outputting notification information, display a notification information guidance message (1101) through a display (260) to notify the occurrence of the first action of the first object.

[0220] According to one embodiment, the electronic device (201) can provide a user interface (1105) capable of controlling the first object as in FIG. 11b when confirming the selection of a guidance message (1103) based on user input while displaying a guidance message (1101) that indicates the occurrence of a first action of the first object through a display (260).

[0221] Referring to FIG. 11b, according to one embodiment, an electronic device (201) (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) may provide a user interface (1105) that can control a first object through a display (260).

[0222] According to one embodiment, the user interface (1105) may include a first menu (1105a) for checking the video and / or audio of a first object in real time, a second menu (1105b) for outputting the user's voice so that the first object can hear it, a third menu (1105c) for outputting audio selected by the user so that the first object can hear it, a fourth menu (1105d) for controlling a sensor of at least one external electronic device located in a first area within a first distance from the first object based on the user's input, a fifth menu (1105e) for checking the operation of the first object stored after notification information announcing the occurrence of a first operation of the first object is output, and / or a sixth menu (1105f) for stopping the monitoring operation of the first object.

[0223] Referring to FIG. 11c, according to one embodiment, an electronic device (201) (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) can switch to a screen that monitors the first object in real time while displaying a user interface (1105) that can control the first object, when it receives input (1107) from a user to switch to a screen that monitors the first object in real time, as in FIG. 10d or FIG. 10e.

[0224] Referring to FIG. 11d, according to one embodiment, an electronic device (201) (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) may display an icon (1109) representing the first object by determining the location and / or operation of the first object based on sensor values ​​received from at least one external electronic device located in a first area within a first distance from the first object, when the monitoring target is designated as the first object and the second object.

[0225] According to one embodiment, the electronic device (201) can display an icon (1111) representing the second object by determining the location and / or operation of the second object based on a sensor value received from at least one external electronic device located in a first area within a first distance from the second object.

[0226] According to one embodiment, in an electronic device that does not have a display (e.g., AI Glass), user input is received via voice or gesture, and objects such as guidance messages and / or buttons can be provided to the user via voice or vibration instead of screen display.

[0227] FIGS. 12a, FIGS. 12b, FIGS. 12c, FIGS. 12d, FIGS. 12e, FIGS. 12f, FIGS. 12g, and FIGS. 12h are drawings for illustrating the provision of a user interface for monitoring an object in an electronic device according to a temporary example.

[0228] FIGS. 12a, FIGS. 12b, FIGS. 12c, FIGS. 12d, FIGS. 12e, FIGS. 12f, FIGS. 12g, and FIGS. 12h illustrate a user interface that can be provided when, according to one embodiment, the electronic device is a wearable electronic device that can be worn on a part of a user's body (e.g., the wearable electronic device (1500) of FIG. 15, the wearable electronic device (1600) of FIG. 15a to FIG. 16b, and / or the wearable electronic device (1701) of FIG. 17).

[0229] Referring to FIG. 12a, according to one embodiment, a wearable electronic device (1230) (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) can provide (display) a first screen (1201) including a button (1203b) that can specify an object to be monitored, and if there is no specified object through a display (1260) (e.g., the display (260) of FIG. 2), a guidance message (1203a) indicating that there is no specified object is displayed, and a button (1203b) that can specify an object to be monitored.

[0230] According to one embodiment, the electronic device (201) can confirm the selection of the button (1203b) based on user input.

[0231] Referring to FIG. 12b, according to one embodiment, a wearable electronic device (1230) (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) may provide (display) a second screen (1205) containing the user's command (1205a) through a display (1260) when it receives a user's command for monitoring an object.

[0232] According to one embodiment, the wearable electronic device (1230) can analyze a user's command using an AI model to identify a first object being monitored (tracked) as "a dog named Beau" and identify a first action of the first object for outputting notification information as "something happening to Beau".

[0233] Referring to FIG. 12c, according to one embodiment, a wearable electronic device (1230) (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) can acquire video or audio containing a first object, "a dog named Beau," based on data of the wearable electronic device (1230) or data of an external electronic device connected to the wearable electronic device (1230).

[0234] According to one embodiment, the wearable electronic device (1230) may provide (display) a third screen (1207) that asks the user to confirm whether the first object to be monitored is correct while displaying an image (1207a) containing the first object, "a dog named Beau," through a display (1260).

[0235] According to one embodiment, the electronic device (201) can provide (display) a third screen (1207) including an image (1207a) containing a first object, a dog named "Beau," a button (1207b) for starting monitoring of the object, and a button (1207c) for adding an object to be monitored, through a display (1260).

[0236] According to one embodiment, the wearable electronic device (1230) can control to turn on at least one sensor of an external electronic device located in a first area within a first distance from the location of the first object, a dog named "Beau," when it confirms the selection of a button (1207b) for starting monitoring of an object based on user input.

[0237] According to one embodiment, the wearable electronic device (1230) can determine the location of the first object and / or the operation of the first object based on sensor values ​​received from at least one external electronic device located in a first area within a first distance from the location of the first object, "a dog named Beau."

[0238] Referring to FIG. 12d, according to one embodiment, a wearable electronic device (1230) (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) may provide a fourth screen (1209) that displays a cross-sectional view of a designated space (e.g., a house) where a first object, a dog named "Beau," can move through a display (1260) in real time, and displays an icon (1209a) at the point where the first object, a dog named "Beau," is located in the cross-sectional view, along with a guidance message (1209b) indicating that the current location of the first object, a dog named "Beau," is the "living room."

[0239] Referring to FIG. 12e, according to one embodiment, a wearable electronic device (1230) (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) may provide a fifth screen (1211) that displays an icon (12011b) representing the first object at the point where the first object "the dog named Beau" is located in a cross-sectional view, along with a guidance message (12011a) indicating that the current location of the first object "the dog named Beau" is the "front door" through a display (1260) when the first object "the dog named Beau" moves from the refusal to the front door.

[0240] According to one embodiment, the wearable electronic device (201) can provide a user interface (1215) that can control the first object as in FIG. 12g when confirming the selection (12011c) of an icon (12011b) representing the first object based on user input.

[0241] Referring to FIG. 12f, according to one embodiment, a wearable electronic device (1230) (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) can detect the action of a dog named Beau, which is a first object, urinating on the floor, and, upon confirming this action as a first action of the first object for outputting notification information, display a notification information guidance message (1213) through a display (1260) to notify the occurrence of the first action of the first object.

[0242] Referring to FIG. 12g, according to one embodiment, a wearable electronic device (1230) (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) may display a sixth screen (1213) including a user interface (1215) that can control a first object through a display (1260).

[0243] According to one embodiment, the user interface (1215) may include a first menu (1215a) for checking the video and / or audio of a first object in real time, a second menu (1215b) for outputting the user's voice so that the first object can hear it, a third menu (1215c) for outputting audio selected by the user so that the first object can hear it, a fourth menu (1215d) for controlling a sensor of at least one external electronic device located in a first area within a first distance from the first object based on the user's input, a fifth menu (1215e) for checking the operation of the first object stored after notification information announcing the occurrence of a first operation of the first object is output, and / or a sixth menu (1215f) for stopping the monitoring operation of the first object.

[0244] Referring to FIG. 12h, according to one embodiment, a wearable electronic device (1230) (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) may provide (display) a seventh screen (1215) that displays an icon (1215a) representing the first object through a display (1260) when the location and / or operation of the first object is confirmed based on sensor values ​​received from at least one external electronic device located in a first area within a first distance from the first object, when the monitoring target is designated as a first object and a second object.

[0245] According to one embodiment, the wearable electronic device (1230) can provide (display) a seventh screen (1215) that displays an icon (1215b) representing the second object through a display (1260) when the location and / or operation of the second object is confirmed based on a sensor value received from at least one external electronic device located in a first area within a first distance from the second object.

[0246] An electronic device according to one embodiment (101 of FIG. 1; 201 of FIG. 2; 1500 of FIG. 15; 1600 of FIG. 16a to 16b; 1701 of FIG. 17) may include a communication circuit (190 of FIG. 1; 1635 of FIG. 16), a user interface (150 of FIG. 1), a display (160 of FIG. 1; 260 of FIG. 2; 1501 of FIG. 15; 1621 of FIG. 16; 1720 of FIG. 17), at least one processor (120 of FIG. 1; 220 of FIG. 2; 1710 of FIG. 17), and a memory for storing instructions (130 of FIG. 1; 230 of FIG. 2a; 1715 of FIG. 17). When the commands according to one embodiment are executed individually or collectively by the at least one processor, the electronic device can identify a first object to be monitored and a first operation of the first object based on a user command obtained through the user interface. When the commands according to one embodiment are executed individually or collectively by the at least one processor, the electronic device can be controlled to turn on a sensor of at least one external device located in a first area within a first distance from the first object when the distance between the electronic device and the first object exceeds a specified distance. When the commands according to one embodiment are executed individually or collectively by the at least one processor, the electronic device can receive a sensor value obtained from the at least one external electronic device through the sensor of the at least one external device via the communication circuit. When the above commands according to one embodiment are executed individually or collectively by the at least one processor, the electronic device can output notification information indicating the occurrence of the first operation of the first object when it identifies the current operation of the first object as the first operation of the first object based on the sensor value.

[0247] The instructions according to one embodiment, when executed individually or collectively by the at least one processor, can control the electronic device to switch a sensor of at least one external electronic device located in a second area within a second distance greater than the first distance from the first object to a standby state. The instructions according to one embodiment, when executed individually or collectively by the at least one processor, can control the electronic device to turn off a sensor of at least one external electronic device located in a third area within a third distance greater than the second distance from the first object.

[0248] When the instructions according to one embodiment are executed individually or collectively by the at least one processor, the electronic device can identify one or more first external electronic devices including the same sensor among the at least one external electronic device included in the first area, and identify the first external electronic device among the one or more first external electronic devices based on the location of the first object.

[0249] According to one embodiment, when the above commands are executed individually or collectively by the at least one processor, the electronic device is controlled to turn on a plurality of sensors of a plurality of external electronic devices registered on a server with the same account as the electronic device when the distance between the electronic device and the first object exceeds the specified distance, and the plurality of external electronic devices may include at least one external electronic device located in the first area, at least one external electronic device located in the second area, and at least one external electronic device located in the third area. According to one embodiment, when the above commands are executed individually or collectively by the at least one processor, the electronic device is controlled to determine the location of the first object based on sensor values ​​obtained through the plurality of sensors of the plurality of external electronic devices. When the instructions according to one embodiment are executed individually or collectively by the at least one processor, the electronic device can be controlled to turn on the sensor of the at least one external electronic device included in the first area based on the location of the first object, switch the sensor of the at least one external electronic device included in the second area to a standby state, and turn off the sensor of the at least one external electronic device included in the third area.

[0250] When the instructions according to one embodiment are executed individually or collectively by the at least one processor, the electronic device can detect the movement of the first object based on the sensor of the at least one external electronic device included in the first region.

[0251] The instructions according to one embodiment, when executed individually or collectively by the at least one processor, can control the electronic device to turn on the sensor of at least one external electronic device included in the first area, switch the sensor of at least one external electronic device included in the second area to a standby state, and turn off the sensor of at least one external electronic device included in the third area through a server configured to manage a plurality of external electronic devices registered under the same account as the electronic device.

[0252] When the instructions according to one embodiment are executed individually or collectively by the at least one processor, the electronic device stores information related to the plurality of external electronic devices received through a server configured to manage the plurality of external electronic devices registered under the same account as the electronic device in the memory, and the information related to the plurality of external electronic devices may include at least one of type information of the external electronic device, function information of the external electronic device, location information of the external electronic device, or sensor information of the external electronic device.

[0253] FIG. 13 is a flowchart illustrating an operation to identify an object in an electronic device according to one embodiment. The operations to identify an object may include operations 1301 through 1315. In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, at least two operations may be performed in parallel, or other operations may be added.

[0254] In operation 1301, an electronic device (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) can receive a user's command to monitor an object.

[0255] According to one embodiment, the electronic device can recognize a user's voice command or user input for setting a command as a user command instructing the performance of monitoring of an object.

[0256] In operation 1303, an electronic device (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) can identify a first object to be monitored and a first operation of the first object for outputting notification information based on a user's command.

[0257] According to one embodiment, when an electronic device confirms the reception of a user's command for performing monitoring of an object, it can use an AI model to analyze the user's command and confirm a first object to be monitored and a first operation of the first object for outputting notification information.

[0258] According to one embodiment, the electronic device may request the user to input additional information for performing monitoring of an object if, as a result of analyzing a user's command using an AI model, only the first object or the first operation is confirmed.

[0259] In operation 1305, an electronic device (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) can receive information from a server regarding multiple external electronic devices registered on the server with the same account as the electronic device.

[0260] According to one embodiment, information related to a plurality of external electronic devices may include at least one of type information of the external electronic device, function information of the external electronic device, location information of the external electronic device, or sensor information of the external electronic device.

[0261] According to one embodiment, the electronic device may store information related to a plurality of external electronic devices received from a server in the memory of the electronic device (e.g., memory (230) of FIG. 2).

[0262] In operation 1307, an electronic device (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) can determine the location of the first object through a sensor of the electronic device.

[0263] According to one embodiment, if the electronic device confirms that the electronic device and at least one external electronic device are included in a first area within a first distance from the first object without the distance between the electronic device and the first object exceeding a specified distance, the location of the first object can be confirmed through the sensor of the electronic device (e.g., sensor (276) of FIG. 2).

[0264] According to one embodiment, the electronic device can identify at least one external electronic device located in a first area within a first distance from a first object using an AI model.

[0265] In operation 1309, an electronic device (e.g., electronic device (101) of FIG. 1 and / or electronic device (201) of FIG. 2) can control to turn on at least one sensor of an external electronic device located in a first area within a first distance from a first object, to switch at least one sensor of an external electronic device located in a second area within a second distance from the first object to a standby state, and to turn off at least one sensor of an external electronic device located in a third area within a second distance from the first object.

[0266] According to one embodiment, the electronic device can control to turn on a sensor of at least one external electronic device located in a first area within a first distance from a first object, to turn off a sensor of at least one external electronic device located in a second area within a second distance from the first object without a standby state, and to turn off a sensor of at least one external electronic device located in a third area within a second distance from the first object.

[0267] According to one embodiment, the electronic device can turn on at least one external electronic device located in a first region.

[0268] According to one embodiment, when an electronic device identifies at least one first external electronic device including a sensor of the same type among at least one external electronic device located in a first region, it can turn on the sensor of the selected first external electronic device using an AI model based on priority conditions.

[0269] According to one embodiment, priority conditions may include the location of the first object and / or the performance of an external electronic device.

[0270] According to one embodiment, the electronic device can use an AI model to check the turn-on of the sensor of the first external electronic device located closest to the first object among at least one first external electronic device located in a first area based on the location of the first object.

[0271] According to one embodiment, the electronic device can control to turn on a first external electronic device selected based on performance and distance from a first object among at least one first external electronic device located in a first area using an AI model.

[0272] According to one embodiment, the electronic device can control at least one sensor of an external electronic device located in a second area within a second distance greater than a first distance from a first object to switch to a standby state.

[0273] According to one embodiment, the electronic device can control the sensor of at least one external electronic device located in a second area within a second distance from a first object to turn off without a standby state.

[0274] According to one embodiment, the second region may represent a region that does not overlap with the first region.

[0275] According to one embodiment, the standby state may represent a state that minimizes power consumption, such as a power saving mode.

[0276] According to one embodiment, the electronic device can control to turn off a sensor of at least one external electronic device located in a third area within a third distance greater than a second distance from a first object. According to one embodiment, the third area may represent an area that does not overlap with the first area and the second area.

[0277] In operation 1311, an electronic device (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) may receive a sensor value obtained through a sensor of at least one external electronic device from at least one external electronic device located in a first area.

[0278] According to one embodiment, the electronic device can identify an image including a first object and / or audio of a first object from a sensor value.

[0279] In operation 1313, an electronic device (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) can determine the current operation of the first object as the first operation of the first object based on the sensor value.

[0280] According to one embodiment, the electronic device can determine the current operation of the first object based on sensor values ​​and output the current operation of the first object in real time through the display of the electronic device (e.g., the display (260) of FIG. 2) or the speaker of the electronic device.

[0281] According to one embodiment, the electronic device can determine the current operation of a first object based on a sensor value and determine the current operation of the first object as a first operation of the first object for outputting notification information.

[0282] In operation 1315, the electronic device (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) may output notification information indicating the occurrence of a first operation of the first object.

[0283] According to one embodiment, the electronic device may output notification information indicating the occurrence of a first action of a first object through haptic feedback (vibration output) using a display, a speaker and / or an actuator.

[0284] According to one embodiment, if the electronic device determines that the distance between the electronic device and the first object is outside a specified distance or determines that the electronic device (201) is not included in the first area, it can determine the position, movement, and / or operation of the first object based on sensor values ​​received from at least one external electronic device located in the first area.

[0285] FIG. 14 is a flowchart illustrating an operation for identifying an object in an electronic device according to one embodiment. The operations for identifying an object may include operations 1401 through 1415. In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, at least two operations may be performed in parallel, or other operations may be added.

[0286] In operation 1401, an electronic device (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) can receive a user's command to monitor an object.

[0287] According to one embodiment, the electronic device can recognize a user's voice command or user input for setting a command as a user command instructing the performance of monitoring of an object.

[0288] In operation 1403, an electronic device (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) can identify a first object to be monitored and a first operation of the first object for outputting notification information based on a user's command.

[0289] According to one embodiment, when an electronic device confirms the reception of a user's command for performing monitoring of an object, it can use an AI model to analyze the user's command and confirm a first object to be monitored and a first operation of the first object for outputting notification information.

[0290] According to one embodiment, the electronic device may request the user to input additional information for performing monitoring of an object if, as a result of analyzing a user's command using an AI model, only the first object or the first operation is confirmed.

[0291] In operation 1405, an electronic device (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) may receive information from a server regarding multiple external electronic devices registered on the server with the same account as the electronic device.

[0292] According to one embodiment, information related to a plurality of external electronic devices may include at least one of type information of the external electronic device, function information of the external electronic device, location information of the external electronic device, or sensor information of the external electronic device.

[0293] According to one embodiment, the electronic device may store information related to a plurality of external electronic devices received from a server in the memory of the electronic device (e.g., memory (230) of FIG. 2).

[0294] In operation 1407, an electronic device (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) can turn on a plurality of sensors of a plurality of external electronic devices to determine the location of the first object.

[0295] According to one embodiment, when an electronic device needs to perform a monitoring operation on a first object when the distance between the electronic device and the first object is outside a specified distance, or is not included in a first area within a first distance from the first object, or is outside a specified space in which the first object can move, the electronic device (201) may temporarily turn on a plurality of sensors of a plurality of external electronic devices registered on a first server with the same account as the electronic device, and can determine the location of the first object based on the plurality of sensors of the plurality of external electronic devices.

[0296] According to one embodiment, the processor (220) can determine that the distance between the electronic device (201) and the first object has exceeded a specified distance if, due to the movement of the electronic device (201) or the first object, it is not possible to obtain a sensor value that can monitor the first object through the sensor (246) or to monitor (confirm) the current operation of the first object based on the sensor value obtained through the sensor (246).

[0297] In operation 1409, an electronic device (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) may control to turn on a sensor of at least one external electronic device located in a first area within a first distance from a first object, to switch to a standby state a sensor of at least one external electronic device located in a second area within a second distance from the first object, and to turn off a sensor of at least one external electronic device located in a third area within a third distance from the first object. According to one embodiment, when the electronic device confirms the location of the first object based on a plurality of sensors of a plurality of external electronic devices, it may control to turn on a sensor of at least one external electronic device located in a first area within a first distance from the first object.

[0298] According to one embodiment, when an electronic device confirms the location of a first object based on a plurality of sensors of a plurality of external electronic devices, it can control at least one sensor of an external electronic device located in a second area within a second distance from the first object to switch to a standby state.

[0299] According to one embodiment, when an electronic device confirms the location of a first object based on a plurality of sensors of a plurality of external electronic devices, it can control at least one sensor of an external electronic device located in a second area within a second distance from the first object to turn off without a standby state.

[0300] According to one embodiment, the second region may represent a region that does not overlap with the first region.

[0301] According to one embodiment, when an electronic device determines the location of a first object based on a plurality of sensors of a plurality of external electronic devices, it can control to turn off at least one sensor of an external electronic device located in a third area within a third distance greater than a second distance from the first object. According to one embodiment, the third area may represent an area that does not overlap with the first area and the second area.

[0302] In operation 1411, an electronic device (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) may receive a sensor value obtained through a sensor of at least one external electronic device from at least one external electronic device located in a first area.

[0303] According to one embodiment, the electronic device can identify an image including a first object and / or audio of a first object from a sensor value.

[0304] In operation 1413, an electronic device (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) can determine the current operation of the first object as the first operation of the first object based on the sensor value.

[0305] According to one embodiment, the electronic device can determine the position, movement, and / or operation of a first object based on sensor values ​​received from at least one external electronic device located in a first area.

[0306] According to one embodiment, the electronic device can determine the current operation of a first object based on a sensor value and output the determined current operation of the first object in real time through the display of the electronic device (e.g., the display (260) of FIG. 2) or the speaker of the electronic device.

[0307] According to one embodiment, the electronic device can determine the current operation of a first object based on a sensor value and determine the determined current operation of the first object as a first operation of the first object for outputting notification information.

[0308] In operation 1415, the electronic device (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) may output notification information indicating the occurrence of a first operation of the first object.

[0309] According to one embodiment, the electronic device may output notification information indicating the occurrence of a first action of a first object through haptic feedback (vibration output) using a display, a speaker and / or an actuator.

[0310] A method for verifying the operation of an object in an electronic device according to one embodiment (101 of FIG. 1; 201 of FIG. 2; 1500 of FIG. 15; 1600 of FIG. 16a to 16b; 1701 of FIG. 17) may include an operation of verifying a first object to be monitored and a first operation of the first object based on a user command obtained through the user interface of the electronic device. The method according to one embodiment may include an operation of controlling to turn on a sensor of at least one external device located in a first area within a first distance from the first object when the distance between the electronic device and the first object exceeds a specified distance. The method according to one embodiment may include an operation of receiving a sensor value obtained from the at least one external electronic device through the sensor of the at least one external device via the communication circuit of the electronic device. The method according to one embodiment may include an operation of outputting notification information indicating the occurrence of the first operation of the first object when the current operation of the first object is identified as the first operation of the first object based on the sensor value.

[0311] The method according to one embodiment may include an operation of controlling a sensor of at least one external electronic device located in a second area within a second distance greater than the first distance from the first object to switch to a standby state. The method according to one embodiment may include an operation of controlling a sensor of at least one external electronic device located in a third area within a third distance greater than the second distance from the first object to turn off.

[0312] The method according to one embodiment may include, when identifying one or more first external electronic devices including the same sensor among the at least one external electronic device included in the first region, an operation of identifying the first external electronic device among the one or more first external electronic devices based on the location of the first object.

[0313] The method according to one embodiment may include an operation in which, when the distance between the electronic device and the first object deviates from the specified distance, a plurality of sensors of a plurality of external electronic devices registered on a server with the same account as the electronic device are turned on, and the plurality of external electronic devices include at least one external electronic device located in the first area, at least one external electronic device located in the second area, and at least one external electronic device located in the third area. The method according to one embodiment may include an operation in which the location of the first object is determined based on sensor values ​​obtained through the plurality of sensors of the plurality of external electronic devices. The method according to one embodiment may include an operation in which, based on the location of the first object, the sensor of the at least one external electronic device included in the first area is turned on, the sensor of the at least one external electronic device included in the second area is switched to a standby state, and the sensor of the at least one external electronic device included in the third area is turned off.

[0314] The method according to one embodiment may include an operation of confirming the movement of the first object based on a sensor of the at least one external electronic device included in the first region.

[0315] The method according to one embodiment may include the operation of turning on a sensor of at least one external electronic device included in the first area, switching a sensor of at least one external electronic device included in the second area to a standby state, and turning off a sensor of at least one external electronic device included in the third area through a server configured to manage a plurality of external electronic devices registered with the same account as the electronic device.

[0316] The method according to one embodiment may include storing information related to a plurality of external electronic devices received through a server configured to manage a plurality of external electronic devices registered with the same account as the electronic device in the memory of the electronic device, and the information related to the plurality of external electronic devices may include at least one of type information of the external electronic device, function information of the external electronic device, location information of the external electronic device, or sensor information of the external electronic device.

[0317] The method according to one embodiment stores information related to a plurality of external electronic devices received through a server that manages a plurality of external electronic devices registered under the same account as the electronic device in the memory, and the information related to the plurality of external electronic devices may include at least one of type information of the external electronic device, function information of the external electronic device, location information of the external electronic device, or sensor information included in the external electronic device.

[0318] FIG. 15 is a perspective view for explaining the internal configuration of a wearable electronic device (1500) according to one embodiment.

[0319] A wearable electronic device (1500) according to one embodiment in FIG. 15 can be implemented in the same way as the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2 and can perform the same operation.

[0320] According to one embodiment, the wearable electronic device (1500) of FIG. 15 (or the wearable electronic device (1600) of FIG. 16 described below) may be substantially identical to the electronic device (101) of FIG. 1 and may be implemented to be wearable on a user's body. In one embodiment, each of the external electronic devices (102, 104) of FIG. 1 may be the same or a different type of device as the electronic device (101) or the wearable electronic device (1500, 1600). According to one embodiment, all or part of the operations performed on the electronic device (101) or the wearable electronic device (1500, 1600) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, when an electronic device (101) or a wearable electronic device (1500, 1600) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) or the wearable electronic device (1500, 1600) may request one or more external electronic devices to perform at least a part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the request may perform at least a part of the requested function or service, or additional functions or services related to the request, and transmit the result of the execution to the electronic device (101) or the wearable electronic device (1500, 1600). The electronic device (101) or the wearable electronic device (1500, 1600) may provide the result as is or additionally processed as at least part of the response to the request.For example, an external electronic device (102) renders content data executed in an application and transmits it to an electronic device (101) or a wearable electronic device (1500, 1600), and the electronic device (101) or the wearable electronic device (1500, 1600) that receives the data can output the content data to a display module. When the electronic device (101) or the wearable electronic device (1500, 1600) detects user movement through an inertial measurement unit sensor, etc., the processor of the electronic device (101) or the wearable electronic device (1500, 1600) (e.g., the processor (120) of FIG. 1) can correct the rendering data received from the external electronic device (102) based on the movement information and output it to a display module. Alternatively, it can transmit the movement information to the external electronic device (102) and request rendering so that the screen data is updated accordingly. According to various embodiments, the external electronic device (102) may be a device of various forms, such as a case device capable of storing and charging the electronic device (101).

[0321] Referring to FIG. 15, a wearable electronic device (1500) according to one embodiment of the present disclosure may include at least one of a light output module (1511), a display member (1501), and a camera module (1550). According to one embodiment, the light output module (1511) may include a light source capable of outputting an image and a lens that guides the image to the display member (1501). According to one embodiment, the light output module (1511) may include at least one of a liquid crystal display (LCD), a digital mirror device (DMD), a liquid crystal on silicon (LCoS), an organic light emitting diode (OLED), or a micro light emitting diode (micro LED).

[0322] According to one embodiment, the display member (1501) may include a light waveguide. According to one embodiment, an image output by a light output module (1511) incident on one end of the light waveguide may propagate within the light waveguide and be provided to a user. For example, the display member (1501) may be understood as an optical system that guides or focuses an image output by the light output module (1511) to the user's eyes along a designated path.

[0323] According to one embodiment, the display member (1501) may include at least one of a diffractive optical element (DOE), a holographic optical element (HOE), or a reflective element (e.g., a reflective mirror) provided in an optical waveguide. For example, the display member (1501) may guide an output image of an optical output module (1511) to the user's eye by including at least one diffractive optical element, a holographic optical element, or a reflective element, and / or by including an optical waveguide.

[0324] According to one embodiment, the camera module (1550) can capture still images and / or videos. According to one embodiment, the camera module (1550) can be placed within a lens frame and around a display member (1501).

[0325] According to one embodiment, the first camera module (1551) can capture and / or recognize the trajectory of the user's eye (e.g., pupil, iris) or gaze. According to one embodiment, the first camera module (1551) can periodically or non-periodically transmit information related to the trajectory of the user's eye or gaze (e.g., trajectory information) to a processor (e.g., processor (120) of FIG. 1).

[0326] According to one embodiment, the second camera module (1553) can capture an external image.

[0327] According to one embodiment, the third camera module (1555) may be used for hand detection and tracking and user gesture (e.g., hand movements) recognition. According to one embodiment of the present disclosure, the third camera module (1555) may be used for 3 degrees of freedom (3DoF), 6DoF head tracking, location (space, environment) recognition, and / or movement recognition. According to one embodiment, the second camera module (1553) may be used for hand detection and tracking and user gesture recognition. According to one embodiment, at least one of the first camera module (1551) to the third camera module (1555) may be replaced with a sensor module (e.g., LiDAR sensor). For example, the sensor module may include at least one of a vertical cavity surface emitting laser (VCSEL), an infrared sensor, and / or a photodiode.

[0328] According to one embodiment, the wearable electronic device (1500) may include a pair of display members (1501) arranged side by side on one side of each other. For example, a user may wear the wearable electronic device (1500) on their face, and while worn on the user's face, the display members (1501) may be positioned to correspond to either of the user's eyes. In one embodiment, when including a pair of display members (1501), the wearable electronic device (1500) may provide visual information to the user through any one of the display members (1501) and / or through each of the display members (1501).

[0329] According to one embodiment, the wearable electronic device (1500) may include at least one wearable member (1502a, 1502b) extending from or rotatably coupled to the display member (1501). In the illustrated embodiment, the wearable member (1502a, 1502b) may be exemplified as a structure rotatably coupled (or connected) to the display member (1501) by a hinge structure (H). For example, the wearable member (1502a, 15015b) may be in an overlapping or folded position with the display member (1501), in which case the user may easily carry or store the wearable electronic device (1500). In one embodiment, the user can easily wear the wearable electronic device (1500) on their face at a position where the wearing member (1502a, 1502b) is rotated by a specified angle (e.g., approximately 90 degrees) from a position overlapping with the display member (1501). For example, the wearable electronic device (1500) can be stably worn by supporting the display member (1501) on the user's face and the wearing member (1502a, 15015b) on the user's head (e.g., ears).

[0330] According to one embodiment, by including a pair of wearable members (1502a, 1502b), the wearable electronic device (1500) can be easily worn on a user's head (or face). In one embodiment, at least some of the components of FIG. 1 (e.g., processor (120), memory (130), acoustic output module (155), battery (189), communication module (190), and / or connection terminal (178)) may be placed on the wearable members (1502a, 1502b). In one embodiment, the wearable electronic device (1500) may be worn on a user's body (e.g., head) with a display member (1501) aligned with one of the user's eyes. For example, when a user wears the wearable electronic device (1500), a lens frame (LF) that supports or secures the display member (1501) may be configured to be positioned facing the user's face.

[0331] According to one embodiment, the wearable members (1502a, 1502b) may be positioned on the side of the user's head when the user wears the wearable electronic device (1500). In one embodiment, an output device for outputting auditory information (e.g., the acoustic output module (155) of FIG. 1 or the speaker module (431) of FIG. 5) may be embedded in at least one of these wearable members (1502a, 1502b). For example, the acoustic output module (155) of FIG. 1 or the speaker module (431) of FIG. 5 may be positioned or aligned adjacent to the user's ear when the user wears the wearable electronic device (1500). In one embodiment, such an arrangement structure of the wearable members (1502a, 1502b), the acoustic output module (155), or the speaker module (431) can provide an environment that simplifies the acoustic path implemented within the wearable electronic device (1500) and / or the wearable members (1502a, 1502b).

[0332] According to one embodiment, if the wearable electronic device (1500) is an augmented reality glass that does not display a virtual object to a user wearing the wearable electronic device (1500), it may include a camera (including at least one of an RGB or IR camera) for confirming external objects or user gestures, a camera (IR) for recognizing the user's gaze, a microphone, and / or a speaker.

[0333] According to one embodiment, if the wearable electronic device (1500) is an augmented reality glass that displays a virtual object to a user wearing the wearable electronic device (1500), it may include a camera for checking external objects (including at least one of an RGB or IR camera), a camera (IR) for recognizing the user's gaze, a microphone, a speaker, and / or a display for displaying a virtual object (a display placed in both eyes or in one eye).

[0334] According to one embodiment, if the wearable electronic device (1500) is a VST (video see-through), it may include a camera for identifying external objects (including at least one of an RGB or IR camera), a camera (IR) for recognizing the gaze of a user wearing the VST, a microphone, a speaker, a display for displaying virtual objects (a display placed on both eyes), and / or an external display (e.g., an external display of VisionPro).

[0335] It may include a camera for verifying gestures of an external object or user (including at least one of an RGB or IR camera), a camera for recognizing the user's gaze (IR), a microphone, and / or a speaker.

[0336] FIGS. 16a and FIGS. 16b are drawings showing the front and rear of a wearable electronic device according to one embodiment.

[0337] The wearable electronic device (1600) according to one embodiment in FIG. 16a and FIG. 16b is implemented in the same way as the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2 and can perform the same operation.

[0338] Referring to FIG. 16a and FIG. 16b, in one embodiment, camera modules (1611, 16115, 1613, 1614, 1615, 1616) and / or a depth sensor (1617) for acquiring information related to the surrounding environment of the wearable electronic device (1600) may be disposed on a first surface (1610) of the housing (e.g., the front of the wearable electronic device (1600)).

[0339] In one embodiment, camera modules (1611, 1612) can acquire images related to the surrounding environment of a wearable electronic device (1600) (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2).

[0340] In one embodiment, camera modules (1613, 1614, 1615, 1616) can acquire images while the wearable electronic device (1600) is worn by a user. The camera modules (1613, 1614, 1615, 1616) can be used for hand detection, tracking, and user gesture (e.g., hand movements) recognition. The camera modules (1613, 1614, 1615, 1616) can be used for 3DoF, 6DoF head tracking, location (space, environment) recognition, and / or movement recognition. In one embodiment, camera modules (1611, 1612) may be used for hand detection and tracking and user gestures.

[0341] In one embodiment, the depth sensor (1617) may be configured to transmit a signal and receive a signal reflected from a subject, and may be used for determining the distance to an object, such as time of flight (TOF). In place of or additionally to the depth sensor (1617), camera modules (1613, 1614, 1615, 1616) may determine the distance to an object.

[0342] According to one embodiment, a face recognition camera module (1625, 1626) and / or a display (1621) (and / or a lens) may be disposed on a second surface (1620) of the housing (e.g., the rear of the wearable electronic device (1600)).

[0343] In one embodiment, a face recognition camera module (1625, 1626) adjacent to the display is used for recognizing the user's face or can recognize and / or track both of the user's eyes.

[0344] In one embodiment, the display (1621) (and / or lens) may be disposed on a second surface (1620) of the wearable electronic device (1600). In one embodiment, the wearable electronic device (1600) may not include camera modules (1615, 1616) among a plurality of camera modules (1613, 1614, 1615, 1616).

[0345] As described above, according to one embodiment, the wearable electronic device (1600) may have a form factor for being worn on a user's head. The wearable electronic device (1600) may further include a strap and / or a wearing member for being secured on a part of the user's body. The wearable electronic device (1600) may provide a user experience based on augmented reality, virtual reality, and / or mixed reality while being worn on the user's head.

[0346] FIG. 17 is a block diagram of a wearable device according to one embodiment.

[0347] A wearable electronic device (1500) according to one embodiment in FIG. 17 can be implemented in the same way as the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2 and can perform the same operation.

[0348] Referring to FIG. 17, a wearable device (1701) according to one embodiment (e.g., electronic device 101 of FIG. 1 and / or electronic device (201) of FIG. 2) may include at least one of a processor (1710), memory (1715), display (1720), camera (1725), sensor (1730), or communication circuit (1735). The processor (1710), memory (1715), display (1720), camera (1725), sensor (1730), and communication circuit (1735) may be electrically and / or operably coupled with each other by an electronic component such as a communication bus (1702). The type and / or number of hardware components included in the wearable device (1701) are not limited to those shown in FIG. 17. For example, the wearable device (1701) may include only some of the hardware components illustrated in FIG. 17. The elements within the memory described below (e.g., layers and / or modules) may be in a logically separated state. The elements within the memory (1715) may be included within a hardware component that is separate from the memory (1715). An operation performed by the processor (1710) using each of the elements within the memory (1715) is one embodiment, and the processor (1710) may perform a different operation different from the above operation through at least one of the elements within the memory (1715).

[0349] A processor (1710) of a wearable device (1701) according to one embodiment may include a hardware component for processing data based on one or more instructions. The hardware component for processing data may include, for example, an arithmetic and logic unit (ALU), a field programmable gate array (FPGA), and / or a central processing unit (CPU). The number of processors (1710) may be one or more. For example, the processor (1710) may have the structure of a multi-core processor such as a dual core, a quad core, or a hexa core.

[0350] The processor (1710) of the wearable device (1701) according to one embodiment can perform the same function as the processor (220) of FIG. 2.

[0351] A memory (1715) of a wearable device (401) according to one embodiment may include a hardware component for storing data and / or instructions that are input and / or output to a processor (1710). The memory (1715) may include, for example, volatile memory such as random-access memory (RAM) and / or non-volatile memory such as read-only memory (ROM). Volatile memory may include, for example, at least one of dynamic RAM (DRAM), static RAM (SRAM), cache RAM, and pseudo SRAM (PSRAM). Non-volatile memory may include, for example, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, hard disk, compact disk, and embedded multi-media card (eMMC).

[0352] In one embodiment, a display (1720) of a wearable device (1701) can output visualized information to a user of the wearable device (1701). For example, the display (1720) can be controlled by a processor (1710) including a circuit such as a GPU (graphic processing unit) to output visualized information to a user. The display (1720) may include a flat panel display (FPD) and / or electronic paper. The FPD may include a liquid crystal display (LCD), a plasma display panel (PDP), and / or one or more light emitting diodes (LEDs). The LED may include an organic LED (OLED).

[0353] In one embodiment, the camera (1725) of the wearable device (1701) may include one or more light sensors (e.g., a charged coupled device (CCD) sensor, a complementary metal oxide semiconductor (CMOS) sensor) that generate an electrical signal indicating the color and / or brightness of light. The plurality of light sensors included in the camera (1725) may be arranged in the form of a two-dimensional grid (2 dimensional array). The camera (1725) may acquire the electrical signals of each of the plurality of light sensors substantially simultaneously to generate two-dimensional frame data corresponding to the light reaching the light sensors of the two-dimensional grid. For example, photo data captured using the camera (1725) may mean one (a) two-dimensional frame data acquired from the camera (1725). For example, video data captured using the camera (1725) may mean a sequence of multiple two-dimensional frame data acquired from the camera (1725) along a frame rate. The camera (1725) may further include a flash light for outputting light in the direction in which the camera (1725) receives light.

[0354] According to one embodiment, the wearable device (1701) may include a plurality of cameras positioned toward different directions as an example of a camera (1725). Among the plurality of cameras, the first camera may be referred to as a motion recognition camera (e.g., motion recognition camera (260-2, 260-3) of FIG. 2b), and the second camera may be referred to as an eye-tracking camera (e.g., eye-tracking camera (260-1) of FIG. 2b). The wearable device (1701) may identify the position, shape, and / or gesture of a hand using an image acquired using the first camera. The wearable device (1701) may identify the direction of gaze of a user wearing the wearable device (1701) using an image acquired using the second camera. For example, the direction in which the first camera faces and the direction in which the second camera faces may be opposite.

[0355] According to one embodiment, a sensor (1730) of a wearable device (1701) can generate electrical information that can be processed by a processor (1710) and / or memory (1715) of the wearable device (1701) from non-electronic information associated with the wearable device (1701). The information may be referred to as sensor data. The sensor (1730) may include a global positioning system (GPS) sensor for detecting the geographic location of the wearable device (1701), an image sensor, an illuminance sensor and / or a time-of-flight (ToF) sensor, and an inertial measurement unit (IMU) for detecting physical motion of the wearable device (1701).

[0356] In one embodiment, the communication circuit (1735) of the wearable device (1701) may include hardware components to support the transmission and / or reception of electrical signals between the wearable device (1701) and an external electronic device. The communication circuit (1735) may include, for example, at least one of a modem, an antenna, and an optic / electronic converter. The communication circuit (1735) may support the transmission and / or reception of electrical signals based on various types of protocols such as Ethernet, LAN (local area network), WAN (wide area network), WiFi (wireless fidelity), Bluetooth, BLE (Bluetooth low energy), ZigBee, LTE (long term evolution), 5G NR (new radio) and / or 6G.

[0357] According to one embodiment, within the memory (1715) of the wearable device (1701), one or more instructions (or commands) representing operations and / or operations to be performed on data by the processor (1710) of the wearable device (1701) may be stored. A set of one or more instructions may be referred to as firmware, an operating system, a process, a routine, a sub-routine, and / or an application. For example, the wearable device (1701), and / or the processor (1710), may perform at least one of the operations of FIG. 6 or FIG. 11 when a set of a plurality of instructions distributed in the form of an operating system, firmware, a driver, and / or an application is executed. In the following, the statement that an application is installed in a wearable device (1701) may mean that one or more instructions provided in the form of an application are stored in memory (1715), and that said one or more applications are stored in an executable format (e.g., a file having an extension specified by the operating system of the wearable device (1701)) by the processor (1710). For example, an application may include a program and / or library related to a service provided to a user.

[0358] Referring to FIG. 17, programs installed on a wearable device (1701) may be classified into any one of different layers based on the target, including an application layer (1740), a framework layer (1750), and / or a hardware abstraction layer (HAL) (1780). For example, within the hardware abstraction layer (1780), programs (e.g., modules, or drivers) designed to target the hardware of the wearable device (1701) (e.g., a display (1720), a camera (1720), and / or a sensor (1730)) may be classified. The framework layer (1750) may be referred to as an XR framework layer in that it contains one or more programs for providing XR (extended reality) services. For example, FIG. 17 illustrates the layers separated within memory (1715), but the layers may be logically separated. However, it is not limited thereto. According to the embodiment, the layers may be stored in a designated area within memory (1715).

[0359] For example, within the framework layer (1750), programs designed to target at least one of the hardware abstraction layer (1780) and / or the application layer (1740) (e.g., location tracker (1771), spatial recognizer (1772), gesture tracker (1773), and / or eye tracker (1774), face tracker (1775)) may be classified. Programs classified into the framework layer (1750) may provide an application programming interface (API) that is executable based on other programs.

[0360] For example, within the application layer (1740), programs designed to target a user controlling a wearable device (1701) may be classified. Examples of programs classified into the application layer (1740) include an XR (extended reality) system UI (user interface) and / or an XR application (1742), but embodiments are not limited thereto. For example, programs classified into the application layer (1740) (e.g., software applications) may call an API (application programming interface) to cause the execution of functions supported by programs classified into the framework layer (1750).

[0361] For example, a wearable device (1701) may display one or more visual objects on a display (1720) to perform interaction with a user for using a virtual space based on the execution of an XR system UI (1741). A visual object may mean an object that can be deployed on a screen for the transmission of information and / or interaction, such as text, images, icons, videos, buttons, checkboxes, radio buttons, text boxes, sliders, and / or tables. A visual object may be referred to as a visual guide, a virtual object, a visual element, a UI element, a view object, and / or a view element. Based on the execution of an XR system UI (1741), the wearable device (1701) may provide a service to the user to control functions available in the virtual space.

[0362] Referring to FIG. 17, a lightweight renderer (1743) and / or an XR plugin (1744) are depicted within the XR system UI (1741), but are not limited thereto. For example, the XR system UI (1741) may cause the execution of supported functions in the lightweight renderer (1743) and / or the XR plugin (1744) included within the application layer (1740).

[0363] For example, a wearable device (1701) may acquire resources (e.g., APIs, system processes and / or libraries) used to define, create, and / or execute a rendering pipeline, which is permitted to be partially modified, based on the execution of a lightweight renderer (1743). The lightweight renderer (1743) may be referred to as a lightweight render pipeline in terms of defining a rendering pipeline, which is permitted to be partially modified. The lightweight renderer (1743) may include a renderer built prior to the execution of a software application (e.g., a pre-built renderer). For example, the wearable device (1701) may acquire resources (e.g., APIs, system processes and / or libraries) used to define, create, and / or execute the entire rendering pipeline based on the execution of an XR plugin (1744). The XR plugin (1744) can be referred to as an open XR native client in terms of defining (or setting) the entire rendering pipeline.

[0364] For example, a wearable device (1701) may display a screen representing at least a portion of a virtual space on a display (1720) based on the execution of an XR application (1742). An XR plugin (1744-1) included in the XR application (1742) may be referenced by an XR plugin (1744) of the XR system UI (1741). Descriptions of the XR plugin (1744-1) that overlap with descriptions of the XR plugin (1744) may be omitted. The wearable device (1701) may trigger the execution of a screen composition manager (1751) based on the execution of the XR application (1742).

[0365] According to one embodiment, a wearable device (1701) may provide a virtual space service based on the execution of a screen composition manager (1751). For example, the screen composition manager (1751) may include a platform (e.g., an Android platform) for supporting the virtual space service. Based on the execution of the screen composition manager (1751), the wearable device (1701) may display on a display the posture of a virtual object representing a rendered user's posture using data acquired through a sensor (1730). The screen composition manager (1751) may be referred to as a composition presentation manager (CPM).

[0366] For example, the screen configuration manager (1751) may include a runtime service (1752). In one example, the runtime service (1752) may be referenced as an OpenXR runtime module. A wearable device (1701) may be used to provide at least one of a user pose prediction function, a frame timing function, and / or a spatial input function through the wearable device (1701) based on the execution of the runtime service (1752). In one example, the wearable device (1701) may be used to perform rendering for a virtual space service for the user based on the execution of the runtime service (1752). For example, an application (e.g., Unity or OpenXR native application) may be implemented based on the execution of the runtime service (1752).

[0367] For example, the screen configuration manager (1751) may include a pass-through library (1753). The wearable device (1701) may, based on the execution of the pass-through library (1753), display another screen representing real space acquired through a camera (1725) superimposed on at least a portion of the screen while displaying a screen representing virtual space on the display (1720).

[0368] For example, the screen composition manager (1751) may include a renderer. The wearable device (1701) can render a screen to be displayed on a display by compositing virtual layers (or virtual nodes) rendered based on sensor data (e.g., sensing data obtained through a camera (1725) or sensor (1730)) and pass-through layers (or pass-through nodes) obtained through a pass-through library (1753) using the renderer through the screen composition manager (1751). The virtual layers may be referred to as virtual nodes and / or virtual surfaces. The wearable device (101) can render each of the virtual layers or render all of the virtual layers through the screen composition manager (1751).

[0369] For example, the screen configuration manager (1751) may include an input manager (1754). The wearable device (1701) may identify acquired data (e.g., sensor data) by executing one or more programs included within the recognition service layer (1770) based on the execution of the input manager (1754). The wearable device (1701) may initiate the execution of at least one of the functions of the wearable device (1701) using the acquired data.

[0370] For example, the perception abstract layer (1760) may be used for data exchange between the screen configuration manager (1751) and the perception service layer (1770). In terms of being used for data exchange between the screen configuration manager (1751) and the perception service layer (1770), the perception abstract layer (1760) may be referred to as an interface. As an example, the perception abstract layer (1760) may be referred to as OpenPX and / or PPAL (perception platform abstract layer). The perception abstract layer (1760) may be used for a perception client and a perception service.

[0371] According to one embodiment, the recognition service layer (1770) may include one or more programs for processing data obtained from a sensor (1730) (or a camera (1725)). The one or more programs may include at least one of a location tracker (1771), a spatial recognizer (1772), a gesture tracker (1773), an eye tracker (1774), and / or a face tracker (1775). The type and / or number of the one or more programs included in the recognition service layer (1770) are not limited to those shown in FIG. 17.

[0372] For example, the wearable device (1701) can identify the posture of the wearable device (1701) using the sensor (1730) based on the operation of the position tracker (1771). The wearable device (1701) can identify the 6 degrees of freedom pose (6 DOF pose) of the wearable device (1701) using data acquired using the camera (1725) and the IMU based on the operation of the position tracker (1771). The position tracker (1771) may be referred to as a head tracking (HeT) module.

[0373] For example, the wearable device (1701) may be used to construct the surrounding environment of the wearable device (1701) (or the user of the wearable device (1701)) into a three-dimensional virtual space based on the execution of the space recognizer (1772). The wearable device (1701) may reconstruct the surrounding environment of the wearable device (1701) in three dimensions using data acquired using a camera (1725) based on the execution of the space recognizer (1772). The wearable device (1701) may identify at least one of a plane, an incline, or a staircase based on the surrounding environment of the wearable device (1701) reconstructed in three dimensions based on the execution of the space recognizer (1772). The space recognizer (1772) may be referred to as a scene understanding (SU) module.

[0374] For example, the wearable device (1701) may be used to identify (or recognize) the pose and / or gesture of the user's hand of the wearable device (1701) based on the execution of the gesture tracker (1773). For example, the wearable device (1701) may identify the pose and / or gesture of the user's hand using data acquired from the sensor (1730) based on the execution of the gesture tracker (1773). For example, the wearable device (1701) may identify the pose and / or gesture of the user's hand based on data (or images) acquired using the camera (1725) based on the execution of the gesture tracker (1773). The gesture tracker (1773) may be referred to as a hand tracking (HaT) module and / or a gesture tracking module.

[0375] For example, the wearable device (1701) can identify (or track) the movement of the user's eyes of the wearable device (1701) based on the execution of the eye tracker (1774). For example, the wearable device (1701) can identify the movement of the user's eyes using data obtained from at least one sensor based on the execution of the eye tracker (1774). For example, the wearable device (1701) can identify the movement of the user's eyes based on data obtained using a camera (1725) and / or an IR LED (infrared light emitting diode) based on the execution of the eye tracker (1774). The eye tracker (1774) may be referred to as an eye tracking (ET) module and / or a gaze tracking module.

[0376] For example, the recognition service layer (1770) of the wearable device (1701) may further include a face tracker (1775) for tracking the user's face. For example, the wearable device (1701) may identify (or track) the movement of the user's face and / or the user's facial expression based on the execution of the face tracker (1775). The wearable device (1701) may estimate the user's facial expression based on the movement of the user's face based on the execution of the face tracker (1775). For example, the wearable device (1701) may identify the movement of the user's face and / or the user's facial expression based on data (e.g., an image) acquired using a camera based on the execution of the face tracker (17675).

[0377] FIG. 18 is a drawing for illustrating a generative artificial intelligence system according to one embodiment.

[0378] According to one embodiment, the user query / response interface (1810) may receive user input. The user input may be in the form of natural language, images, and / or videos, but is not limited thereto. Additionally, context information may be transmitted along with the user input. The context information may include various additional information at the time of user input. For example, the additional information may include information about the application currently being used by the user or the user's location information. Additionally, the user input may be in a mixed form of the aforementioned natural language, images, sounds, and context information. Furthermore, the user input may be in a non-natural language form, such as selecting a menu. The user query / response interface (1810) may output results from a generative artificial intelligence system to the user. The output may be in the form of natural language or specific content, and may also be provided in the form of actions requested by the user. The user query / response interface (1810) may output results from a generative artificial intelligence system to the user. The output can be in the form of natural language or specific content, and it may also be provided in the form of actions requested by the user.

[0379] The AI ​​framework (1840) can receive input from the user and coordinate and control each component necessary to perform the user's intent based on the user's query.

[0380] User input received from the user query / response interface (1810) can be transmitted to a prompt design component (1841). The prompt design component (1841) can be used to generate prompts suitable for inputting user input into a large language model (LLM) or a large multimodal model (LMM). The prompt design component (1841) may be an AI component that uses machine learning algorithms or neural networks to develop better prompts over time. The prompt design component (1841) can generate prompts by accessing a knowledge component containing user preference data, a prompt library, and prompt examples based on user input, and can transmit the generated prompts to the LLM or LMM.

[0381] The API / Plug-in management component (1842) can perform the role of communicating with external information when there is a request for additional information when passing user input as input to a generative model. The API / Plug-in management component (1842) establishes a channel to communicate with the outside of the AI ​​Interface via the API, and through the established channel, it can enable access to various data sources (e.g., knowledge repository (1820)). Additionally, the API / Plug-in management component (1842) can request the application / service component (1830) via the API to perform an action that ultimately executes the user input, rather than an intermediate result, in cases where the application or service needs to perform such action. The information obtained from the outside can be used to generate a prompt in the prompt design component (1841) along with the user input, or it can be passed as input to the generative model.

[0382] The output modification component (or refiner component) (1843) can fine-tune the output of the generative model. For example, the output modification component (1843) can verify whether the content generated through the LLM and / or LMM is irrelevant, contains biased content, or contains harmful content. Additionally, the output modification component (1843) can determine the extent to which the output matches the user's desired result and, if additional processing is required, proceed with that process. Furthermore, the output modification component (1843) can configure and provide hints to the user to avoid unwanted output.

[0383] A generative AI model (1860) generally refers to an artificial intelligence neural network that generates new forms of data based on user input information. A generative AI model (1860) may include models that generate images and / or models that generate language. Models that generate images include, but are not limited to, GANs (generative adversarial networks) and VAEs (variational autoencoders), and examples include Diffusion-based generative models that use VAEs and Transformer structures. Models that generate language are models trained to output the most statistically appropriate output value based on input values, and examples include models such as CHAT-GPT 3 and CHAT-GPT 4. There are also LMMs (large multimodal models) that can recognize various forms of data input, such as text, images, and voice, and generate new data corresponding to them.

[0384] The electronic device according to one embodiment disclosed in this document may be of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance. The electronic device according to the embodiment of this document is not limited to the aforementioned devices.

[0385] One embodiment of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, each of phrases such as "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C", and "at least one of A, B, or C" may include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as “first,” “second,” or “first” or “second” may be used simply to distinguish a component from another component and do not limit the components in any other aspect (e.g., importance or order). Where any (e.g., first) component is referred to as “coupled” or “connected” to another (e.g., second) component, with or without the terms “functionally” or “communicationally,” it means that said component may be connected to said other component directly (e.g., wired), wirelessly, or through a third component.

[0386] The term "module" as used in an embodiment of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to an embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0387] One embodiment of the present document may be implemented as software (e.g., program (140)) comprising one or more instructions stored in a storage medium (e.g., internal memory (136) or external memory (138)) readable by a machine (e.g., electronic device (101) or electronic device (301)). For example, a processor (e.g., processor (520)) of the machine (e.g., electronic device (301)) may call at least one of the one or more instructions stored in the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0388] According to one embodiment, the method according to one embodiment disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or an application store (e.g., Play Store). TM It can be distributed online (e.g., downloaded or uploaded) through ) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0389] According to one embodiment, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to one embodiment, one or more of the components or operations among the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to one embodiment, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In an electronic device (101 of FIG. 1; 201 of FIG. 2; 1500 of FIG. 15; 1600 of FIG. 16a to 16b; 1701 of FIG. 17), Communication circuit (190 in FIG. 1; 1735 in FIG. 17); User interface (150 in FIG. 1); Display (160 in FIG. 1; 260 in FIG. 2; 1501 in FIG. 15; 1621 in FIG. 16; 1720 in FIG. 17); At least one processor (120 in FIG. 1; 220 in FIG. 2; 1710 in FIG. 17); and It includes memory for storing instructions (130 in FIG. 1; 230 in FIG. 2a; 1715 in FIG. 17), and When the above instructions are executed individually or collectively by the at least one processor, the electronic device, Based on the user's command obtained through the above user interface, the first object to be monitored and the first operation of the first object are identified, and When the distance between the electronic device and the first object exceeds a specified distance, control to turn on a sensor of at least one external device located in a first area within a first distance from the first object, and Through the communication circuit above, a sensor value obtained from at least one external electronic device through a sensor of at least one external device is received, and An electronic device that outputs notification information indicating the occurrence of the first operation of the first object when the current operation of the first object is identified as the first operation of the first object based on the sensor value.

2. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, Controlling to switch a sensor of at least one external electronic device located in a second area within a second distance further than the first distance from the first object to a standby state, and An electronic device that controls to turn off a sensor of at least one external electronic device located in a third area within a third distance further than the second distance from the first object.

3. In any one of paragraphs 1 to 2, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, An electronic device that identifies one or more first external electronic devices including the same sensor among the at least one external electronic device included in the first area, and identifies the first external electronic device among the one or more first external electronic devices based on the location of the first object.

4. In any one of paragraphs 1 to 3, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, When the distance between the electronic device and the first object exceeds the specified distance, control is made to turn on multiple sensors of multiple external electronic devices registered on the server with the same account as the electronic device, and the multiple external electronic devices include at least one external electronic device located in the first area, at least one external electronic device located in the second area, and at least one external electronic device located in the third area. Based on sensor values ​​obtained through the plurality of sensors of the plurality of external electronic devices, the location of the first object is determined, and An electronic device that controls, based on the location of the first object, to turn on the sensor of the at least one external electronic device included in the first area, switch the sensor of the at least one external electronic device included in the second area to a standby state, and turn off the sensor of the at least one external electronic device included in the third area.

5. In any one of paragraphs 1 through 4, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, An electronic device that confirms the movement of the first object based on a sensor of at least one external electronic device included in the first area.

6. In any one of paragraphs 1 through 5, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, An electronic device that controls turning on a sensor of at least one external electronic device included in the first area, switching a sensor of at least one external electronic device included in the second area to a standby state, and turning off a sensor of at least one external electronic device included in the third area through a server configured to manage a plurality of external electronic devices registered with the same account as the electronic device.

7. In any one of paragraphs 1 through 6, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, Information related to the plurality of external electronic devices received through a server configured to manage the plurality of external electronic devices registered with the same account as the electronic device is stored in the memory, and The electronic device, wherein the information related to the plurality of external electronic devices includes at least one of type information of the external electronic device, function information of the external electronic device, location information of the external electronic device, or sensor information of the external electronic device.

8. A method for verifying the operation of an object in an electronic device (101 of FIG. 1; 201 of FIG. 2; 1500 of FIG. 15; 1600 of FIG. 16a to 16b; 1701 of FIG. 17), An operation to verify a first object to be monitored and a first operation of the first object based on a user command obtained through the user interface of the electronic device; An operation to control turning on a sensor of at least one external device located in a first area within a first distance from the first object when the distance between the electronic device and the first object exceeds a specified distance; The operation of receiving a sensor value obtained from at least one external electronic device through a sensor of at least one external device via a communication circuit of the electronic device; and A method comprising the operation of outputting notification information indicating the occurrence of the first operation of the first object when the current operation of the first object is identified as the first operation of the first object based on the sensor value.

9. In Paragraph 8, An operation to control a sensor of at least one external electronic device located in a second area within a second distance greater than the first distance from the first object to switch to a standby state; and A method further comprising the operation of controlling to turn off a sensor of at least one external electronic device located in a third area within a third distance further than the second distance from the first object.

10. In any one of paragraphs 8 to 9, A method further comprising the operation of identifying one or more first external electronic devices including the same sensor among the at least one external electronic device included in the first region, and identifying the first external electronic device among the one or more first external electronic devices based on the location of the first object.

11. In any one of paragraphs 8 through 10, When the distance between the electronic device and the first object exceeds the specified distance, the operation controls the turning on of multiple sensors of multiple external electronic devices registered on the server with the same account as the electronic device, wherein the multiple external electronic devices include at least one external electronic device located in the first area, at least one external electronic device located in the second area, and at least one external electronic device located in the third area; An operation to determine the location of the first object based on sensor values ​​obtained through the plurality of sensors of the plurality of external electronic devices; and A method further comprising, based on the location of the first object, controlling to turn on the sensor of the at least one external electronic device included in the first area, switch the sensor of the at least one external electronic device included in the second area to a standby state, and turn off the sensor of the at least one external electronic device included in the third area.

12. In any one of paragraphs 8 through 11, A method further comprising the operation of confirming the movement of the first object based on a sensor of at least one external electronic device included in the first area.

13. In any one of paragraphs 8 through 12, A method further comprising the operation of controlling to turn on a sensor of at least one external electronic device included in the first area, switch a sensor of at least one external electronic device included in the second area to a standby state, and turn off a sensor of at least one external electronic device included in the third area through a server configured to manage a plurality of external electronic devices registered with the same account as the electronic device.

14. In any one of paragraphs 8 through 13, Information related to the plurality of external electronic devices received through a server configured to manage the plurality of external electronic devices registered with the same account as the electronic device is stored in the memory of the electronic device, and A method further comprising an operation in which the information related to the plurality of external electronic devices includes at least one of the type information of the external electronic device, the function information of the external electronic device, the location information of the external electronic device, or the sensor information of the external electronic device.

15. In a non-volatile storage medium storing instructions, said instructions are configured to cause said electronic device to perform at least one operation when executed by said electronic device, said at least one operation being, An operation to verify a first object to be monitored and a first operation of the first object based on a user command obtained through the user interface of the electronic device; An operation to control turning on a sensor of at least one external device located in a first area within a first distance from the first object when the distance between the electronic device and the first object exceeds a specified distance; The operation of receiving a sensor value obtained from at least one external electronic device through a sensor of at least one external device via a communication circuit of the electronic device; and A storage medium comprising an operation that outputs notification information indicating the occurrence of the first operation of the first object when the current operation of the first object is identified as the first operation of the first object based on the sensor value.

Images