Electronic device and control method therefor

Abstract

The present disclosure provides a method for controlling an electronic device. The method may comprise the operations of: obtaining identification information of a subject through a first application; on the basis of the identification information of the subject being obtained, identifying whether to establish relationship information regarding the subject; on the basis of identifying that the relationship information regarding the subject is to be established, obtaining first information associated with a relationship with the subject by using data of the first application including the identification information; obtaining at least one image including an image region corresponding to the subject by using the first information; obtaining the relationship information regarding the subject by using the identification information of the subject and the at least one image; and storing the relationship information regarding the subject.

Description

Electronic device and control method thereof

[0001] The present disclosure relates to an electronic device and method capable of establishing relationship information regarding an object.

[0002] With the advancement of mobile communication technology, the use of portable or mobile electronic devices (e.g., smartphones, wearable devices) has become widespread, and the functions provided through these electronic devices are becoming increasingly diverse.

[0003] Users of electronic devices can communicate with a large number of people. Electronic devices can acquire various information about people during the process of users communicating with them. Information about people can be updated in real time through various applications. If users manage this information about people manually, it can be time-consuming. Therefore, a method for efficiently managing information about people is required.

[0004] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. None of the foregoing is to be claimed as prior art related to the present disclosure, nor is it to be used to determine prior art.

[0005] The present disclosure provides an electronic device and method capable of establishing relationship information regarding an object.

[0006] According to one embodiment, the electronic device comprises a memory including at least one storage medium for storing instructions; and at least one processor including a processing circuit, wherein when the instructions are executed individually or collectively by the at least one processor, the electronic device may be caused to perform at least one operation. The at least one operation may include an operation of obtaining identification information of a subject through a first application. The at least one operation may include an operation of determining whether to establish relationship information regarding the subject based on the acquisition of the identification information of the subject. The at least one operation may include an operation of obtaining first information associated with a relationship with the subject using data of the first application including the identification information based on the determination of establishing relationship information regarding the subject. The at least one operation may include an operation of obtaining at least one image including an image region corresponding to the subject using the first information. The at least one operation may include an operation of obtaining relationship information regarding the subject using the identification information of the subject and the at least one image. The at least one operation may include an operation of storing the relationship information regarding the subject.

[0007] According to one embodiment, a method for controlling an electronic device may include at least one operation. At least one operation may include an operation of obtaining identification information of a subject through a first application. At least one operation may include an operation of determining whether to set relationship information regarding the subject based on the acquisition of the identification information of the subject. At least one operation may include an operation of obtaining first information associated with a relationship with the subject using data of the first application containing the identification information, based on the determination that relationship information regarding the subject is set. At least one operation may include an operation of obtaining at least one image including an image region corresponding to the subject using the first information. At least one operation may include an operation of obtaining relationship information regarding the subject using the identification information of the subject and the at least one image. At least one operation may include an operation of storing the relationship information regarding the subject.

[0008] According to one embodiment, a storage medium may be provided for storing at least one instruction readable by a computer. The at least one instruction may cause the electronic device to perform at least one operation when executed by at least a part of at least one processor of the electronic device. The at least one operation may include an operation of obtaining identification information of a subject through a first application. The at least one operation may include an operation of determining whether to establish relationship information regarding the subject based on the acquisition of the identification information of the subject. The at least one operation may include an operation of obtaining first information associated with a relationship with the subject using data of the first application containing the identification information based on the determination of establishing relationship information regarding the subject. The at least one operation may include an operation of obtaining at least one image including an image region corresponding to the subject using the first information. The at least one operation may include an operation of obtaining relationship information regarding the subject using the identification information of the subject and the at least one image. The at least one operation may include an operation of storing the relationship information regarding the subject.

[0009] In one embodiment, the electronic device can establish relationship information regarding an object based on various data received by the electronic device. The electronic device can automatically cluster images corresponding to the object into a single group based on the various data.

[0010] The effects obtainable from the exemplary embodiments of the present disclosure are not limited to those mentioned above, and other unmentioned effects can be clearly derived and understood by those skilled in the art to which the exemplary embodiments of the present disclosure belong from the description below. That is, unintended effects resulting from the implementation of the exemplary embodiments of the present disclosure can also be derived by those skilled in the art from the exemplary embodiments of the present disclosure.

[0011] FIG. 1 is a block diagram of an electronic device in a network environment according to various embodiments.

[0012] FIGS. 2a and 2b illustrate a method for setting relationship information regarding an object according to one embodiment.

[0013] FIG. 3 is a flowchart illustrating the operation of an electronic device according to one embodiment.

[0014] FIGS. 4 and 5 illustrate a method for providing a proposal regarding a registration object according to one embodiment.

[0015] FIG. 6 is a flowchart illustrating the operation of an electronic device according to one embodiment.

[0016] FIGS. 7a and 7b illustrate a method for receiving user input to identify an object according to one embodiment.

[0017] FIG. 8 is a block diagram showing a configuration for setting relationship information regarding an object according to one embodiment.

[0018] FIG. 9 illustrates a block diagram of an AI agent according to one embodiment.

[0019] FIG. 10 illustrates a method for setting up an application used to set up relationship information regarding an object, according to one embodiment.

[0020] FIG. 11 illustrates a method for setting relationship information regarding an object according to one embodiment.

[0021] FIG. 12 illustrates a method for providing recommendation information according to one embodiment.

[0022] FIG. 13 is a block diagram of a generative artificial intelligence system according to one embodiment.

[0023] FIG. 14 is a block diagram of an AI framework according to one embodiment.

[0024] Hereinafter, embodiments of the present disclosure are described in detail with reference to the drawings so that those skilled in the art can easily practice them. However, the present disclosure may be embodied in various different forms and is not limited to the embodiments described herein. In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and brevity.

[0025] An embodiment of the present disclosure will be described below with reference to the attached drawings. In the present disclosure, the term "user" may refer to a person using an electronic device or a device using an electronic device (e.g., an artificial intelligence electronic device).

[0026] FIG. 1 is a block diagram of an electronic device (101) in a network environment (100) according to various embodiments.

[0027] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or with at least one of an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through a server (108). According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (160), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)).

[0028] The processor (120) can control at least one other component (e.g., hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., program (140)), for example, and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., sensor module (176) or communication module (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., central processing unit or application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., graphics processing unit, neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use lower power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.

[0029] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0030] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, input data or output data for software (e.g., program (140)) and related commands. The memory (130) may include volatile memory (132) or non-volatile memory (134).

[0031] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0032] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0033] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0034] The display module (160) can visually provide information to an external (e.g., user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0035] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (101).

[0036] The sensor module (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0037] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multi-media interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0038] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0039] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that can be perceived by the user through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0040] The camera module (180) can capture still images and video. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0041] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0042] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0043] The communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber First Identification Information (IMSI)) stored in the subscriber identification module (196).

[0044] The wireless communication module (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or largescale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), external electronic device (e.g., electronic device (104)), or network system (e.g., second network (199)). According to one embodiment, the wireless communication module (192) can support a Peak data rate (e.g., 20 Gbps or more) for eMBB realization, loss coverage (e.g., 164 dB or less) for mMTC realization, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for URLLC realization.

[0045] An antenna module (197) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197).

[0046] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0047] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0048] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the request may perform at least part of the requested function or service, or additional functions or services related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the above result as is or additionally and provide it as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (104) or the server (108) may be included within the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0049] The electronic device according to the various embodiments disclosed in this document may be of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. The electronic device according to the embodiments of this document is not limited to the devices described above.

[0050] FIGS. 2a and 2b illustrate a method for setting relationship information regarding an object according to one embodiment.

[0051] In FIGS. 2a and 2b, an electronic device (e.g., the electronic device (101) of FIG. 1) can automatically establish relationship information regarding an object based on input data. Establishing relationship information regarding an object may mean acquiring and storing relationship information regarding an object. The object may include people and / or animals. The relationship information regarding an object may include information that is substantially meaningful regarding the relationship between a user (e.g., a user of the electronic device (101)) and the object. The electronic device (101) may acquire the relationship information regarding an object in a table format (e.g., see FIG. 11). The relationship information regarding an object may include the object's attributes (e.g., name, nickname, relationship title, related event, related content, etc.) and the attribute value of each attribute. For example, the related event may include a trip to the xx campsite on x month x day. For example, the related content may include camping photos and / or message content between the user and the object. The electronic device (101) may store relationship information regarding an object while leaving some attributes of the object empty. For example, the attributes of the object may include information regarding the object or identification information of the object. Information regarding the object may include, but is not limited to, at least one of any information regarding the object, such as, the object's name, nickname, relationship term (e.g., dad, mom), SNS ID, image, voice, contact (e.g., phone number), information regarding activities between the object and the user, interests, age, gender, schedule, memo, conversation content, and / or location.The identification information of an object is an attribute that can uniquely distinguish the object, and may include, for example, at least one of the object's name, nickname, relationship term (e.g., dad, mom), SNS ID, image, voice, contact (e.g., phone number), or information regarding activities between the object and the user, but is not limited thereto. The electronic device (101) may acquire relationship information regarding the object and store it in the user's personal database (DB), for example, a relationship information database. The user's personal database may be stored in the electronic device (101). The user's personal database may be stored in a server (108) and / or an external electronic device (102 and / or 104) based on the user's personal account. The electronic device (101) may build a database regarding the object by establishing relationship information regarding the object. Establishing relationship information regarding the object may be referred to as defining a person relationship or establishing a person relationship.

[0052] In one embodiment, the screen (210) of FIG. 2a illustrates the execution screen of a first application according to one embodiment. An electronic device (101) can execute the first application. The first application may be any application capable of receiving information about an object. For example, the first application may be a messenger application and / or a text message application. The electronic device (101) may obtain data of the first application (e.g., text data and / or image data). For example, the data of the first application may include text data (215) containing conversational content such as "Do you remember my uncle John?" (215-1), "Of course! He used to take me on camping trips often" (215-2), "Yeah, it was a really good time!" (215-3), and "Do you have photos from the camping trip? If so, can you share them with me?" (215-4).

[0053] In one embodiment, the electronic device (101) may obtain first identification information (211) of an object through a first application. For example, the first identification information (211) may include at least one of the object's name, nickname, relationship term, SNS ID, image, voice, contact (e.g., phone number), or information regarding activities between the object and the user (e.g., the object and the user have previously gone on a camping trip together), but is not limited thereto. For example, as shown in the screen (210) of FIG. 2a, the first identification information (211) may include the object's relationship term (e.g., uncle) and / or name (e.g., John). Based on the acquisition of the object's first identification information (211), the electronic device (101) may determine whether to establish relationship information regarding the object. The electronic device (101) may determine whether data substantially matching the object's first identification information (211) is stored. The electronic device (101) may identify that it has established relationship information regarding an object when data substantially matching the first identification information (211) of the object is not stored. The electronic device (101) may identify that it has not established relationship information regarding an object when data substantially matching the first identification information (211) of the object is stored. Data substantially matching the first identification information (211) may include data substantially identical to the first identification information (211) (e.g., relationship title of the object (e.g., uncle) and / or name (e.g., John). Data substantially matching the first identification information (211) may include information stored in correspondence with the first identification information (211) (e.g., image and / or voice of the object). The electronic device (101) may identify an object for which relationship information has not been established as an unregistered object.

[0054] In one embodiment, the electronic device (101) may determine whether to set relationship information regarding an object if data substantially matching the first identification information (211) of the object is not stored. For example, as shown in the screen (220) of FIG. 2a, the electronic device (101) may determine whether to set relationship information regarding an object through a user interface (e.g., a graphical user interface). The electronic device (101) may determine whether to search for and collect information regarding an object by notifying the user through the user interface that relationship information regarding an object has not been set (or registered). For example, the user interface may display the identification information (e.g., relationship title and / or name) (222) of an object identified as an unregistered object among the data (215) of the first application in a specific color (e.g., blue). For example, the user interface may include a message (221) such as "Unregistered person detected. Would you like to start person search?" The electronic device (101) can receive a user input (U) (e.g., a touch input to a YES button) that approves the search for information about an object through a user interface. The electronic device (101) can identify the user input that approves the search for information about an object as a user input requesting the setting of relationship information about an object. The electronic device (101) can identify the setting of relationship information about an object in response to the reception of the user input requesting the setting of relationship information about an object.

[0055] In one embodiment, based on being identified as setting relationship information regarding an object, the electronic device (101) may display the screen (230) of FIG. 2b. As illustrated in the screen (230) of FIG. 2b, the electronic device (101) may provide a user interface indicating that relationship information regarding an object is being set. The user interface may include a message (231). For example, the message (231) may include "Defining person relationships. Please wait a moment."

[0056] In one embodiment, the electronic device (101) may acquire and store relationship information regarding an object while providing a user interface. The electronic device (101) may acquire first information associated with a relationship with an object using data of a first application including first identification information (211), and may acquire at least one image (242) including an image area (244) corresponding to the object using the acquired first information. For example, the data of the first application including the first identification information (211) may be text data (215) regarding conversation content including the first identification information (211). The first information may include information regarding the relationship between the user and the object. For example, the first information may include information regarding activities between the object and the user (e.g., information regarding a camping trip). The electronic device (101) may acquire the first information using an artificial intelligence model (e.g., a generative artificial intelligence model). A generative artificial intelligence model (e.g., a large language model (LLM), a large language model (LVM)) can take data of a first application (e.g., text, screen image, etc.) containing first identification information (211) as input and output first information. The electronic device (101) can acquire at least one image (e.g., a camping trip photo) associated with the first information. The at least one image associated with the first information may be an image similar to the content of the first information. The at least one image associated with the first information may include an image region corresponding to an object. The electronic device (101) can acquire at least one image (242) including an image region (244) corresponding to an object. The electronic device (101) can determine whether the at least one image (242) includes an image region corresponding to an object.The electronic device (101) can use the first information as a search term to search for an image corresponding to the first information among images stored in the user's personal database (e.g., an image database of a gallery application). The image corresponding to the first information may include an image region corresponding to a specific object regarding the first information. The electronic device (101) can search for images by transmitting a search term (e.g., the first information) and a prompt to an image clustering AI agent (e.g., the first AI agent (831) of FIG. 8). For example, the prompt may include a command such as 'search for an image corresponding to the first information in the image database of the gallery application.' The electronic device (101) can acquire and store relationship information regarding an object using the first identification information (211) of the object and at least one image (242).

[0057] In one embodiment, the electronic device (101) may obtain data of a second application containing second information associated with a relationship with an object by using at least one of the first identification information (211) of an object or at least one image (242). The second application may be the same as the first application. The second application may be different from the first application. For example, the second application may be a calendar application. For example, the second information may be information associated with a relationship with an object (e.g., a past camping schedule with John (November 12)). The electronic device (101) may obtain the second information by sending a prompt using the first identification information (211) and / or at least one image (242) to an AI agent (e.g., the second AI agent (832) of FIG. 8). The electronic device (101) may transmit a prompt to an AI agent (e.g., the second AI agent (832) of FIG. 8) to obtain information (e.g., date information on which the image was taken) and second information regarding the first identification information (211) and / or at least one image (242). For example, the prompt may include a command to 'obtain second information corresponding to the first identification information (211) and / or at least one image (242) from the user's personal database.' The AI ​​agent may receive the first identification information (211) and / or at least one image (242) as input, and obtain and output data associated with the first identification information (211) and / or at least one image (242) from the user's personal database. The electronic device (101) may obtain relationship information regarding the object using the first identification information (211) and second information of the object.For example, the electronic device (101) can compare information regarding at least one image (242) (e.g., metadata indicating that it was taken on November 12) with second information (e.g., past camping schedule with John (November 12)) to determine (or verify) whether at least one image (242) includes an image area corresponding to an object.

[0058] In one embodiment, the electronic device (101) may acquire at least one image (242) including an image region (244) corresponding to an object based on user input. For example, as shown in the screen (210) of FIG. 2a, the user may transmit at least one image including an image region corresponding to the object in response to a message (215-4) requesting the transmission of a photo of a specific object (e.g., John). The electronic device (101) may acquire at least one image (242) including an image region corresponding to the object based on the transmitted image using an AI agent (e.g., the first AI agent (831) of FIG. 8). The AI ​​agent (e.g., the first AI agent (831) of FIG. 8) may identify features of the image region corresponding to the object (e.g., features of the object's face) and cluster the image region corresponding to the object. In one embodiment, the electronic device (101) may acquire at least one image of an image region corresponding to an object based on user input selecting an image region corresponding to an object (e.g., see FIG. 7a and 7b). In one embodiment, the electronic device (101) may acquire information about an object using a plurality of AI agents and establish relationship information about an object based on the acquired information (e.g., see FIG. 8). During the nth cycle (n is 1 or greater), the electronic device (101) may acquire information about an object using a first AI agent (e.g., the first AI agent (831) of FIG. 8) and a second AI agent (e.g., the second AI agent (832) of FIG. 8). For example, in the first cycle, the electronic device (101) may acquire at least one first image associated with first information (e.g., information about a camping trip) from a user's first personal database (e.g., an image database of a gallery application) using the first AI agent (e.g., the first AI agent (831) of FIG. 8).The electronic device (101) can use a second AI agent (e.g., the second AI agent (832) of FIG. 8) to obtain information (e.g., image, voice, and / or location information) associated with at least one first image from the user's second personal database (e.g., a database of all applications installed on the electronic device (101)). The electronic device (101) can obtain a knowledge graph using the various types of data obtained. Based on the knowledge graph, the electronic device (101) can obtain sub-data that constitutes relationship information regarding an object. For example, the sub-data may include information such as the object's name, gender, age, voice, SNS ID, and / or interests. For example, in the second cycle, the electronic device (101) can perform the same operation as in the first cycle. In the second cycle, the electronic device (101) can use a first AI agent (831) to obtain at least one second image associated with the sub-data from the user's first personal database. The electronic device (101) can obtain information (e.g., image, voice, and / or location information) associated with at least one second image from the user's second personal database (e.g., a database of all applications installed on the electronic device (101)) using the second AI agent (832). The first AI agent (e.g., the first AI agent (831) of FIG. 8) and the second AI agent (e.g., the second AI agent (832) of FIG. 8) can be managed by a management AI agent (e.g., the AI ​​agent management model (820) of FIG. 8). The first AI agent (831) and the second AI agent (832) can obtain information about objects sequentially using a Chain-of-Thought (CoT) method.The management AI agent (e.g., the AI ​​agent management model (820) of FIG. 8) can terminate the interaction cycle between the first AI agent (831) and the second AI agent (832) when the termination condition is satisfied.

[0059] In one embodiment, an electronic device (101) (e.g., the AI ​​agent management model (820) of FIG. 8) may be configured to set a termination condition that can terminate the setting of relationship information regarding an object, and may be configured to terminate the setting of relationship information regarding an object when the termination condition is satisfied. The termination condition may be set in advance. The electronic device (101) may set the termination condition by user input. The electronic device (101) may identify whether the termination condition is satisfied. Based on the identification that the termination condition is satisfied, the electronic device (101) may terminate the setting of relationship information regarding an object. The termination condition will be described later with reference to FIG. 8.

[0060] In one embodiment, the electronic device (101) may display, through a user interface, which of the applications (e.g., applications installed on the electronic device (101)) is searching for information about an object. For example, as shown in the screen (230) of FIG. 2b, the electronic device (101) may display, in real time, the icons (233: 233-1, 233-2, 233-3) of the application being used to search for information about an object through the user interface. The user may identify the application being used to search for information about an object through the user interface. The electronic device (101) may control the user interface using an AI agent management model (e.g., the AI ​​agent management model (820) of FIG. 8). The user interface may display a user interface element (232) regarding work time. For example, the user interface element (232) regarding work time may include a user interface element (234) representing a timer and / or the remaining time (235) expected until the setting of relationship information about the object is completed. The electronic device (101) can display tasks to be performed during the remaining time based on receiving user input that selects (e.g., touches) a user interface element (234) representing a timer. For example, the electronic device (101) can display a summary of tasks to be performed during the remaining time in the form of a list.

[0061] In one embodiment, the electronic device (101) may provide a user interface that indicates that the relationship information setting regarding an object has been terminated when the relationship information setting regarding an object has been terminated. For example, as shown in the screen (240) of FIG. 2b, the user interface may include a message (241) and / or data used for the relationship information setting regarding an object (e.g., an image (242) including an image area (244) corresponding to the object). The message (241) may indicate that the relationship information setting regarding an object has been terminated. The message (241) may include identification information of the object (e.g., relationship title and / or name) (243). For example, the message (241) may include a message such as "Uncle John has been identified. You can now search for this person on the device." The electronic device (101) may display the identification information (243) of the object in a specific color (e.g., blue).

[0062] In one embodiment, the electronic device (101) acquires various types of data regarding an object (e.g., image data, text data, and / or sound data) and can cluster images of the object with relatively high accuracy using the acquired data. The electronic device (101) could cluster images of a specific object into one group using an artificial intelligence model (e.g., a face recognition model). The face recognition model had a limitation that it could only receive image data as input. The face recognition model had a limitation that it required n or more images to perform clustering. The electronic device (101) received relationship information from the user regarding the clustered objects, while not receiving relationship information regarding the non-clustered objects. In one embodiment, when identification information of an object for which relationship information has not been set is detected (e.g., when the name of the object is searched in a search bar), the electronic device (101) can acquire at least one image including an image area corresponding to the object (e.g., at least one image associated with the input text) using the detected identification information, and can set relationship information regarding the object using the object's identification information and at least one image. The electronic device (101) can establish relationship information regarding an object using various types of data (including images) regarding an object obtained while performing various operations performed through the electronic device (101). The electronic device (101) can establish relationship information regarding an object using multi-AI agents (e.g., see FIG. 8).

[0063] FIG. 3 is a flowchart illustrating the operation of an electronic device (101) according to one embodiment.

[0064] In the following operational embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, or at least two operations may be performed in parallel.

[0065] In one embodiment, operations 310 to 360 may be understood as being performed by an AI agent of the electronic device (101) (e.g., the AI ​​agent management model (820) and the AI ​​agent unit (830) of FIG. 8).

[0066] In one embodiment, the electronic device (101) can obtain identification information of a subject through a first application in operation 310. For a description of the first application, the subject, and the identification information of the subject, refer to FIGS. 2a and 2b. The electronic device (101) can obtain identification information of the subject by constantly identifying content input through input via a keyboard and / or drawing input using a pen. The electronic device (101) can obtain identification information of the subject based on user input selecting (e.g., touch) a specific area of ​​a display (e.g., display module (160) of FIG. 1).

[0067] In one embodiment, the electronic device (101) can identify whether to set relationship information regarding an object in operation 320. The electronic device (101) can identify whether to set relationship information regarding an object based on the acquisition of identification information of the object. The electronic device (101) can determine whether data substantially matching the identification information of the object is stored. The electronic device (101) identifies whether data substantially matching the identification information of the object is stored based on the acquisition of identification information of the object, and can identify whether to set relationship information regarding the object based on the identification that data substantially matching the identification information of the object is not stored. The electronic device (101) can confirm with the user whether to set relationship information regarding the object based on the identification that data substantially matching the identification information of the object is not stored. The electronic device (101) can receive user input requesting to set relationship information regarding the object. The electronic device (101) can identify whether to set relationship information regarding the object based on the user input.

[0068] In one embodiment, the electronic device (101) may, in operation 330, obtain first information associated with a relationship with an object by using data of a first application containing identification information. The electronic device (101) may obtain first information associated with a relationship with an object by using data of a first application containing identification information based on being identified as establishing relationship information regarding an object. For example, the data of a first application containing identification information may include at least one of text data, image data, or sound data containing identification information. For example, the text data may include memo content and / or conversation content (e.g., "Do you remember going camping with Uncle John?") including the name of the object and / or relationship designation. For example, the image data may include an image including an image area corresponding to the object. For example, the sound data may include the voice of the object and / or a conversation including the name of the object. The first information obtained by using data of a first application containing identification information may include text data. The first information may include text data associated with a relationship with an object. For example, the first information may include information regarding activities between an object and a user (e.g., information regarding a camping trip). The electronic device (101) may acquire text data using image data. The electronic device (101) may generate text describing the content of an image (e.g., a camping trip photo) by captioning the image. For example, the first information may include text data related to the relationship with the object among the text describing the content of the image. The electronic device (101) may acquire text data using sound data. The electronic device (101) may convert voice data included in the sound data into text data based on speech recognition technology (e.g., speech-to-text (STT) technology).For example, the first information may include text data associated with the relationship with the object among the text data converted from voice data.

[0069] In one embodiment, the electronic device (101) may, in operation 340, acquire at least one image including an image region corresponding to an object using the first information. The electronic device (101) may acquire at least one image associated with the first information (e.g., a photo of a camping trip) using the first information (e.g., information about a camping trip). The electronic device (101) may acquire at least one image associated with the first information from the user's first personal database (e.g., an image database of a gallery application). The electronic device (101) may search for at least one image associated with the content of the first information from the user's first personal database using the first AI agent (e.g., the first AI agent (831) of FIG. 8). At least one image associated with the first information may include an image region corresponding to an object. The image region corresponding to the object may include features of the object image (e.g., part or all of a face).

[0070] In one embodiment, the electronic device (101) can obtain relationship information regarding an object using identification information of an object and at least one image in operation 350. The relationship information regarding an object may include identification information of an object (e.g., name of the object and / or relationship designation) and at least one image including an image area corresponding to the object.

[0071] In one embodiment, the electronic device (101) may store relationship information regarding an object in operation 360. For example, the electronic device (101) may store relationship information regarding an object in a table format. By acquiring and storing relationship information regarding an object, relationship information regarding an object is established. The electronic device (101) may establish relationship information regarding an object, that is, construct a database regarding an object. The database regarding an object may include identification information of the object and at least one image. An object for which relationship information is established may be referred to as a "registered object." An object for which relationship information is not established may be referred to as an "unregistered object."

[0072] FIGS. 4 and 5 illustrate a method for providing a proposal regarding a registration object according to one embodiment.

[0073] In FIGS. 4 and 5, the electronic device (101) may provide a user interface representing a registered object. The user can identify whether an object is a registered object through the user interface. The electronic device (101) enables the user to perform interactions regarding the registered object through the user interface.

[0074] In one embodiment, the electronic device (101) may provide a user interface capable of receiving user input selecting at least one of the identification information of an object or an image region corresponding to an object, based on the fact that relationship information regarding an object is stored. The electronic device (101) may provide a suggestion regarding an object based on the fact that user input selecting at least one of the identification information of an object or an image region corresponding to an object is received. The suggestion regarding an object may be determined based on the user's usage pattern of the electronic device (101). The electronic device (101) may provide a suggestion based on functions frequently used by the user. The electronic device (101) may provide a user interface including user interface elements regarding the suggestion. The electronic device (101) may execute a suggestion corresponding to the selected user interface element based on the fact that user input selecting at least one of the user interface elements is received. In one embodiment, the electronic device (101) may provide a user interface including user interface elements corresponding to the source application of the relationship information regarding an object based on the fact that user input selecting at least one of the identification information of an object or an image region corresponding to an object is received. The electronic device (101) can execute a source application corresponding to a selected user interface element based on receiving a user input selecting at least one of the user interface elements.

[0075] In one embodiment, as illustrated in the screen (410) of FIG. 4, the electronic device (101) can identify a query (e.g., "Can I go fishing with Uncle John this week?") containing identification information (e.g., "Uncle John") (411) of a registered object. In this case, the electronic device (101) can activate the identification information (411) based on the identification of the registration object's identification information (411). For example, the electronic device (101) can display the identification information (411) in a specific color (e.g., blue). The electronic device (101) can display at least one image (412) corresponding to the identification information (411). The image (412) may be a thumbnail image. The image (412) may include an image area corresponding to the registration object. For example, the image (412) may include a face image of the registration object.

[0076] In one embodiment, the electronic device (101) may receive user input selecting at least one of identification information (411) or an image (412) of an object. Based on the reception of user input, the electronic device (101) may provide one or more suggestions regarding the object. The electronic device (101) may provide a user interface including one or more user interface elements corresponding to one or more suggestions regarding the object.

[0077] In one embodiment, the electronic device (101) may provide a user interface based on the user's electronic device (101) usage pattern. The screen (420) of FIG. 4 illustrates a user interface (421) that provides suggestions regarding objects based on the user's electronic device (101) usage pattern according to one embodiment. For example, the user interface (421) may include a first user interface element (421-1), a second user interface element (421-2), and a third user interface element (421-3), but the number of user interface elements (421-1, 421-2, and 421-3) is not limited to a specific number. For example, if the user frequently performs gallery searches, SMS sending, and schedule checks regarding people, the electronic device (101) may identify such usage patterns of the user's electronic device (101). The electronic device (101) may provide a user interface including a first user interface element (421-1) regarding gallery search, a second user interface element (421-2) regarding sending SMS, and a third user interface element (421-3) regarding checking a schedule, based on the user's usage pattern of the electronic device (101). For example, the electronic device (101) may receive user input selecting (e.g., touching) any one of one or more user interface elements (421-1, 421-2, and 421-3). Based on the reception of user input, the electronic device (101) may execute a suggestion corresponding to the selected user interface element (421-1, 421-2, and 421-3). For example, the electronic device (101) may execute a gallery search for an object based on the reception of user input selecting the first user interface element (421-1). A gallery search may mean searching for an image corresponding to an object in a gallery application.

[0078] In one embodiment, the electronic device (101) may provide suggestions regarding an object based on relationship information regarding the object. For example, as illustrated in FIG. 5, the electronic device (101) may provide a user interface (501) that includes suggestions regarding an object. In FIG. 5, the electronic device (101) may receive user input selecting (e.g., long touch) an image area (500) corresponding to an object. The object may be a registered object. Based on the reception of user input, the electronic device (101) may provide a user interface that allows the user to perform simple tasks on the image area (500), such as adding a sticker, copying an image, or sharing an image. According to one embodiment, the electronic device (101) may provide a user interface (501) that provides suggestions regarding an object based on the reception of user input selecting an image area (500) corresponding to a registered object. The electronic device (101) may provide a user interface (501) using relationship information regarding an object. For example, the electronic device (101) can identify that a trip with an object is scheduled using relationship information regarding an object. The electronic device (101) can obtain one or more proposals using relationship information regarding an object. One or more proposals may include a first proposal (e.g., travel reservation) (510), a second proposal (e.g., accommodation reservation) (520), and / or a third proposal (e.g., adding to the schedule (530)), but the number of proposals is not limited to a specific number. The electronic device (101) may provide a user interface (511, 521, and 531) regarding an application capable of executing each of the proposals (510, 520, and 530). In one embodiment, the electronic device (101) can identify a source application of data used to establish relationship information regarding an object. The electronic device (101) can obtain one or more proposals (510, 520, and / or 530) using the identified source application.The electronic device (101) may provide user interfaces (511, 521, and 533) corresponding to proposals (510, 520, and 530). A first user interface (511) corresponding to a first proposal (510) may include one or more user interface elements (511-1, 511-2, 511-3). A second user interface (521) corresponding to a second proposal (520) may include one or more user interface elements (521-1, 521-2, 521-3). A third user interface (531) corresponding to a third proposal (530) may include one or more user interface elements (531-1). Each of the user interface elements (511-1, 511-2, 511-3, 521-1, 521-2, 521-3, and 531-1) may correspond to a source application of data used to establish relationship information regarding an object. Each of the user interface elements (511-1, 511-2, 511-3, 521-1, 521-2, 521-3, and 531-1) may be an icon of the source application. The electronic device (101) may receive user input selecting (e.g., touching) any one of the user interface elements (511-1, 511-2, 511-3, 521-1, 521-2, 521-3, 531-1). The electronic device (101) can execute an application corresponding to a selected user interface element (511-1, 511-2, 511-3, 521-1, 521-2, 521-3, 531-1) based on the reception of user input. The user can execute an application capable of performing operations on an object through an image area corresponding to an object in a gallery application without intentionally needing to access an application capable of performing operations on an object.

[0079] FIG. 6 is a flowchart illustrating the operation of an electronic device (101) according to one embodiment.

[0080] In FIG. 6, the electronic device (101) can use multi agents to set relationship information regarding an object (or establish person relationships) and provide a user interface representing a registered object for which relationship information has been set.

[0081] In the following operational embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, or at least two operations may be performed in parallel.

[0082] In one embodiment, operations 610 to 670 may be understood to be performed by an electronic device (101) (e.g., the search term filtering model (810) of FIG. 8, the AI ​​agent management model (820), the AI ​​agent unit (830), and the augmented content generator (840)).

[0083] In one embodiment, an electronic device (101) (e.g., a search term filtering model (810)) can detect identification information of an object in operation 610. Refer to FIGS. 2a and 2b for a description of the identification information of an object. The identification information of an object may include text corresponding to the object. For example, the text corresponding to the object may include at least one of the object's name, nickname, relationship title, or SNS (social network service) ID, but is not limited thereto. The electronic device (101) may receive a search term (query) entered into an input window (e.g., a search window), analyze the search term, and output a filtered result. The filtered result may include identification information of an object (e.g., text corresponding to the object).

[0084] In one embodiment, an electronic device (101) (e.g., an AI agent management model (820)) can identify whether relationship information regarding an object is set in operation 620. The electronic device (101) can identify whether relationship information regarding an object is set using the identification information of the object. The set relationship information regarding the object may be stored in the user's personal database. The electronic device (101) can identify whether data matching the identification information of the object is stored in the user's personal database. If data matching the identification information of the object is stored, the electronic device (101) can identify that relationship information regarding the object is set. If data matching the identification information of the object is not stored, the electronic device (101) can identify that relationship information regarding the object is not set. If relationship information is set for the object, the electronic device (101) can perform operation 670.

[0085] In one embodiment, when relationship information regarding an object is not set, the electronic device (101) (e.g., the AI ​​agent management model (820) of FIG. 8) may, in operation 630, transmit commands to subordinate AI agents (e.g., the first AI agent (831) and the second AI agent (832) included in the AI ​​agent unit (830) of FIG. 8). The commands may include tasks for setting relationship information regarding an object. The electronic device (101) (e.g., the AI ​​agent management model (820) of FIG. 8) may transmit a request signal to the first AI agent (831) to extract information regarding an object from the user's personal database. The electronic device (101) may transmit tasks for setting relationship information regarding an object to each of the subordinate AI agents (831, 832) using the AI ​​agent management model (e.g., the AI ​​agent management model (820) of FIG. 8). Each of the subordinate AI agents (831, 832) can independently perform tasks to establish relationship information regarding the object.

[0086] In one embodiment, an electronic device (101) (e.g., the second AI agent (832) of FIG. 8) may, in operation 640, obtain and update information regarding an object in a user's personal database. The electronic device (101) (e.g., the second AI agent (832) of FIG. 8) may obtain and update information regarding an object or identification information regarding an object in a user's personal database. Information regarding an object may include, for example, images, voice, schedules, memos, conversation content, and / or location information regarding the object. The electronic device (101) may obtain and update information regarding an object in a user's personal database using the second AI agent (832).

[0087] In one embodiment, an electronic device (101) (e.g., the first AI agent (831) of FIG. 8) can establish relationship information regarding an object based on an image in operation 650. The electronic device (101) (e.g., the first AI agent (831) of FIG. 8) can cluster images similar to an image region corresponding to an object in a user's personal database into object groups. The electronic device (101) (e.g., the first AI agent (831) of FIG. 8) can obtain features of the object groups (e.g., features of a face). The electronic device (101) can use the first AI agent (831) to cluster image regions corresponding to an object and identify features of image regions corresponding to an object (e.g., features of an object's face). The electronic device (101) can cluster images similar to an image region corresponding to an object into object groups and obtain features of the object groups. An electronic device (101) (e.g., the first AI agent (831) of FIG. 8) can establish relationship information regarding an object based on the image region itself corresponding to the object and / or the features of the image region corresponding to the object. The electronic device (101) can establish relationship information regarding an object based on the identification information of the object and the image region itself corresponding to the object and / or the features of the image region corresponding to the object. The electronic device (101) (e.g., the first AI agent (831) of FIG. 8) can store relationship information regarding an object including the image region itself corresponding to the object and / or the features of the image region corresponding to the object. The electronic device (101) can store the identification information of the object and the image region itself corresponding to the object and / or the features of the image region corresponding to the object by mapping them.

[0088] In one embodiment, an electronic device (101) (e.g., an AI agent management model (820)) can identify whether a termination condition is satisfied in operation 660. The termination condition may serve as a criterion for terminating the setting of relationship information regarding an object. The electronic device (101) (e.g., an AI agent management model (820)) can verify whether the inferred relationship information is the same as the relationship information regarding the actual object. To verify the integrity of the relationship information, the electronic device (101) (e.g., an AI agent management model (820)) can perform verification using pre-set verification data. In one embodiment, the electronic device (101) (e.g., an AI agent management model (820)) can obtain a matching score (or similarity score) by comparing at least a portion of the image area corresponding to the object with the verification data. The electronic device (101) (e.g., an AI agent management model (820)) can determine that the termination condition is satisfied based on the fact that the matching score is above a threshold. In one embodiment, an electronic device (101) (e.g., an AI agent management model (820)) may calculate a matching score with an object included in verification data using one or more object images and / or at least some of the features of the object images included in the relationship information regarding the object. If the matching score is high, the electronic device (101) (e.g., an AI agent management model (820)) may determine that the termination condition is satisfied. When the termination condition is satisfied, the electronic device (101) (e.g., an AI agent management model (820)) may terminate the operation of obtaining relationship information regarding the object.

[0089] In one embodiment, when the termination condition is satisfied, the electronic device (101) (e.g., augmented content generator (840)) may provide a user interface representing an object in operation 670. When the setting of relationship information regarding the object is finished, the electronic device (101) (e.g., augmented content generator (840)) may augment and provide interactive user interface elements to the identification information of the object searched by the user (e.g., the name and / or image of the object). In one embodiment, when the termination condition is not satisfied, the electronic device (101) may return to operation 640.

[0090] FIGS. 7a and 7b illustrate a method for receiving user input to identify an object according to one embodiment.

[0091] In FIGS. 7a and 7b, the electronic device (101) can automatically acquire information about an object and establish relationship information about the object based on the acquired information about the object. Part or all of the operation of establishing relationship information about the object may be performed based on user input to identify information about the object (e.g., information about an image region corresponding to the object). The electronic device (101) may receive user input to identify an image region corresponding to the object. The electronic device (101) may identify an image region corresponding to the object based on the reception of user input to identify an image region corresponding to the object. The electronic device (101) may acquire at least one image including an image region corresponding to the object. The "manual definition of a person" process may include receiving user input to identify information about the object (e.g., information about an image region corresponding to the object).

[0092] In one embodiment, the electronic device (101) can obtain the identification information (713) of an object. The electronic device (101) can obtain the identification information (713) of an object through a first application. For example, the first application may be a messenger application. For example, as shown in the screen (710) of FIG. 7a, the electronic device (101) can detect the identification information (713) of an object from text data (715). For example, the text data (715) may include conversational content such as "Do you remember my uncle John?". For example, the identification information (713) of an object may include the object's relational title (e.g., Uncle) and / or name (e.g., John). Based on the detection of the identification information (713) of an object, the electronic device (101) can determine whether data matching the identification information (713) is set (or registered). The electronic device (101) can automatically acquire information about an object based on the determination that data matching the identification information (713) is not set. Instead of automatically acquiring information about an object, the electronic device (101) can acquire information about an object based on user input.

[0093] In one embodiment, the electronic device (101) may, based on the determination that data matching identification information (713) is not stored, notify the user through a user interface that information regarding the object is not stored and determine whether to automatically search for and collect information regarding the object using the electronic device (101). For example, the user interface may include a message (711) such as "An unregistered person has been detected. Would you like to start automatic person search?". The electronic device (101) may receive user input (e.g., a touch input to a YES button) that approves automatic search for information regarding the object (or "automatic person search") through the user interface. Based on the receipt of user input approving automatic search for information regarding the object, the electronic device (101) may operate as described in FIGS. 2a and 2b. The electronic device (101) may receive user input (e.g., a touch input to a NO button) that requests not to perform automatic search for information regarding the object through the user interface.

[0094] In one embodiment, based on receiving user input requesting not to perform automatic search for information regarding an object, the electronic device (101) may request the user to input information regarding an object. In this case, for example, as shown in the screen (720) of FIG. 7a, the electronic device (101) may check through a user interface whether the user will manually input information regarding an object (or perform a "manual person definition"). For example, the user interface may include a message (721) such as "Would you like to perform a manual person definition?". The electronic device (101) may receive user input (e.g., a touch input to a YES button) indicating that it will perform a "manual person definition" through the user interface. The electronic device (101) may receive user input (e.g., a touch input to a NO button) indicating that it will not perform a "manual person definition" through the user interface.

[0095] In one embodiment, based on receiving user input indicating that "manual person definition" will be performed, the electronic device (101) may request the user to input information regarding an object. The electronic device (101) may provide a user interface for receiving user input to identify an image region corresponding to an object using data of a first application containing identification information of an object. The electronic device (101) may obtain first information (e.g., information regarding a camping trip) associated with a relationship with an object using data of the first application. The electronic device (101) may provide a user interface for receiving user input to identify an image region corresponding to an object using the first information. The electronic device (101) may search the user's personal database for whether there is an image (e.g., an image regarding a camping trip) (731) associated with the first information using the first information. The electronic device (101) may search for whether there is an image (731) associated with the first information using, for example, a large language model (LLM). As illustrated in the screen (730) of FIG. 7b, the electronic device (101) may provide a user interface including an image (731) associated with first information. The user interface may include a message (732) requesting the user to select an image area corresponding to an object. For example, the message (732) may include "Image corresponding to search term found. Select Uncle John."

[0096] In one embodiment, the electronic device (101) can perform segmentation on the retrieved image (731). The electronic device (101) can identify binary mask information for one or more objects (e.g., people, trees, and / or food) within the image (731) and x, y axis coordinate values ​​representing the location of the objects. The electronic device (101) can perform AI-based segmentation before receiving user input to identify image regions corresponding to the objects. By performing segmentation in advance, the electronic device (101) can accurately identify the objects based on user input to identify image regions corresponding to the objects.

[0097] In one embodiment, as illustrated in the screen (740) of FIG. 7b, the electronic device (101) may receive user input to identify an image region (741) corresponding to an object in an image (731). The electronic device (101) may identify that the user has selected the object when the user input (e.g., touch input) is detected within a binary mask corresponding to the object obtained through segmentation. Based on receiving user input to identify the image region (741) corresponding to the object, the electronic device (101) may display the image region (741) to be distinct from the area of ​​the image (731) excluding the image region (741). For example, the electronic device (101) may surround the image region (741) with a border of a specific color (e.g., yellow) and / or overlay a specific color (e.g., yellow). The user can verify whether the user's intention is accurately reflected through the distinctly displayed image region (741). The electronic device (101) may display a message (742) to the user to confirm whether the image area (741) correctly represents an object. For example, the message (742) may include "Is the displayed area Uncle John?". The electronic device (101) may receive user input (e.g., touch input to a YES button) indicating that the image area (741) represents an object. The electronic device (101) may receive user input (e.g., touch input to a NO button) indicating that the image area (741) does not represent an object.

[0098] In one embodiment, the electronic device (101) may set relationship information regarding an object (or define person relationships). The electronic device (101) may set relationship information regarding an object based on receiving user input indicating that an image area (741) represents an object. As shown in the screen (750) of FIG. 7b, the electronic device (101) may provide a user interface indicating that relationship information regarding an object is being set. The user interface may include a message (751). The message (751) may be substantially the same as the message (231) shown in the screen (230) of FIG. 2b. The user interface may include a user interface element (752) regarding work time. The user interface element (752) may be substantially the same as the user interface element (232) shown in the screen (230) of FIG. 2b.

[0099] In one embodiment, the process of establishing relationship information regarding an object may involve user intervention. For example, a user's personal database may contain an image (e.g., a picture of someone enjoying a barbecue at a campsite) that is associated with a query containing information about a person with the same name as the object and / or identification information of the object, but does not include an image area corresponding to the object. In this case, it may be difficult for the electronic device (101) to accurately identify information regarding the object without user intervention. The electronic device (101) can identify information regarding the object (e.g., an image area corresponding to the object) based on user input. The electronic device (101) can identify an image area corresponding to the object based on user input for identifying a specific area of ​​the display.

[0100] In one embodiment, the electronic device (101) may perform a manual person definition process that receives user input for identifying an object. In the manual person definition process, the electronic device (101) may receive user input that explicitly selects a person to be defined in the electronic device (101). In this case, the electronic device (101) can define person relationships with high accuracy without a process of inferring information about the object. Even when receiving user input for identifying an object, the first AI agent (e.g., the first AI agent (831) of FIG. 8) and the second AI agent (e.g., the second AI agent (832) of FIG. 8) may set relationship information about the object using a Chain-of-Thought (CoT) method, just as in the case where relationship information about the object is automatically set. In the case of the manual person definition process, the first AI agent (831) and the second AI agent (832) can obtain relatively more accurate information compared to the case where relationship information about the object is automatically extracted. In the case of a manual person definition process, the first AI agent (831) and the second AI agent (832) can extract information relatively more efficiently, and the conversation between the first AI agent (831) and the second AI agent (832) can proceed relatively quickly. In the case of a manual person definition process, the time required to finally establish relationship information regarding the object may be shorter.

[0101] FIG. 8 is a block diagram showing a configuration for setting relationship information regarding an object according to one embodiment.

[0102] In FIG. 8, the electronic device (101) may include or be composed of at least one of a query correction model (810), an AI agent management model (820), an AI agent unit (830), or an augmented content generator (840). The electronic device (101) is based on a multi-agent structure and utilizes an AI agent management model (820) that manages each AI agent of the AI ​​agent unit (830) to enable each AI agent to efficiently distribute and perform tasks required to set relationship information regarding an object. Once relationship information regarding an object is set, the electronic device (101) can provide convenience functions by providing augmented content corresponding to the registered object for which relationship information regarding the object has been set.

[0103] In one embodiment, the AI ​​agent management model (820) and / or the AI ​​agent unit (830) may include an AI agent. An AI agent is a program that performs specific tasks based on artificial intelligence and can operate autonomously on behalf of a user and / or other system. The AI ​​agent can analyze data and make decisions by utilizing machine learning and / or natural language processing.

[0104] In one embodiment, a search term filtering model (810) may receive a search term (query) entered through an input window (e.g., a search window), analyze the search term, and output a filtered result. For example, a user may search for an object in the main search window of the electronic device (101) (e.g., displayed by a slide-up or slide-down gesture input while running a home application) and / or in the search window of an application (e.g., a gallery application, a messenger application, and / or a social media application). The search term may be input data in the form of natural language entered by the user. The search term may be text data. The search term filtering model (810) may receive search terms in various languages. The search term filtering model (810) may take image data and / or text data as input and output text data. The search term filtering model (810) may filter the search term (query) using an artificial intelligence model and output filtered text data. The search term filtering model (810) may filter text corresponding to an object using an artificial intelligence model. The output text data may include words or sentences in the form of natural language. The search term filtering model (810) may receive image data or text data, or receive both different types of data simultaneously. The search term filtering model (810) may receive text data describing the image data along with the image data. Text data describing the image data may be obtained through image captioning. The search term filtering model (810) may receive text data converted from voice data.

[0105] In one embodiment, the search term filtering model (810) may analyze the search term independently or analyze the search term based on image data input along with the search term, and correct errors in the search term. The search term filtering model (810) may include a large language model (LLM). The LLM may perform pre-training using a vast amount of corpus data consisting of images and text. For example, the LLM may include, but is not limited to, GPT, LLaMA, and / or Bard.

[0106] In one embodiment, the AI ​​agent management model (820) may refer to an AI agent capable of managing subordinate AI agents. The AI ​​agent management model (820) may assign tasks to subordinate AI agents so that they can set relationship information regarding objects, and may collect data output by each subordinate AI agent. The subordinate AI agents may be AI agents (831, 832) included in the AI ​​agent unit (830). The AI ​​agent management model (820) may manage the AI ​​agents (831, 832) within a cyclic structure in which the AI ​​agents (831, 832) update information.

[0107] In one embodiment, the AI ​​agent management model (820) may receive a command agenda received from a user, a rule for controlling the AI ​​agent unit (830), and / or output data of the AI ​​agent unit (830). The command agenda may include a request instructing the setting of relationship information regarding an object based on the acquisition of identification information of the object (e.g., the name of the object and / or relationship designation). The rule for controlling the AI ​​agent unit (830) may include a termination condition indicating that the setting of relationship information regarding the object is terminated. The rule for controlling the AI ​​agent unit (830) may be set in advance. The rule for controlling the AI ​​agent unit (830) may be set by user input. The output data of the AI ​​agent unit (830) may include output data of AI agents (831, 832) included in the AI ​​agent unit (830). The AI ​​agent management model (820) can obtain a matching score (or similarity score) by comparing at least a portion of the image area corresponding to the object with pre-set verification data, and determine that the termination condition is satisfied based on whether the obtained matching score is above a threshold. A case can be assumed where the user manually inputs the relationship title regardless of whether clustering is generated. In this case, the pre-set verification data may include the image area corresponding to the object and the relationship title set corresponding to the image area (e.g., dad, mom, uncle, aunt, brother). There is a trade-off between the speed of setting relationship information regarding the object and the accuracy of the relationship information regarding the object, and the termination condition can be adjusted by the AI ​​agent management model (820) or by user input to select whether to prioritize speed or accuracy.

[0108] In one embodiment, the AI ​​agent management model (820) may instruct subordinate AI agents (e.g., first AI agent (831) and second AI agent (832)) to perform tasks to obtain information about an object during the nth cycle (n is 1 or more). During the nth cycle, the subordinate AI agents (e.g., first AI agent (831) and second AI agent (832)) may sequentially analyze the data they manage using the Chain-of-Thought (CoT) method and perform tasks instructed by the AI ​​agent management model (820). For example, if a user enters a query such as "I had a barbecue with Uncle John at the campsite..." through an input window (e.g., a search window or a chat window), the first AI agent (831) may use a large language model (LLM) to search for images associated with the query (e.g., camping photos and / or barbecue photos) in the user's first personal database (e.g., an image database in a gallery application). If an image associated with a search term is found, the first AI agent (831) may request the second AI agent (832) to extract information associated with the image (e.g., camping schedule) from the user's second personal database. The user's personal database may store information such as images, voice, and / or location information. When the nth cycle ends, the subordinate AI agents (831, 832) output the work results to the AI ​​agent management model (820), and the AI ​​agent management model (820) may determine whether the termination condition is satisfied based on the work results.

[0109] In one embodiment, the AI ​​agent management model (820) can determine the progress of the work performed by the AI ​​agents (831, 832) based on the output data of the AI ​​agent unit (830). Based on the output data of the AI ​​agent unit (830), if the termination condition of the work performed by the AI ​​agents (831, 832) is not satisfied, the AI ​​agent management model (820) can output a command prompt to be delivered to each of the AI ​​agents (831, 832). If the termination condition of the work performed by the AI ​​agents (831, 832) is satisfied, the AI ​​agent management model (820) can output the result obtained by the AI ​​agents (831, 832) (e.g., relationship information regarding an object) in text form.

[0110] In one embodiment, the AI ​​agent management model (820) may be based on a multi-agent architecture responsible for task instructions and progress tracking to manage AI agents (831, 832). The AI ​​agent management model (820) may plan to process tasks and collect data required for each operation cycle using data required for the task and a model trained through pre-training. The AI ​​agent management model (820) may obtain task progress for each cycle of the operation plan of the AI ​​agents (831, 832) and provide feedback thereon. The AI ​​agent management model (820) may check whether the AI ​​agents (831, 832) have satisfied the task completion conditions as they were pre-trained (or as specified by the user). If the AI ​​agents (831, 832) have not completed the task, the AI ​​agent management model (820) may assign the task to some or all of the AI ​​agents (831, 832) to complete. The AI ​​agent management model (820) can update information based on the output data of the AI ​​agent (831 and / or 832) when the AI ​​agent (831 and / or 832) completes the task, and can repeat this action until the entire task is completed. If the AI ​​agent management model (820) determines that the AI ​​agents (831, 832) do not satisfy the task completion conditions, it can establish a new plan (e.g., obtaining other information about the object and image clustering using the other information obtained) and have the AI ​​agents (831, 832) re-execute the task.For example, the AI ​​agent management model (820) can transmit tasks (e.g., setting relationship information regarding an object) to subordinate AI agents (e.g., first AI agent (831) and second AI agent (832)) and determine whether a task termination condition (e.g., the face feature point matching score of a target object among images in a gallery application is above a threshold) is satisfied based on the results that the subordinate AI agents (831, 832) reply to after the task. The subordinate AI agents (831, 832) can perform specialized functions (e.g., in the case of the first AI agent (831), detecting image regions corresponding to the object and / or extracting face feature points) based on the task transmitted by the AI ​​agent management model (820) and provide data so that the AI ​​agent management model (820) can determine whether the termination condition is satisfied.

[0111] In one embodiment, the AI ​​agent unit (830) may include one or more AI agents. For example, the AI ​​agent unit (830) may include at least one of a first AI agent (831) or a second AI agent (832), or may be composed of these. The first AI agent (831) may manage the user's first personal database (e.g., an image database of a gallery application). The second AI agent (832) may manage the user's second personal database (e.g., a database of all applications installed on the electronic device (101). Each AI agent (831 and 832) may perform tasks independently depending on the type of applications or data types installed on the electronic device (101). When using AI agents that independently perform specialized functions, processing various types of data may be relatively more effective compared to using AI models that perform only specific purposes. Each AI agent (831 and 832) may perform training using a predefined dataset. In this case, the first and second AI agents (831 and 832) can adaptively perform special tasks according to the input data. The AI ​​agent unit (830) can activate the AI ​​agents (831, 832) according to the situation and provide a pair of training data appropriate to the situation in order to perform tasks for different applications and / or functions according to the user's request.

[0112] In one embodiment, the first AI agent (831) can establish relationship information regarding an object based on images stored in the user's first personal database (e.g., images from a gallery application). The first AI agent (831) can perform clustering using images stored in the user's first personal database (e.g., images from a gallery application). The first AI agent (831) can accurately cluster people and / or animals by performing pre-training using training data pairs in which various people and / or animals are matched with specific numbers. The first AI agent (831) can cluster image regions corresponding to objects in the user's first personal database. The first AI agent (831) can identify features of image regions corresponding to objects (e.g., features of the object's face).

[0113] In one embodiment, the second AI agent (832) can manage raw data received by the electronic device (101). The second AI agent (832) can manage all raw data that may be obtained through a communication module (e.g., Wi-Fi and / or Bluetooth) and / or an external electronic device (e.g., a wearable device) while the user performs a specific task through the electronic device (101) or obtains and / or generates specific information using an application. The user's personal database in which the raw data is stored may be referred to as the second personal database. The second AI agent (832) can manage the second personal database. The second AI agent (832) can efficiently manage data through a data processing unit (DPU) instead of a central processing unit (CPU).

[0114] In one embodiment, the second AI agent (832) may receive a data extraction request signal. The second AI agent (832) may receive a data extraction request signal when a query regarding an object for which relationship information is not established is detected. In response to the receipt of the data extraction request signal, the second AI agent (832) may extract all data associated with the query and / or images regarding the object from the user's personal database. The second AI agent (832) may use a large language model (LLM) to extract data from the user's personal database. The LLM may be one of the modules constituting the user's personal database. The second AI agent (832) may transmit words entered by the user to the LLM in the form of a prompt. The LLM may obtain, for example, images, text, voice, time information, location information, and / or health information from the user's personal database. Since the AI ​​agent management model (820) operates based on a circular structure and can reference data acquired by the second AI agent (832), the data acquired by the second AI agent (832) can be stored and managed in the form of a JSON file to facilitate management. The second AI agent (832) can extract information about objects from the user's personal database in the form of a JSON file. For example, the data acquired by the second AI agent (832) can be stored and managed in the following file format.

[0115] "Image"=[DCIM / users / sub_img_1.jpg, DCIM / users / sub_img2.jpg, … ]

[0116] "Text"={"Kakao": 2024_10_30_talk.txt, 2024_11_02_talk.txt

[0117] "Instagram": 2024_10_29_dm.txt, 2024_11_13_dm.txt}

[0118] “GeoDB”=[…] / users / 20241013_loc.db, … / users / 20241017_loc.db]

[0119] In one embodiment, since the data acquired by the second AI agent (832) is managed in the form of a JSON file, it may be easy to modify the original data stored in the user's personal database. Since the data acquired by the second AI agent (832) is used exclusively when setting relationship information regarding an object, there is no need to include the original data. In this case, the memory usage required when setting relationship information regarding an object can be minimized.

[0120] In one embodiment, the second AI agent (832) can transmit the acquired data to the AI ​​agent management model (820). The AI ​​agent management model (820) can analyze the received data (e.g., classify data types) and transmit the data along with work details to each of the AI ​​agents (831 and 832).

[0121] In one embodiment, the augmented content generator (840) may provide a user interface that activates and displays content (e.g., text or image) entered by a user using relationship information regarding an object obtained through the AI ​​agent unit (830). The augmented content generator (840) may notify the user that relationship information regarding an object has been set.

[0122] In one embodiment, when relationship information regarding an object is set, the augmented content generator (840) may receive text data (relationship information regarding an object) output by the AI ​​agent management model (820) and / or the AI ​​agent unit (830). The augmented content generator (840) may output augmented content using a generative AI model. The generative AI model may perform pre-training using a dataset consisting of pairs of relationship information regarding an object (e.g., text-to-text (or image)) and user interface images (e.g., application-applied design-based user interface images). The augmented content generator (840) may receive text data (relationship information regarding an object) and a query and / or image, and output a result that mixes user interface elements compatible with the query and / or image.

[0123] In one embodiment, a user interface element generated by an augmented content generator (840) can not only provide visual information to the user but also receive user input (e.g., touch input). User input to the user interface element can enable performing operations on the object.

[0124] In one embodiment, the augmented content generator (840) can augment identification information of a registered object (e.g., the name and / or relationship designation of the object) to provide augmented content (e.g., an augmented text user interface (UI) element). The augmented content element may include text corresponding to the object (e.g., the name and / or relationship designation of the object) and / or an image corresponding to the object. In one embodiment, the augmented content generator (840) can augment text corresponding to the registered object onto an image corresponding to the object to provide an augmented text UI element. When user input (e.g., touch, sliding) is received for the augmented text UI element, a suggestion to additionally verify or supplement information about the object, such as 'search in gallery', 'register image in phonebook', and / or 'search on SNS', may be provided.

[0125] In one embodiment, the electronic device (101) may additionally include a personal DB updater (not shown) and / or an input box manager (not shown). The personal DB updater receives data acquired by the AI ​​agent unit (830) and can update (or update) relationship information regarding objects. The input box manager can control an input box (e.g., a search box and / or a chat box). The input box manager can receive a search term entered into the input box. If the search term contains text corresponding to a registered object, the input box manager can display an augmented text UI element acquired using an augmented content generator (840).

[0126] FIG. 9 illustrates a block diagram of an AI agent (900) according to one embodiment.

[0127] In FIG. 9, the AI ​​agent (900) may include or be composed of at least one of a configurator module (910), a memory module (920), a perception module (930), an execution module (940), or a critique-reflection module (950). For example, the AI ​​agent (900) may be a management AI agent (the AI ​​agent management model (820) of FIG. 8). The AI ​​agent (900) may be the first AI agent (831) and / or the second AI agent (832) of FIG. 8.

[0128] In one embodiment, the configuration module (910) can set the role that the AI ​​agent (900) will perform within the device. The configuration module (910) can set initial conditions that cause the AI ​​agent (900) to perform an operation.

[0129] In one embodiment, the memory module (920) may provide a function to store and retrieve data change and / or generation processes. The memory module (920) may acquire data flow processes within the AI ​​agent (900) as data, thereby enabling the AI ​​agent (900) to share data with other AI agents.

[0130] In one embodiment, the recognition module (930) can observe the situation regarding data that changes during the operation of the AI ​​agent (900) and obtain the situation regarding the changing data as text data.

[0131] In one embodiment, the execution module (940) can substantially execute the action that the AI ​​agent (900) has decided to execute.

[0132] In one embodiment, the feedback module (950) can update the model parameters on its own. The AI ​​agent (900) can converse with other AI agents (900). Through the conversation, the feedback module (950) can provide feedback to the AI ​​agent (900) to identify problems that occur during operation and to ensure that the operation runs smoothly. For example, a management AI agent (the AI ​​agent management model (820) of FIG. 8) can receive output data from the AI ​​agents (831, 832) and determine whether the termination condition is satisfied.

[0133] FIG. 10 illustrates a method for setting up an application used to set up relationship information regarding an object, according to one embodiment.

[0134] In FIG. 10, an electronic device (e.g., the electronic device (101) of FIG. 1) can set up an application used to set up relationship information regarding an object. The electronic device (101) can receive user input selecting an application used to set up relationship information regarding an object. The user input can add or remove applications used to set up relationship information regarding an object. The electronic device (101) can receive user input selecting an application used to set up relationship information regarding an object through a user interface. Based on the reception of user input, the electronic device (101) can set up relationship information regarding an object using data from the selected application.

[0135] In one embodiment, the electronic device (101) may be a foldable device. For example, the foldable device may include one or more hinge structures, a foldable housing comprising two or more housing portions rotatably connected by one or more hinge structures, and a flexible display disposed on the front of the foldable housing. For example, as illustrated in FIG. 10, the electronic device (101) may include a first display area and a second display area separated by a single hinge structure.

[0136] In one embodiment, as illustrated in FIG. 10, an electronic device (101) may display data containing identification information of an unregistered object (e.g., image data (1012) containing an image (1011) of an unregistered object) in a first display area (1010). The electronic device (101) may display a user interface in a second display area (1020) indicating that relationship information regarding the unregistered object displayed in the first display area (1010) is being established. For example, the electronic device (101) may display icons (1023: 1023-1, 1023-2, 1023-3) of one or more applications used to establish relationship information regarding the object in the second display area (1020). The electronic device (101) may inform the user through the user interface that one or more applications used to establish relationship information regarding the object can be selected. For example, the user interface may include messages (1022) such as “Defining person relationships. Please wait a moment” and / or “You can add or remove applications for defining person relationships”.

[0137] In one embodiment, the electronic device (101) may receive user input approving the addition of an application icon (1023) for defining person relationships. For example, as illustrated in FIG. 10, the electronic device (101) may provide a user interface element (e.g., "Add Application" button (1022)) for receiving user input approving the addition of an application icon (1023). The electronic device (101) may identify a touch on the user interface element (1022) as user input approving the addition of an application icon (1023). The electronic device (101) may set relationship information regarding an object using data of the application corresponding to the added application icon.

[0138] In one embodiment, the electronic device (101) may receive user input approving the deletion of an application icon (1023) for defining person relationships. For example, as illustrated in FIG. 10, the electronic device (101) may identify a long touch on the application icon (1023) as user input approving the deletion of the corresponding application icon (1023). For example, as illustrated in FIG. 10, the electronic device (101) may provide a user interface element (e.g., a "-" button (1024)) for receiving user input approving the deletion of an application. The electronic device (101) may identify a touch on the user interface element (1024) as user input approving the deletion of the corresponding application icon (1023-2). The electronic device (101) may set relationship information regarding an object without using the data of the application corresponding to the deleted application icon.

[0139] In one embodiment, the electronic device (101) can establish relationship information regarding an object using data of an application corresponding to an application icon displayed through a user interface. The electronic device (101) allows the user to add or delete applications used to establish relationship information regarding an object through the user interface. The user can increase the reliability of the conversation process between AI agents (e.g., AI agents (831, 832) of FIG. 8) by adding applications. The user can delete applications that provide relatively inaccurate information for establishing relationship information regarding an object. The electronic device (101) can establish relationship information regarding an object based on accurate data by receiving user input to select applications used to establish relationship information regarding an object through the user interface.

[0140] In one embodiment, the electronic device (101) may be a bar type, a sliderable device, and / or a rollable device. The user interface illustrated in FIG. 10 may be provided in a similar manner in a bar type, a sliderable device, and / or a rollable device.

[0141] FIG. 11 illustrates a method for setting relationship information regarding an object according to one embodiment.

[0142] In FIG. 11, an electronic device (e.g., the electronic device (101) of FIG. 1) can establish relationship information regarding an object. The electronic device (101) can establish relationship information regarding an object based on at least one of information regarding an object, such as a name, nickname, relationship title, SNS ID, image, voice, contact (e.g., phone number), or information regarding activities between the object and the user. The electronic device (101) can establish relationship information regarding an object based on various data received by the electronic device (101).

[0143] In one embodiment, the electronic device (101) can detect identification information of the object (1110) while the user (1120) is making a call with the object (1110) through a call application installed on the electronic device (101). For example, the identification information of the object (1110) may include voice. The electronic device (101) can determine whether data matching the voice is stored based on the detection of the voice of the object (1110). The electronic device (101) can identify the characteristics of the voice of the object (1110). The characteristics of the voice may be represented as a voice feature vector. The electronic device (101) can extract the voice feature vector of the voice of the object (1110). The electronic device (101) can determine whether data matching the voice feature vector of the object (1110) is stored in the user's personal database. The electronic device (101) can check whether there exists a voice feature vector among the voice feature vectors stored in the user's personal database that has a similarity to the voice feature vector of the object (1110) that is greater than or equal to a certain threshold value. Based on the determination that data matching the voice of the object (1110) is not stored in the user's personal database, the electronic device (101) can obtain first information related to the relationship with the object (1110) using data from a voice-containing call application. The data from the voice-containing call application may include data regarding the call, for example, the phone number of the object (1110) and / or the content of the call. For example, the content of the call may include the utterance, "Hi, it's Kevin who went out with OO yesterday, took pictures, and had dinner. Do you remember?" (1112). For example, the electronic device (101) can obtain at least one of identification information of an object (1110), such as the name of the object (1110) or information regarding an activity between the object (1110) and the user (1120) (e.g., taking a picture with OO) from the content of the call (1112).The electronic device (101) can obtain information about an object (1110), including identification information (voice) of the object (1110) and first information (e.g., name or phone number), from data regarding a call between the object (1110) and the user (1120). The electronic device (101) can store information about the object (1110). For example, the electronic device (101) can store information about the object (1110) in a table format. For example, as illustrated in the table (1130), the electronic device (101) can obtain and store information about the object (1110), such as the name, relational title, and voice of the object (1110), from data regarding a call between the object (1110) and the user (1120). Since the amount and type of data obtainable through a call are limited, the electronic device (101) can obtain relationship information regarding an object from the data regarding the call and store the obtained relationship information regarding the object in a database to set it to an initial state. The electronic device (101) can additionally obtain information regarding the object (e.g., social media account, interests, age, and / or gender). The electronic device (101) can update the relationship information regarding the object based on the additionally obtained information.

[0144] FIG. 12 illustrates a method for providing recommendation information according to one embodiment.

[0145] In FIG. 12, an electronic device (e.g., the electronic device (101) of FIG. 1) may provide a user interface that includes recommendation information for an object based on relationship information regarding the object.

[0146] In one embodiment, the electronic device (101) may acquire data used in a specific organization or group. In a specific organization or group, communication may be conducted by assigning a separate name to specific objects or individuals according to internal regulations. For example, this applies to a company distinguishing job titles as Manager and Deputy Manager. The electronic device (101) may identify the name used in the specific organization or group as the object's identification information from the acquired data. The electronic device (101) may use the name to search for and collect information regarding the name from the data used in the specific organization or group. For example, the information regarding the name may include at least one of qualifications and / or exams that a person with the name can acquire, relevant personnel information, essential technical terms, or information regarding business meetings that can be attended. The electronic device (101) may establish relationship information regarding the name based on the name used in the specific organization or group and the information regarding the name. Based on the relationship information regarding the name, the electronic device (101) may provide a user interface that includes recommendation information for the name. If there are no documents regarding relationship information regarding names, personnel newly joining a specific organization or group may find it difficult to identify information regarding names within that organization or group. The electronic device (101) can use multi-AI agents to obtain relationship information regarding names from documents and / or images where information regarding names is scattered, and provide recommendation information to the object to which the name is assigned.

[0147] For example, as illustrated in the screen (1210) of FIG. 12, the electronic device (101) can acquire data (1211) used by a specific organization or group. For example, the data (1211) may include information regarding qualifications and / or exams, personnel information, technical terms, or business meetings. The electronic device (101) can provide the data (1211) to a user who has logged in with a user account (or has been granted access to the data of the specific organization or group). For example, the user can log in to the electronic device (101) using an ID and password.

[0148] In one embodiment, as illustrated in the screen (1220) of FIG. 12, the electronic device (101) can identify a user logged in with a user account. The electronic device (101) can obtain user identification information (1222) based on information registered in the user account. For example, the user identification information (1222) may include at least one of the group (or team) to which the user belongs (e.g., AI team), job (e.g., AI model training), or title (or rank) (e.g., OO engineer). Based on detecting the user identification information (1222), the electronic device (101) can provide recommendation information corresponding to the user identification information (1222). In this case, the electronic device (101) can provide a user interface that asks the user whether they want recommendation information. For example, as shown in the screen (1220) of FIG. 9, the user interface may include a message (1221) such as "User information has been verified. Would you like to regenerate data based on user information?" and / or user identification information (1222).

[0149] In one embodiment, the electronic device (101) may provide a user interface that provides recommendation information based on relationship information regarding an object. The electronic device (101) may obtain relationship information regarding user identification information (1222). The electronic device (101) may provide recommendation information based on relationship information regarding user identification information through the user interface. For example, the recommendation information may include at least one of qualifications and / or exams that can be obtained, relevant personnel information, required technical terms, or information regarding business meetings that can be attended. For example, as illustrated in the screen (1230) of FIG. 12, the user interface may include at least one of first recommendation information (1231) regarding qualifications and / or exams, second recommendation information (1232) regarding technical terms, or third recommendation information (1233) regarding personnel information. For example, the first recommendation information (1231) may include a message such as "You can obtain the following qualifications and exams: best reviewer, and / or associate architecture." For example, the second recommendation information (1232) may include a message such as “The following terms may be required to perform AI model training tasks,” “Backbone: refers to an encoder structure for extracting feature vectors from an AI model,” and / or “Fine-tuning: refers to retraining some of the model's parameters using a separately prepared dataset to operate a pre-trained AI model for user purposes.” For example, the third recommendation information (1233) may include a message such as “To be promoted from the OO Engineer position to the next position, you must satisfy the following requirements.”

[0150] In one embodiment, the electronic device (101) may enable a user to participate in the communication process between AI agents (e.g., the AI ​​agent management model (820) of FIG. 8 and / or AI agents (831, 832)). The electronic device (101) may use a knowledge graph to display the data flow of a first private database (e.g., an image database of a gallery application) and a second private database (e.g., an image database of all applications installed on the electronic device (101)) exchanged during the communication process between AI agents. The electronic device (101) may allow the user to directly control the data flow exchanged during the communication process between AI agents. In this case, the time required for AI agents to output results may be reduced, and the reliability of the results output by AI agents may be higher.

[0151] FIG. 13 is a block diagram of a generative artificial intelligence (AI) system (1300) according to one embodiment.

[0152] In FIG. 13, the generative AI system (1300) may include a user interface (1310), an AI framework (1320), a generative AI model (1330), a knowledge repository (1340), and an application / service module (1350). These components may be operated on one or more of an electronic device (101), an external electronic device (102 or 104), or a server (108). For example, the user interface (1310) and the AI ​​framework (1320) may be operated on the electronic device (101), and the knowledge repository (1340) and the generative AI model (1330) may be operated on the server (108).

[0153] According to one embodiment, a user interface (1310) may receive user input (e.g., user query). User input may be received in the form of text, images, voice (e.g., natural language), video, menu selection, or a combination thereof. The user interface (1310) may include various context information (e.g., running application or user location) related to the generative artificial intelligence system (1300) at the time the user input is received, in addition to or instead of the user input. The user interface (1310) may provide the user input or the context information to the AI ​​framework (1320) and provide the result of processing therefrom to the user, for example, through the AI ​​framework (1320). According to one embodiment, in addition to user input, the electronic device may provide context information obtained using information included on the screen to the AI ​​framework (1320). The result may be provided in the form of text, images, voice, video, an action requested by the user (e.g., execution of a specified function or app), or a combination thereof.

[0154] According to one embodiment, the AI ​​framework (1320) can identify (e.g., estimate) a user intent based on at least some user input or context information received from a user interface (1310), control each of the relevant modules (e.g., 1321, 1323, or 1325) to perform a function or action corresponding to the identified user intent, and coordinate collaboration between two or more modules. The AI ​​framework (1320) may include a prompt design module (1321), an API / plugin management module (1323), and an output modification module (1325), as illustrated in FIG. 13.

[0155] According to one embodiment, the prompt design module (1321) can generate a prompt to be input to a generative AI model (1330) based at least partially on user input or context information received from a user interface (1310). For example, the prompt design module (1321) can generate a prompt using user preferences, a prompt library, or prompt examples stored in a knowledge repository (1340) based at least partially on user input or context information.

[0156] According to one embodiment, an API / plug-in management module (1323) may communicate, for example, via an API, with various resources (e.g., a knowledge repository (1340)) that provide said additional information when there is a request for said additional information in relation to user input. Additionally or alternatively, when a specified action (e.g., a function, app, or service) is performed in response to said user input, the API / plug-in management module (1323) may request the application / service module (1350) to perform said specified action via a corresponding API. The API / plug-in management module (1323) may provide information obtained from the knowledge repository (1340), the application / service module 1350, or another external resource to the prompt design module (1321). The obtained information may be used by the prompt design module (1321) to generate a prompt together with the user input, or provided to a generative AI model (1330).

[0157] According to one embodiment, an output processing module (1325) can fine-tune the results obtained through a generative AI model (1330) as at least part of the response to a user input (e.g., user query). For example, the output processing module (1325) can determine whether the content of the response obtained through the generative AI model (1330) is appropriate as a response to a request made by the user input. For example, the output processing module (1325) can determine the degree of relevance of the difference between the response obtained through the generative AI model (1330) and the user input, the degree of bias (e.g., political or social bias), or the degree of harmfulness (e.g., sexual or profanity). Additionally or generally, the output processing module (1325) can request that additional AI processing be performed on the obtained response, or provide the user with a hint to avoid unwanted output. For example, the response can be obtained again through the generative AI model (1330) by generating additional prompts through a prompt design module.

[0158] According to one embodiment, the generative AI model (1330) may form at least part of an artificial intelligence neural network and may include a model that generates images or a model that generates language. The image generation model may include, for example, a generative adversarial network (GAN), a variational autoencoder (VAE), or a Diffusion-based model using a VAE and a Transformer. The language generation model may include, for example, a large language model (LLM), a large multimodal model (LMM), a large vision model (LVM), or a large action model (LAM). The LAM may automatically generate actions for an environment (e.g., a robot, a car, an electronic device (101), or a program (140)). Additionally, for at least some AI models (e.g., LLM), there may be a low-rank adaptation (LoRA) adapter fine-tuned for, for example, a specific task or a specific situation.

[0159] FIG. 14 is a block diagram of an AI framework (1320) according to one embodiment.

[0160] In FIG. 14, the AI ​​framework (1320) may have on-device AI processing capabilities. In this case, the AI ​​framework (1320) may generate and learn a response to the user input using resources within the device, instead of sending the user input received through a user interface (1310) operating on the same device (e.g., electronic device (101)) to a generative AI model (1330) operating on an external device (e.g., server (108)), or additionally. The AI ​​framework (1320) may include a cross-application action module (1410), a personal data management module (1430), an on-device AI model (1450), and / or an orchestration module (1470).

[0161] According to one embodiment, the cross-application action module (1410) determines one or more additional applications required for the operation of an executed application (e.g., an assistant app) and may connect or suggest operations between the app and at least one additional application, or between a plurality of additional applications. For example, the cross-application action module (1410) may execute one or more additional applications to be used to respond to a user request through the assistant app sequentially or at least partially and simultaneously. Additionally, the cross-application action module (1410) may communicate with the additional applications so that the result of the execution of one additional application (e.g., content) can be shared with other additional applications.

[0162] According to one embodiment, the personal data management module (1430) may provide personal information (e.g., schedule, contact, or message information) about a user of the application (e.g., assistant app) or the additional application running on the device (e.g., electronic device 101) or other related individuals (e.g., family or friends) to another module of the AI ​​framework (1320) or a related module (e.g., generative AI model (1330)) running on another device.

[0163] According to one embodiment, the on-device AI model (1450) may include at least one model among one or more AI models (e.g., GAN, VAE, LLM, LMM, LVM, or LAM) operated on an external device (e.g., server (108)) or a corresponding lightweight AI model. Additionally, for said model or said lightweight model, there may be, for example, a LoRA adapter.

[0164] According to one embodiment, the orchestration module (1470) may select one or more AI models to be used to obtain a response to user input (e.g., user query). For example, the orchestration module (1470) may select one or more AI models from an on-device AI model (1450), an AI model operating on an external device (e.g., server (108)) (e.g., generative AI model (1330)), or a 14th AI model (not shown) operating on another external device. When multiple AI models are selected, the orchestration module (1470) may communicate with the selected models or devices so that the operation between the selected AI models and the processing of the results thereof can be coordinated between the relevant models or devices.

[0165] According to one embodiment, two or more modules of a generative AI system (1300) (e.g., a cross-application action module (1410) and an orchestration module (1470)) may be implemented as a single module to maintain the same functionality. Various variations are possible.

[0166] The ‘artificial intelligence model’ in the present disclosure may be identical or similar to the configuration of the generative AI model (1330) of FIG. 13 or the on-device AI model (1450) of FIG. 14, or may be an artificial intelligence model included in the generative AI model (1330) of FIG. 13 or the on-device AI model (1450) of FIG. 14. In the present disclosure, the ‘artificial intelligence system’ may be the generative artificial intelligence system (1300) of FIG. 13.

[0167] In one embodiment, the electronic device (101) comprises a memory (130) including at least one storage medium for storing instructions; and at least one processor (120) including a processing circuit, and when the instructions are executed individually or collectively by at least one processor, the electronic device may be caused to perform: an operation of obtaining identification information of a subject through a first application; an operation of determining whether to set relationship information regarding the subject based on the acquisition of the identification information of the subject; an operation of obtaining first information associated with a relationship with the subject using data of the first application including the identification information based on the determination that relationship information regarding the subject is set; an operation of obtaining at least one image including an image region corresponding to the subject using the first information; an operation of obtaining relationship information regarding the subject using the identification information of the subject and the at least one image; and an operation of storing the relationship information regarding the subject.

[0168] In one embodiment, when the instructions are executed individually or collectively by at least one processor, the electronic device may be caused to perform: an operation of obtaining data of a second application including second information associated with a relationship with the object using the first information and / or information regarding the at least one image; and an operation of obtaining relationship information regarding the object using the second information further.

[0169] In one embodiment, the operation of obtaining relationship information regarding the object by additionally using the second information may include comparing information regarding the at least one image and the second information to determine whether the at least one image includes the image region corresponding to the object.

[0170] In one embodiment, when the instructions are executed individually or collectively by at least one processor, the electronic device may be caused to perform an operation of providing a user interface capable of receiving user input for selecting an application used to obtain the relationship information regarding the object based on the acquisition of the first information.

[0171] In one embodiment, the operation of acquiring at least one image including an image region corresponding to the object using the first information may include: acquiring an image associated with the first information from a user's personal database using the first information; providing a user interface including the image associated with the first information; receiving user input for identifying the image region corresponding to the object in the image associated with the first information; and identifying the image region corresponding to the object based on the reception of the user input for identifying the image region corresponding to the object.

[0172] In one embodiment, when the instructions are executed individually or collectively by at least one processor, the electronic device may be caused to perform: an operation of providing a user interface capable of receiving a user input selecting at least one of the identification information of the object or an image region corresponding to the object based on the fact that the relationship information regarding the object is stored; an operation of providing a user interface including one or more user interface elements regarding a proposal based on the fact that the user input selecting at least one of the identification information of the object or an image region corresponding to the object is received; and an operation of executing a proposal corresponding to the selected user interface element based on the fact that the user input selecting at least one of the one or more user interface elements regarding the proposal is received.

[0173] In one embodiment, when the instructions are executed individually or collectively by at least one processor, the electronic device may be caused to perform an operation of providing a user interface that includes recommendation information for the object based on the relationship information regarding the object.

[0174] In one embodiment, the operation of acquiring at least one image including an image region corresponding to the object using the first information includes the operation of searching for the at least one image associated with the content of the first information in the user's first personal database using a first AI (artificial intelligence) agent, and the operation of acquiring relationship information regarding the object using the identification information of the object and the at least one image may include the operation of searching for data associated with at least one of the identification information of the object or the at least one image in the user's second personal database using a second AI agent.

[0175] In one embodiment, when the instructions are executed individually or collectively by at least one processor, the electronic device may be caused to perform: an operation of obtaining a matching score by comparing at least a portion of the image region corresponding to the object with verification data; and an operation of terminating the operation of obtaining the relationship information regarding the object based on the matching score being greater than or equal to a threshold.

[0176] In one embodiment, the relationship information regarding the object may include at least one of the object's name, nickname, relationship title, SNS ID, image, voice, contact, phone number, information regarding activities between the object and the user, interests, age, or gender.

[0177] In one embodiment, the control operation of the electronic device (101) may include: an operation of obtaining identification information of a subject through a first application; an operation of determining whether to set relationship information regarding the subject based on the acquisition of the identification information of the subject; an operation of obtaining first information associated with a relationship with the subject using data of the first application including the identification information based on the determination that relationship information regarding the subject is set; an operation of obtaining at least one image including an image area corresponding to the subject using the first information; an operation of obtaining relationship information regarding the subject using the identification information of the subject and the at least one image; and an operation of storing the relationship information regarding the subject.

[0178] In one embodiment, the control method may include: an operation of obtaining data of a second application including second information associated with a relationship with the object using the first information and / or information regarding the at least one image; and an operation of obtaining relationship information regarding the object using the second information further.

[0179] In one embodiment, the control method may include an operation of obtaining relationship information regarding the object by additionally using the second information, which involves comparing information regarding the at least one image with the second information to determine whether the at least one image includes the image region corresponding to the object.

[0180] In one embodiment, the control method may include an operation of providing a user interface capable of receiving a user input for selecting an application used to acquire the relationship information regarding the object based on the acquisition of the first information.

[0181] In one embodiment, the operation of acquiring at least one image including an image region corresponding to the object using the first information may include: acquiring an image associated with the first information from a user's personal database using the first information; providing a user interface including the image associated with the first information; receiving user input for identifying the image region corresponding to the object in the image associated with the first information; and identifying the image region corresponding to the object based on the reception of the user input for identifying the image region corresponding to the object.

[0182] In one embodiment, the control method may include: providing a user interface capable of receiving a user input selecting at least one of the identification information of the object or an image area corresponding to the object based on the fact that the relationship information regarding the object is stored; providing a user interface including one or more user interface elements regarding a proposal based on the fact that the user input selecting at least one of the identification information of the object or an image area corresponding to the object is received; and executing a proposal corresponding to the selected user interface element based on the fact that the user input selecting at least one of the one or more user interface elements regarding the proposal is received.

[0183] The above control method may include an operation of providing a user interface that includes recommendation information for the object based on the relationship information regarding the object.

[0184] The above control method includes, in the operation of acquiring at least one image including an image region corresponding to the object using the first information, an operation of searching for the at least one image associated with the content of the first information in the user's first personal database using a first AI agent, and in the operation of acquiring relationship information regarding the object using the identification information of the object and the at least one image, an operation of searching for data associated with at least one of the identification information of the object or the at least one image in the user's second personal database using a second AI agent.

[0185] In one embodiment, the control method may include: an operation of obtaining a matching score by comparing at least a portion of the image area corresponding to the object with verification data; and an operation of terminating the operation of obtaining the relationship information regarding the object based on the fact that the matching score is greater than or equal to a threshold.

[0186] In one embodiment, the relationship information regarding the object may include at least one of the object's name, nickname, relationship title, SNS ID, image, voice, contact, phone number, information regarding activities between the object and the user, interests, age, or gender.

[0187] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as "coupled" or "connected" to another (e.g., 2nd) component, with or without the terms "functionally" or "communicationly," it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0188] The term “module” as used in the various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0189] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0190] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0191] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In an electronic device (101), Memory (130) including at least one storage medium for storing instructions; and It includes at least one processor (120) including a processing circuit, and When the above instructions are executed individually or collectively by at least one processor, the electronic device: The operation of obtaining identification information of an object (subject) through the first application; An operation to determine whether to set relationship information regarding the object based on the acquisition of the identification information of the object; An operation to obtain first information associated with a relationship with the object using data of the first application including the identification information, based on being identified as establishing relationship information regarding the object; An operation of obtaining at least one image including an image region corresponding to the object using the first information above; An operation to obtain relationship information regarding the object using the identification information of the object and at least one image; and The operation of storing the above relationship information regarding the above object An electronic device that causes to perform.

2. In Paragraph 1, When the above instructions are executed individually or collectively by at least one processor, the electronic device: An operation of obtaining data of a second application including second information associated with a relationship with the object using the first information and / or information regarding the at least one image; and The operation of obtaining relationship information regarding the object by additionally using the second information above. An electronic device that causes to perform.

3. In Paragraph 2, The operation of obtaining relationship information regarding the object by additionally using the second information above is, An operation to determine whether the at least one image includes an image region corresponding to the object by comparing information regarding the at least one image and the second information. An electronic device including 4. In any one of paragraphs 1 through 3, When the above instructions are executed individually or collectively by at least one processor, the electronic device: Operation of providing a user interface capable of receiving user input for selecting an application used to acquire the relationship information regarding the object based on the acquisition of the first information. An electronic device that causes to perform.

5. In any one of paragraphs 1 through 4, The operation of acquiring at least one image including an image region corresponding to the object using the first information above is, The operation of obtaining an image associated with the first information from the user's personal database using the first information; An operation to provide a user interface including an image associated with the first information above; An operation of receiving user input to identify the image region corresponding to the object in an image associated with the first information; and An operation to identify the image region corresponding to the object based on the reception of the user input for identifying the image region corresponding to the object. An electronic device including 6. In any one of paragraphs 1 through 5, When the above instructions are executed individually or collectively by at least one processor, the electronic device: An operation of providing a user interface capable of receiving user input to select at least one of the identification information of the object or an image region corresponding to the object, based on the fact that the relationship information regarding the object is stored; An operation of providing a user interface comprising one or more user interface elements regarding a proposal based on receiving a user input selecting at least one of the identification information of the object or an image region corresponding to the object; and An operation to execute a proposal corresponding to a selected user interface element based on receiving a user input selecting at least one of one or more user interface elements regarding the above proposal. An electronic device that causes to perform.

7. In any one of paragraphs 1 through 6, When the above instructions are executed individually or collectively by at least one processor, the electronic device: Operation of providing a user interface including recommendation information for the object based on the relationship information regarding the object. An electronic device that causes to perform.

8. In any one of paragraphs 1 through 7, The operation of acquiring at least one image including an image region corresponding to the object using the first information above is, The operation of searching for at least one image associated with the content of the first information in the user's first personal database using a first AI (artificial intelligence) agent. Includes, The operation of obtaining relationship information regarding the object using the identification information of the object and at least one image is, The operation of searching for data associated with at least one of the identification information of the object or at least one of the at least one image in the user's second personal database using a second AI agent. An electronic device including 9. In any one of paragraphs 1 through 8, When the above instructions are executed individually or collectively by at least one processor, the electronic device: An operation to obtain a matching score by comparing at least a portion of the image region corresponding to the object with verification data; and An operation to terminate the operation of obtaining the relationship information regarding the object based on the fact that the matching score is greater than or equal to a threshold. An electronic device that causes to perform.

10. In any one of paragraphs 1 through 9, An electronic device comprising at least one of the relationship information regarding the object, the name, nickname, relationship title, social network service (SNS) ID, image, voice, contact, phone number, information regarding activity between the object and the user, interests, age, or gender.

11. In the control operation of the electronic device (101), The operation of obtaining identification information of an object (subject) through the first application; An operation to determine whether to set relationship information regarding the object based on the acquisition of the identification information of the object; An operation to obtain first information associated with a relationship with the object using data of the first application including the identification information, based on being identified as establishing relationship information regarding the object; An operation of obtaining at least one image including an image region corresponding to the object using the first information above; An operation to obtain relationship information regarding the object using the identification information of the object and at least one image; and The operation of storing the above relationship information regarding the above object A control method including 12. In Paragraph 11, An operation of obtaining data of a second application including second information associated with a relationship with the object using the first information and / or information regarding the at least one image; and The operation of obtaining relationship information regarding the object by additionally using the second information above. A control method including 13. In Paragraph 12, The operation of obtaining relationship information regarding the object by additionally using the second information above is, An operation to determine whether the at least one image includes an image region corresponding to the object by comparing information regarding the at least one image and the second information. A control method including 14. In any one of paragraphs 11 through 13, Operation of providing a user interface capable of receiving user input for selecting an application used to acquire the relationship information regarding the object based on the acquisition of the first information. A control method including 15. In any one of paragraphs 11 through 14, The operation of acquiring at least one image including an image region corresponding to the object using the first information above is, The operation of obtaining an image associated with the first information from the user's personal database using the first information; An operation to provide a user interface including an image associated with the first information above; An operation of receiving user input to identify the image region corresponding to the object in an image associated with the first information; and An operation to identify the image region corresponding to the object based on the reception of the user input for identifying the image region corresponding to the object. A control method including

Images