Electronic device for performing user-requested task by using multi-agent, and operating method thereof

Abstract

The present disclosure provides an electronic device. The electronic device can: receive a user query through an input device; allow a first agent from among a plurality of agents to identify a requested task from the user query; decompose the requested task into at least one sub-task; request, from a second agent for managing the metadata about the plurality of agents, metadata about at least one agent capable of processing the at least one sub-task; receive an answer including the metadata from the second agent; configure a multi-agent by selecting, on the basis of the answer, at least one agent for processing the at least one sub-task from among the plurality of agents; perform a natural language dialogue for delegating the at least one sub-task to each agent of the multi-agent; and determine, on the basis of the result of the natural language dialogue, whether to reconfigure the multi-agent.

Description

Electronic device for performing user request tasks using multi-agents and method of operation thereof

[0001] The present disclosure relates to an electronic device that performs a user request task using a multi-agent and a method of operation thereof.

[0002] Large Language Models (LLMs) are artificial intelligence models that learn from massive amounts of text data to understand and generate natural language. LLMs are primarily based on the Transformer architecture, which enables parallel processing and is therefore suitable for training large-scale data. The core of the Transformer is the 'Attention Mechanism,' which allows it to effectively identify relationships between words in input sentences and understand context. Through this, LLMs can perform not only simple sentence generation but also various tasks such as answering complex questions, summarizing documents, and translation.

[0003] LLM is trained using billions of parameters, and these large-scale parameters contribute to learning various patterns and grammatical structures of language. Training requires high-performance computing resources and large datasets, after which the LLM can be further adjusted to suit specific tasks through 'fine-tuning'. Recently, services such as chatbots, content creation, and data analysis utilizing LLM are being provided.

[0004] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art related to the present disclosure.

[0005] An electronic device according to one embodiment of the present disclosure comprises: an input device; a display; a memory comprising a plurality of programs and instructions corresponding to a plurality of agents; and at least one processor comprising a processing circuit; wherein, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be caused to: receive a user query through the input device; for a first agent among the plurality of agents to identify a requested task from the user query and break down the requested task into at least one subtask; for a second agent managing metadata for the plurality of agents to request metadata for at least one agent capable of processing the at least one subtask; receive a response including the metadata from the second agent; for a multi-agent to be formed by selecting at least one agent among the plurality of agents to process the at least one subtask based on the response; perform a natural language conversation to delegate the at least one subtask to each agent of the multi-agent; and determine whether to reconfigure the multi-agent based on the result of the natural language conversation.

[0006] A method of an electronic device according to one embodiment of the present disclosure may include: receiving a user query through an input device; a first agent among a plurality of agents included in the memory of the electronic device identifying a requested task from the user query; breaking down the requested task into at least one subtask; requesting metadata for at least one agent capable of processing the at least one subtask from a second agent managing metadata for the plurality of agents; receiving a response including the metadata from the second agent; forming a multi-agent by selecting at least one agent among the plurality of agents to process the at least one subtask based on the response; performing a natural language conversation to delegate the at least one subtask to each agent of the multi-agent; and determining whether to reconfigure the multi-agent based on the result of the natural language conversation.

[0007] A non-transient computer-readable storage medium recording instructions according to one embodiment of the present disclosure, wherein the instructions, when executed individually or collectively by one or more processors, enable the one or more processors to: receive a user query through an input device; a first agent among a plurality of agents identify a requested task from the user query; break down the requested task into at least one subtask; request metadata for at least one agent capable of processing the at least one subtask from a second agent managing metadata for the plurality of agents; receive a response including the metadata from the second agent; configure a multi-agent by selecting at least one agent among the plurality of agents to process the at least one subtask based on the response; perform a natural language conversation to delegate the at least one subtask to each agent of the multi-agent; and determine whether to reconfigure the multi-agent based on the result of the natural language conversation.

[0008] The above and other aspects, features, and advantages of specific embodiments of the present disclosure may become more apparent from the following description, which is referenced together with the accompanying drawings.

[0009] FIG. 1 illustrates an electronic device in a network environment according to one or more embodiments.

[0010] FIG. 2 shows an LLM-based agent framework according to one embodiment of the present disclosure.

[0011] FIG. 3 illustrates a collaborative operation using a multi-agent of an electronic device according to one embodiment of the present disclosure.

[0012] FIG. 4 is a flowchart illustrating the operation of an electronic device processing a user request task using a multi-agent according to one embodiment of the present disclosure.

[0013] FIGS. 5a and 5b are examples of conversation screens of multiple agents according to one embodiment of the present disclosure.

[0014] FIG. 6 is a flowchart illustrating an operation to search for an agent of an electronic device according to one embodiment of the present disclosure.

[0015] FIG. 7 is an example of agent groups according to one embodiment of the present disclosure.

[0016] FIG. 8 is a flowchart illustrating the operation of processing a user request task of an electronic device according to one embodiment of the present disclosure.

[0017] FIG. 9 is an example of an activation agent group according to one embodiment of the present disclosure.

[0018] FIG. 10 is a flowchart illustrating the operation of processing a user request task of an electronic device according to one embodiment of the present disclosure.

[0019] FIG. 11 is a flowchart illustrating an agent generation operation of an electronic device according to one embodiment of the present disclosure.

[0020] FIG. 12a is an example of an agent selection screen of an electronic device according to one embodiment of the present disclosure.

[0021] FIG. 12b is an example of a new agent creation screen of an electronic device according to one embodiment of the present disclosure.

[0022] FIG. 12c is an example of a user agent list screen of an electronic device according to one embodiment of the present disclosure.

[0023] FIG. 13 is an example of a tool call operation of LLM-based agents according to one embodiment of the present disclosure.

[0024] FIG. 14a is an example of a conversation screen of a user configuration agent team according to one embodiment of the present disclosure.

[0025] FIG. 14b is an example of an action screen of a user configuration agent team according to one embodiment of the present disclosure.

[0026] Hereinafter, embodiments of the present disclosure are described in detail with reference to the drawings so that those skilled in the art can easily practice them. However, the present disclosure may be embodied in various different forms and is not limited to the embodiments described herein. In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and brevity.

[0027] An embodiment of the present disclosure will be described below with reference to the attached drawings.

[0028] FIG. 1 illustrates an electronic device in a network environment according to one or more embodiments.

[0029] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or with at least one of an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through a server (108). According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (160), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)).

[0030] The processor (120) can control at least one other component (e.g., hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., program (140)), and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., sensor module (176) or communication module (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., central processing unit or application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., graphics processing unit, neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use lower power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.

[0031] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0032] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, software (e.g., program (140)) and input data or output data for related commands. The memory (130) may include volatile memory (132) or non-volatile memory (134).

[0033] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0034] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0035] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0036] The display module (160) can visually provide information to an external (e.g., user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0037] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly (e.g., via a wire) or wirelessly to the electronic device (101).

[0038] The sensor module (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0039] The interface (177) may support one or more specified protocols that can be used to connect the electronic device (101) to an external electronic device (e.g., electronic device (102)) directly (e.g., via a wire) or wirelessly. According to one embodiment, the interface (177) may include, for example, a high definition multi-media interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0040] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0041] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0042] The camera module (180) can capture still images and video. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0043] The power management module (188) can manage the power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0044] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0045] The communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).

[0046] The wireless communication module (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or largescale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), external electronic device (e.g., electronic device (104)), or network system (e.g., second network (199)). According to one embodiment, the wireless communication module (192) can support a Peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mMTC, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for realizing URLLC.

[0047] An antenna module (197) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197).

[0048] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0049] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0050] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (104) or the server (108) may be included within the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0051] The electronic device according to the various embodiments disclosed in this document may be of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. The electronic device according to the embodiments of this document is not limited to the devices described above.

[0052] FIG. 2 shows an LLM-based agent framework according to one embodiment of the present disclosure.

[0053] According to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1) can process user request tasks using an LLM-based agent and generate a response corresponding to the user request task. The electronic device (101) can recognize information from input data obtained through an input device (e.g., microphone, camera, display) and identify a query. The electronic device (101) can refine data related to the query and store and manage it in short-term or long-term memory (e.g., memory (130) of FIG. 1). To process the query, the electronic device (101) can derive a response through a conversation between multiple agents that interact with each other without boundaries using various APIs (application programming interfaces) based on the data stored in memory. A framework (200) for an LLM-based agent may include an input data recognition step, an information memory step, and an agent action step for each agent to interact with the environment and perform functions.

[0054] According to one embodiment, an electronic device (101) can receive text or voice (201) through an input device (e.g., an input module (150) of FIG. 1). When voice is input, the electronic device (101) can convert the voice into text using automatic speech recognition (ASR) (211) to identify a user query (214).

[0055] According to one embodiment, the electronic device (101) may receive additional data (202) in addition to user queries. For example, the electronic device (101) may receive documents and queries and generate answers to queries based on document analysis. The additional data (202) may include documents, photos, videos, location information, or sensor data. The electronic device (101) may obtain information (212) obtained by analyzing images, videos, or documents acquired as additional data (202). For example, the electronic device (101) may recognize text within an image using OCR (213). The electronic device (101) may process modality data using a large multimodal model (LMM) (215). The electronic device (101) may analyze the additional data (202) using the LMM (215), extract necessary data, and store it in memory.

[0056] According to one embodiment, the electronic device (101) stores the recognized information in a short-term or long-term memory and can refer to past data related to a user query.

[0057] Short-term memory (221) can maintain information to complete a task through several stages. The electronic device (101) can classify the subject of the input data through the LLM (215) of the input data recognition stage and store it in the short-term memory (221) to maintain the context of the current conversation. The electronic device (101) can recognize a change in the subject of the input data and retrieve the information stored in the short-term memory (221). The electronic device (101) can use the LMM (215) to merge the recognized information with the information stored in the long-term memory (222) or update it in case of a conflict. The electronic device (101) can maintain the classified information (e.g., user preferences) in the long-term memory (222). The classified information can be embedded by a tool-based LLM (231) and stored in a vector database (223).

[0058] According to one embodiment, an electronic device (101) can generate an answer to a user query (214) using a tool-based LLM (231). The tool-based LLM (231) can perform specific tasks in combination with a modular tool. An LLM that processes text-based queries can handle more complex tasks by calling or controlling a modular tool. The tool of the tool-based LLM (231) may be an API. The tool may include, for example, a calendar, phone, contacts, web search, user interface guide, notification, message, calculator, translator, or document generator. To generate an answer (235) to the user query (214), the tool-based LLM (231) may configure a multi-agent through planning, reasoning, and agent selection, and call each of the multi-agents (233) to handle detailed tasks. The action of calling each of the multi-agents (233) may be performed in the form of a natural language conversation (question and answer) between the multi-agents. The electronic device (101) can output the question and answer between the multi-agents through an output device (e.g., a display) as a process for deriving an answer (235) to a user query (214).

[0059] According to one embodiment, an LLM-based agent is a software program capable of conversing in natural language with a person or another agent. For example, the agent may be a chatbot capable of natural language conversation. The agent may be programmed to perform defined functions. For example, the agent may be programmed to search for information, summarize documents, provide natural language conversations, understand natural language questions and generate corresponding answers, perform actions based on problem-solving by understanding context, provide descriptions and examples for a specific domain, generate text according to given requirements, translate language, analyze documents, and classify, decompose, and automate tasks consisting of one or more actions. The agent may have a goal to be achieved and may be optimized for a specific domain (or specific function) through fine-tuning. For example, a search agent may provide text or image search functions, and the search target may be set to web data, a specific database, a cloud server, etc. According to one embodiment, each agent may be configured with a separate LLM and may share a single LLM. When sharing a single LLM, a specific agent can serve as an interface that performs the functions of a specific domain. For example, an electronic device (101) is connected to a server containing an LLM, and a search agent and a travel agent included in the multi-agent can each perform their respective tasks using the server's LLM. In this disclosure, for convenience of explanation, an LLM-based agent may be referred to as an "agent."

[0060] According to one embodiment, an electronic device (101) may have LLM-based agents perform detailed tasks to achieve a goal corresponding to a user query. For example, a manager agent may perform task decomposition to plan complex tasks. The manager agent may analyze the user's intent from the user query and establish a task plan. The manager agent may configure a multi-agent. The multi-agent may include one or more agents to perform the detailed tasks of the task plan. The manager agent may call the multi-agent (e.g., web search agent, phone call agent) to delegate detailed tasks. Based on the answers from the multi-agent, the manager agent may identify errors in the plan or execution. In response to errors, the manager agent may change the detailed tasks or change the agent to execute the corresponding detailed tasks. In one embodiment, the manager agent may be referred to by other names. For example, it may be a main agent or a host agent. Alternatively, the name of the agent acting as the manager agent may be set or modified by the user.

[0061] The electronic device (101) can generate a final answer and provide it to the user when the goal is achieved according to the conversation of multi-agents processing detailed tasks that have broken down the task corresponding to the user query. The electronic device (101) can provide the user with the process of configuring the multi-agents and the conversation between the multi-agents. The conversation between the multi-agents can consist of questions and answers in the form of natural language and may also include information about the agent who is the speaker. The electronic device (101) can provide a summary of the conversation between the multi-agents along with the final answer. The summary may include the main process of deriving an answer to the user query. The electronic device (101) can output the conversation between the multi-agents, the summary, and the final answer through a display.

[0062] FIG. 3 illustrates a collaborative operation using a multi-agent of an electronic device according to one embodiment of the present disclosure.

[0063] According to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1) can derive an answer through the collaboration of multiple agents when it receives a user query. The electronic device (101) can provide the user with a conversation between the multiple agents in the process of reaching the final answer. The electronic device (101) can output the process of the multiple agents collaborating in the form of a conversation in which they exchange questions and answers with each other.

[0064] According to one embodiment, the electronic device (101) may include a plurality of agents. The plurality of agents may include a manager agent (310), an orchestrator (320), and agents that process tasks (331, 332, 333). The agents illustrated in FIG. 3 are merely examples, and there is no limit to the types or number of agents. They are distinguished and referred to simply to explain the operation of the plurality of agents, and the distinction between agents may differ in various embodiments.

[0065] As illustrated in FIG. 3, the manager agent (310), orchestrator (320), and agents (331, 332, 333) may each include an LLM and may request and perform tasks through natural language conversation with each other. In one embodiment, the LLM may be implemented in each agent, or multiple agents may be implemented to share and use a single LLM. In the latter case, each agent may act as an interface to request tasks from the LLM and receive responses. The LLM may be included within an electronic device (101) or may be included in a server or other device.

[0066] According to one embodiment, an electronic device (101) may configure multiple agents to process user queries using a manager agent (310). The manager agent (310) may obtain agent information suitable for processing user queries from an orchestrator (320). The orchestrator (320) may be a type of agent that stores and manages metadata about the agents. The orchestrator (320) may search for an agent to process a request task or a detailed task corresponding to the user query. The orchestrator (320) may be included in the electronic device (101) or located in a separate storage device or server.

[0067] The manager agent (310) can classify or break down a user request task into one or more detailed tasks in order to request agent search from the orchestrator (320). For example, a user request task may perform detailed tasks A and B sequentially, such as "Do A and B." A user request task may be "Do C," but to achieve task C, it may need to infer and sequentially perform detailed tasks D, E, and F. The manager agent (310) can classify tasks by referring to information stored in short-term memory or long-term memory related to the user request task.

[0068] The manager agent (310) can search the orchestrator (320) for agents to perform each detailed task. The orchestrator (320) can transmit metadata about the discovered agents to the manager agent (310). The manager agent (310) can configure multiple agents including at least one agent to process user request tasks.

[0069] Multi-agents can process detailed tasks sequentially through conversation. A manager agent (310) can generate prompts to delegate detailed tasks to agents. The manager agent (310) can break down a user request task into three detailed tasks, specify three agents (first agent (331), second agent (332), and third agent (333)) to handle each detailed task, and then generate three prompts (prompt1, prompt2, prompt3) to delegate each detailed task. The manager agent (310) can request a first detailed task from the first agent (331) using the first prompt (prompt 1) and receive a corresponding response. For example, the first agent (331) can use a phone API or a calendar API to process the first prompt. If the first agent (331) determines that additional data is needed to perform the first detailed task, it can select an agent among the multi-agents to request additional data from and request additional data from the selected agent. At this time, the first agent (331) can generate a natural language prompt to request additional data, transmit the generated prompt to the selected agent, and receive a response containing additional data from the selected agent.

[0070] FIG. 4 is a flowchart illustrating the operation of an electronic device processing a user request task using a multi-agent according to one embodiment of the present disclosure.

[0071] According to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1) may perform an action as a result of processing a user request task based on a conversation between multiple agents. The action may be one or more execution results, including providing a response. The electronic device (101) may provide the user with a basis for judgment regarding the performance of an action by summarizing the natural language conversation between multiple agents, i.e., the transmission of a prompt and the reception of a response. In the following embodiments, each operation may be performed sequentially, but in other embodiments, each operation may not be performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0072] In operation 410, an electronic device (101) according to one embodiment may receive a user query through an input device (e.g., input module (150) of FIG. 1) (e.g., keyboard, microphone). The electronic device (101) may receive a query in the form of natural language, i.e., a user request task. For example, the electronic device (101) may receive voice such as "Help me plan my summer vacation" through a microphone. The electronic device (101) may extract the query from the voice through an ASR (e.g., ASR (211) of FIG. 2).

[0073] In operation 420, an electronic device (101) according to one embodiment can identify a request task from a user query using a first agent (e.g., manager agent (310) of FIG. 3) among a plurality of agents stored in the memory of the electronic device (101) (e.g., memory (130) of FIG. 1). The manager agent (310) can identify a user request task by analyzing a natural language query.

[0074] In operation 430, the electronic device (101) according to one embodiment may break down or classify the request task into at least one detailed task. The detailed task may be a unit that each agent can process. For example, a detailed schedule for a summer vacation plan may be specified as planning a travel itinerary and collecting travel-related information. Planning a travel itinerary can be processed by a travel agent, and collecting travel-related information can be processed by a search agent.

[0075] In operation 440, an electronic device (101) according to one embodiment may request metadata for at least one agent capable of handling at least one detailed task from a second agent (e.g., an orchestrator (320) of FIG. 3) that manages metadata for a plurality of agents. The manager agent (310) may generate a prompt requesting agent information capable of handling detailed tasks and transmit it to the orchestrator (320).

[0076] In operation 450, an electronic device (101) according to one embodiment may receive a response containing metadata from a second agent. The orchestrator (320) may transmit the metadata of agents capable of handling detailed tasks to the manager agent (310) as a response.

[0077] In operation 460, the electronic device (101) according to one embodiment may configure a multi-agent by selecting at least one agent to handle at least one detailed task among a plurality of agents based on the answer. For example, a manager agent (310) may configure a multi-agent including a travel agent and a search agent to establish a summer vacation plan.

[0078] In operation 470, an electronic device (101) according to one embodiment may perform a natural language conversation to delegate at least one detailed task to each agent. A multi-agent including a manager agent (310) may perform a natural language conversation by generating a natural language prompt for at least one detailed task and sending the prompt to each agent. The natural language conversation may include a query using the prompt and a response thereto. For example, if Agent A sends a natural language prompt to Agent B, Agent B may generate a response to the natural language prompt and respond to Agent A. Agent A may proceed to the next detailed task based on Agent B's response. The electronic device (101) may continue the conversation of the multi-agents until all detailed tasks are completed.

[0079] In operation 480, an electronic device (101) according to one embodiment may determine whether to reconfigure the multi-agent based on the result of a natural language conversation. For example, the electronic device (101) may determine to reconfigure the multi-agent based on determining that a third agent cannot generate an answer for a first detailed task during a conversation between the multi-agents. The manager agent (310) may send a prompt to the orchestrator (320) to request metadata for another agent to handle the first detailed task. The manager agent (310) may receive an answer containing metadata for a fourth agent and reconfigure the multi-agent to include the fourth agent. For example, the fourth agent may be included in the multi-agent as a replacement for the third agent, or the fourth agent may be added while the third agent remains included in the multi-agent.

[0080] The electronic device (101) can process the remaining detailed tasks, including the first detailed task, through a conversation between the reconfigured multi-agents.

[0081] The electronic device (101) may change the remaining detailed tasks or change the multi-agent configuration depending on the answers of the multi-agents. If the multi-agents determine that there is additional information needed to generate an answer, they may ask another agent or user for additional information.

[0082] The electronic device (101) can generate a final answer and generate and output information summarizing the conversation between the multi-agents when the conversation between the multi-agents is completed using the manager agent (310). The electronic device (101) can output the final answer and summary information through an output device (e.g., the display module (160) of FIG. 1). The electronic device (101) can explain the answer generation process to the user by providing summary information along with the answer to the user request task.

[0083] An electronic device (101) according to one embodiment of the present disclosure comprises: an input device (150); a display (160); a memory (130) including a plurality of programs and instructions corresponding to a plurality of agents; and at least one processor (120) including a processing circuit; When the above commands are executed individually or collectively by the at least one processor (120), the electronic device (101) can: receive a user query through the input device (150); a first agent among the plurality of agents identifies a requested task from the user query and breaks down the requested task into at least one subtask; request metadata for at least one agent capable of handling the at least one subtask from a second agent managing metadata for the plurality of agents; receive a response containing the metadata from the second agent; configure a multi-agent by selecting at least one agent among the plurality of agents to handle the at least one subtask based on the response; perform a natural language conversation to delegate the at least one subtask to each agent of the multi-agent; and determine whether to reconfigure the multi-agent based on the result of the natural language conversation.

[0084] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor (120), the electronic device (101) may cause: the first agent to request metadata from the second agent regarding another agent capable of handling the first detailed task based on the determination that the third agent cannot generate an answer regarding the first detailed task during a conversation between the multi-agents, and to receive an answer from the second agent containing metadata regarding a fourth agent capable of handling the first detailed task, and the first agent to reconfigure the multi-agents including the fourth agent.

[0085] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor (120), the electronic device (101) may be caused to: process the remainder of the at least one detailed task through the reconstructed conversation between the multi-agents, generate a final answer and summary information regarding the conversation between the multi-agents based on the completion of the conversation between the multi-agents, and output the final answer and the summary information through the display.

[0086] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor (120), the electronic device (101) may cause: the first agent to reflect each response according to the conversation between the multi-agents in the remaining detailed work.

[0087] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor (120), the electronic device (101) may cause: the first agent to determine whether the multi-agent can handle the at least one changed detailed task when at least one of the remaining detailed tasks is changed in response to each response of the multi-agent conversation, and based on the determination that the multi-agent cannot handle the at least one changed detailed task, to request metadata for additional agents to handle the changed detailed tasks from the second agent, receive metadata for additional agents from the second agent, and cause the first agent to reconfigure the multi-agent including the additional agents.

[0088] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor (120), the electronic device (101) may cause: the first agent to display information about the multi-agent through the display, and, based on the conversation between the multi-agents, to sequentially output the natural language prompt query and response through the display in the form of a conversation between each agent.

[0089] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor (120), the electronic device (101) may: receive user input through the input device during a conversation between the multi-agents, and cause the first agent to reflect the user input in the remaining details of the at least one detail.

[0090] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor (120), the electronic device (101) may cause: the fourth agent to be added to the multi-agent or to replace the third agent, and to output guidance for the multi-agent reconfiguration through the display.

[0091] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor (120), the electronic device (101) may be caused to: receive information about agent creation through the input device, create a new agent based on the information, and store the new agent and information about the new agent in the memory.

[0092] According to one embodiment, the information may include at least one of an agent name, a start condition, a goal, a function, a tool, an API, training data, LLM information, or a generation condition.

[0093] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor (120), the electronic device (101) may be caused to: cause the plurality of agents to generate answers to input prompts based on a large language model.

[0094] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor (120), the electronic device (101) may be caused to: cause the plurality of agents to generate the answer or perform an action related to the answer using a tool corresponding to one or more functions stored in the memory.

[0095] FIGS. 5a and 5b are examples of conversation screens of multiple agents according to one embodiment of the present disclosure.

[0096] According to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1) can output a conversation of multi-agents in the process of processing a user request task through a display screen. FIG. 5a and 5b show a series of first to fourth screens (510, 520, 530, 540) displaying a conversation of multi-agents in the process of processing a user request task of "Help me plan my summer vacation."

[0097] The first screen (510) to the fourth screen (540) may include a first part (511, 521, 531, 541) displaying multi-agents, a second part (512, 522, 532, 542) displaying conversations between multi-agents and a user, and a third part (513, 523, 533, 543) corresponding to an input window for receiving user input.

[0098] The first part (511) of the first screen (510) may include a manager agent (e.g., the manager agent (310) of FIG. 3) as a default agent that configures multi-agents and receives user queries. The manager agent (310) may receive a request operation as user input and determine an agent to process the request operation.

[0099] In the second part (512) of the first screen (510), it may indicate receiving user input (5001) of "Help me plan my summer vacation." The second part (512) may include the manager agent (310) configuring the travel agent into a multi-agent to process the user input. In response to the action of including the travel agent in the multi-agent, the electronic device (101) may display a conversation (5002) such as "I would like to request help from the Travel Agent to plan my vacation" in the second part (512).

[0100] In the first part (521) of the second screen (520), multi-agents including a manager agent (310) and a travel agent can be displayed. In the second part (522), a message indicating that a Travel Agent has been added can be output. The manager agent (310) can send a first prompt (5003) of "Please plan a vacation considering the calendar schedule" to delegate detailed tasks to the travel agent, and can display the first prompt (5003) in the second part (522). The travel agent can generate a first response (5004) to the first prompt (5003) of "Your vacation schedule is the last week of July. Places frequently visited by Koreans recently include Jeju Island domestically and Guam, Phu Quoc, etc. overseas," and output the first response (5004) in the second part (522). The electronic device (101) can receive "Search and let me know which of them is better" as a user conversation (5005) through the third part (523) during a multi-agent conversation, and output the user conversation (5005) entered in the second part (522). The manager agent (310) can identify a search agent as an agent to process user requests in response to the input of the user conversation (5005), and include the search agent in the multi-agent. In response to the operation of including the search agent in the multi-agent, the electronic device (101) can display a conversation (5006) such as "I will request help from the Search Agent to search for travel destination information" in the second part (522).

[0101] In the first part (531) of the third screen (530), multi-agents including a manager agent (310), a travel agent, and a search agent may be displayed. In the second part (532), a message indicating that a Search Agent has been added may be displayed. The manager agent (310) may send a second prompt (5007) of "Search for recent travel destination information for Jeju Island, Guam, and Phu Quoc and tell me where to visit" to delegate detailed tasks to the search agent, and may display the second prompt (5007) in the second part (532). The search agent may generate "Guam is not recommended as tourists have recently evacuated due to the influence of a typhoon" as a second response to the second prompt (5007). The search agent may generate a conversation of "Please check if there are flights to Jeju Island and Phu Quoc" as a third prompt to delegate detailed tasks to the travel agent in relation to the second response. As a conversation between a search agent and a travel agent, a second response and a third prompt (5008) can be output in the second part (532). The travel agent can generate a third response (5009) to the third prompt (5008) stating, "There are no available flights for Jeju Island on that schedule. For Phu Quoc, there are products available for booking both flights and accommodations," and output the third response (5009) in the second part (532). In response to the third response (5009), the manager agent (310) can generate a fourth prompt (5010) to delegate detailed tasks to the travel agent stating, "The usual cost is within 3 million won. Since I travel frequently based on a family of three, please recommend based on this standard. Don't forget the king bed room." and output it in the second part (532). The travel agent can generate "Jeju Air flight on XXth at XX o'clock, Phu Quoc Vinpearl Resort reservation available" as a fourth answer (5011) to the fourth prompt (5010), and output the fourth answer (5011) in the second part (532).

[0102] In the first part (541) of the fourth screen (540), current multi-agents including a manager agent (310), a travel agent, and a search agent may be displayed. In the second part (542), the multi-agent (310) may output a conversation, "Jeju Air flight on XXth at XX o'clock, Phu Quoc Vinpearl Resort reservation available," as a final response (5012) to user input (5001). The electronic device (101) may receive "Summarize why that was decided" as user input (5013) for the final response (5012) through the third part (543), and output the received user input (5013) in the second part (542). The manager agent may generate information summarizing the conversation between the multi-agents and a final response (5014) as a response to the user input (5013). The final answer (5014) may be, "Guam has evacuated tourists due to a typhoon, and there are no available flights to Jeju Island. Phu Quoc has a flight available on XXth at XX:00, and Vinpearl Resort is available for booking with a budget of XXX. Shall we proceed like this?" In the second part (542), summarized information and the final answer (5014) may be printed.

[0103] The electronic device (101) may receive a final decision regarding the final answer (5014) as user input (5015). User input (5015) may include an intention to accept, such as "Yes, please do that." In response to receiving the final decision from the user, the electronic device (101) may perform an action for a detailed task (e.g., execute flight reservation, execute accommodation reservation). The manager agent (310) may output a conversation stating "Both flight and accommodation have been reserved. Reservation details have been sent via email" as the result of the action (5016) for the detailed task in the second area (542). Based on the action result (5016), the manager agent (310) may store user preference information in long-term memory and output a conversation stating "Saving user preference information" in the second area (542).

[0104] The first screen (510), the second screen (520), the third screen (530), and the fourth screen (540) are examples of multi-agent configurations and conversations output to display screens to derive a response to the user input (5001) of "Help me plan my summer vacation." The electronic device (101) can output natural language conversations between multi-agents to the screen and provide final answers (5011, 5014). The electronic device (101) can summarize the conversation content and output it to the screen either by user request (5013) or automatically.

[0105] FIG. 6 is a flowchart illustrating an operation to search for an agent of an electronic device according to one embodiment of the present disclosure.

[0106] According to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1) may search for an agent to process a user task request using an orchestrator (e.g., the orchestrator (320) of FIG. 3). In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0107] In operation 610, an electronic device (101) according to one embodiment can identify a domain corresponding to a task request using a manager agent (e.g., the manager agent (310) of FIG. 3). For example, the manager agent (310) can distinguish a domain from a user query as corresponding to health, education, productivity, entertainment, shopping, music, movies, or social media. The type of domain can be determined based on the metadata of the agents. Similar metadata may be included in the same domain group. Domains may be divided into upper groups and lower groups. Depending on the degree of granularity of the task request, the domain may also be determined in detail. For example, if the task request is "recommend a daily workout," it may be classified as exercise. If the task request is "recommend a good stretching pose to do every morning," it may be classified into a granular domain such as home training stretching.

[0108] In operation 620, an electronic device (101) according to one embodiment may request a domain search from an orchestrator (320). The orchestrator (320) may search for agents included in the domain.

[0109] In operation 630, the electronic device (101) according to one embodiment may succeed in domain search if there is an agent included in the domain. If the orchestrator (320) succeeds in search, it may select an agent included in the domain. For example, if there are multiple agents included in the domain, the orchestrator (320) may select one based on user preferences stored in memory.

[0110] In operation 640, if the electronic device (101) according to one embodiment fails to search in the orchestrator (320) (operation 630 No), it can search for the corresponding domain in the agent store. The agent store may represent a platform where agents can be uploaded or downloaded. The agent store may provide metadata information about linked agents. The agent store may serve as a public orchestrator.

[0111] In operation 650, an electronic device (101) according to one embodiment may succeed in domain search if there is an agent included in the domain in the agent store. If there are multiple agents included in the domain, the electronic device (101) may select one by considering the metadata of the searched agents based on user preferences stored in memory.

[0112] In operation 660, an electronic device (101) according to one embodiment may download an agent found in an agent store. The electronic device (101) may notify the user that an agent download is required and may download the agent depending on the user's final permission. The downloaded agent may be made available for use in the electronic device (101), and metadata for the agent may be stored in the orchestrator (320) of the electronic device (101).

[0113] In operation 670, an electronic device (101) according to one embodiment may output information about an agent that is retrieved by an orchestrator (320) (630 operation example) or retrieved by an agent store (650 operation example) and downloaded. The agent information may include metadata about the agent.

[0114] In operation 680, the electronic device (101) according to one embodiment may output a search failure if the search for the corresponding domain fails in the orchestrator (320) and the agent store. Since there is no agent suitable for performing the task, the electronic device (101) may suggest changing the domain or creating an agent for the task.

[0115] FIG. 7 is an example of agent groups according to one embodiment of the present disclosure.

[0116] According to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1) may configure a multi-agent for processing user queries. Among the agents (downloaded agents) (720) stored in the memory (e.g., the memory (130) of FIG. 1) of the electronic device (101), the multi-agents currently in operation may be referred to as an active agent group (710). Metadata regarding the agents (720) contained in the memory (130) may be stored and managed in an orchestrator of the electronic device (101) (e.g., the orchestrator (320) of FIG. 3). The electronic device (101) may search for an agent to perform a task from among the agents (720) contained in the memory (130) using the orchestrator (320). If the electronic device (101) fails to search for an agent by the orchestrator (320), it may search for an agent in an agent store (730).

[0117] For example, the agents (720) included in the memory (130) may include a manager agent, agent 1a, agent 2a, and agent 3a. The agent store (730) may include agent 1a, agent 1b, agent 2a, agent 2b, agent 3a, and agent 3b. The electronic device (101) may search for an agent included in the first domain from the downloaded agents (720). As a result of searching the first domain, the electronic device (101) may output agent 3a. If the electronic device (101) determines that agent 3a needs to be replaced, it may search for another agent included in the first domain from the agent store (730) and output agent 3b as a result of the search. The electronic device (101) may download agent 3b from the agent store (730) and include it in the multi-agent (710).

[0118] The electronic device (101) can search for and download necessary agents by referring to metadata regarding agents contained in memory (130) and agents contained in agent store (730). The electronic device (101) can create new agents based on user input. The electronic device (101) can store the newly created agents in memory (130) (720) and can also upload them to agent store (730) upon user request. The electronic device (101) can also delete agents from memory (130).

[0119] FIG. 8 is a flowchart illustrating the operation of processing a user request task of an electronic device according to one embodiment of the present disclosure.

[0120] According to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1) may derive a response while automatically or manually reconfiguring a multi-agent during the processing of a user request task. In the following embodiments, each operation may be performed sequentially, but in other embodiments, each operation is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0121] In operation 810, an electronic device (101) according to one embodiment can receive a user request operation.

[0122] In operation 820, the electronic device (101) according to one embodiment may configure multiple agents to process user request tasks.

[0123] In operation 830, an electronic device (101) according to one embodiment may process a request task through a conversation between multiple agents and output a conversation between multiple agents. The electronic device (101) may break down a user request task into one or more subtasks, select an agent to process each subtask, and generate a natural language prompt to delegate the subtask to the agent. The multiple agents may transmit the prompt and generate a response to the prompt according to the processing order of the subtasks. The electronic device (101) may provide the conversation between the multiple agents to a user through an output device (e.g., a display module (160) of FIG. 1). The user may perceive the processing of the user request task through the conversation between the multiple agents.

[0124] In operation 840, an electronic device (101) according to one embodiment can determine whether the accuracy of an answer generated by a first agent is above a predetermined standard during a conversation among multiple agents. Each agent based on LLM generates an answer to an input prompt, and if the accuracy of the answer is lower than the predetermined standard, it can output a result indicating that an answer cannot be generated. If the accuracy of the answer during a conversation among multiple agents is lower than the predetermined standard, the electronic device (101) can reconfigure the multiple agents.

[0125] In operation 845, an electronic device (101) according to one embodiment may reconfigure a multi-agent when the response of the first agent is lower than a predetermined standard. The electronic device (101) may search for another agent capable of handling the detailed task to be performed by the first agent and include the searched agent in the multi-agent. The added agent may handle the prompt for the detailed task for which the first agent failed to generate a response. The electronic device (101) may modify the prompt in part based on the newly added agent.

[0126] In operation 850, an electronic device (101) according to one embodiment may output a summary of the conversation and the final answer derived from the conversation of multi-agents. The conversation summary may include a summary based on the entire conversation between multi-agents in operation 830. The electronic device (101) may summarize the conversation to include sentences containing the basis for judgment in reaching the final answer among the entire conversation between multi-agents.

[0127] In operation 860, an electronic device (101) according to one embodiment can verify a user's final decision through an input device (e.g., an input module (150) of FIG. 1). The final decision may be a user intention including whether to accept a final answer to a user request task. The user may request a modification or ask additional questions regarding the final answer to a user request task. If the final decision is not completed, the electronic device (101) may reconfigure the multi-agents. The agent to handle the user's modification request or additional question may be reconfigured to handle the modification request or additional question. This process is repeated, and finally, the user may accept the final answer.

[0128] In operation 870, an electronic device (101) according to one embodiment may perform one or more actions included in the final answer as the user's final decision is completed. The actions may be one or more execution results. For example, if the user request action corresponds to "help me plan a summer vacation," and the electronic device (101) provides a travel plan as the final answer, and the user makes a final decision, a reservation action may be required to confirm the travel plan. The reservation action may include booking a flight and booking accommodation.

[0129] FIG. 9 is an example of an activation agent group according to one embodiment of the present disclosure.

[0130] According to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1) may configure a multi-agent group to process user request tasks. Agents included in the multi-agent group may be referred to as an activation agent group. The electronic device (101) may reconfigure the multi-agent group while processing user request tasks. For example, if agent 3a is added to the first activation agent group (910), it may become the second activation agent group (920). Conversely, if agent 3a is deleted from the second activation agent group (920), it may become the first activation agent group (910). If agent 3a is replaced with agent 3b in the second activation agent group (920), it may become the third activation agent group (930). The activation agent group may add, modify, or delete agents while processing user request tasks. However, the activation agent group is configured to process user request tasks and does not permanently add, replace, or delete agents stored in memory managed by the orchestrator.

[0131] FIG. 10 is a flowchart illustrating the operation of processing a user request task of an electronic device according to one embodiment of the present disclosure.

[0132] According to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1) can process user request tasks through real-time collaborative operations of multiple agents. In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0133] In operation 1010, an electronic device (101) according to one embodiment can receive a work request.

[0134] Operations 1020 and 1021 interact with each other, and an electronic device (101) according to one embodiment may analyze a requested task and break it down into one or more subtasks. The processing order of the one or more subtasks may be determined. To analyze the requested task, the electronic device (101) may extract information related to the requested task from memory (e.g., short-term memory (221) and long-term memory (222) of FIG. 2). For example, the electronic device (101) may access data stored in the short-term memory (221) in relation to a task request currently being processed. The electronic device (101) may refer to memories stored in the long-term memory (222), such as user preferences.

[0135] Operations 1030 and 1031 interact with each other, and an electronic device (101) according to one embodiment may select agents to execute a detailed task by referring to metadata about the agents. The metadata about the agents includes information about the goals and functions that each agent can perform, and may also include information about the tools or APIs that each agent utilizes. As a type of agent managing the metadata about the agents, there may be an orchestrator (e.g., the orchestrator (320) of FIG. 3). The orchestrator knows the metadata about at least one agent stored in the electronic device (101) and can search for an agent capable of handling a detailed task. If there are multiple agents searched to handle the first detailed task, the electronic device (101) may select one agent based on user preference or recent usage history.

[0136] In operation 1040, an electronic device (101) according to one embodiment can generate natural language queries and responses between agents according to the processing order of detailed tasks. The natural language query may be a prompt in the input data format of an LLM.

[0137] In operation 1050, an electronic device (101) according to one embodiment can determine whether a final answer has been derived when each detailed task is processed and a response is generated or when processing for all detailed tasks is completed. While a final answer has not been derived, the electronic device (101) can determine whether to change the detailed task or the agent based on the response generated by the agent in response to the detailed task at each step.

[0138] Each agent based on LLM may determine that it cannot generate an answer if it determines that the accuracy is lower than the first threshold criterion for generating an answer to the prompt. In this case, the electronic device (101) may decide to change the detailed task (Example of operation 1055). The electronic device (101) may change the detailed task and regenerate the prompt to delegate the changed detailed task to the agent (Operations 1030 to 1040).

[0139] Each agent based on LLM has generated an answer to the prompt, but the electronic device (101) may determine that the answer is insufficient to proceed with the next detailed task. In this case, the electronic device (101) does not need to change the detailed task (action 1055 No) but may decide to change the agent (action 1045 Yes). The electronic device (101) may select a new agent to handle the detailed task (action 1030).

[0140] If the answer generated by each agent based on the LLM according to the delegated prompt determines that no detailed work or agent change is required, the electronic device (101) may proceed to the next detailed work (no in the 1050 action, 1055 action and 1045 action).

[0141] The electronic device (101) can derive a final answer to the requested task by repeating 1020 to 1050 operations.

[0142] In operation 1060, the electronic device (101) according to one embodiment may output a final answer when all detailed tasks are completed or when a final answer is derived according to the conversation of the multi-agents.

[0143] FIG. 11 is a flowchart illustrating an agent generation operation of an electronic device according to one embodiment of the present disclosure.

[0144] According to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1) can generate a new agent by user input. In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0145] In operation 1101, an electronic device (101) according to one embodiment may start creating an agent. For example, the electronic device (101) may receive user input (e.g., touch an icon to create an icon) to start the agent creation function.

[0146] In operation 1102, an electronic device (101) according to one embodiment may receive meta information for a new agent. The meta information includes one or more pieces of information about the agent and may include metadata for defining the agent. The metadata may include a goal to be achieved through the agent, functions provided, tools or APIs used, training data, LLM information including implementation specifications (e.g., LLM type), and instructions describing creation conditions. In addition to the metadata, the meta information may further include a profile image, start conditions, and agent name for the new agent.

[0147] In operation 1103, an electronic device (101) according to one embodiment can create a new agent based on meta-information. For example, the tool or API used by the electronic device (101) may be matched among the tools or APIs provided by the electronic device (101). The electronic device (101) can be programmed to use an LLM that meets the specifications required by the new agent among the supported LLMs. The electronic device (101) can acquire training data of meta-information and train the LLM of the new agent using the training data.

[0148] In operation 1104, an electronic device (101) according to one embodiment may register new agent information in a metadata DB. The metadata DB may be managed by an agent that manages metadata (e.g., an orchestrator (320) of FIG. 3). The electronic device (101) can search for agents based on metadata for newly created agents.

[0149] FIG. 12a is an example of an agent selection screen of an electronic device according to one embodiment of the present disclosure.

[0150] FIG. 12b is an example of a new agent creation screen of an electronic device according to one embodiment of the present disclosure.

[0151] FIG. 12c is an example of a user agent list screen of an electronic device according to one embodiment of the present disclosure.

[0152] According to one embodiment, the electronic device (101) includes at least one agent in memory (e.g., memory (130) of FIG. 1), can search for and download agents from an agent store as needed, and can create a new agent upon user request.

[0153] In one embodiment, the electronic device (101) may provide a user interface screen that allows a user to check information about an agent and manually configure a multi-agent by selecting an agent.

[0154] The first screen (1201) may include icons for selecting basic agents, an agents store, and a custom agent provided by the electronic device (101).

[0155] The second screen (1202) is a basic agent screen and can display some of the agents included in the electronic device (101). For example, it may include a search agent, a calendar agent, and a health agent.

[0156] The third screen (1203) is an agent store screen and can display downloadable agents. For example, it can display popular agents and recommended agents.

[0157] The fifth screen (1205) is a custom agent screen and is a user input screen for creating a new agent. The fifth screen (1205) may include items for inputting one or more pieces of information for creating a new agent. For example, it may include a new agent name, trigger condition, tools, knowledge, LLM model, and goals and instructions to be achieved. The electronic device (101) can create a new agent based on the input data obtained through the fifth screen (1205). For example, the electronic device (101) can create an agent program that satisfies the metadata of the new data.

[0158] The seventh screen (1207) is a screen that displays agents included in the electronic device (101). Multiple agents may be sorted and displayed at the top according to usage frequency, importance, or user preference. For example, the seventh screen (1207) may display a manager agent at the top and sequentially list a planner agent, a calendar agent, a search agent, and an SNS agent.

[0159] FIG. 13 is an example of a tool call operation of LLM-based agents according to one embodiment of the present disclosure.

[0160] According to one embodiment, an electronic device (101) may enable at least one agent (1320) to use at least one tool (1310) and at least one LLM (1330). The electronic device (101) may include multiple agents (1320) having various goals and functions. Each agent (1320) may be capable of natural language conversation based on the LLM (1330) and may provide one or more functions to achieve a set goal. Each agent (1320) may use basic tools (1310) included in the electronic device (101) to execute functions. Basic tools (1310) represent modular tools and may include, for example, a calendar, phone, contacts, web search, user interface guide, notifications, messages, a calculator, a translator, or a document generator. Depending on updates to the electronic device (101) or the creation of modular tools, basic tools (1310) may be added or modified. The electronic device (101) may include at least one LLM (1330), but may physically have only one LLM (1330). Alternatively, the LLM (1330) may be multiple LLMs fine-tuned to specific functions in the form of on-device LLMs.

[0161] Each agent (1320) can converse using the LLM (1330) with natural language prompts and responses. For example, the first agent can query the third agent by sending a prompt to the LLM (1330), and the third agent's response can be output in natural language form and delivered to the first agent. The first agent can use the first tool and the third tool to perform functions. The third agent can use the third tool and the fifth tool to perform functions. The first agent may belong to the first domain, and the third agent may belong to the second domain. Agents may belong to any one of multiple domains. In the case of an agent that provides multiple functions, it may belong to multiple domains.

[0162] Each tool (1310) can execute a function and return a result in response to a call from an agent. The LLM (1330) can generate a response and return it in response to a prompt sent by an agent. The electronic device (101) can perform multi-agent collaboration through conversation through interaction between each tool (1310), each agent (1320), and the LLM (1330).

[0163] FIG. 14a is an example of a conversation screen of a user configuration agent team according to one embodiment of the present disclosure.

[0164] FIG. 14b is an example of an action screen of a user configuration agent team according to one embodiment of the present disclosure.

[0165] According to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1) can manually create an agent team by user input. Unlike an embodiment (e.g., FIG. 3) in which a manager agent (e.g., the manager agent (310) of FIG. 3) configures multiple agents to handle the user request tasks described above, an agent team configured by user input allows the agents within the team to handle tasks delegated by the user.

[0166] The first screen (1401) may include agent teams configured by the user. The agent teams may include a team name and icons representing the included agents. The first agent team (14011) is a time management team and may include schedule management, social media, health, books, video, and calendar agents. The second agent team (14012) is a work efficiency team and may include productivity and search agents.

[0167] The second screen (1402) illustrates a conversation in which a user delegates a task to the first agent team (14011). The electronic device (101) can receive user input (14021) regarding a task request. For example, the user may request a task such as, "These are things I usually do. Morning exercise, reading books, watching TV with my wife, studying math with my child, playing games with my child, looking at social media, classify them according to the time management matrix." The scheduling agent of the first agent team (14011) can classify the user's requested tasks according to importance and urgency. As a result of the classification, 1. Morning exercise and studying math with my child can be classified as important and urgent. 2. Reading books and playing games with my child can be classified as important but not urgent. 3. Watching TV with my wife and looking at social media can be classified as neither important nor urgent. The scheduling agent can output a response (14022) summarizing the classification content to the second screen (1402). The schedule management agent can collaborate with the health agent and the calendar agent to derive a schedule of 1. morning exercise every morning at 6:00 AM and math study with child every Monday evening at 8:00 PM. The schedule management agent can output a natural language response (14023) to the second screen (1402) for the derived schedule, asking, "Since morning exercise and math study with child are important and urgent, should we increase their priority and register them as the schedule: Morning exercise: every morning at 6:00 AM, Math study with child: every Monday evening at 8:00 PM?" On the second screen (1402), it can output that the user has received a final decision (14024) from the user, saying, "Yes, please make math study on Wednesday." The calendar agent can reflect the user's final decision (14024) to register the schedule "Morning exercise every morning at 6:00 AM, Math study with child every Wednesday evening at 8:00 PM" and output the action content (14025) on the second screen (1402).

[0168] Sequentially, the schedule management agent can collaborate with the calendar agent and the book agent to derive a schedule to register during weekend leisure time for 2. Reading books and playing games with children. The schedule management agent can output a natural language response (14026) to the second screen (1402) of the derived schedule, stating, "Reading books and playing games with children are important but not urgent. I will register the schedule during weekend leisure time so that it can be continued consistently. Would you like me to let you know when a new book comes out in the genre of novels you are usually interested in?" On the second screen (1402), it can output that "Yes, okay" is received as a final decision (14027) from the user. The schedule management agent can collaborate with the movie agent, the calendar agent, and the SNS agent to derive a schedule to reduce viewing time to 1 hour per week and reduce SNS time for 3. Watching TV and SNS with wife. The schedule management agent can output a natural language response (14028) to the second screen (1402) for the derived schedule, stating, "Watching TV with my wife is not important or urgent, so should we watch it for 1 hour a week? I think it would be good to reduce SNS usage as well." On the second screen (1402), the agent can output a receipt for the final decision (14029) from the user, stating, "I would like to watch TV for 2 hours a week. I won't look at SNS. However, I am curious about news of acquaintances going on trips, so please let me know only those kinds of news." The calendar agent and the SNS agent can reflect the user's final decision (14029) and output an action content (140210) to the second screen (1402), stating, "I will register the TV schedule and let you know the travel news on SNS."

[0169] An electronic device (101) according to one embodiment can recognize a situation through an agent and generate a message according to the situation and provide it to the user. For example, a first agent team (14011) can register a schedule as an action for a user request task and output a notification related to the schedule according to time.

[0170] In the third screen (1405), the first agent team (114011) can output a situational message to the first alarm (14051). For example, it can output a message such as, "It is 6 o'clock in the morning, time to exercise. Shall we try running for 30 minutes, the same amount of time as yesterday?" or, "You haven't used social media for a week and you are keeping to your goal well. OO is going on a trip to Jeju Island, so we are letting you know."

[0171] The embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as “coupled” or “connected” to another (e.g., 2nd) component, with or without the terms “functionally” or “communicationly,” it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0172] The term “module” as used in the embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0173] One embodiment of the present document may be implemented as software (e.g., program (140)) comprising one or more instructions stored in a storage medium (e.g., internal memory (136) or external memory (138)) readable by a machine (e.g., electronic device (101)). For example, a processor (e.g., processor (120)) of the machine (e.g., electronic device (101)) may call at least one of the one or more instructions stored in the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0174] According to one embodiment, the method according to the embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0175] According to one embodiment, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to one embodiment, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to one embodiment, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In an electronic device, Input device; display; Memory containing multiple programs and instructions corresponding to multiple agents; and It includes at least one processor including a processing circuit; When the above instructions are executed individually or collectively by the at least one processor, the electronic device: Through the above input device, a user query is received, and Among the plurality of agents above, the first agent identifies a request task from the user query, and Break down the above requested task into at least one detailed task, and Requesting metadata for at least one agent capable of processing at least one detailed task from a second agent managing metadata for the plurality of agents, and Receive a response containing the metadata from the second agent, and Based on the above answer, a multi-agent is configured by selecting at least one agent to handle at least one detailed task from among the plurality of agents, and Performing a natural language conversation to delegate at least one of the above detailed tasks to each agent of the multi-agent, An electronic device that causes to determine whether to reconfigure the multi-agent based on the result of the above natural language conversation.

2. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device: In response to the third agent determining during the conversation between the multi-agents that it cannot generate an answer regarding the first detailed task, the first agent requests metadata from the second agent regarding another agent to handle the first detailed task, and Receive a response from the second agent containing metadata about a fourth agent capable of processing the first detailed task, and An electronic device that causes the first agent to reconfigure the multi-agent, including the fourth agent.

3. In Paragraph 2, When the above instructions are executed individually or collectively by the at least one processor, the electronic device: The remainder of the at least one detailed task is processed through the conversation between the reconstructed multi-agents, and Based on the completion of the conversation between the multi-agents mentioned above, generate a final answer and summary information regarding the conversation between the multi-agents mentioned above, and An electronic device that causes the final answer and summary information to be output through the display.

4. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device: The above first agent is an electronic device that causes each response according to the conversation between the above multi-agents to be reflected in the remaining detailed tasks.

5. In Paragraph 4, When the above instructions are executed individually or collectively by the at least one processor, the electronic device: The first agent determines whether the multi-agent can process at least one of the remaining detailed tasks when at least one of the remaining detailed tasks is changed by reflecting each response of the multi-agent conversation, and Based on the determination that the multi-agent cannot process at least one of the changed detailed tasks, the second agent requests metadata for additional agents to process the changed detailed tasks, and Receive metadata about an additional agent from the second agent, and An electronic device that causes the first agent to reconfigure the multi-agent, including the additional agent.

6. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device: The first agent displays information about the multi-agent through the display, and An electronic device that causes the natural language prompt questions and responses to be sequentially output in the form of a conversation between each agent through the display, based on the conversation between the multi-agents.

7. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device: During a conversation between the multi-agents mentioned above, user input is received through the input device, and An electronic device that causes the first agent to reflect the user input in the remaining details of at least one detail task.

8. In Paragraph 2, When the above instructions are executed individually or collectively by the at least one processor, the electronic device: The above-mentioned fourth agent is added to the above-mentioned multi-agent or replaces the above-mentioned third agent, and An electronic device that causes to output guidance on the multi-agent reconfiguration through the above display.

9. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device: Information regarding agent creation is received through the above input device, and Based on the above information, create a new agent, and An electronic device that causes information about the new agent and the new agent to be stored in the memory.

10. In Paragraph 9, The above information includes at least one of an agent name, start condition, goal, function, tool, API, training data, LLM information, or generation condition, an electronic device.

11. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device: An electronic device in which the above plurality of agents cause to generate an answer to an input prompt based on a large language model.

12. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device: An electronic device in which the plurality of agents are caused to generate the answer or perform an action related to the answer using a tool corresponding to one or more functions stored in the memory.

13. In a method of an electronic device, The operation of receiving a user query through an input device; The operation of a first agent among a plurality of agents included in the memory of the electronic device identifying a requested task from the user query; The action of breaking down the above-mentioned request task into at least one detailed task; An action of requesting metadata for at least one agent capable of processing at least one detailed task from a second agent managing metadata for the plurality of agents; The operation of receiving an answer including the metadata from the second agent; Based on the above answer, the operation of configuring a multi-agent by selecting at least one agent among the plurality of agents to handle at least one detailed task; An action of performing a natural language conversation to delegate at least one detailed task to each agent of the multi-agent; and A method comprising an operation to determine whether to reconfigure the multi-agent based on the result of the above natural language conversation.

14. In Paragraph 13, The operation of determining whether the above multi-agent requires reconfiguration is, An action in which the first agent requests metadata from the second agent regarding another agent to handle the first detailed task, based on the third agent determining during the conversation between the multi-agents that it cannot generate an answer for the first detailed task; The action of receiving a response from the second agent containing metadata about a fourth agent capable of processing the first detailed task; and A method in which the first agent further includes the operation of reconfiguring the multi-agent including the fourth agent.

15. A non-transitory computer-readable storage medium that records instructions, When the above instructions are executed individually or collectively by one or more processors, the one or more processors: Receive user queries through an input device; Among the plurality of agents, the first agent identifies a request task from the user query; Break down the above requested task into at least one detailed task; Requesting metadata for at least one agent capable of processing at least one detailed task from a second agent managing metadata for the plurality of agents; Receive a response containing the metadata from the second agent; Based on the above answer, a multi-agent is configured by selecting at least one agent to handle at least one detailed task from among the plurality of agents; Performing a natural language conversation to delegate at least one of the above detailed tasks to each agent of the multi-agent; and A storage medium that determines whether to reconfigure the multi-agent based on the result of the above natural language conversation.

Images