Selection device, selection method, and selection program

Abstract

This selection device receives input of a task. The selection device then selects a prompt for a persona suitable for executing the task from a set of prompts for each persona. The selection device also selects a base model suitable for executing the task from a set of base models. The selection device then uses the combination of the selected prompt and base model to create an agent for executing the task, and stores the created agent in a storage unit. The selection device then calculates the diversity of answers output by each agent stored in the storage unit, on the basis of answers to a prescribed question for each agent, and selects and outputs a group of agents which are among a set of a prescribed number of agents, and in which the diversity of the answers between the agents is maximized.

Description

Selection Device, Selection Method, and Selection Program

[0001] The present invention relates to a selection device, a selection method, and a selection program for selecting an LLM (Large Language Models) agent used for task execution.

[0002] Conventionally, there is a multi-agent debating framework that uses multiple LLMs (Large Language Models) to discuss tasks. According to this multi-agent debating framework, the accuracy of task execution results (hereinafter, appropriately abbreviated as "accuracy") can be improved compared to a single LLM.

[0003] Also, in the above multi-agent debating framework, it has been reported that the accuracy increases as the number of agents (LLMs used for task execution) increases, and there is a possibility that the accuracy increases when different prompts are given. Also, in the multi-agent debating framework, it has been reported that the accuracy increases when different LLMs are combined.

[0004] Y. Du et al, “Improving Factuality and Reasoning in Language Models through Multiagent Debate,” CoRR, vol. abs / 2305.14325, 2023.Justin Chih-Yao Chen et al, 2023a. “RECONCILE: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs,”arXiv preprint arXiv:2309.13007.2024.

[0005] However, the multi-agent debating framework has a problem that the total number of inferences of the LLM increases depending on the number of agents and the number of times of repeating the discussion (round number), and accordingly the inference cost increases.

[0006] Therefore, the present invention aims to solve the above-mentioned problems and reduce the inference cost required for task execution while maintaining the accuracy of the task execution results in a multi-agent debating framework.

[0007] To solve the above-mentioned problems, the present invention includes a task reception unit that receives a task input, a model selection unit that selects a model suitable for executing the task from a set of models, a prompt selection unit that selects a prompt for a persona suitable for executing the task from a set of prompts for each persona, an agent creation unit that creates an agent based on a combination of the selected model and the prompt for the persona and stores the created agent in a storage unit, and a calculation unit that calculates the diversity of the answers output by each agent based on the answers to a predetermined question for each agent stored in the storage unit and selects a group of agents such that the diversity of the answers output by each agent within a set of a predetermined number of agents is maximized.

[0008] According to the present invention, in a multi-agent debating framework, it is possible to reduce the inference cost required for task execution while maintaining the accuracy of the task execution results.

[0009] FIG. 1 is a diagram for explaining the outline of the selection device. FIG. 2 is a diagram showing a configuration example of the selection device. FIG. 3 is a flowchart showing an example of the processing procedure executed by the selection device. FIG. 4 is a flowchart for explaining an example of the processing procedure executed by the diversity index calculation unit. FIG. 5 is a diagram for explaining an example of the task execution procedure using a plurality of agents. FIG. 6 is a diagram showing an example of a computer that executes the selection program.

[0010] Hereinafter, embodiments for implementing the present invention will be described with reference to the drawings. The present invention is not limited to this embodiment.

[0011] [Overview] The overview of the selection device of this embodiment will be explained using Figure 1. In the following, the agent will be described as being realized by a combination of a base model (model) and a prompt for the persona. The base model is, for example, a natural language processing model such as LLM (Large Language Models). Here, we will explain using the example where the task that the agent should perform is a "question".

[0012] First, when the selection device receives a question, it prompts the persona best suited to answering that question. size Select one option. For example, if the question is "Solve the following math problem...", the selection device will select prompts such as "You are a mathematician" or "You are a student who excels at mathematics."

[0013] Furthermore, the selection device selects a suitable base model for the question (e.g., an LLM such as ChatGPT®). size Select one. For example, if the question is "Solve the following math problem...", the selection device will select a base model that excels at solving math problems, or a base model that has been fine-tuned for solving math problems.

[0014] Next, the selection device creates an agent based on the selected prompt and base model combination and saves it to the agent DB (database). For example, the selection device creates an agent by combining the base model: GPT-4 and the prompt for the persona: "You are a mathematician," and saves it to the agent DB. Here, the selection device, for example, l=p size xm size Create individual agents and save them to the agent database.

[0015] Subsequently, the selection device selects k agents from the agents in the agent database to be used to answer the question. Here, the selection device selects a set A consisting of k agents. s Diversity index f(A s Calculate the diversity index f(A) (see formula (1)) and sSelect k agents from the agent database that maximize the given value. Note that k is a hyperparameter.

[0016] Details of the diversity indicator will be described later, but set A s This index represents the degree of diversity in the responses (task execution results) output by the k agents that make up the system. This diversity index is calculated by inputting a diversity evaluation prompt to each agent and using the difference in the probability distribution of the output responses (distance between agents).

[0017] Subsequently, the selection device has the k selected agents discuss the above question and outputs the answer obtained from the discussion.

[0018] Such a selection mechanism allows for the selection of a set of k agents with the highest possible diversity within a multi-agent debating framework. This reduces the number of foundational models, personas, and agents required to obtain a certain level of task execution results within the multi-agent debating framework. As a result, the inference cost required for task execution can be reduced.

[0019] [Configuration Example] Next, an example of the configuration of the selection device 10 will be described using Figure 2. The selection device 10 includes, for example, a task reception unit 11, a base model selection unit 12, a prompt selection unit 13, an agent creation unit 14, a diversity index calculation unit 15, and a task execution unit 16.

[0020] Furthermore, the selection device 10 includes a storage unit (not shown) that stores a set of base models, prompts for personas, a set of prompts for diversity evaluation, an agent database, and a set of selected agents.

[0021] Note that the set of base models is, for example, a set of LLMs such as ChatGPT. Also, the prompts for personas are, for example, a set of prompts with various personas set, such as "You are a mathematician". The set of prompts for diversity evaluation is a set of question texts for obtaining the characteristics of each agent. The agent DB is a DB that stores information on agents (combinations of base models and prompts for personas) created by the agent creation unit 14. The set of selected agents is the set of agents selected by the diversity index calculation unit 15.

[0022] Note that the prompts for personas, the set of base models, and the set of prompts for diversity evaluation may be deployed in an external device of the selection device 10, for example.

[0023] The task reception unit 11 receives an input of a task. For example, the task reception unit 11 receives an input of a query such as "Please solve the problem of mathematics...".

[0024] The base model selection unit 12 selects m size base models suitable for executing the above task from the set of base models. For example, when the task is a query such as "Please solve the problem of mathematics...", the base model selection unit 12 selects a base model that is good at mathematics problems, a base model that has been fine-tuned to solve mathematics problems, etc. from the set of base models.

[0025] The prompt selection unit 13 selects p size prompts for personas suitable for executing the above task from the set of prompts for personas. For example, when the task is a query such as "Please solve the problem of mathematics...", the prompt selection unit 13 selects prompts such as "You are a mathematician" and "You are a student who is good at mathematics" from the set of prompts for personas.

[0026] The agent creation unit 14 creates an agent by combining the base model selected by the base model selection unit 12 and the prompt for the persona selected by the prompt selection unit 13, and saves it in the agent DB. In this embodiment, an agent is realized by combining a base model and the prompt for the persona to be applied to that base model. Therefore, creating an agent means creating a combination of a base model and the prompt for the persona to be applied to that base model.

[0027] For example, if the agent creation unit 14 creates an agent combining the base model: GPT-4 and the persona prompt: "You are a mathematician", it saves information indicating the combination of the base model: GPT-4 and the persona prompt: "You are a mathematician" in the agent DB. For example, if the base model selected by the base model selection unit 12 is m size The prompt for the person selected by the prompt selection unit 13 is p size If it's individual, then m size ×p size Create individual agents and save their information to the agent database.

[0028] The diversity index calculation unit 15 calculates the diversity index f(A s The diversity index calculation unit 15 selects k agent groups that maximize the following (see equation (1)): The diversity index calculation unit 15 calculates the diversity of responses output by each agent based on the answers to predetermined questions for each agent, and selects an agent group that maximizes the diversity of responses output by each agent within a set of predetermined agents. For example, the diversity index calculation unit 15 inputs the above diversity evaluation prompt to each agent and calculates the distance d between each agent based on the answers obtained from each agent. The diversity index calculation unit 15 then selects and outputs k agent groups from the agent DB that maximize the distance d between each agent in the set of agents. Details of this diversity index calculation unit 15 will be described later.

[0029] The task execution unit 16 executes a task using k agent groups selected by the diversity index calculation unit 15 and outputs the execution results of that task. For example, the task execution unit 16 inputs a "question" that it wants the selected k agent groups to discuss and has them conduct the discussion. Then, the task execution unit 16 outputs the answers to the "question".

[0030] [Example of Processing Procedure] Next, an example of a processing procedure performed by the selection device 10 will be explained using Figure 3.

[0031] First, the task reception unit 11 receives the task input (S1). Next, the base model selection unit 12 selects a base model from the set of base models that is suitable for the task received in S1. size Select one (S2: Selection of base model). The prompt selection unit 13 selects a persona prompt from the persona prompts that is suitable for the task received in S1. size Select an item (S3: Select prompt for persona).

[0032] Next, the agent creation unit 14 creates an agent by combining the base model selected in S2 and the prompt for the persona selected in S3, and saves the created agent (information about the created agent) to the agent DB (S4).

[0033] After S4, the diversity index calculation unit 15 calculates the diversity index (f(A) s Based on the above, k agents are selected from the agent DB (S5). Then, the task execution unit 16 executes the task using the agents selected in S5 (S6) and outputs the task execution result (S7).

[0034] [Details of the Diversity Index Calculation Unit] Next, the diversity index calculation unit 15 will be explained in detail. First, the information input to the diversity index calculation unit 15 will be explained. The information input to the diversity index calculation unit 15 is the agent set A of the agent DB, hyperparameter k, and the set of prompts Q for diversity evaluation. Each agent has the following two elements: ・Prompt (prompt for persona) prompt i∈P, p size =|P| ・Base model m i ∈M, m size =|M|

[0035] Furthermore, the total number of elements in agent set A is l: l = p size xm size That is the case.

[0036] Furthermore, the hyperparameter k is a parameter that indicates the number of agents ultimately selected by the diversity index calculation unit 15.

[0037] Furthermore, the set of prompts Q for diversity evaluation is a set of question sentences (Q = {question1, question2, ..., question r})

[0038] Next, an example of the processing procedure performed by the diversity index calculation unit 15 will be explained using Figure 4. The diversity index calculation unit 15 sequentially selects agents for the required number of agents k by following the procedure below, for example.

[0039] First, the empty set A s Prepare the following. Then, the diversity index calculation unit 15 randomly selects one agent from agent set A, and A s Add to (S11). Then, the diversity index calculation unit 15 calculates the diversity index (f(A s Agent a that maximizes )) * Select A sequentially. s Add to (A s ←A s ∪a * ) (S12).

[0040] Note that agent set A has already been selected. s In contrast, the diversity index (f(A s )) is defined by the following equations (1) and (2) using the distance function d. Note that A s represents the set of agents a that have already been selected. Also, a i ,a j This indicates each agent, and a i ,a j ∈As That is the case.

[0041]

[0042] After S12, the diversity index calculation unit 15 performs |A s If we determine that |=k (Yes in S13), A s The set of selected agents is output (S14). Meanwhile, the diversity index calculation unit 15 calculates |A s If the result is not k (No in S13), the process returns to S12.

[0043] Next, we will explain the distance function d in equation (1). The distance function d is a function used to calculate the difference (distance) between the characteristics of each agent, and for example, KL divergence can be used. For example, by assuming that the results output by each agent (e.g., the answer to the question) have a predetermined probability distribution, the distance between each agent can be calculated using KL divergence.

[0044] For example, agent a i We consider this as the probability distribution for generating text x (answer), and define it as shown in equation (3) below. Note that in equation (3), m i Agent A i This shows the underlying model. Also, prompt i Agent A i The prompt used is shown.

[0045]

[0046] For example, the diversity index calculation unit 15 samples the results (text x) output by each agent and determines agent a i ,a j KL-divergence between (KL[a i ||a j ]) is calculated, and using the KL-divergence, the diversity index (f(A)) is calculated using the following formula (4). s )) calculate.

[0047]

[0048] The following describes an example of calculating the distance between agent a1 and agent a2 using KL-divergence. Agent a1 and agent a2 are defined by the following equation (5) based on the probability distribution of the text x generated by each agent.

[0049]

[0050] The KL-divergence between agent a1 and agent a2 is calculated using the results of sampling conditioned with the aforementioned diversity evaluation prompt (question). The distance between agent a1 and agent a2 (d(a i ,a j )) can be calculated as shown in equation (6) below. Note that Q is the set of question sentences, and Q = {question1, question2, ..., question r}. X is the set of sentences x output by the agent, where X = {x1, x2, ..., x N}

[0051]

[0052] The diversity index using the above KL divergence (f(A s An example of the calculation of )) is shown in the following equation (7).

[0053]

[0054] [Example of Task Execution] Next, an example of task execution by an agent selected by the diversity index calculation unit 15 will be described. The task execution unit 16 causes the selected agent to execute the task, for example, according to the procedure shown in Figure 5.

[0055] For example, if the selected agent is one of two agents (Agent A i ,a j If this is the case, the task execution unit 16 will execute agent a as follows: i ,a j Let them discuss it.

[0056] First, the task execution unit 16 executes agent a i Enter the task into agent ai The output obtained from is output i round This is assumed (see equation (8) below). Note that `round` indicates the number of times the output of one agent is input to another agent.

[0057]

[0058] Task execution unit 16 is agent a i The output obtained from (output i round ) to agent a j Enter into agent a j The output is obtained from (see equation (9) below).

[0059]

[0060] Note that the instruction in equation (9) is a prompt that incorporates the obtained output. For example, the prompt may look like this:

[0061] "These are the solutions to the problem from other agents: [“output”] Based off the opinion of other agents, can you give an updated response ..."

[0062] For example, if the selected agent is one of three agents (Agent A i ,a j ,a k If this is the case, there are several ways to discuss, for example, with two agents (agent a i ,a j The output from ) is passed to the remaining agent (Agent A k There is a way to input it into ). In this case, the task execution unit 16 will input agent a i ,a j The output obtained from agent a k Enter into agent a k From this, the output shown in equation (10) is obtained.

[0063]

[0064] According to the selection device 10 described above, in a multi-agent debating framework, it is possible to reduce the inference cost required to execute a task while maintaining the accuracy of the task execution results.

[0065] [Other Embodiments] In the above-described selection device 10, the device is configured to select a set of k agents that maximizes the diversity index in the multi-agent debating framework, but it is not limited to this. For example, the selection device 10 may select a set of k agents that minimizes the diversity index.

[0066] For example, in the multi-agent debating framework described above, if there are too many agents, the discussion may not converge, or convergence may be slow. In such cases, or when a conclusion is to be reached quickly, the user instructs the selection device 10 to select a set of k agents that minimizes the diversity index. Based on this instruction, the selection device 10 selects a set of k agents that minimizes the diversity index. The selection device 10 then conducts a discussion with the selected set of k agents. This makes it possible to avoid the discussion not converging or to reduce the number of rounds in the multi-agent debating framework when there are too many agents or when a conclusion is to be reached quickly.

[0067] [System Configuration, etc.] Furthermore, the components of each part shown in the diagram are functional concepts and do not necessarily need to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those shown in the diagram, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions. In addition, all or any part of the processing functions performed by each device can be realized by a CPU and the program executed on that CPU, or by hardware using wired logic.

[0068] Furthermore, among the processes described in the embodiments described above, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by known methods. In addition, the processing procedures, control procedures, specific names, and information including various data and parameters shown in the above document and drawings can be arbitrarily changed unless otherwise specified.

[0069] [Program] The selection device 10 described above can be implemented by installing a program (selection program) as packaged software or online software on a desired computer. For example, by having the above program run on an information processing device, the information processing device can be made to function as the selection device 10. The information processing device referred to here includes mobile communication terminals such as smartphones, mobile phones and PHS (Personal Handyphone System), as well as terminals such as PDA (Personal Digital Assistant).

[0070] Figure 6 shows an example of a computer running a selection program. Computer 1000 has, for example, memory 1010 and a CPU 1020. Computer 1000 also has a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0071] Memory 1010 includes ROM (Read Only Memory) 1011 and RAM (Random Access Memory) 1012. ROM 1011 stores, for example, a boot program such as BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to the hard disk drive 1090. The disk drive interface 1040 is connected to the disk drive 1100. For example, a removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to, for example, a mouse 1110 and a keyboard 1120. The video adapter 1060 is connected to, for example, a display 1130.

[0072] The hard disk drive 1090 stores, for example, the OS 1091, application program 1092, program module 1093, and program data 1094. That is, the program that defines each process executed by the selection device 10 is implemented as a program module 1093 in which executable code for a computer is written. The program module 1093 is stored, for example, in the hard disk drive 1090. For example, a program module 1093 for executing a process similar to the functional configuration of the selection device 10 is stored in the hard disk drive 1090. Note that the hard disk drive 1090 may be replaced by an SSD (Solid State Drive).

[0073] Furthermore, the data used in the processing of the above-described embodiment is stored as program data 1094 in, for example, memory 1010 or hard disk drive 1090. The CPU 1020 then reads the program module 1093 and program data 1094 stored in memory 1010 or hard disk drive 1090 into RAM 1012 as needed and executes them.

[0074] Furthermore, the program module 1093 and program data 1094 are not limited to being stored in the hard disk drive 1090; for example, they may be stored in a removable storage medium and read by the CPU 1020 via a disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (LAN (Local Area Network), WAN (Wide Area Network), etc.). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via a network interface 1070.

[0075] 10 Selection device 11 Task reception unit 12 Base model selection unit 13 Prompt selection unit 14 Agent creation unit 15 Diversity index calculation unit 16 Task execution unit

Claims

1. A selection device comprising: a task reception unit that accepts task input; a model selection unit that selects a model suitable for executing the task from a set of models; a prompt selection unit that selects a prompt for a persona suitable for executing the task from a set of prompts for each persona; an agent creation unit that creates an agent using the selected combination of model and persona prompts and stores the created agent in a memory unit; and a calculation unit that calculates the diversity of answers output by each agent based on the answers to predetermined questions for each agent stored in the memory unit, and selects a group of agents that maximizes the diversity of answers output by each agent within a set of a predetermined number of agents.

2. The selection device according to claim 1, characterized in that the calculation unit calculates the diversity of responses output by each agent based on the distance between the responses output by each agent.

3. The selection device according to claim 1, characterized in that the calculation unit calculates the diversity of responses output by each agent based on the probability distribution of responses to predetermined questions for each agent.

4. The selection device according to claim 2, characterized in that the calculation unit randomly selects one agent from the storage unit and adds it to the set, then selects an agent from the storage unit that maximizes the distance between each agent in the set and adds it to the set, repeating this process until the number of agents in the set reaches a predetermined number, thereby selecting a group of agents that maximizes the distance between each agent in the set of the predetermined number of agents.

5. The selection device according to claim 1, further comprising a task execution unit that executes the task using the group of agents selected by the calculation unit and outputs the execution result of the task.

6. A selection method performed by a selection device, comprising the steps of: receiving input for a task; selecting a model suitable for executing the task from a set of models; selecting a prompt for a persona suitable for executing the task from a set of prompts for each persona; creating an agent using the selected model and prompt for the persona, and storing the created agent in a memory unit; and calculating the diversity of responses output by each agent based on the answers to predetermined questions for each agent stored in the memory unit, and selecting a group of agents such that the diversity of responses for each agent in a set of a predetermined number of agents is maximized.

7. A selection program that causes a computer to perform the following steps: receiving input for a task; selecting a model suitable for executing the task from a set of models; selecting a prompt for a persona suitable for executing the task from a set of prompts for each persona; creating an agent using the selected model and persona prompt combination and storing the created agent in memory; and calculating the diversity of responses output by each agent based on the answers to predetermined questions for each agent stored in memory, and selecting a group of agents that maximizes the diversity of responses for each agent within a predetermined number of agents.

Images