Selection device and selection method

Abstract

A selection device 10 inputs a prompt to an LLM server 30, the prompt including specification data of available APIs, a request from a user, and an instruction sentence for instructing the extraction of APIs having functions that meet the request from the user, thereby extracting API candidates. When a plurality of API candidates have been extracted, the selection device 10 analyzes the API specification data of the plurality of extracted API candidates, the request from the user, and functional differences among the plurality of API candidates and inputs a prompt to the LLM server 30, the prompt including an instruction sentence for instructing the generation of a question sentence for obtaining additional information from the user on the basis of the functional differences, thereby generating a question sentence. The selection device 10 generates a new request from the user, the new request including a request from the user, a question sentence, and an answer, and repeats the extraction of API candidates.

Description

Selection Apparatus and Selection Method

[0001] The present disclosure relates to a selection apparatus and a selection method.

[0002] In recent years, large language models (LLMs) have been increasingly utilized to improve the efficiency of daily life and business tasks. However, with current standalone LLMs, it is difficult to obtain the latest information and solve tasks that require complex calculations. Accordingly, in order to enable handling such tasks, a method has been studied in which an LLM analyzes a request described by a user in natural language and autonomously uses an appropriate application programming interface (API) based on the intention of the request.

[0003] Wenxuan Wang, Juluan Shi, Chaozheng Wang, Cheryl Lee, Youliang Yuan, and Jen-tse Huang, "Learning to Ask: When LLMs Meet Unclear Instruction," arXiv preprint arXiv:2409.00557, 2024, https: / / arxiv.org / abs / 2409.00557

[0004] In the technique of Non-Patent Document 1, a user's description and specifications of an API selected based on the description are input to an LLM, thereby detecting a lack of information necessary for generating API parameters for the request described by the user, and requesting the user to provide the missing information.

[0005] However, if there are multiple API candidates that meet the user's request, only a few of them may actually satisfy the user's original intent. In this case, there is a risk that the LLM may select an inappropriate API. For example, even if the user's original intent was to "get the release date of an artist's album," if the user describes the request as "I want to get artist information," the LLM may mistakenly select an API that can get artist information but not album release dates. Non-patent document 1 employs a method of requesting additional information from the user if there is insufficient information when generating the parameters for the selected API. However, this method does not involve requesting missing information from the user in the API selection process itself.

[0006] This disclosure is made in light of the above and aims to select an appropriate API.

[0007] A selection device in one aspect of this disclosure extracts API candidates that can solve the first user's request from the specification data of available APIs, outputs the API candidate if there is one extracted API candidate, generates a question to narrow down the API candidates based on the functional differences between the multiple API candidates, generates a second user's request from the first user's request and the user's answer to the question, and repeats the extraction of API candidates with the second user's request as the first user's request. The selection device inputs an instruction to a large-scale language model, which includes an instruction to extract APIs that have the function to solve the first user's request, the first user's request, and the specification data of the available APIs, to extract API candidates, and if there are multiple extracted API candidates, it inputs an instruction to a large-scale language model, which includes an instruction to analyze the functional differences between the multiple API candidates and generate a question to obtain additional information from the user based on the functional differences, the first user's request, and the specification data of the multiple extracted API candidates, to generate a question.

[0008] According to this disclosure, it is possible to select an appropriate API.

[0009] Figure 1 shows an example of the configuration of the selection device. Figure 2 shows an example of API specification data. Figure 3 shows an example of the first prompt template. Figure 4 shows an example of the second prompt template. Figure 5 shows an example of a newly generated user request. Figure 6 is a flowchart showing an example of the API selection process flow. Figure 7 is a flowchart showing an example of the process flow for extracting API candidates. Figure 8 is a flowchart showing an example of the process flow for generating a question. Figure 9 is a flowchart showing an example of the process flow for obtaining answers from the user. Figure 10 shows an example of the hardware configuration of the selection device.

[0010] [Configuration of the API Selection Device] Figure 1 shows an example of the configuration of the selection device 10 in this embodiment. The selection device 10 takes user requests as input and selects an API to resolve those requests. The selection device 10 shown in Figure 1 comprises an input unit 11, an API candidate extraction unit 12, a question text generation unit 13, a user request generation unit 14, an API specification data storage unit 15, and a prompt template storage unit 16, and is connected to the LLM server 30.

[0011] The input unit 11 receives requests written by the user in natural language (hereinafter referred to as user requests). User requests are linguistic information such as text intended to request tasks.

[0012] The API candidate extraction unit 12 extracts API candidates that can solve the user request from all the API specification data held by the API specification data storage unit 15. Specifically, the API candidate extraction unit 12 creates a prompt by inserting the user request and all the API specification data into a first prompt template that includes an instruction to extract APIs that have the function to solve the user request, and sends the created prompt to the LLM server 30.

[0013] Figure 2 shows an example of API specification data. The API specification data 100 in Figure 2 is text data for each API, including the API name, API ID, and a description of the API function. The API name is text representing the name of the API. The API ID is a number uniquely assigned to each API. The API function description is text describing the API function. In addition to the description of the API function, the API function description in Figure 2 includes the main items that can be obtained by the API. The API function description can be generated based on API specifications stored in API repositories, etc. For example, a publicly available API specification can be input into LLM, and the API function description can be generated based on all the information contained in the API specification (e.g., name, parameters, description). This makes it possible to reflect even detailed functions that are not described in the API description.

[0014] Figure 3 shows an example of the first prompt template. The first prompt template 110 in Figure 3 has an instruction statement 111, an insertion point 112 for inserting a user request, and an insertion point 113 for inserting all API specification data. The instruction statement 111 includes instructions to extract APIs that have the function to resolve the user request and instructions for the output format. The text of the user request is inserted into the insertion point 112, and the text of all API specification data is inserted into the insertion point 113 to generate the prompt. Note that the first prompt template 110 in Figure 3 is just an example, and the content and format of the instruction statement can be set arbitrarily.

[0015] The API candidate extraction unit 12 obtains API candidates from the output data returned from the LLM server 30.

[0016] If there is only one API candidate, the API candidate extraction unit 12 outputs that API candidate as the selected API. The selection device 10 or LLM server 30 may call the API to generate a response to the user's request and return the result to the user. Another device may receive the API information, use that API to obtain the information desired by the user, and return the result to the user.

[0017] If there are multiple API candidates, the system obtains additional information from the user, generates new user requests, and repeatedly extracts API candidates. For example, each API in Figure 2 retrieves different items, but all of them retrieve artist information. If the user request is "I want to retrieve artist information," the four APIs in Figure 2 are extracted as API candidates. If the user's original intention was "to retrieve the release date of an artist's album," then the artist-albums API, which can retrieve the release date, should be selected from the four API candidates. In this embodiment, in order to select the appropriate API, the selection device 10 obtains additional information from the user based on the functional differences between the API candidates, generates new user requests, and repeatedly extracts API candidates. The acquisition of additional information and the generation of new user requests are performed by the following question generation unit 13 and user request generation unit 14.

[0018] Even if there are multiple API candidates, the selection device 10 may select and output multiple APIs if multiple APIs are needed to solve the user's request. For example, if a user requests information on multiple items, and one API cannot obtain information on all items, requiring the use of multiple APIs, the selection device 10 may select multiple APIs. Also, even if there are multiple API candidates, if there is little difference in functionality between the API candidates, the selection device 10 may select and output any of the APIs.

[0019] The question generation unit 13 identifies additional information necessary to select an appropriate API based on the difference functions between API candidates and generates a question that asks the user for that additional information. Specifically, the question generation unit 13 creates a prompt by inserting the user request and API candidates into a second prompt template that includes an instruction to analyze the difference functions between API candidates and an instruction to generate a question that inquires for additional information based on the difference functions, and then sends the created prompt to the LLM server 30.

[0020] Figure 4 shows an example of a second prompt template. The second prompt template 120 in Figure 4 has an instruction statement 121, an insertion point 122 for inserting user requests, and an insertion point 123 for inserting API candidates. The instruction statement 121 includes instructions to generate a question to inquire about missing information needed for API selection, a procedure for generating the question, and instructions for the output format. The instruction statement 121 also describes the procedure for generating the question, which involves extracting the difference functions between API candidates, identifying the missing information based on the difference functions, confirming that the missing information is not included in the user requests, and generating a question to identify the missing information. The text of the user requests is inserted into the insertion point 122, and the text of the API specification data for multiple API candidates is inserted into the insertion point 123 to generate the prompt. Note that the second prompt template 120 in Figure 4 is just an example, and the content and format of the instruction statement can be set arbitrarily.

[0021] The question generation unit 13 obtains question texts from the output data returned from the LLM server 30. The question text is text data consisting of one or more questions, and includes questions to obtain missing information necessary for selecting an API.

[0022] The user request generation unit 14 presents the user with a question and generates a new user request based on the answer received. Specifically, the user request generation unit 14 generates a new user request by sequentially adding the question and answer to the existing user request. By writing the information in this order, the LLM server 30 can recognize that information has been narrowed down based on the user request.

[0023] Figure 5 shows an example of a newly generated user request. User request 130 in Figure 5 includes the text data of the original user request 131, question 132, and answer 133. Question 132 is the question generated by the question generation unit 13. Answer 133 is the user's response to question 132.

[0024] A new user request is sent to the API candidate extraction unit 12, and the extraction of API candidates is repeated. When the extraction of API candidates is repeated, a new user request 130 is inserted into the insertion location 112 of the first prompt template 110 where the user request is inserted. Instead of all the API specification data, the API specification data of the multiple API candidates that have been narrowed down up to that point may be inserted into the insertion location 113 of the first prompt template 110 where the API specification data is inserted.

[0025] The API specification data storage unit 15 stores specification data for available APIs.

[0026] The prompt template storage unit 16 stores various prompt templates for generating prompts to send to the LLM server 30.

[0027] [API Selection Process] Refer to the flowchart in Figure 6 to explain an example of the API selection process flow.

[0028] In step S11, the input unit 11 receives user requests. User requests may be directly input by the user to the selection device 10, or they may be received by another device from the user.

[0029] In step S12, the API candidate extraction unit 12 extracts API candidates that can solve the user's request from all API specification data.

[0030] In step S13, the API candidate extraction unit 12 determines whether or not there are multiple extracted API candidates.

[0031] If there is only one API candidate, in step S21, the API candidate extraction unit 12 outputs information about the selected API. The selection device 10 or LLM server 30 may call the API to obtain the information desired by the user, generate a response to the user's request, and return the result to the user.

[0032] If there are multiple API candidates, in step S14, the question generation unit 13 generates a question that asks the user for additional information necessary to select the appropriate API.

[0033] In step S15, the user request generation unit 14 presents a question to the user and obtains an answer from the user.

[0034] In step S16, the user request generation unit 14 generates a new user request from the user request, question, and user's answer, returns to step S12, and extracts API candidates that can resolve the new user request.

[0035] Next, we will refer to the flowchart in Figure 7 and explain an example of the process flow for extracting API candidates.

[0036] In step S121, the API candidate extraction unit 12 receives the user request input, retrieves a first prompt template from the prompt template storage unit 16, and retrieves all API specification data from the API specification data storage unit 15. It then inserts the user request and all specification data into the first prompt template to create a prompt. The API candidate extraction unit 12 sends the created prompt to the LLM server 30.

[0037] In step S122, the API candidate extraction unit 12 extracts the API ID of an API that can resolve the user request from the output data received from the LLM server 30. For example, in the first prompt template 110 shown in Figure 3, the output format is specified as a list of API IDs separated by three backticks. The API candidate extraction unit 12 can obtain the API ID by using a regular expression to extract a list of numbers separated by three backticks.

[0038] In step S123, the API candidate extraction unit 12 retrieves API specification data corresponding to the API ID from the API specification data storage unit 15 and creates API candidate data. The API candidate data created here is text data that contains one or more API specification data that can solve the user request. If there is one API candidate, that is, if the API candidate data contains one API specification data, the API candidate extraction unit 12 outputs that API specification data. If there are multiple API candidates, that is, if the API candidate data contains multiple API specification data, the API candidate extraction unit 12 sends the user request and the API candidate data to the question generation unit 13.

[0039] Next, referring to the flowchart in Figure 8, we will explain an example of the process flow for generating question sentences.

[0040] In step S141, the question generation unit 13 retrieves a second prompt template from the prompt template storage unit 16, and creates a prompt by inserting the user request and API candidate data into the second prompt template. The question generation unit 13 then sends the created prompt to the LLM server 30.

[0041] In step S142, the question generation unit 13 extracts the question from the output data received from the LLM server 30. For example, in the second prompt template 120 shown in Figure 4, the output format specifies that the question should be separated by three backticks. The question generation unit 13 can obtain the question by extracting the string separated by three backticks using a regular expression. Multiple question sentences may be obtained from the output data.

[0042] Next, referring to the flowchart in Figure 9, we will explain an example of the process flow for obtaining responses from the user.

[0043] In step S151, the user request generation unit 14 presents the question generated by the question generation unit 13 to the user.

[0044] In step S152, the user request generation unit 14 receives a response from the user.

[0045] When there are a plurality of questions, steps S151 and S152 may be repeated, or question sentences may be collectively presented in step S151, and answers may be collectively obtained in step S152. If the selection device 10 is not provided with a user interface, the user request generation unit 14 may transmit the question sentence to another device, and the other device may obtain an answer from the user and return it to the selection device 10.

[0046] When an answer is obtained from the user, the user request generation unit 14 generates a new user request from the user request, the question sentence, and the user's answer.

[0047] Note that, in the above description, the API candidate extraction unit 12 and the question sentence generation unit 13 use the LLM server 30 to perform extraction of API candidates and generation of question sentences, but these processes may be performed without using the LLM server 30. For example, the API candidate extraction unit 1 specifies information desired by the user from the user request (for example, artist information, album information, etc.), refers to API specification data, and extracts API candidates capable of acquiring the information desired by the user. The question sentence generation unit 13 specifies a difference in acquirable items between a plurality of API candidates, and generates a question sentence asking whether information of the specified item is necessary.

[0048] As described above, the selection device 10 of the present embodiment inputs a prompt including specification data of available APIs, a user's request, and an instruction for instructing extraction of an API having a function of solving the user's request to the LLM server 30 to extract API candidates. When there are a plurality of extracted API candidates, the selection device 10 analyzes the API specification data of the plurality of extracted API candidates, the user's request, and functional differences between the plurality of API candidates, and inputs a prompt including an instruction for instructing generation of a question sentence to obtain additional information from the user based on the functional differences to the LLM server 30 to generate the question sentence. The selection device 10 generates a new user request including the user's original request, the question sentence, and the answer, and repeats the extraction of API candidates. This makes it possible to select an appropriate API that satisfies the user's original intention.

[0049] The selection device 10 described above can be a general-purpose computer system, such as the one shown in Figure 10, which includes a central processing unit (CPU) 901, memory 902, storage 903, communication device 904, input device 905, and output device 906. In this computer system, the selection device 10 is realized when the CPU 901 executes a predetermined program loaded onto the memory 902. This program can be recorded on a computer-readable non-temporary recording medium such as a magnetic disk, optical disk, or semiconductor memory, or it can be distributed via a network.

[0050] 10 Selection device 11 Input unit 12 API candidate extraction unit 13 Question text generation unit 14 User request generation unit 15 API specification data storage unit 16 Prompt template storage unit 30 LLM server

Claims

1. A selection device that extracts API candidates that can solve the requirements of a first user from the specification data of available APIs; outputs the API candidate if there is only one extracted API candidate; generates a question to narrow down the API candidates based on the functional differences between the multiple API candidates if there are multiple extracted API candidates; generates a second user's requirements from the requirements of the first user and the user's answers to the question; and repeats the extraction of API candidates using the second user's requirements as the requirements of the first user.

2. A selection device according to claim 1, wherein a first prompt including an instruction statement that instructs the extraction of APIs having the function of resolving the request of the first user, the request of the first user, and specification data of the available APIs is input to a large language model to extract API candidates, and if there are multiple extracted API candidates, a second prompt including an instruction statement that instructs the generation of a question statement to analyze the functional differences between the multiple API candidates and to generate a question statement to obtain additional information from the user based on the functional differences, the request of the first user, and specification data of the multiple extracted API candidates is input to the large language model to generate a question statement.

3. A selection device according to claim 2, which holds a first prompt template comprising an instruction statement that instructs the extraction of an API having a function to resolve the request of the first user, a location for inserting the request of the first user, and a location for inserting specification data of the available APIs, and generates the first prompt by inserting the request of the first user and the specification data of the available APIs into the first prompt template.

4. A selection device according to claim 2, which holds a second prompt template comprising an instruction statement for instructing the analysis of functional differences between a plurality of API candidates, an instruction statement for generating a question statement for obtaining additional information from the user based on the functional differences, a place to insert the first user's request, and a place to insert the specification data of the plurality of API candidates, and generates the second prompt by inserting the first user's request and the specification data of the plurality of API candidates into the second prompt template.

5. A selection device according to claim 2, wherein the request of the second user is text data in which the request of the first user, the question, and the user's answer are listed in order.

6. A selection method comprising: a computer extracting API candidates that can solve the requirements of a first user from the specification data of available APIs; outputting the API candidate if there is only one extracted API candidate; generating a question to narrow down the API candidates based on the functional differences between the multiple API candidates if there are multiple extracted API candidates; and repeating the extraction of API candidates by generating the requirements of a second user from the requirements of the first user and the user's answers to the question.

7. A selection method according to claim 6, comprising: inputting an instruction statement to a large-scale language model, which instructs the extraction of APIs having the function of resolving the request of the first user, the request of the first user, and specification data of the available APIs, to extract API candidates; and, if there are multiple extracted API candidates, inputting an instruction statement to the large-scale language model, which instructs the generation of a question statement to analyze the functional differences between the multiple API candidates and to generate a question statement to obtain additional information from the user based on the functional differences, the request of the first user, and specification data of the multiple extracted API candidates, to generate a question statement.

Images